Stop acquiring BGL for LUXI queries
Short description: This fixes an issue whereby masterd would become unresponsive on the LUXI socket, leading to client timeouts. While made worse in 2.5, the underlying issue was already present in 2.4. Longer description: Until now all LUXI queries would acquire the BGL (big Ganeti lock) in shared mode. With the exception of OpNodeAdd and OpNodeRemove, this was also the case for all opcodes before version 2.5. In 2.5 we split OpClusterVerify into multiple opcodes, one of which (OpClusterVerifyConfig) now acquires the BGL in exclusive mode. Whether or not doing so is good is a separate discussion: OpNodeAdd and OpNodeRemove, as of this writing, still require an exclusive BGL. OpClusterVerifyConfig is run more often than OpNodeAdd or OpNodeRemove in normal clusters, which is why we only recognized this issue in 2.5. What would happen is that once OpClusterVerifyConfig tried to acquire its exclusive BGL while it was actually held by other opcodes (e.g. OpInstanceReplaceDisks), the locking code would not grant shared acquires for the BGL, even when the exclusive acquire is removed from the queue for a short amount of time after a timeout. This is necessary to prevent lock starvation. In this situation further LUXI queries requiring the BGL in shared mode, e.g. OpClusterQuery, would block and the client eventually time out. Over time they fill the client request workerpool's queue and at that point even requests not requiring the BGL stop working. Once the long-running operation(s) holding the BGL in shared mode finished, OpClusterVerifyConfig gets it in exclusive mode and everything returns to normal. LUXI recovers very soon too. I'd like to thank Bernardo Dal Seno for his contribution to this bugfix. Signed-off-by:Michael Hanselmann <hansmi@google.com> Reviewed-by:
Bernardo Dal Seno <bdalseno@google.com>
Loading
Please register or sign in to comment