Commit e6e17529 authored by Hrvoje Ribicic's avatar Hrvoje Ribicic
Browse files

Improve job status assert affected by race condition



In the sliver of time between choosing a waiting job to be executed and
trying to acquire locks for its execution, the status of the job can be
changed to canceling. An assert checking the job status neglected to
take this into account, and raised an error that managed to perpetually
lock the job in the canceling state. This patch resolves the issue by
making the assert accept the canceling state as well, and exiting if
the job was cancelled.
Signed-off-by: default avatarHrvoje Ribicic <riba@google.com>
Reviewed-by: default avatarKlaus Aehlig <aehlig@google.com>
parent 0f68f7fa
......@@ -1104,7 +1104,12 @@ class _JobProcessor(object):
"""
op = opctx.op
assert op.status == constants.OP_STATUS_WAITING
assert op.status in (constants.OP_STATUS_WAITING,
constants.OP_STATUS_CANCELING)
# The very last check if the job was cancelled before trying to execute
if op.status == constants.OP_STATUS_CANCELING:
return (constants.OP_STATUS_CANCELING, None)
timeout = opctx.GetNextLockTimeout()
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment