Commit c0785d26 authored by Christos Stavrakakis's avatar Christos Stavrakakis
Browse files

Fix reconciliation for building VMs

Since commit 482c645, a VM exists in the Cyclades DB before the
corresponding job is enqueued in the Ganeti backend. Therefore, there is
a corner case, where Cyclades can not enqueue the job to Ganeti and
Cyclades crashes before deleting the VM from the DB, which results in a
stale VM entry in a building state. This commit adds the necessary check
to guarantee that these VMs will be removed by the reconciliation
process.
parent ae6964c7
......@@ -147,8 +147,13 @@ def instances_with_build_errors(D, G):
for i in idD & idG:
if not G[i] and D[i] == 'BUILD':
vm = VirtualMachine.objects.get(id=i)
# Check time to avoid many rapi calls
if datetime.now() > vm.backendtime + timedelta(seconds=5):
if not vm.backendjobid: # VM has not been enqueued in the backend
if datetime.now() > vm.created + timedelta(seconds=120):
# If a job has not been enqueued after 2 minutues, then
# it must be a stale entry..
failed.add(i)
elif datetime.now() > vm.backendtime + timedelta(seconds=30):
# Check time to avoid many rapi calls
with pooled_rapi_client(vm) as c:
try:
job_info = c.GetJobStatus(job_id=vm.backendjobid)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment