locking.rst 3.18 KB
Newer Older
1 2 3 4 5
Ganeti locking
==============

Introduction
------------
6 7 8 9 10

This document describes lock order dependencies in Ganeti.
It is divided by functional sections


11 12
Opcode Execution Locking
------------------------
13 14 15 16 17 18 19 20 21

These locks are declared by Logical Units (LUs) (in cmdlib.py) and acquired by
the Processor (in mcpu.py) with the aid of the Ganeti Locking Library
(locking.py). They are acquired in the following order:

  * BGL: this is the Big Ganeti Lock, it exists for retrocompatibility. New LUs
    acquire it in a shared fashion, and are able to execute all toghether
    (baring other lock waits) while old LUs acquire it exclusively and can only
    execute one at a time, and not at the same time with new LUs.
22
  * Instance locks: can be declared in ExpandNames() or DeclareLocks() by an LU,
23 24
    and have the same name as the instance itself. They are acquired as a set.
    Internally the locking library acquired them in alphabetical order.
25
  * Node locks: can be declared in ExpandNames() or DeclareLocks() by an LU, and
26 27 28 29 30 31 32 33 34 35 36 37 38
    have the same name as the node itself. They are acquired as a set.
    Internally the locking library acquired them in alphabetical order. Given
    this order it's possible to safely acquire a set of instances, and then the
    nodes they reside on.

The ConfigWriter (in config.py) is also protected by a SharedLock, which is
shared by functions that read the config and acquired exclusively by functions
that modify it. Since the ConfigWriter calls rpc.call_upload_file to all nodes
to distribute the config without holding the node locks, this call must be able
to execute on the nodes in parallel with other operations (but not necessarily
concurrently with itself on the same file, as inside the ConfigWriter this is
called with the internal config lock held.

39 40 41 42 43

Job Queue Locking
-----------------

The job queue is designed to be thread-safe. This means that its public
44 45 46
functions can be called from any thread. The job queue can be called from
functions called by the queue itself (e.g. logical units), but special
attention must be paid not to create deadlocks or an invalid state.
47 48

The single queue lock is used from all classes involved in the queue handling.
49 50 51 52 53
During development we tried to split locks, but deemed it to be too dangerous
and difficult at the time. Job queue functions acquiring the lock can be safely
called from all the rest of the code, as the lock is released before leaving
the job queue again. Unlocked functions should only be called from job queue
related classes (e.g. in jqueue.py) and the lock must be acquired beforehand.
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69

In the job queue worker (``_JobQueueWorker``), the lock must be released before
calling the LU processor. Otherwise a deadlock can occur when log messages are
added to opcode results.


Node Daemon Locking
-------------------

The node daemon contains a lock for the job queue. In order to avoid conflicts
and/or corruption when an eventual master daemon or another node daemon is
running, it must be held for all job queue operations

There's one special case for the node daemon running on the master node. If
grabbing the lock in exclusive fails on startup, the code assumes all checks
have been done by the process keeping the lock.
70 71

.. vim: set textwidth=72 :