Reviewed-by: iustinp

- Introduce abstraction class on client side
- Use constants for method names
- Adopt legacy function SubmitOpCode to use it

Reviewed-by: iustinp

- Use constants for dict entries
- Handle exceptions on server side
- Rename client function to CallMethod to match server side naming

Reviewed-by: iustinp

Reviewed-by: iustinp

Reviewed-by: iustinp

This is to discourage even more whoever may think that this requirement
is not really useful and can be lifted, and to at least know where it's
used before trying to break it.

Reviewed-by: imsnah

In order to do so:
  - We set REQ_BGL to False
  - We implement ExpandNames

That's it, really.

Reviewed-by: iustinp

If we're running in a "new style" LU we may need some locks, as required
by the ExpandNames function, to be able to run. We'll walk up the lock
levels present in the needed_locks dictionary and acquire them, then run
the actual LU. LUs can release some or all the acquired locks, if they
want, before terminating, provided they update their needed_locks
dictionary appropriately, so that we know not to release a level if they
have already done so.

Reviewed-by: iustinp

New concurrent LUs will need to call ExpandNames so that any names
passed in by the user are canonicalized, and can be used by hooks,
locking and other parts of the code. This was done in CheckPrereq
before, but it's now splitted out, as it's needed for locking, which in
turn CheckPrereq needs. Old LUs can be converted gradually.

Reviewed-by: iustinp

This makes the try...finally code simplier, and helps adding a more
complex locking structure before the actual execution. It also fixes a
concurrency bug caused by the fact that write_count was read before
acquiring the BGL, and thus spurious config update hooks run could have
been triggered. This doesn't solve the issue of running config update
hooks for concurrent LUs.

Reviewed-by: iustinp

This test checks the LockSet behaviour when an empty list is passed.
The current behaviour is expected, but since this is a corner case,
we're safer to keep it under a check, and if we need a different one
monitor that everything is as we expect it to be.

Reviewed-by: imsnah

Reviewed-by: iustinp

Workers have to notify their pool if they finished a task to make
the WorkerPool.Quiesce function work. This is done in the finally:
clause to notify even in case of an exception. However, before
we notified on each run, even if there was no task, thereby creating
some sort of an endless loop of notifications. In a future patch
we should split the single condition object into several to
produce less spurious notifications.

While we're at this, this patch also adds two new functions to
BaseWorker to query whether it's currently running a task and then
uses one of these functions in the WorkerPool instead of querying
the internal variable directly.

Reviewed-by: iustinp

Rather than just creating BDEV_CACHE_DIR we loop through the
SUB_RUN_DIRS list and create all its childs.

Reviewed-by: iustinp

This patch creates a base RUN_GANETI_DIR and then moves the other run
dir constants to use that (even if just setting BDEV_CACHE_DIR as equal
to it, rather than putting it deeper, for now).

Also we create a constant list of all the subdirs we need in RUN_DIR to
work properly, which we'll use when creating them in ganeti-noded.

Reviewed-by: iustinp

The DISK_LINKS_DIR points to the RUN_DIR/ganeti/instance-disks
directory, which will contain symlinks to the instances' disks. These
provide a stable name accross all nodes for them, and permit
live-migration to happen.

Unfortunately RUN_DIR/ganeti/instance-disks happens to be below ganeti
1.2's BDEV_CACHE_DIR, which will we need to address at some point
(possibly in 2.0).

Reviewed-by: iustinp

Reviewed-by: iustinp

If the simplejson module supports indentation, it's always used. There
are cases where we might not want to use it or enable it only for
debugging purposes, such as in RPC.

Reviewed-by: iustinp

cmdlib uses some constants from locking (ie. locking levels) but doesn't
import it. This patch fixes the issue.

Reviewed-by: iustinp

Since the context is passed to LogicalUnit, rather than the cfg, we can
only access the cfg as self.cfg, self.context.cfg, or context.cfg (in
the constructor). cfg is not valid anymore.

Reviewed-by: iustinp

Whenever we add an instance or node to the cluster (i.e. to the config
and whenever we remove them we should add/remove locks as well). In the
future we may want to optimize this so that the configwriter does it, or
it's handled at the context level, but till we're adding/removing
instances and nodes with the BGL held it doesn't matter too much.

Reviewed-by: iustinp

Rather than passing a ConfigWriter to the LUs we'll pass the whole
context, from which a ConfigWriter can be extracted, but we can also
access the GanetiLockManager. This also fixes the places where a FakeLU
is created.

Reviewed-by: iustinp

This will be passed to FakeLUs

Reviewed-by: iustinp

Reviewed-by: iustinp

Since the ConfigWriter now handles its own locking it's not necessary to
have a specific level for the config in the Locking Manager anymore.
This patch thus removes it, and all the unittest calls that used it, or
depended on it being present.

Reviewed-by: iustinp

Since we share the ConfigWriter we need somehow to make sure that
accessing it is properly synchronized. We'll do it using the
locking.ssynchronized decorator and a module-private shared lock.

This patch also renames a few functions, which were called inside the
ConfigWriter, to a private version _UnlockedFunctionName, and exports
the synchronized public ones. The internal callers, which are already
synchronized, are then changed to use the _Unlocked version, to prevent
double locking.

Reviewed-by: iustinp

This patch creates a new decorator function ssynchronized in the locking
library, which takes as input a SharedLock, and synchronizes access to
the decorated functions using it. The usual SharedLock semantics apply,
so it's possible to call more than one synchronized function at the same
time, when the lock is acquired in shared mode, and still protect
against exclusive access.

The patch also adds a few unit test to check the basic decorator's
functionality, and to provide an example on how to use it.

Reviewed-by: iustinp

Remove empty function _ReleaseLock and all its calls. Since we only
have one configwriter per cluster the locking needs to cover all the
data in the object, and not just the file contents. Locking in
ConfigWriter will be handled using the ganeti locking library.

Reviewed-by: iustinp

This patch fixes two bugs:
  - the state file is not saved because we use the method for checking
    for udpated data
  - in two places 'Error' was used instead of 'Exception', which breaks
    error handling

Additionally:
  - the unused 're' import has been removed
  - a variable named 'id' which collides with a builtin function has
    been renamed

Note that comparing the serialized forms might create false negatives
(due to the dicts being reordered) but that will just cause an extra
write of the file, which is sub-optimal but harmless.

Reviewed-by: ultrotter

Reviewed-by: ultrotter

By a mistake, we don't reuse the luxi client. As such, we open and close
the connection at each poll cycle and spam the server logs.

Reviewed-by: ultrotter

It's better for daemons if:
  - they log only to one log file
  - the log level is included
  - for debug runs, the filename/line number is included

This patch moves the custom formatter from the watcher to the logging
module and generalizes it; then it changes the master daemon to use this
function instead of the generic logging (which might be deprecated
anyway in the future).

Reviewed-by: imsnah

The gnt-instance script doesn't run in the same process anymore, so we
can't and don't have to unlock.

Reviewed-by: ultrotter

Reviewed-by: iustinp, ultrotter

Reviewed-by: ultrotter, iustinp

Make the GanetiLockManager instance of GanetiContext lowercase

Reviewed-by: imsnah

At least docbook2man inserts a date formatted using the current
locale into its output.

Reviewed-by: iustinp

Reviwed-by: imsnah

This function will return the first unused integer based on a list of
used integers (e.g. [0, 1, 3] will return 2).

Reviewed-by: imsnah

Now that we use the locking library to make sure running opcodes cannot
step on each other toes we can have a bigger thread size, and
potentially process many opcodes in a parallel manner.

Reviewed-by: iustinp