Reviewed-by: iustinp

Queries don't create jobs and are more efficient. Log messages
are not yet stored anywhere.

Reviewed-by: iustinp

Newly added nodes are not yet taken care of. Queue locking on
non-master nodes is not yet correct.

Reviewed-by: iustinp

Reviewed-by: iustinp

This will be used to move common code out of jqueue.

Reviewed-by: iustinp

This patch adds the infrastructure for executing a job in background,
instead of foreground, via a new “--submit” option. The behaviour is
that the job ID is printed and the script will immediately exit.

The patch also converts gnt-node list to this model (yes, this will be a
query in the future).

Reviewed-by: imsnah

Reviewed-by: imsnah

Reviewed-by: imsnah

This reduces code duplication. A later patch will modify the job queue
a bit more and will need a change of this assert. The assertion is
also removed from all class-internal functions.

Reviewed-by: iustinp

The current SubmitOpCode function is not flexible enough to be used for
submitters that don't want to wait for the job finish.

The patch splits this in two, a SendJob function and a PollJob one, and
the old SubmitOpCode becomes a wrapper. Note that the new SendJob takes
a list of opcodes (and not a single opcode anymore).

Reviewed-by: imsnah

This is needed for job queue replication.

Reviewed-by: iustinp

This class is a wrapper around fcntl.flock and abstracts opening and
closing the lockfile. It'll used for the job queue.

(The patch also removes a duplicate import of tempfile into the unittest)

Reviewed-by: iustinp

The job queue will need to access to configuration, which is provided
through the context object, to get a list of nodes.

Reviewed-by: iustinp

Reviewed-by: imsnah

Add instance tag handling, improved error logging.
...oh, yes adopt instance listing for RAPI2!

Reviewed-by: iustinp

With the recent startup/shutdown changes (and with the master daemon in
place), the cluster destroy needs some fixing.

This patch moves the finalization of the destroy out from cmdlib into
bootstrap, so we can nicely shutdown the rapi and master daemons.

Reviewed-by: ultrotter

It's python, isn't it?

Reviewed-by: iustinp

With the recent changes, I forgot the extra parameter to this rpc call.
Also the rpc call needs to be done after we setup the config data, for
the master daemon to be able to start, so we move it after all other
init steps.

Reviewed-by: ultrotter

This patch adds a check that we are on the master after failing to
connect to the socket, and log nicely the master name.

Reviewed-by: ultrotter

Reviewed-by: iustinp

Prevent mistakes with an assert.

Reviewed-by: iustinp

This is my fault, sorry.

Reviewed-by: imsnah

This is mostly:
  - whitespace fix (space at EOL in some files, not all, broken
    indentation, etc)
  - variable names overriding others (one is a real bug in there)
  - too-long-lines
  - cleanup of most unused imports (not all)

Reviewed-by: ultrotter

Reviewed-by: imsnah

The 'old-style' info, error, debug logs do not make much sense. This
patch unifies the SetupLogging and SetupDaemon functions. As a result,
all the commands logs to a 'commands.log' file.

The patch also changes the log setup to keep going if there's an error
in setting up the file logging but we're logging to stderr.

Also, burnin now logs to its own file (burnin.log).

Reviewed-by: ultrotter

The patch changes the log constants by moving the slash to the end of
the log dir instead of at the beginning of *each* log file name.

It also adds a new LOG_COMMANDS constant (to be used in a next patch).

Reviewed-by: ultrotter

This is special in the sense that it can run on any node. As such, we
just instantiate ssconf and read the data from it.

Reviewed-by: ultrotter

Reviewed-by: iustinp

self.recalculate_locks[locking.LEVEL_NODE] could have any value and
everything would work anyway. We'll use the string 'replace' by
convention because in the future we might want an 'append' mode.

Reviewed-by: iustinp

This function is used to lock instances' primary and secondary nodes
after locking instances themselves.

Reviewed-by: iustinp

LUs can declare which locks they need by populating the
self.needed_locks dictionary, but those locks are always acquired as
exclusive. Make it possible to acquire shared locks as well, by
declaring a particular level as shared in the self.share_locks
dictionary. By default this dictionary is populated so that all locks
are acquired exclusively.

Reviewed-by: iustinp

This additional LogicalUnit function is optional to implement, but lets
you change your locking needs for one level just before locking it, but
after the previous levels have been already locked. It is useful for
example to calculate what nodes to lock after locking an instance.

Reviewed-by: iustinp

LURenameInstance forgot to remove the old lock name and add the new one,
making it impossible for parallel LUs to act on the instance (without a
master daemon restart). This also fixes burning+rename with the
parallelization of {Start,Stop}Instance.

Reviewed-by: iustinp

We found several issues in the old job queue implementation. It had race
conditions, deadlocks and other deficiencies.

Short summary:
- _QueuedOpCode and _QueuedJob are now more or less data structures with a few
  utility functions. __Setup is gone.
- DiskJobStorage and JobQueue classes merged into one to reduce code complexity.
- One lock in JobQueue for almost everything. There's also a lock per opcode
  for log messages.

Reviewed-by: iustinp

Reviewed-by: iustinp

This (big) patch reworks the master startup/shutdown and the fixes the
master failover.

What does the patch do?

For master start/stop:
  - remove the old ganeti-master script and its associated man page
  - moves the ip start/stop directly into the backend.(Start|Stop)Master
  - adds start/stop of the master/rapi daemon into these functions,
    selectively based on the start/stop arguments
  - makes the master call via rpc StartMaster(start_daemons=False) to
    the local node so that the master IP is started
  - and finally changes the example init.d script to directly start and
    stop all three daemons, since they do the right thing (depending on
    master/not master role)

For master failover:
  - moves the code from LUMasterFailover into bootstrap.MasterFailover,
    since we need to start/stop the master during this operation and
    thus it can't be executed from the master
  - removes the LUMasterFailover and its associated opcode

Notes: ubuntu's /etc/lsb-base-logging.sh is dumb, so the messages 'not
master' are not seen during startup on non-master nodes.

Reviewed-by: ultrotter

Since we need to compute this from outside utils.py, we change this to a
public function.

Reviewed-by: ultrotter

This patch moves the CheckMaster function from ganeti-masterd to ssconf
(most logical place, it cannot go in utils since we would have recursive
imports between ssconf and utils) and changes ganeti-rapi to also call
this function.

This is needed so that starting ganeti-rapi on a non-master node does
the right thing.

Reviewed-by: ultrotter

This patch adds a new, unused for now, parameter to the start and stop
master operations in backend. The idea behind it is that we need to be
able to control whether the IP (de)activation is coupled with daemon
startup/shutdown.

The callers are also modified to pass this parameter (even if unused for
now).

Reviewed-by: ultrotter

Reviewed-by: iustinp