The patch changes the log constants by moving the slash to the end of
the log dir instead of at the beginning of *each* log file name.

It also adds a new LOG_COMMANDS constant (to be used in a next patch).

Reviewed-by: ultrotter

This is special in the sense that it can run on any node. As such, we
just instantiate ssconf and read the data from it.

Reviewed-by: ultrotter

Reviewed-by: iustinp

self.recalculate_locks[locking.LEVEL_NODE] could have any value and
everything would work anyway. We'll use the string 'replace' by
convention because in the future we might want an 'append' mode.

Reviewed-by: iustinp

This function is used to lock instances' primary and secondary nodes
after locking instances themselves.

Reviewed-by: iustinp

LUs can declare which locks they need by populating the
self.needed_locks dictionary, but those locks are always acquired as
exclusive. Make it possible to acquire shared locks as well, by
declaring a particular level as shared in the self.share_locks
dictionary. By default this dictionary is populated so that all locks
are acquired exclusively.

Reviewed-by: iustinp

This additional LogicalUnit function is optional to implement, but lets
you change your locking needs for one level just before locking it, but
after the previous levels have been already locked. It is useful for
example to calculate what nodes to lock after locking an instance.

Reviewed-by: iustinp

LURenameInstance forgot to remove the old lock name and add the new one,
making it impossible for parallel LUs to act on the instance (without a
master daemon restart). This also fixes burning+rename with the
parallelization of {Start,Stop}Instance.

Reviewed-by: iustinp

We found several issues in the old job queue implementation. It had race
conditions, deadlocks and other deficiencies.

Short summary:
- _QueuedOpCode and _QueuedJob are now more or less data structures with a few
  utility functions. __Setup is gone.
- DiskJobStorage and JobQueue classes merged into one to reduce code complexity.
- One lock in JobQueue for almost everything. There's also a lock per opcode
  for log messages.

Reviewed-by: iustinp

Reviewed-by: iustinp

This (big) patch reworks the master startup/shutdown and the fixes the
master failover.

What does the patch do?

For master start/stop:
  - remove the old ganeti-master script and its associated man page
  - moves the ip start/stop directly into the backend.(Start|Stop)Master
  - adds start/stop of the master/rapi daemon into these functions,
    selectively based on the start/stop arguments
  - makes the master call via rpc StartMaster(start_daemons=False) to
    the local node so that the master IP is started
  - and finally changes the example init.d script to directly start and
    stop all three daemons, since they do the right thing (depending on
    master/not master role)

For master failover:
  - moves the code from LUMasterFailover into bootstrap.MasterFailover,
    since we need to start/stop the master during this operation and
    thus it can't be executed from the master
  - removes the LUMasterFailover and its associated opcode

Notes: ubuntu's /etc/lsb-base-logging.sh is dumb, so the messages 'not
master' are not seen during startup on non-master nodes.

Reviewed-by: ultrotter

Since we need to compute this from outside utils.py, we change this to a
public function.

Reviewed-by: ultrotter

This patch moves the CheckMaster function from ganeti-masterd to ssconf
(most logical place, it cannot go in utils since we would have recursive
imports between ssconf and utils) and changes ganeti-rapi to also call
this function.

This is needed so that starting ganeti-rapi on a non-master node does
the right thing.

Reviewed-by: ultrotter

This patch adds a new, unused for now, parameter to the start and stop
master operations in backend. The idea behind it is that we need to be
able to control whether the IP (de)activation is coupled with daemon
startup/shutdown.

The callers are also modified to pass this parameter (even if unused for
now).

Reviewed-by: ultrotter

Reviewed-by: iustinp

The passed parameters were not correct.

Reviewed-by: iustinp, ultrotter

Reviewed-by: imsnah

Reviewed-by: imsnah

We cannot depend on all environments to have a start-stop-daemon or
similar tool. We instead implement a KillProcess function that behaves
similar to “start-stop-daemon --retry”.

Note that the attached unittest can hang in foreground if the child
misbehaves (doesn't write to the internal pipe). Since unittest are
either run in the foreground or are run with a timeout from an automated
framework, I think this is an acceptable trade-off (against of using
hardcoded timeouts in the test).

Reviewed-by: imsnah

We already have a function to test if a PID is alive, so it makes more
sense to use function composition that force calling (since we need to
read PIDs from files in other places too). Now IsProcessAlive returns
False for PIDs <= 0, since this is the error return from ReadPidFile.

The patch also adds a unittest for checking that WriteFile raises the
correct exception, and checks that an invalid or missing file causes
ReadPidFile to return zero. The unittest tearDown method will try to
cleanup the temp directory too (otherwise it leaves stuff after it).

Reviewed-by: ultrotter

This is needed for controlling it cleanly with start-stop daemon.

Reviewed-by: ultrotter

The RESTHTTPServer module went the way of the dodo.

Reviewed-by: iustinp

Reviewed-by: iustinp

All other daemons have their main code in themselves and not in a module.
This patch does the same to ganeti-rapi by moving the code from
lib/rapi/RESTHTTPServer.py to daemons/ganeti-rapi.

Reviewed-by: iustinp

The generic HTTP server doesn't know about httperror based exceptions
and would treat them as unknown exceptions, thereby not doing the right
thing with HTTP errors.

Reviewed-by: iustinp

Reviewed-by: ultrotter

Locking is not completeley right due to a deadlock when the job calls
UpdateJob after changing its status.

Reviewed-by: ultrotter

Reviewed-by: iustinp

Reviewed-by: ultrotter

The old API is no longer working.

Reviewed-by: ultrotter

It might come in handy at some point and makes the code a bit easier
to read.

Reviewed-by: iustinp

The set triggers exception on a list-tags command and RAPI calls for tags
since it is not serializable by JSON.

Reviewed-by: iustinp

Reviewed-by: imsnah

Reviewed-by: iustinp

So far no error reporting to the client is done. Clients don't get
noticed if a job doesn't exist or couldn't be archived because of
its current status.

The internal cache is always cleaned when the preconditions didn't
fail to make sure that the actual disk status will be reread next
time.

Reviewed-by: iustinp

Reviewed-by: iustinp

They aren't be tuples on the client side.

Reviewed-by: iustinp

The submitted WritePidFile, RemovePidfile and IsPidFileAlive functions
miss unit tests. Adding a simple one which covers their basic
functionality.

Reviewed-by: iustinp

A later patch will add a memory based job storage class, hence this
code is going into a separate class. It also changes the number format
to always use at least 10 digits, allowing up to 9'999'999'999 jobs to
be sorted without using a custom function.

Reviewed-by: iustinp

Rather than searching for the ganeti daemons by name we'll use the
pidfile they create to stop them. This change also adds the --oknodo
option to start-stop-daemon when stopping ganeti (which means it won't
give an error if it wasn't started).

Reviewed-by: iustinp