The 'old-style' info, error, debug logs do not make much sense. This
patch unifies the SetupLogging and SetupDaemon functions. As a result,
all the commands logs to a 'commands.log' file.

The patch also changes the log setup to keep going if there's an error
in setting up the file logging but we're logging to stderr.

Also, burnin now logs to its own file (burnin.log).

Reviewed-by: ultrotter

This (big) patch reworks the master startup/shutdown and the fixes the
master failover.

What does the patch do?

For master start/stop:
  - remove the old ganeti-master script and its associated man page
  - moves the ip start/stop directly into the backend.(Start|Stop)Master
  - adds start/stop of the master/rapi daemon into these functions,
    selectively based on the start/stop arguments
  - makes the master call via rpc StartMaster(start_daemons=False) to
    the local node so that the master IP is started
  - and finally changes the example init.d script to directly start and
    stop all three daemons, since they do the right thing (depending on
    master/not master role)

For master failover:
  - moves the code from LUMasterFailover into bootstrap.MasterFailover,
    since we need to start/stop the master during this operation and
    thus it can't be executed from the master
  - removes the LUMasterFailover and its associated opcode

Notes: ubuntu's /etc/lsb-base-logging.sh is dumb, so the messages 'not
master' are not seen during startup on non-master nodes.

Reviewed-by: ultrotter

This patch moves the CheckMaster function from ganeti-masterd to ssconf
(most logical place, it cannot go in utils since we would have recursive
imports between ssconf and utils) and changes ganeti-rapi to also call
this function.

This is needed so that starting ganeti-rapi on a non-master node does
the right thing.

Reviewed-by: ultrotter

Reviewed-by: imsnah

They aren't be tuples on the client side.

Reviewed-by: iustinp

Reviewed-by: iustinp

This patch adds distribution of the queue serial file after each write
to it (but before a new job is created and written with that ID, and
before a response is returned, so we should be safe from crashes in
between).

Currently it only logs if a node cannot be contacted, it should abort if
> 50% errors are seen.

Reviewed-by: imsnah

Reviewed-by: ultrotter

The function to stop a worker pool is TerminateWorkers(), not Shutdown().

Reviewed-by: iustinp

Reusing threads instead of starting one for each request is more efficient.

Reviewed-by: iustinp

Apparently I forgot to this code when removing the rest.

Reviewed-by: iustinp

Currently, in debug mode, both the logfile handler and the stderr
handler will log debug messages. Since the stderr is redirected to the
same logfile (to catch non-logged errors), it means log entries are
doubled.

The patch adds an extra parameter to the logger.SetupDaemon() function
that allows disabling of the stderr logging. The master and node daemon
will use this to enable stderr logging only when running in foreground.

Reviewed-by: imsnah

This removes (hopefully) all traces of the old locking functions and
uses.

Reviewed-by: imsnah

Reviewed-by: iustinp

- Introduce abstraction class on client side
- Use constants for method names
- Adopt legacy function SubmitOpCode to use it

Reviewed-by: iustinp

- Use constants for dict entries
- Handle exceptions on server side
- Rename client function to CallMethod to match server side naming

Reviewed-by: iustinp

Reviewed-by: iustinp

It's better for daemons if:
  - they log only to one log file
  - the log level is included
  - for debug runs, the filename/line number is included

This patch moves the custom formatter from the watcher to the logging
module and generalizes it; then it changes the master daemon to use this
function instead of the generic logging (which might be deprecated
anyway in the future).

Reviewed-by: imsnah

Reviewed-by: iustinp, ultrotter

Reviewed-by: ultrotter, iustinp

Make the GanetiLockManager instance of GanetiContext lowercase

Reviewed-by: imsnah

Now that we use the locking library to make sure running opcodes cannot
step on each other toes we can have a bigger thread size, and
potentially process many opcodes in a parallel manner.

Reviewed-by: iustinp

The processor used to create a new ConfigWriter when it was initialized.
We now have one in the context, so we'll just recycle it. First of all
we'll pass the context in when creating a new Processor object, then
we'll just use context.cfg, which is granted to be initialized, wherever
we used self.cfg, and stop checking whether the config is already
initialized or not.

In the future the Processor will be able to use the context also to
acquire the BGL for LUs that require it, and to push the context down to
LUs that don't in order for them to manage their own locking.

Reviewed-by: iustinp

This patch creates a new GanetiContext class, which is used to hold
context common to all ganeti worker threads. As for the
GanetiLockingManager class it is paramount that there is only one such
class throughout the execution of Ganeti, so the class checks for that,
and also forbids its own modification after it's been initialized. The
context for now contains a ConfigWriter and a GanetiLockingManager and
is created by the daemon and propagated to PoolWorker(s) and
JobRunner(s).

Reviewed-by: iustinp

If an uncaught exception is thrown currently it destroys the calling
thread. This patch changes the behaviour to failing the current job,
logging a message, but trying to keep the daemon up.

Reviewed-by: imsnah

When a Job raises a ganeti exception a message is printed but nothing is
reported in the job itself. It's better to update the job status, thus
notifying the client, possibly polling for the job result, of what went
wrong.

Reviewed-by: iustinp

- Add a docstring to IOServer's constructor
- Add argument description to PoolWorker's and JobRunner's ones

Reviewed-by: iustinp

This patch adds a mechanism to disable utils.RunCmd in selected
programs. This is needed in the master daemon unless we confirm
threading doesn't pose any problems.

This makes cluster init fail, but creating new trunk clusters is anyway
broken at the moment.

Reviewed-by: ultrotter

This patch removes the lock and the lock options from cli.py and moves
them to the master.

Later during development we can remove it completely, but for now it's
good to protect any other tool that uses the lock directly.

Reviewed-by: imsnah,ultrotter

This patch converts the cli.py SubmitOpCode method to use the unix
protocol and thus execute the opcodes via the master.

The patch allows a partial burnin to work with the master. Currently the
query opcodes, since they are executed via the SubmitOpCode, are
executed inside a job too, which is suboptimal, but they work fine.

The cmd lock has been removed from the master, but the cli.py still
takes the lock. This is ok for this in-progress patch (since the master
still has only one executor thread). This will be fixed in a future
patch.

Reviewed-by: ultrotter

This patch changes the definition of a job and introduces per-opcode
results.

First, the result and status fields of a job are condensed into a single
'status' attribute. Then, we introduce an opcode status and one result
list, that allow jobs to return values.

The gnt-job script is also modified to allow these new fields to be
queried.

Note that the patch changes the opcode field to op_list, and it changes
its return value from string to a list of (serialized) opcodes.

Reviewed-by: ultrotter

This patch adds checks for the master role and daemonize support to
ganeti-masterd.

The patch modifies the startup/shutdown of the server because:
  - we want bind()/listen() to the master socket to occur before forking
    so that we can return a correct exit code and write messages to
    stderr
  - but we want thread startup to occur after fork(), otherwise python
    threading gets confused

The patch also has some small cleanups:
  - remove the unix socket after closing it, so we don't need to remove
    it manually
  - instead of just telling the threads to terminate via the new_queue,
    we also join() them so that the logs show what thread clinging to
    life
  - the daemon logs to its own logfile now
  - there is command line parameter support :)

Reviewed-by: imsnah

This patch adds a very basic gnt-job script that allows job querying.
This goes on top of the previous master daemon patches.

Currently, because of the not-changed cmd lock, you can't query the jobs
as long as a job is running - you have to rm the cmd lock and then you
can query the jobs.

Reviewed-by: imsnah

This patch adds a very in-progress master daemon. This needs to be
launched manually, does not background itself, but can be used for
opcode execution.

Also parts of this code should be moved to luxi.py.

Reviewed-by: ultrotter