This patch splits the single threading.Condition object used in the
worker pool for synchronization into three.

- worker_to_pool: Notified if a worker wants to notify the pool
- pool_to_worker: Notified if the pool wants to notify a single
  or all workers
- pool_to_pool: Used for synchronization in Quiesce

Reviewed-by: ultrotter

This also fixes a TODO added by ultrotter by killing the parent
process when QuitGanetiException is raised.

Reviewed-by: ultrotter

Reviewed-by: ultrotter

This signal handler class abstracts some of the code previously
used in other places. It also uninstalls its handler when Reset()
is called or the class is destructed, thereby restoring the
previous behaviour.

Reviewed-by: iustinp

Reviewed-by: imsnah

Reviewed-by: imsnah

Create a new ForkingHTTPServer in ganeti-noded by deriving both from
NodeDaemonHttpServer and ForkingMixin. This will allow us to process
concurrent requests.

Reviewed-by: imsnah

Reviewed-by: imsnah

Reviewed-by: imsnah

The patch adds the apidoc target and the epydoc config file for it. Note
that this is for epydoc 3.0 and that it will put the docs into
./doc/api/.

The patch also adds a new .gitignore rule for the auto-generated rapi
fragment.

Reviewed-by: imsnah

Since we don't have for now a job definition object anymore, we rename
this class to BaseOpCode. It's still useful (and not merged with OpCode)
since it holds all the 'pure' logic (no custom field handling, etc.)
whereas OpCode holds opcode specific data (OP_ID handling, etc).

The patch also fixes the module's docstring.

Reviewed-by: imsnah

Since the IDs are integers, we can simply sort them.

Reviewed-by: imsnah

The function to stop a worker pool is TerminateWorkers(), not Shutdown().

Reviewed-by: iustinp

I broke gnt-backup in rev 1035, sorry :(

Reviewed-by: imsnah

Reusing threads instead of starting one for each request is more efficient.

Reviewed-by: iustinp

Note that since RAPI itself doesn't use luxi.Client yet, nothing works,
but at least it can startup now.

Reviewed-by: imsnah

This was forgot on the forward-porting of RAPI.

Reviewed-by: imsnah

Currently, the feedback_fn argument to SubmitOpCode is no longer used.
We still need it in burnin, so we re-enable it by making the code call
that function with the msg argument in case feedback_fn is callable. The
patch also modifies burnin to accept the new argument format (msg is not
a triple instead of a string).

The patch also removes the ‘proc’ argument as it's obsolete; instead we
can accept a luxi.Client instance (noone uses this right now).

Reviewed-by: ultrotter

This patch contains a raw version for fixing feedback_fn.

The new mechanism works as follows:
  - instead of a per-Processor feedback_fn, there's one for each
    ExecOpCode, so that feedback for different opcodes go via possibly
    different functions
  - each _QueuedOpCode gets a message buffer, a method for adding
    feedback and a method for retrieving (parts of) the feedback
  - the _QueuedJob object gets a new attribute that is equal to the
    index of the currently executing opcode
  - job queries get an extra parameter called 'ticker' that will return
    the latest message on the current executing opcode
  - the cli.py job completion poll will show the new status if different
    from the old one

Of course, quick messages will be lost, as currently only the latest one
is available. Also changes between opcodes are not represented at all.

Reviewed-by: imsnah

This patch adds a caching mechanisms to the JobStorage. Note that is
does not make the memory cache authoritative.

The algorithm is:
  - all jobs loaded from disks are entered in the cache
  - all new jobs are entered in the cache
  - at each job save (in UpdateJobUnlocked), jobs which are not
    executing or queued are removed from the cache

The end effect is that running jobs will always be in the cache (which
will fix the opcode log changes) and finished jobs will be kept for a
while in the cache after being loaded.

Reviewed-by: imsnah

The job ID returned must be an integer (and the regex enforces that),
but we didn't convert it manually.

Reviewed-by: imsnah

Currently some of the functions in JobStorage work with filenames (which
is an implementation detail and should only be used when dealing with
the storage) and not with job IDs. We need to change this in order to
implement a job cache.

Reviewed-by: ultrotter

It's not perfect and it's not finished, but it's a start.

- Serial number is read only once, but written on each update
- Jobs are kept only on disk (caching will be implemented)

Reviewed-by: iustinp

The patch also switches some of the exception logs to use
logging.exception (and therefore the log message will have a diferent
format).

(Note that this might not be a good choice in all cases, though)

Reviewed-by: imsnah

This patch (for trunk) adds the PID to all daemon logs.

Reviewed-by: imsnah

As a nice side-effect this also fixes "make distcheck".

The way used to include the generated rapi-resources.sgml file only
works if it's built in the same directory. That's not the case during
"make distcheck". The patch changes the sed script used for variables
to include the file using its absolute path.

Reviewed-by: iustinp

This is the same fix as for GetVolumeList.

I've checked manually and all other places that call lvm commands are
already checking the output validity in terms of correct number of
fields.

Reviewed-by: ultrotter

Reviewed-by: imsnah

Sometimes ‘lvs’ can spit error messages on stdout, even when one wants
to parse the output:
...
Inconsistent metadata copies found - updating to use version 2776
...

So we need to validate the output to guard against such cases.

The patch converts the split on the separater to match against a regex
and extract the fields via groups. The original separator choice is a
bad one now :(

Reviewed-by: imsnah

Reviewed-by: iustinp

Some of the code is adopted from the 1.2 branch
(lib/rapi/RESTHTTPServer.py). This code can be used as a base for the
various HTTP servers in Ganeti.

Reviewed-by: iustinp

"gnt-job list" was broken after my recent changes in the RPC
between clients and the master. This patch makes it work again.

Reviewed-by: iustinp

Reviewed-by: iustinp

Apparently I forgot to this code when removing the rest.

Reviewed-by: iustinp

Reviewed-by: iustinp

Right now, the queued opcode doesn't have a lock, and instead relies on
the parent QueuedJob's lock.

This is not good for logging feedback, so it's better to have a lock for
each queuedopcode.

Reviewed-by: ultrotter

This is just a simple, hardcoded decorator for object methods needing
synchronization on the _lock instance attribute.

Reviewed-by: ultrotter

Reviewed-by: iustinp

Currently, in debug mode, both the logfile handler and the stderr
handler will log debug messages. Since the stderr is redirected to the
same logfile (to catch non-logged errors), it means log entries are
doubled.

The patch adds an extra parameter to the logger.SetupDaemon() function
that allows disabling of the stderr logging. The master and node daemon
will use this to enable stderr logging only when running in foreground.

Reviewed-by: imsnah

The patch adds some more logging to the node daemon:

- log methods at beggining not only at the end
- log method parameters (they are very verbose, but useful)

A separate change is to initialize the global variable in the global
scope, not inside main().

Reviewed-by: imsnah