Currently, the feedback_fn argument to SubmitOpCode is no longer used.
We still need it in burnin, so we re-enable it by making the code call
that function with the msg argument in case feedback_fn is callable. The
patch also modifies burnin to accept the new argument format (msg is not
a triple instead of a string).

The patch also removes the ‘proc’ argument as it's obsolete; instead we
can accept a luxi.Client instance (noone uses this right now).

Reviewed-by: ultrotter

This patch contains a raw version for fixing feedback_fn.

The new mechanism works as follows:
  - instead of a per-Processor feedback_fn, there's one for each
    ExecOpCode, so that feedback for different opcodes go via possibly
    different functions
  - each _QueuedOpCode gets a message buffer, a method for adding
    feedback and a method for retrieving (parts of) the feedback
  - the _QueuedJob object gets a new attribute that is equal to the
    index of the currently executing opcode
  - job queries get an extra parameter called 'ticker' that will return
    the latest message on the current executing opcode
  - the cli.py job completion poll will show the new status if different
    from the old one

Of course, quick messages will be lost, as currently only the latest one
is available. Also changes between opcodes are not represented at all.

Reviewed-by: imsnah

This patch adds a caching mechanisms to the JobStorage. Note that is
does not make the memory cache authoritative.

The algorithm is:
  - all jobs loaded from disks are entered in the cache
  - all new jobs are entered in the cache
  - at each job save (in UpdateJobUnlocked), jobs which are not
    executing or queued are removed from the cache

The end effect is that running jobs will always be in the cache (which
will fix the opcode log changes) and finished jobs will be kept for a
while in the cache after being loaded.

Reviewed-by: imsnah

The job ID returned must be an integer (and the regex enforces that),
but we didn't convert it manually.

Reviewed-by: imsnah

Currently some of the functions in JobStorage work with filenames (which
is an implementation detail and should only be used when dealing with
the storage) and not with job IDs. We need to change this in order to
implement a job cache.

Reviewed-by: ultrotter

It's not perfect and it's not finished, but it's a start.

- Serial number is read only once, but written on each update
- Jobs are kept only on disk (caching will be implemented)

Reviewed-by: iustinp

The patch also switches some of the exception logs to use
logging.exception (and therefore the log message will have a diferent
format).

(Note that this might not be a good choice in all cases, though)

Reviewed-by: imsnah

This patch (for trunk) adds the PID to all daemon logs.

Reviewed-by: imsnah

As a nice side-effect this also fixes "make distcheck".

The way used to include the generated rapi-resources.sgml file only
works if it's built in the same directory. That's not the case during
"make distcheck". The patch changes the sed script used for variables
to include the file using its absolute path.

Reviewed-by: iustinp

This is the same fix as for GetVolumeList.

I've checked manually and all other places that call lvm commands are
already checking the output validity in terms of correct number of
fields.

Reviewed-by: ultrotter

Reviewed-by: imsnah

Sometimes ‘lvs’ can spit error messages on stdout, even when one wants
to parse the output:
...
Inconsistent metadata copies found - updating to use version 2776
...

So we need to validate the output to guard against such cases.

The patch converts the split on the separater to match against a regex
and extract the fields via groups. The original separator choice is a
bad one now :(

Reviewed-by: imsnah

Reviewed-by: iustinp

Some of the code is adopted from the 1.2 branch
(lib/rapi/RESTHTTPServer.py). This code can be used as a base for the
various HTTP servers in Ganeti.

Reviewed-by: iustinp

"gnt-job list" was broken after my recent changes in the RPC
between clients and the master. This patch makes it work again.

Reviewed-by: iustinp

Reviewed-by: iustinp

Apparently I forgot to this code when removing the rest.

Reviewed-by: iustinp

Reviewed-by: iustinp

Right now, the queued opcode doesn't have a lock, and instead relies on
the parent QueuedJob's lock.

This is not good for logging feedback, so it's better to have a lock for
each queuedopcode.

Reviewed-by: ultrotter

This is just a simple, hardcoded decorator for object methods needing
synchronization on the _lock instance attribute.

Reviewed-by: ultrotter

Reviewed-by: iustinp

Currently, in debug mode, both the logfile handler and the stderr
handler will log debug messages. Since the stderr is redirected to the
same logfile (to catch non-logged errors), it means log entries are
doubled.

The patch adds an extra parameter to the logger.SetupDaemon() function
that allows disabling of the stderr logging. The master and node daemon
will use this to enable stderr logging only when running in foreground.

Reviewed-by: imsnah

The patch adds some more logging to the node daemon:

- log methods at beggining not only at the end
- log method parameters (they are very verbose, but useful)

A separate change is to initialize the global variable in the global
scope, not inside main().

Reviewed-by: imsnah

Reviewed-by: imsnah

... as it was intended from the beggining, but by mistake left in the
top run dir.

Reviewed-by: ultrotter

Currently, the 'public' functions of bdev (FindDevice and
AttachOrAssemble) will call the Attach() method right after class
instantiation.

But the constructor itself calls this function, and therefore we have
duplicate Attach() calls (which are not cheap at all).

The patch introduces a new 'attached' instance attribute that tells if
the last Attach() was successful. The public functions reuse this so
that we only do the minimum required number of calls.

Reviewed-by: imsnah

This does not enhance in any way the messages; it just switches to the
new module.

Reviewed-by: imsnah

The patch also logs all commands executed from RunCmd when we are at
debug level.

Reviewed-by: imsnah

This removes (hopefully) all traces of the old locking functions and
uses.

Reviewed-by: imsnah

Reviewed-by: iustinp

- Introduce abstraction class on client side
- Use constants for method names
- Adopt legacy function SubmitOpCode to use it

Reviewed-by: iustinp

- Use constants for dict entries
- Handle exceptions on server side
- Rename client function to CallMethod to match server side naming

Reviewed-by: iustinp

Reviewed-by: iustinp

Reviewed-by: iustinp

This is to discourage even more whoever may think that this requirement
is not really useful and can be lifted, and to at least know where it's
used before trying to break it.

Reviewed-by: imsnah

In order to do so:
  - We set REQ_BGL to False
  - We implement ExpandNames

That's it, really.

Reviewed-by: iustinp

If we're running in a "new style" LU we may need some locks, as required
by the ExpandNames function, to be able to run. We'll walk up the lock
levels present in the needed_locks dictionary and acquire them, then run
the actual LU. LUs can release some or all the acquired locks, if they
want, before terminating, provided they update their needed_locks
dictionary appropriately, so that we know not to release a level if they
have already done so.

Reviewed-by: iustinp

New concurrent LUs will need to call ExpandNames so that any names
passed in by the user are canonicalized, and can be used by hooks,
locking and other parts of the code. This was done in CheckPrereq
before, but it's now splitted out, as it's needed for locking, which in
turn CheckPrereq needs. Old LUs can be converted gradually.

Reviewed-by: iustinp

This makes the try...finally code simplier, and helps adding a more
complex locking structure before the actual execution. It also fixes a
concurrency bug caused by the fact that write_count was read before
acquiring the BGL, and thus spurious config update hooks run could have
been triggered. This doesn't solve the issue of running config update
hooks for concurrent LUs.

Reviewed-by: iustinp

This test checks the LockSet behaviour when an empty list is passed.
The current behaviour is expected, but since this is a corner case,
we're safer to keep it under a check, and if we need a different one
monitor that everything is as we expect it to be.

Reviewed-by: imsnah