This will be needed for master failover. If we don't have a valid queue
directory, we need to reinitialize it, but we should keep the existing
serial number.

As such, we abstract the reading of the serial and if we find a valid
serial, we do not reset it.

Reviewed-by: imsnah

This was a TODO for 2.0

Reviewed-by: iustinp

Grab a lock for the instance we're working on, and update its params.

Reviewed-by: iustinp

When we set the instance params we're not adding a new instance, but
just updating an existing one, so why using AddInstance?

Reviewed-by: iustinp

For ConnectConsole we just need to lock the instance we're connecting
to. We make a few rpcs to its primary node, but node daemons can now
handle multiple queries and nodes cannot be removed till they have
instances on them anyway. Note that since we return the ssh command, and
that's executed outside of the ganeti daemon, without any locks held,
the instance can then be subject to operations while we're connected to
it, but that was the previous behavior as well.

Reviewed-by: iustinp

LUs that take an instance name as input and need to expand its name and
lock it can use it to simplify their ExpandNames call. Possibly, and
_ExpandAndLockNode will come as well.

Reviewed-by: iustinp

LUQueryClusterInfo and LUDumpClusterConfig can be made concurrent and
don't need to acquire any locks. In fact they don't interact with the
cluster at all, but just with its configuration, which is thread-safe by
design.

Reviewed-by: iustinp

Two top level definitions were separated only by one empty line.
Fixing this.

Reviewed-by: imsnah

Reviewed-by: imsnah

Not passing the argument means it has the value None. Iterating None
doesn't work:
  >>> "123" in None
  Traceback (most recent call last):
    File "<stdin>", line 1, in ?
  TypeError: iterable argument required

Hence I rename it to "exclude" instead of "exceptions", which may be
confusing, and make it mandatory. If one wants to clean all cache
entries, an empty list can be passed.

Reviewed-by: iustinp

Reviewed-by: iustinp

This patch splits the single threading.Condition object used in the
worker pool for synchronization into three.

- worker_to_pool: Notified if a worker wants to notify the pool
- pool_to_worker: Notified if the pool wants to notify a single
  or all workers
- pool_to_pool: Used for synchronization in Quiesce

Reviewed-by: ultrotter

This also fixes a TODO added by ultrotter by killing the parent
process when QuitGanetiException is raised.

Reviewed-by: ultrotter

Reviewed-by: ultrotter

This signal handler class abstracts some of the code previously
used in other places. It also uninstalls its handler when Reset()
is called or the class is destructed, thereby restoring the
previous behaviour.

Reviewed-by: iustinp

Reviewed-by: imsnah

Reviewed-by: imsnah

Create a new ForkingHTTPServer in ganeti-noded by deriving both from
NodeDaemonHttpServer and ForkingMixin. This will allow us to process
concurrent requests.

Reviewed-by: imsnah

Reviewed-by: imsnah

Reviewed-by: imsnah

The patch adds the apidoc target and the epydoc config file for it. Note
that this is for epydoc 3.0 and that it will put the docs into
./doc/api/.

The patch also adds a new .gitignore rule for the auto-generated rapi
fragment.

Reviewed-by: imsnah

Since we don't have for now a job definition object anymore, we rename
this class to BaseOpCode. It's still useful (and not merged with OpCode)
since it holds all the 'pure' logic (no custom field handling, etc.)
whereas OpCode holds opcode specific data (OP_ID handling, etc).

The patch also fixes the module's docstring.

Reviewed-by: imsnah

Since the IDs are integers, we can simply sort them.

Reviewed-by: imsnah

The function to stop a worker pool is TerminateWorkers(), not Shutdown().

Reviewed-by: iustinp

I broke gnt-backup in rev 1035, sorry :(

Reviewed-by: imsnah

Reusing threads instead of starting one for each request is more efficient.

Reviewed-by: iustinp

Note that since RAPI itself doesn't use luxi.Client yet, nothing works,
but at least it can startup now.

Reviewed-by: imsnah

This was forgot on the forward-porting of RAPI.

Reviewed-by: imsnah

Currently, the feedback_fn argument to SubmitOpCode is no longer used.
We still need it in burnin, so we re-enable it by making the code call
that function with the msg argument in case feedback_fn is callable. The
patch also modifies burnin to accept the new argument format (msg is not
a triple instead of a string).

The patch also removes the ‘proc’ argument as it's obsolete; instead we
can accept a luxi.Client instance (noone uses this right now).

Reviewed-by: ultrotter

This patch contains a raw version for fixing feedback_fn.

The new mechanism works as follows:
  - instead of a per-Processor feedback_fn, there's one for each
    ExecOpCode, so that feedback for different opcodes go via possibly
    different functions
  - each _QueuedOpCode gets a message buffer, a method for adding
    feedback and a method for retrieving (parts of) the feedback
  - the _QueuedJob object gets a new attribute that is equal to the
    index of the currently executing opcode
  - job queries get an extra parameter called 'ticker' that will return
    the latest message on the current executing opcode
  - the cli.py job completion poll will show the new status if different
    from the old one

Of course, quick messages will be lost, as currently only the latest one
is available. Also changes between opcodes are not represented at all.

Reviewed-by: imsnah

This patch adds a caching mechanisms to the JobStorage. Note that is
does not make the memory cache authoritative.

The algorithm is:
  - all jobs loaded from disks are entered in the cache
  - all new jobs are entered in the cache
  - at each job save (in UpdateJobUnlocked), jobs which are not
    executing or queued are removed from the cache

The end effect is that running jobs will always be in the cache (which
will fix the opcode log changes) and finished jobs will be kept for a
while in the cache after being loaded.

Reviewed-by: imsnah

The job ID returned must be an integer (and the regex enforces that),
but we didn't convert it manually.

Reviewed-by: imsnah

Currently some of the functions in JobStorage work with filenames (which
is an implementation detail and should only be used when dealing with
the storage) and not with job IDs. We need to change this in order to
implement a job cache.

Reviewed-by: ultrotter

It's not perfect and it's not finished, but it's a start.

- Serial number is read only once, but written on each update
- Jobs are kept only on disk (caching will be implemented)

Reviewed-by: iustinp

The patch also switches some of the exception logs to use
logging.exception (and therefore the log message will have a diferent
format).

(Note that this might not be a good choice in all cases, though)

Reviewed-by: imsnah

This patch (for trunk) adds the PID to all daemon logs.

Reviewed-by: imsnah

As a nice side-effect this also fixes "make distcheck".

The way used to include the generated rapi-resources.sgml file only
works if it's built in the same directory. That's not the case during
"make distcheck". The patch changes the sed script used for variables
to include the file using its absolute path.

Reviewed-by: iustinp

This is the same fix as for GetVolumeList.

I've checked manually and all other places that call lvm commands are
already checking the output validity in terms of correct number of
fields.

Reviewed-by: ultrotter

Reviewed-by: imsnah

Sometimes ‘lvs’ can spit error messages on stdout, even when one wants
to parse the output:
...
Inconsistent metadata copies found - updating to use version 2776
...

So we need to validate the output to guard against such cases.

The patch converts the split on the separater to match against a regex
and extract the fields via groups. The original separator choice is a
bad one now :(

Reviewed-by: imsnah