Reviewed-by: iustinp

This (big) patch moves the hypervisor type from the cluster to the
instance level; the cluster attribute remains as the default hypervisor,
and will be renamed accordingly in a next patch. The cluster also gains
the ‘enable_hypervisors’ attribute, and instances can be created with
any of the enabled ones (no provision yet for changing that attribute).

The many many changes in the rpc/backend layer are due to the fact that
all backend code read the hypervisor from the local copy of the config,
and now we have to send it (either in the instance object, or as a
separate parameter) for each function.

The node list by default will list the node free/total memory for the
default hypervisor, a new flag to it should exist to select another
hypervisor. Instance list has a new field, hypervisor, that shows the
instance hypervisor. Cluster verify runs for all enabled hypervisor
types.

The new FIXMEs are related to IAllocator, since now the node
total/free/used memory counts are wrong (we can't reliably compute the
free memory).

Reviewed-by: imsnah

This patch changes formatting and the DRBD shared secret details, and
adds master daemon socket details to the security doc.

Reviewed-by: imsnah

Reviewed-by: imsnah

Reviewed-by: imsnah

This is just some additions of not-yet-mentioned docs.

Reviewed-by: ultrotter

Merged r1777 from branches/ganeti/ganeti-1.2

Reviewed-by: imsnah

Reviewed-by: imsnah

Currently the call_instance_migrate call only passes the instance name;
we need to pass the whole object for the hypervisor_type changes (all
the other individual instance rpc calls already pass the instance
object).

Reviewed-by: imsnah

Reviewed-by: iustinp

Reviewed-by: ultrotter

Background: when we have multiple jobs in the queue (more than just a
few), many of the jobs (up to the number of threads) will be in state
'running', although many of them could be actually blocked, waiting for
some locks. This is not good, as one cannot easily see what is
happening.

The patch extends the opcode/job possible statuses with another one,
waiting, which shows that the LU is in the acquire locks phase. The
mechanism for doing so is simple, we initialize (in the job queue) the
opcode with OP_STATUS_WAITLOCK, and when the processor is ready to give
control to the LU's Exec, it will call a notifier back into the
_JobQueueWorker that sets the opcode status to OP_STATUS_RUNNING (with
the proper queue locking). Because this mechanism does not save the job,
all opcodes on disk will be in status WAITLOCK and not RUNNING anymore,
so we also change the load sequence to consider WAITLOCK as RUNNING.

With the patch applied, creating in parallel (via burnin) five instances
on a five node cluster shows that only two are executing, while three
are waiting for locks.

Reviewed-by: imsnah

Reviewed-by: imsnah

Reviewed-by: imsnah

This patch adds a new luxi call that implements auto-archiving of jobs
older than a certain age (or -1 for all completed jobs), and the gnt-job
command that makes use of this (with 'all' for -1).

Reviewed-by: imsnah

This function will be used for auto-archiving jobs via the command line.
The function is pretty simple, we only support up to weeks since months
and higher are not 'precise' entities, and dealing with them would
require us to start using calendar functions.

Reviewed-by: imsnah

Currently there are three function in backend that need the cluster name
in order to instantiate an SshRunner. The patch changes these to get the
cluster name from the master in the rpc call; once the multi-hypervisor
change is implemented, then very few places in which we need the SCR
remain in the backend.

Reviewed-by: killerfoxi, imsnah

Since the objects read from the config file are passed to the various
threads, it's unsafe to re-read the config file (and throw away
ConfigWriter._config_data). As such, we disable the re-reading of the
file (since now the master is the owner the file, it makes not sense to
re-read it), and any modifications to the file must be done offline,
otherwise they will be overwritten.

Reviewed-by: imsnah

Reviewed-by: iustinp

This patch introduces a simple framework for executing jobs in parallel
in burnin (the ExecJobSet function) and the "--parallel" command line
flag.

The patch also changes the instance creation to run in parallel when the
above flag is given. Error handling/instance removal is currently flacky
with this options if there are errors in the instance creation.

We also modify burnin to reuse a single client.

Reviewed-by: imsnah

In case the job object doesn't have a timestamp (which is a separate
issue), the listing should not break. We fix this by changing the
FormatTimstamp function itself to return '?' in case the timestamp
doesn't look good (note that it still can break if non-integers are
returned, but this is unlikely).

Reviewed-by: imsnah

Since our locks are not gathered nicely, we can have jobs that are
actually blocking on locks (parallel burnin shows this), so at least we
need to increase the number of threads above the usual number of jobs we
could have in a such a case.

Reviewed-by: imsnah

Reviewed-by: iustinp

More places actually use the SshRunner than just the gnt-cluster
commands.

Reviewed-by: ultrotter

Currently the SshRunner uses a SimpleConfigReader instance, however this
is not best. We change it to use the cluster name directly (and its
constructor now takes this as parameter, instead of SCR), and its
callers are change to pass the name directly.

As a consequence, we can now remove the initialization of SCR in
gnt-cluster (copyfile and command), and instead we query the master for
the cluster name).

Reviewed-by: imsnah

It may need further updates, but here's a start.

Reviewed-by: ultrotter

The ssconf migration left this out.

Reviwed-by: imsnah,ultrotter

The actual location is /export, not /exports.

Reviewed-by: ultrotter

docbook-wrapper had the names for the docbook2* programs hardcoded. This
patch changes Makefile.am and the wrapper script to pass them via
another argument.

Another issue where rapi.in was built before rapi-resources.sgml is
also fixed.

Reviewed-by: iustinp

Reviewed-by: iustinp

Remove leftovers from ssconf.

Reviewed-by: iustinp

sstore is no longer used in LUs.

Reviewed-by: iustinp

Use simpleconfig instead of ssconf.

Reviewed-by: iustinp

Use RPC calls instead of ssconf.

Reviewed-by: iustinp

Replace ssconf with utility functions.

Reviewed-by: iustinp

Replace ssconf with configuration.

Reviewed-by: iustinp

Replace ssconf with configuration.

Reviewed-by: iustinp

Replacing ssconf with configuration. Cluster rename is broken and stays
that way.

Reviewed-by: iustinp

Get rid of ssconf and convert to configuration instead.

Reviewed-by: iustinp

Replacing ssconf with utility functions.

Reviewed-by: iustinp