We have a problem with the current model of combining instance lists
from multiple hypervisors: we don't allow duplicates, but "xm list"
gives the same output for both pvm and hvm. This is a lack in the actual
xen hypervisor implementation/split between pvm and hvm, but for now we
implement a weak workaround: identical instance params will be allowed,
and merged. This breaks because there is a delta in listing, and should
be treated as temporary workaround only.

Note that there are two cases for duplicate instance: the above one (xen
is the same, whether pvm or hvm), and the other case, the real error,
when we have two different hypervisors reporting the same instance name.
The latter case needs to be handled better (not by refusing to list the
instances in the backend).

Reviewed-by: ultrotter

We fill the instance hvparams with cluster defaults, and send a modified
dict over the wire to the node methods - they don't know anything about
cluster defaults versus instance parameters.

Reviewed-by: ultrotter

The newly-added node-specific ValidateParams hypervisor method is
exported over RPC, using the semi-standard (success, message) return
value. Multi-node call, so that we call on both primary and secondary at
once.

Reviewed-by: ultrotter

This patch adds the new methods CheckParameterSyntax and the
ValidateParameters in the base and xen hypervisors. It also changes the
xen hypervisor to use the new hvparams field on instance.

Reviewed-by: imsnah

This was a leftover from the early days of the remote API in Ganeti 1.2.

Reviewed-by: ultrotter

This patch adds paramter removal in SplitKeyVal, by prefixing a
value-less key with "-"; this is needed in resetting parameters back to
cluster defaults, but care must be applied now that None can come from
the parser.

Reviewed-by: imsnah

This patch adds the instance and cluster be/hvparams attributes, and the
Fill* methods on the cluster object. Nothing else is changes, except
that these attributes exist now on the objects.

Reviewed-by: ultrotter

Since we don't want the string values of the parameters (e.g.
“kernel_path”) spread over the code, we introduce constants for these.

Reviewed-by: ultrotter,schreiberal

This fixes:
  - whitespace change, double lines between methods
  - duplication of call_upload_file, introduced by mistake in rev 1795
    and which went undetected because of the many changes in that ref
    (only diff -b shows it clearly)
  - call_instance_info didn't pass the hypervisor name parameter, but
    the backend requires it

Reviewed-by: ultrotter

For the new 2.0-style command line options, we need to parse strings of
the type:
  ident:key=val[,...]
and
  key=val[,...]

This patch adds two new option builders for these two, which return
(ident, {key=val,}) and {key=val,} for the above two formats. It also
handles specially constructs of type “key” (val is set to True) and
“no_key” (val is set to False, and the ‘no_’ prefix is stripped).

Reviewed-by: imsnah

Currently, we check if we have a given ip address (i.e. it's alive on
one of our interfaces) but manually calling TcpPing(source=localhost).
This works, but having it spread all over the code makes it hard to
change the implementation.

The patch abstracts this into a separate utils.OwnIpAddress(addr)
function. We add a rpc call for it, which we use instead of the
(single-use of) call_node_tcp_ping. We leave node_tcp_ping in, as seems
useful and eventually it should be removed in a separate patch.

Reviewed-by: imsnah

Reviewed-by: iustinp

Allow multiple api versions in an OS. This is according to the OS API
changes design doc, by which an OS can support multiple versions of the
Ganeti API and if one is supported by Ganeti it will work. Since up to
version 5 of the API mandates an OS could support only one version, this
change is retrocompatible with it and requires no version bump up.

Reviewed-by: iustinp

When calling node_verify leads to an error _VerifyNodes tries to iterate
over a non-sequence. Catch the error before and avoid this from
happening.

Reviewed-by: iustinp

This patch adds another implementation of an HTTP server. It's
based on code of Python's BaseHTTPServer, from both version
2.4 and 3k. In the future we can write code to decide whether
we should fork for a request or not. Keep-alive is not supported.

Reviewed-by: iustinp

This mainloop can be used in daemons like ganeti-noded.

Reviewed-by: iustinp

This big patch changes the call model used in internode-rpc from
standalong function calls in the rpc module to via a RpcRunner class,
that holds all the methods. This can be used in the future to enable
smarter processing in the RPC layer itself (some quick examples are not
setting the DiskID from cmdlib code, but only once in each rpc call,
etc.).

There are a few RPC calls that are made outside of the LU code, and
these calls are left as staticmethods, so they can be used without a
class instance (which requires a ConfigWriter instance).

Reviewed-by: imsnah

This patch is a cleanup of the standalone functions in cmdlib. Many of
them too as argument a ConfigWriter instance, but some also took other
parameters from the lu (e.g. proc), and in the future, if we want to
also pass the RpcRunner, we would have to add yet another parameter.

One option is to make all these methods of top-level LogicalUnit class.
I took another approach, and made (almost) all these functions take as
first parameter the lu instance. It's like methods, just not declared
under LogicalUnit.

Reviewed-by: imsnah

Indentation in bootstrap was wrong and some names in cmdlib.py were not
right.

Reviewed-by: imsnah

This check can be done earlier, in ExpandNames, and is needed here for
the hypervisor parameter check.

Reviewed-by: ultrotter

Reviewed-by: ultrotter

Since in 2.0 the user will possibly have more interaction with the
hypervisor names, we sanitize them by removing the version numbers
(the version can be a prerequisite for the ganeti installation, we
shouldn't document it in variable names).

Reviewed-by: schreiberal

Reviewed-by: iustinp

This (big) patch moves the hypervisor type from the cluster to the
instance level; the cluster attribute remains as the default hypervisor,
and will be renamed accordingly in a next patch. The cluster also gains
the ‘enable_hypervisors’ attribute, and instances can be created with
any of the enabled ones (no provision yet for changing that attribute).

The many many changes in the rpc/backend layer are due to the fact that
all backend code read the hypervisor from the local copy of the config,
and now we have to send it (either in the instance object, or as a
separate parameter) for each function.

The node list by default will list the node free/total memory for the
default hypervisor, a new flag to it should exist to select another
hypervisor. Instance list has a new field, hypervisor, that shows the
instance hypervisor. Cluster verify runs for all enabled hypervisor
types.

The new FIXMEs are related to IAllocator, since now the node
total/free/used memory counts are wrong (we can't reliably compute the
free memory).

Reviewed-by: imsnah

Currently the call_instance_migrate call only passes the instance name;
we need to pass the whole object for the hypervisor_type changes (all
the other individual instance rpc calls already pass the instance
object).

Reviewed-by: imsnah

Background: when we have multiple jobs in the queue (more than just a
few), many of the jobs (up to the number of threads) will be in state
'running', although many of them could be actually blocked, waiting for
some locks. This is not good, as one cannot easily see what is
happening.

The patch extends the opcode/job possible statuses with another one,
waiting, which shows that the LU is in the acquire locks phase. The
mechanism for doing so is simple, we initialize (in the job queue) the
opcode with OP_STATUS_WAITLOCK, and when the processor is ready to give
control to the LU's Exec, it will call a notifier back into the
_JobQueueWorker that sets the opcode status to OP_STATUS_RUNNING (with
the proper queue locking). Because this mechanism does not save the job,
all opcodes on disk will be in status WAITLOCK and not RUNNING anymore,
so we also change the load sequence to consider WAITLOCK as RUNNING.

With the patch applied, creating in parallel (via burnin) five instances
on a five node cluster shows that only two are executing, while three
are waiting for locks.

Reviewed-by: imsnah

This patch adds a new luxi call that implements auto-archiving of jobs
older than a certain age (or -1 for all completed jobs), and the gnt-job
command that makes use of this (with 'all' for -1).

Reviewed-by: imsnah

This function will be used for auto-archiving jobs via the command line.
The function is pretty simple, we only support up to weeks since months
and higher are not 'precise' entities, and dealing with them would
require us to start using calendar functions.

Reviewed-by: imsnah

Currently there are three function in backend that need the cluster name
in order to instantiate an SshRunner. The patch changes these to get the
cluster name from the master in the rpc call; once the multi-hypervisor
change is implemented, then very few places in which we need the SCR
remain in the backend.

Reviewed-by: killerfoxi, imsnah

Since the objects read from the config file are passed to the various
threads, it's unsafe to re-read the config file (and throw away
ConfigWriter._config_data). As such, we disable the re-reading of the
file (since now the master is the owner the file, it makes not sense to
re-read it), and any modifications to the file must be done offline,
otherwise they will be overwritten.

Reviewed-by: imsnah

In case the job object doesn't have a timestamp (which is a separate
issue), the listing should not break. We fix this by changing the
FormatTimstamp function itself to return '?' in case the timestamp
doesn't look good (note that it still can break if non-integers are
returned, but this is unlikely).

Reviewed-by: imsnah

Since our locks are not gathered nicely, we can have jobs that are
actually blocking on locks (parallel burnin shows this), so at least we
need to increase the number of threads above the usual number of jobs we
could have in a such a case.

Reviewed-by: imsnah

More places actually use the SshRunner than just the gnt-cluster
commands.

Reviewed-by: ultrotter

Currently the SshRunner uses a SimpleConfigReader instance, however this
is not best. We change it to use the cluster name directly (and its
constructor now takes this as parameter, instead of SCR), and its
callers are change to pass the name directly.

As a consequence, we can now remove the initialization of SCR in
gnt-cluster (copyfile and command), and instead we query the master for
the cluster name).

Reviewed-by: imsnah

The ssconf migration left this out.

Reviwed-by: imsnah,ultrotter

Remove leftovers from ssconf.

Reviewed-by: iustinp

sstore is no longer used in LUs.

Reviewed-by: iustinp

Replace ssconf with configuration.

Reviewed-by: iustinp

Replacing ssconf with configuration. Cluster rename is broken and stays
that way.

Reviewed-by: iustinp

Get rid of ssconf and convert to configuration instead.

Reviewed-by: iustinp