This patch adds the two attributes file_storage_dir and file_driver
to the CreateInstance opcode in the ImportExport method, since
otherwise it will fail with the file storage patches.

Reviewed-by: iustinp

When the allocator or a non-mirrored template are used the message shown by
burnin at instance creation time was wrong. Fixing it.

Reviewed-by: iustinp

Make burnin remove exports after importing the instance back.

Reviewed-by: iustinp

This patch also fixes the LUExportInstance Prereq docstring.

Reviewed-by: iustinp

This patch adds the possibility to execute instance renaming when
doing a cluster burnin. The optional parameter --rename takes
one 'spare' instance name.
Given this option it renames all given instances to the spare one
and back to their original name.

Reviewed-by: iustinp

Currently the replace_secondary mode is too restrictive. This patch
changes this to a general 'relocate' mode where the node(s) to be
changed are specified via a new key in the request dict ('nodes') so
that we can change any of the instance's nodes.

Note that for the relocate mode, len(nodes) == required_nodes, so the
required nodes field is redundant, but it is provided for consistency
with the allocate mode.

Reviewed-by: ultrotter

This patch allows the dumb allocator to perform the replace secondary
request type. This is done simply by not allowing the current instances
nodes as a valid selection, and using the existing instance data instead
of the request (which doesn't have the instance info when doing the
replace_secondary).

Reviewed-by: ultrotter

This patch adds the 'required_nodes' field in the request dict for the
iallocator.

This means that the handmade-checks in the create instance can be
simplified, and that the dumb allocator can be made simple. Therefore
the patch also modifies it.

The patch also sends the disk_space_total to the script in realocate
mode and a small fix for showing errors (include stderr too).

Reviewed-by: ultrotter

This patch adds iallocator support to burnin, currently only in instance
creation. This means that, depending on the external algorithm, not all
nodes might get instances.

The patch also restricts node selection and iallocator options together.

Reviewed-by: ultrotter

This patch moves all the iallocator function into a separate class that
is then somewhat easier to use. It doesn't bring any new functionality.

The patch also changes the way the iallocator is called - the
OpTestAllocator opcode is no longer needed, and all its parameters
should be passed directly to the IAllocator constructor.

Reviewed-by: ultrotter

Reviewed-by: ultrotter

This patch makes the _SetInstanceStatus only change the status of the
instance (in the config file) if it's not already in the desired state.
This skips the increase of the config serial no without actual data
changes.

Reviewed-by: ultrotter

This patch creates a new method ConfigWriter._SetInstanceStatus that
consists of the common code between MarkInstanceUp and MarkInstanceDown.
These two methods become wrappers over the _SetInstanceStatus.

Reviewed-by: ultrotter

This simple patch adds a new module that holds the simplejson functions
for serialization/deserialization. This reduces the amount of redundant
code.

The patch also adds some normalizations to the json output:
  - the output text will always have an EOL as last char
  - extra spaces before EOL are removed

Reviewed-by: ultrotter

This allows us to readd a node after it failed and required a
reinstallation or replacement.

Reviewed-by: iustinp

Reviewed-by: amishchenko

This (final) patch allows the instance's nodes to be selected
automatically based on the passed allocator algorithm.

The patch changes the pnode opcode parameter from required to optional,
now either the pnode or the iallocator must be passed.

A possible improvement could be to organize all the _IAllocator
functions into a separate class, but that can come later and the current
version is functionally ok.

Reviewed-by: ultrotter

This patch reorders the checks in the instance create prereq so that all
checks and normalisations that are not node-dependent are done before
the node dependent checks.

This is done so that, after the instance-related opcode parameters are
checked and fixed, we can run the allocator and compute the primary (and
any secondary) nodes, and only then proceed with node-related checks.

Reviewed-by: ultrotter

This patch adds the paths for searching for instance allocators and
makes the LUTestAllocator code run the allocator and return the results
if the direction specified is 'out'. 'out' means that the opcode will
return the result of the allocator run, instead of the allocator input
file ('in').

The patch unifies all names to refer to 'iallocator' instead of plain
allocator.

The patch also adds an example allocator that can be used for testing
this new functionality.

Reviewed-by: ultrotter

In preparation for the introduction of automatic instance allocator,
this patch adds an allocator simulation opcode, that based on the input
parameters, will return either the input message to the allocator
(implemented) or the result of the allocator run (not yet implemented).

This allows algorithm tests against simulated allocations and the
current cluster state.

The patch adds the following:
  - a function that generates the generic cluster information for the
    allocator
  - a function that generates the 'new instance' information
  - a function that generates the 'replace_secondary' information

These three functions will be used by the allocator framework later to
generate the actual information for the external algorithms. Currently
we just return the json-serialized text.

Reviewed-by: imsnah

Reviewed-by: iustinp

Reviewed-by: iustinp

This patch does two things:
- adjusts burnin to work with the file storage backend patches
- adds the possibilty to do a burnin for file based instances

Reviewed-by: iustinp

This is currently hard-coded for the two drive case and will need to be
reworked for multi-disk support.

The patch is needed to support passing the total required size to the
iallocator interface.

Reviewed-by: ultrotter

When we release a shared lock if there are no exclusive waiter then the number
of shared waiters must be exactly equal to the ones scheduled to pass.

Reviewed-by: iustinp

Reviewed-by: ultrotter

Add a general way to skip some checks at cluster-verify time and make the N+1
memory redundancy check optional.

Reviewed-by: iustinp

For every node we check that we can host all the instances it's currently
secondary for belonging to the same primary. This ensures that if a node fails
all its instances can fit on their secondary node. The code only works when
failover is forced to go to the secondary node, and cannot go to an arbitrary
node in the cluster, which is the case in Ganeti 1.2.

Reviewed-by: iustinp

Save the instance config after we queried it in an instance_cfg dict.  This can
be used later by any function that wants it, without reloading it from the
configuration module. It will be used for N+1 memory resilience checking.

Reviewed-by: iustinp

The sisnt-by-pnode field contains all secondary instances of a node, grouped by
their primary node. This information allows us to see quickly if when a node
dies some of its instances cannot be started on their secondary node.

Reviewed-by: iustinp

With this patch node_info is changed to store information about which primary
and secondary instances are configured on a node. This information is useful to
check memory and disk allocation. A list of non-redundant instances is also
collected at this stage.

Reviewed-by: iustinp

During information gathering we collect information from call_node_info, and
then when we cycle trough the nodes add it into a node_info dict containing a
node's free memory and disk. This will be useful later to verify that the
cluster is N+1 redundant. The disk space is saved as well because it can be
useful for checks about disk space redundancy.

Reviewed-by: iustinp

Currently, the opcode DiagnoseOS is the only opcode that return a
structure of objects.OS (which is a custom class, and not a simple
python object) and furthermore all the processing of OS validity across
nodes is left to the clients of this opcode.

It would be more logical to have this opcode be similar to list
instances/nodes, in the sense that:
  - it should return a table of results
  - the fields in the table should be selectable

This patch does the above. The possible fields are:
  - name (os name)
  - valid (bool representing validity across all nodes)
  - node_status, which is a complicated structure required for ‘gnt-os
    diagnose’

With this patch, gnt-os list becomes a very simple iteration over the
list of results, filtering out non-valid ones. gnt-os diagnose is still
complicated, but no more than before.

The burnin tool has also been modified to work with the modified
results, and is simpler because of this (it only needs to know if an OS
is valid or not, not the per-node details).

Reviewed-by: imsnah

This patch enhances gnt-debug to be able to submit any opcodes. The
opcodes are input from a json file containing a list of opcodes.

This allows enhanced testing of opcodes until the other gnt-* commands
are converted to use the master daemon.

Reviewed-by: ultrotter

Currently the luxi.client.SubmitJob and Query methods return the unserialized
result without processing it at all. This patch changes this by adding a
'RequestException' error that is raised if the query itself or the
submission of the job failed, and (if not) returning only the 'result'
field from the message.

The patch also does processing on the result of a query if we queried
for jobs, as the 'op_list' field in the result has serialized opcodes
and we need the de-serialized.

Reviewed-by: ultrotter

This patch changes the definition of a job and introduces per-opcode
results.

First, the result and status fields of a job are condensed into a single
'status' attribute. Then, we introduce an opcode status and one result
list, that allow jobs to return values.

The gnt-job script is also modified to allow these new fields to be
queried.

Note that the patch changes the opcode field to op_list, and it changes
its return value from string to a list of (serialized) opcodes.

Reviewed-by: ultrotter

Based on the previous OS search code changes, we can now move the OS
search code into a generic look-for-file function in utils.py. This
means that the allocator code can use the same function.

Reviewed-by: ultrotter

Currently, the function backend._OSSearch() returns the (first) base dir
in which this OS can be found. Thereafter the full actual path to the OS
dir is built in the backend.OSFromDisk() function.

This patch changes this so that _OSSearch() always returns the full path
to the OS directory, and OSFromDisk uses that as returned (it will only
build it if it gets a base dir in the first place).

This patch is needed before we can abstract the _OSSearch into a generic
'look for file object' functionality that can be used for allocator
plugins search too.

Reviewed-by: ultrotter

Currently, the example scripts (the init.d and the cron script) do not
depend on the Makefile, so if configure is rerun with different
parameters these scripts are not rebuild.

The patch adds Makefile as a dependency so changes to installation paths
or other parameters trigger the rebuild.

Reviewed-by: imsnah

The list of instances passed to _VerifyInstance is the one coming from
self.cfg.GetInstanceList(). So there's no point, inside that function, in
checking whether the current instance is a member of that list. Moreover
orphaned instance verification is already done in a separate step.

Reviewed-by: imsnah