In preparation for the introduction of automatic instance allocator,
this patch adds an allocator simulation opcode, that based on the input
parameters, will return either the input message to the allocator
(implemented) or the result of the allocator run (not yet implemented).

This allows algorithm tests against simulated allocations and the
current cluster state.

The patch adds the following:
  - a function that generates the generic cluster information for the
    allocator
  - a function that generates the 'new instance' information
  - a function that generates the 'replace_secondary' information

These three functions will be used by the allocator framework later to
generate the actual information for the external algorithms. Currently
we just return the json-serialized text.

Reviewed-by: imsnah

Reviewed-by: iustinp

Reviewed-by: iustinp

This patch does two things:
- adjusts burnin to work with the file storage backend patches
- adds the possibilty to do a burnin for file based instances

Reviewed-by: iustinp

This is currently hard-coded for the two drive case and will need to be
reworked for multi-disk support.

The patch is needed to support passing the total required size to the
iallocator interface.

Reviewed-by: ultrotter

When we release a shared lock if there are no exclusive waiter then the number
of shared waiters must be exactly equal to the ones scheduled to pass.

Reviewed-by: iustinp

Reviewed-by: ultrotter

Add a general way to skip some checks at cluster-verify time and make the N+1
memory redundancy check optional.

Reviewed-by: iustinp

For every node we check that we can host all the instances it's currently
secondary for belonging to the same primary. This ensures that if a node fails
all its instances can fit on their secondary node. The code only works when
failover is forced to go to the secondary node, and cannot go to an arbitrary
node in the cluster, which is the case in Ganeti 1.2.

Reviewed-by: iustinp

Save the instance config after we queried it in an instance_cfg dict.  This can
be used later by any function that wants it, without reloading it from the
configuration module. It will be used for N+1 memory resilience checking.

Reviewed-by: iustinp

The sisnt-by-pnode field contains all secondary instances of a node, grouped by
their primary node. This information allows us to see quickly if when a node
dies some of its instances cannot be started on their secondary node.

Reviewed-by: iustinp

With this patch node_info is changed to store information about which primary
and secondary instances are configured on a node. This information is useful to
check memory and disk allocation. A list of non-redundant instances is also
collected at this stage.

Reviewed-by: iustinp

During information gathering we collect information from call_node_info, and
then when we cycle trough the nodes add it into a node_info dict containing a
node's free memory and disk. This will be useful later to verify that the
cluster is N+1 redundant. The disk space is saved as well because it can be
useful for checks about disk space redundancy.

Reviewed-by: iustinp

Currently, the opcode DiagnoseOS is the only opcode that return a
structure of objects.OS (which is a custom class, and not a simple
python object) and furthermore all the processing of OS validity across
nodes is left to the clients of this opcode.

It would be more logical to have this opcode be similar to list
instances/nodes, in the sense that:
  - it should return a table of results
  - the fields in the table should be selectable

This patch does the above. The possible fields are:
  - name (os name)
  - valid (bool representing validity across all nodes)
  - node_status, which is a complicated structure required for ‘gnt-os
    diagnose’

With this patch, gnt-os list becomes a very simple iteration over the
list of results, filtering out non-valid ones. gnt-os diagnose is still
complicated, but no more than before.

The burnin tool has also been modified to work with the modified
results, and is simpler because of this (it only needs to know if an OS
is valid or not, not the per-node details).

Reviewed-by: imsnah

This patch enhances gnt-debug to be able to submit any opcodes. The
opcodes are input from a json file containing a list of opcodes.

This allows enhanced testing of opcodes until the other gnt-* commands
are converted to use the master daemon.

Reviewed-by: ultrotter

Currently the luxi.client.SubmitJob and Query methods return the unserialized
result without processing it at all. This patch changes this by adding a
'RequestException' error that is raised if the query itself or the
submission of the job failed, and (if not) returning only the 'result'
field from the message.

The patch also does processing on the result of a query if we queried
for jobs, as the 'op_list' field in the result has serialized opcodes
and we need the de-serialized.

Reviewed-by: ultrotter

This patch changes the definition of a job and introduces per-opcode
results.

First, the result and status fields of a job are condensed into a single
'status' attribute. Then, we introduce an opcode status and one result
list, that allow jobs to return values.

The gnt-job script is also modified to allow these new fields to be
queried.

Note that the patch changes the opcode field to op_list, and it changes
its return value from string to a list of (serialized) opcodes.

Reviewed-by: ultrotter

Based on the previous OS search code changes, we can now move the OS
search code into a generic look-for-file function in utils.py. This
means that the allocator code can use the same function.

Reviewed-by: ultrotter

Currently, the function backend._OSSearch() returns the (first) base dir
in which this OS can be found. Thereafter the full actual path to the OS
dir is built in the backend.OSFromDisk() function.

This patch changes this so that _OSSearch() always returns the full path
to the OS directory, and OSFromDisk uses that as returned (it will only
build it if it gets a base dir in the first place).

This patch is needed before we can abstract the _OSSearch into a generic
'look for file object' functionality that can be used for allocator
plugins search too.

Reviewed-by: ultrotter

Currently, the example scripts (the init.d and the cron script) do not
depend on the Makefile, so if configure is rerun with different
parameters these scripts are not rebuild.

The patch adds Makefile as a dependency so changes to installation paths
or other parameters trigger the rebuild.

Reviewed-by: imsnah

The list of instances passed to _VerifyInstance is the one coming from
self.cfg.GetInstanceList(). So there's no point, inside that function, in
checking whether the current instance is a member of that list. Moreover
orphaned instance verification is already done in a separate step.

Reviewed-by: imsnah

The instance configuration is grabbed both in the _VerifyInstance function and
in the loop that calls it. Clean this up by passing the configuration as a
parameter.

Reviewed-by: imsnah

Currently if ganeti-noded doesn't respond on a node gnt-cluster verify will die
when verifying primary instances for that node. Fix this by just emitting an
error message if no information about running instances is returned from the
node.

Reviewed-by: iustinp

All ERROR messages in cluster verify are indented by four spaces, this one is
indented by two. Fixing this skew.

Reviewed-by: imsnah, iustinp

Of course instance creation don't have any modem, and the comment was just
talking about modes. Sorry to everybody expecting whistles.

Reviewed-by: imsnah

This patch adds the previously introduced lvm storage toggling
info to the gnt-cluster man page. It adds some info the the
gnt-cluster init -g option and adds the gnt-cluster modify command.

Reviewed-by: iustinp

Reviewed-by: imsnah

os.path.join does not like None as argument and fails with
AttributeError: 'NoneType' object has no attribute 'startswith'.

This patch makes sure the passed argument is a string in any case.

Reviewed-by: iustinp

Reviewed-by: imsnah

Reviewed-by: ultrotter

Reviewed-by: ultrotter, iustinp

This patch does two things:
- Modify LURenameInstance.Exec to rename directory
  when a file-based instance is renamed
- Modify config.RenameInstance() to replace the directory name in
  config.data for file devices

Reviewed-by: iustinp

- Modfiy _GenerateDiskTemplate to support file-based disk template
- Modify _CreateDisks to create directory needed for file-based
  instances before creating the actual files
- Modify _RemoveDisks to delete directory for file-based instances
  after deleting their VBDs
- Add Prereq-check to check if given file-driver is valid
- Add Prereq-check to check if given file-storage-dir path is relative

Reviewed-by: iustinp

Reviewed-by: iustinp

Having the individual parts in the code allows us to build version
numbers like "1.2" while leaving "3" out in a clean fashion, that is
without regular expressions or the like. This might be used together
with configuration format versions.

Why m4 code? AM_INIT_AUTOMAKE, which could take a shell variable, is
considered deprecated[1] and should be replaced by AC_INIT. Unfortunately,
AC_INIT is expanded at build time, so one has to use m4 to build
composite values like this version number[2].

[1] http://www.gnu.org/software/libtool/manual/automake/Public-macros.html
[2] http://www.mail-archive.com/autoconf@gnu.org/msg16720.html

Reviewed-by: iustinp

The driver in the xen config file needs to be changed when dealing with
files rather then bdevs.

This patch does two things:
- Add _GetConfigFileDiskData to XenHypervisor which returns the correct
  disk xen config line. This function checks the logical disk type of
  every given block device, such that also hybrid (e.g. mixed drbd and
  file VBDs) are possible
- Make Xen[Pvm|Hvm]Hypervisor._WriteConfigFile() a classmethod to be
  able to call the helper function _GetConfigFileDiskData() in their
  parent XenHypervisor

Reviewed-by: iustinp

Reviewed-by: iustinp

Reviewed-by: iustinp

Reviewed-by: iustinp

This patch implements query-ing of only selected jobs instead of all.

Reviewed-by: ultrotter