This (big) patch moves the hypervisor type from the cluster to the
instance level; the cluster attribute remains as the default hypervisor,
and will be renamed accordingly in a next patch. The cluster also gains
the ‘enable_hypervisors’ attribute, and instances can be created with
any of the enabled ones (no provision yet for changing that attribute).

The many many changes in the rpc/backend layer are due to the fact that
all backend code read the hypervisor from the local copy of the config,
and now we have to send it (either in the instance object, or as a
separate parameter) for each function.

The node list by default will list the node free/total memory for the
default hypervisor, a new flag to it should exist to select another
hypervisor. Instance list has a new field, hypervisor, that shows the
instance hypervisor. Cluster verify runs for all enabled hypervisor
types.

The new FIXMEs are related to IAllocator, since now the node
total/free/used memory counts are wrong (we can't reliably compute the
free memory).

Reviewed-by: imsnah

This can be used to retrieve certain cluster config values from
within clients.

OpDumpClusterConfig was not used anywhere, hence I'm just reusing
it. The way ConfigWriter.DumpConfig returned the configuration
was not thread-safe, anyway (no deepcopy).

Reviewed-by: iustinp

The watcher has one last use of ganeti commands as opposed to sending
requests via luxi. The patch changes this to use the cli functions.

The patch also has two other changes:
  - fix the docstring for OpVerifyDisks (found out while converting
    this)
  - enable stderr logging on the watcher when “-d” is passes

Reviewed-by: imsnah

It is not currently possibly to show a summary of the job in the output
of “gnt-job list”. The closes is listing the whole opcode(s), but that
is too verbose. Also, the default output (id, status) is not very
useful, unless one looks for (and knows about) an exact job ID.

The patch adds a “summary” description of a job composed of the list of
OP_ID of the individual opcodes. Moreover, if an opcode has a ‘logical’
target in a certain opcode field (e.g. start instance has the instance
name as the target), then it is included in the formatting also. It's
easier to explain via a sample output:

gnt-job list
ID Status  Summary
1  error   NODE_QUERY
2  success NODE_ADD(gnta2)
3  success CLUSTER_QUERY
4  success NODE_REMOVE(gnta2.example.com)
5  error   NODE_QUERY
6  success NODE_ADD(gnta2)
7  success NODE_QUERY
8  success OS_DIAGNOSE
9  success INSTANCE_CREATE(instance1.example.com)
10 success INSTANCE_REMOVE(instance1.example.com)
11 error   INSTANCE_CREATE(instance1.example.com)
12 success INSTANCE_CREATE(instance1.example.com)
13 success INSTANCE_SHUTDOWN(instance1.example.com)
14 success INSTANCE_ACTIVATE_DISKS(instance1.example.com)
15 error   INSTANCE_CREATE(instance2.example.com)
16 error   INSTANCE_CREATE(instance2.example.com)
17 success INSTANCE_CREATE(instance2.example.com)
18 success INSTANCE_ACTIVATE_DISKS(instance1.example.com)
19 success INSTANCE_ACTIVATE_DISKS(instance2.example.com)
20 success INSTANCE_SHUTDOWN(instance1.example.com)
21 success INSTANCE_SHUTDOWN(instance2.example.com)

This is done by a simple change to the opcode classes, which allows an
opcode to format itself. The additional function is small enough that it
can go in opcodes.py, where it could also be used by a client if needed.

Reviewed-by: imsnah

It was already allowed in gnt-instance modify, but ignored.
It will be used to force skipping parameter checks.

This is a forward-port from branches/ganeti-1.2

Original-Reviewed-by: imsnah
Reviewed-by: iustinp

Add HVM device type flags 2/3

Reviewed-by: ultrotter

Reviewed-by: iustinp

This (big) patch reworks the master startup/shutdown and the fixes the
master failover.

What does the patch do?

For master start/stop:
  - remove the old ganeti-master script and its associated man page
  - moves the ip start/stop directly into the backend.(Start|Stop)Master
  - adds start/stop of the master/rapi daemon into these functions,
    selectively based on the start/stop arguments
  - makes the master call via rpc StartMaster(start_daemons=False) to
    the local node so that the master IP is started
  - and finally changes the example init.d script to directly start and
    stop all three daemons, since they do the right thing (depending on
    master/not master role)

For master failover:
  - moves the code from LUMasterFailover into bootstrap.MasterFailover,
    since we need to start/stop the master during this operation and
    thus it can't be executed from the master
  - removes the LUMasterFailover and its associated opcode

Notes: ubuntu's /etc/lsb-base-logging.sh is dumb, so the messages 'not
master' are not seen during startup on non-master nodes.

Reviewed-by: ultrotter

Reviewed-by: imsnah

Since we don't have for now a job definition object anymore, we rename
this class to BaseOpCode. It's still useful (and not merged with OpCode)
since it holds all the 'pure' logic (no custom field handling, etc.)
whereas OpCode holds opcode specific data (OP_ID handling, etc).

The patch also fixes the module's docstring.

Reviewed-by: imsnah

Reviewed-by: iustinp

Since the disabling of forking in the master daemon, the two ssh-based
subcommands were not working anymore. However, there is no need at all
for the commands to be run from the master daemon (permissions to read
the cluster private ssh key notwithstanding), they can be run directly
from the command line utilities.

The patch removes the two opcodes OpRunClusterCommand and
OpClusterCopyFile (and their associated LUs) and changes the code in
‘gnt-cluster’ to query the list of nodes and run directly the SshRunner
over the list. As such, all forking is done from the gnt-cluster script,
and the commands are working again.

Reviewed-by: imsnah

This patch adds a new opcode and LU for growing an instance's disk.

The opcode allows growing only one disk at time, and will throw an error
if the operation fails midway (e.g. on the primary node after it has
been increased on the secondary node). As such, it might actually leave
different sized LVs on different nodes, but this will not create
problems.

Reviewed-by: imsnah

This allows us to initialize a new cluster. The code certainly contains
bugs and hooks aren't implemented yet.

Reviewed-by: iustinp

This patch adds the commandline extensions and the code to store
and display the extended HVM features.

Author: schreiberal
Reviewed-by: iustinp

This patch implements secondary replace via the iallocator. The new
opcode parameter 'iallocator' behaves like this: if passed, it will
always compute and assign a new secondary, behaving in effect as if the
secondary node has been passed. It conflicts with actually giving the
secondary too.

[Note: not tested with remote_raid1, but the code should behave the
same, we only touch CheckPrereq and we assign a node.]

The patch also adds burnin support for the replace secondary operation;
with this in place, burnin can fully work with auto-assigned nodes.

Reviewed-by: ultrotter

This patch also fixes the LUExportInstance Prereq docstring.

Reviewed-by: iustinp

This allows us to readd a node after it failed and required a
reinstallation or replacement.

Reviewed-by: iustinp

This (final) patch allows the instance's nodes to be selected
automatically based on the passed allocator algorithm.

The patch changes the pnode opcode parameter from required to optional,
now either the pnode or the iallocator must be passed.

A possible improvement could be to organize all the _IAllocator
functions into a separate class, but that can come later and the current
version is functionally ok.

Reviewed-by: ultrotter

In preparation for the introduction of automatic instance allocator,
this patch adds an allocator simulation opcode, that based on the input
parameters, will return either the input message to the allocator
(implemented) or the result of the allocator run (not yet implemented).

This allows algorithm tests against simulated allocations and the
current cluster state.

The patch adds the following:
  - a function that generates the generic cluster information for the
    allocator
  - a function that generates the 'new instance' information
  - a function that generates the 'replace_secondary' information

These three functions will be used by the allocator framework later to
generate the actual information for the external algorithms. Currently
we just return the json-serialized text.

Reviewed-by: imsnah

Add a general way to skip some checks at cluster-verify time and make the N+1
memory redundancy check optional.

Reviewed-by: iustinp

Currently, the opcode DiagnoseOS is the only opcode that return a
structure of objects.OS (which is a custom class, and not a simple
python object) and furthermore all the processing of OS validity across
nodes is left to the clients of this opcode.

It would be more logical to have this opcode be similar to list
instances/nodes, in the sense that:
  - it should return a table of results
  - the fields in the table should be selectable

This patch does the above. The possible fields are:
  - name (os name)
  - valid (bool representing validity across all nodes)
  - node_status, which is a complicated structure required for ‘gnt-os
    diagnose’

With this patch, gnt-os list becomes a very simple iteration over the
list of results, filtering out non-valid ones. gnt-os diagnose is still
complicated, but no more than before.

The burnin tool has also been modified to work with the modified
results, and is simpler because of this (it only needs to know if an OS
is valid or not, not the per-node details).

Reviewed-by: imsnah

This patch changes the definition of a job and introduces per-opcode
results.

First, the result and status fields of a job are condensed into a single
'status' attribute. Then, we introduce an opcode status and one result
list, that allow jobs to return values.

The gnt-job script is also modified to allow these new fields to be
queried.

Note that the patch changes the opcode field to op_list, and it changes
its return value from string to a list of (serialized) opcodes.

Reviewed-by: ultrotter

Reviewed-by: ultrotter, iustinp

This small patch fixed the opcodes.OpCode.LoadOpcode capitalization to
what was intented to be (as the comment says): LoadOpCode.

Reviewed-by: ultrotter

- Substitute all occurences of name 'parms' with 'params'
- Small codestyle fix

Reviewed-by: ultrotter

Reviewed-by: iustinp

These two operations are related to md/drbd7 code (remote_raid1). Remove
them as part of the md/drbd7 removal.

Reviewed-by: imsnah

This patch adds a new top-level class (BaseJO) that is used for both
opcodes and a new Job class.

This new class and the related changes to the OpCode abstract class are
used to implement simple to-dict/from-dict transformations, so that we
can easily serialize the classes using json.

Reviewed-by: imsnah

Author: manuel.franceschini
Reviewed-by: iustinp

Reviewed-by: ultrotter

Reviewed-by: ultrotter

This can be used for testing purposes.

Reviewed-by: ultrotter,imsnah

This patch adds support for specifying and changing the boot device order for
HVM instances. The boot device order specification is ignored for non HVM
instances.

Reviewed-by: iustinp

This patch adds support for defining the kernel/initrd at instance
creation time, using the same interface as in instance modify.

Reviewed-by: imsnah

This patch adds support in ‘gnt-instance modify’ to set the kernel and
initrd paths. The user can pass either 'default' or 'none' (none is not
valid for kernel).

Reviewed-by: imsnah

This patch improves the ‘gnt-cluster verify-disks’ command by adding
support for detecting broken volume groups and missing logical volume
names.

As such, we don't try anymore to activate disks for instances that are
not likely to succeed anyway, and instead report them.

Reviewed-by: schreiberal

Allow the MAC address of an instance to be specified optionally during
instance creation and later to be changed via instance modify.

Reviewed-by: iustinp

This patch adds the definition of a new opcode that will be used to
compute the list of instances with not-online disks.

Reviewed-by: imsnah

This patch adds a search command for locating tags on all objects of the
cluster using a regex pattern.

Reviewed-by: aat