This patch changes the multi-instance gnt-* commands (gnt-instance
start/stop, gnt-node evacuate/failover) such that the individual
operations are submitted in parallel, ideally improving the speed of the
execution.

The patch does this by abstracting the job set functionality into a new
class in cli.py, that takes care of the job submit, job poll and error
handling.

Reviewed-by: ultrotter

In case we submit multiple instances via batcher, it's nicer to have the
sorted nicely.

Reviewed-by: imsnah

This patch fixes the gnt-instance batch-create command, and in doing so
also slightly changes two other functions:
  - we change utils.ParseUnit so that it accepts integer values also
    (both ParseUnit(5) and ParseUnit("5") return the same value)
  - a bridge 'None' in LUCreateInstance will be converted to the default
    bridge; currently only missing bridges will be accepted to mean the
    default one

The main changes to batcher were the change to variable number of disks
and NICs.

The patch also adds a batcher-instances.json example file copied from
the 1.2 branch and properly modified.

Reviewed-by: imsnah, killerfoxi

This patch changes the iallocator framework to work with and properly
export to plugins offline nodes. It does this by only exporting the
static configuration data for those nodes, and not attempting to parse
the runtime data.

The patch also fixes bugs in iallocator related to the RpcResult
conversion, changes the should_run to admin_up attribute name (as per
the internals change), and adds “-I” as a short option for
“--iallocator” in gnt-instance, gnt-backup and burnin.

Reviewed-by: ultrotter

This removes some constraints:
  - only two disks supported, this is no longer true as the underlying
    functions can now compute size for a variable number of disks
  - error when the hypervisor was not being passed
  - typo error

Reviewed-by: imsnah

Reviewed-by: ultrotter

Instead of having the default live in the gnt-cluster script, we move it
to the constants file. The patch also fixes a typo on constants.py.

Reviewed-by: ultrotter

This is forward port via copy (and not individual patches cherry-pick)
of the latest code on the 1.2 branch related to the migration.

The changes compared to 1.2 are the fact that we don't need the
IdentifyDisks step anymore (the drbd rpc calls are independent now), and
the rpc module improvements.

Reviewed-by: ultrotter

This patch makes gnt-cluster copyfile and command skip the offline
nodes.

Reviwed-by: ultrotter, imsnah

In 2.0, we have more parameters in drbd's logical_id, and passing the
results over json makes them unicode which looks worse with the default
formatting. As such, a redo of the output is needed.

This patch:
  - adds a separate function to format the logical_id of devices
  - moves the actual indentation format out of _FormatBlockDevInfo,
    which now just generates a list of items
  - adds a function _FormatList that recursively formats the list
  - formats specially key,value tuples

The result is that the output is nicer, and the code in
_FormatBlockDevInfo somewhat cleaner (as it doesn't deal with spacing
and such issues).

Reviewed-by: ultrotter

This LU can be used to force a push of the config in case it's needed,
for example after an upgrade to update the ssconf_release_version file.

Reviewed-by: imsnah

In 1.2, due to the md+drbd7 legacy, we had a complex choice of replace
modes, and the new drbd8 modes where forced into this syntax, with some
complicated rules of transition from one mode to another (if REPLACE_ALL
but not new node passed, switch to REPLACE_SEC, etc.).

This patch cleans this situation by making a clear separation between
the two main modes:
  - replace on current nodes (with the two sub-cases on primary and on
    secondary)
  - change to a new node (either via manually specified node or via
    iallocator)

Reviewed-by: imsnah

With a large job queue, auto-archiving jobs can take a very long time,
causing timeouts on the luxi RPC layer. With this change, auto-
archive returns after half of the RPC timeout has passed. The user
will see how many jobs are left unchecked.

Reviewed-by: ultrotter

This patch should fix all outstanding epydoc parsing errors; as such, we
switch epydoc into verbose mode so that any new errors will be visible.

Reviewed-by: imsnah

This patch changes gnt-node modify and the associated opcode/lu to allow
modification of the node offline attribute.

Setting a node into offline mode automatically demotes it from the
master role.

Reviewed-by: ultrotter

Reviewed-by: ultrotter

- Add a new command line option, defaulting to the constant value
- Pass the value to bootstrap.InitCluster
- Use it to init the new Cluster object

Reviewed-by: imsnah

Currently we have to update both the ListNodes headers and the online
help for the full field list. This patch uses the headers keys for the
help, thus removing duplicating places to update, and adding hope that
we'll have things in sync. As a downside we lose ordering of the
non-default fields in the online help.

Reviewed-by: imsnah

On cluster init if the user specifies a default hypervisor (with -t)
which is not in the default list of enabled hypervisors (currently just
xen-pvm) without explicitely specifying the list we silently override
the choice.

With this patch we set the list by default to just the required one, and
we bail out should the list be hand-specified and not contain the
default one. This still has an issue when the user doesn't specify a
default hypervisor but specifies a list which doesn't include xen-pvm:
in this case though we give an error, rather than silently discarding
choices.

Reviewed-by: imsnah

If the hv/be parameter lowercase value is set to "default" we'll pass
constants.VALUE_DEFAULT, if it's set to "none" we'll pass
constants.VALUE_NONE.

Reviewed-by: imsnah

--src-node and --src-dir are not mandatory anymore

Reviewed-by: iustinp

This patch adds a new cluster paramater "candidate_pool_size" which
tracks the desired size of the list of nodes with the master_candidate
flag set.

Reviewed-by: imsnah

This patch adds the OpCode, LogicalUnit and gnt-node command for
modifying node parameters, more specifically the master candidate flag
for a node.

Reviewed-by: imsnah

This patch adds listing of the master_candidate field (as Y/N) and of
the master role (again Y/N) for nodes.

Reviewed-by: imsnah

They're not used anymore.

Reviewed-by: ultrotter

Reviewed-by: iustinp

- Add new "canceling" status
- Notify clients when job is canceled
- Give a return value from CancelJob
- Handle it in the client library

Reviewed-by: iustinp

Currently, the node add operation uses a job to query the node name and
the bootstrap function directly reads the config file for the cluster
name.

This patch changes to that both the cluster name and the verification of
the node is done via queries to the master.

Reviewed-by: ultrotter

This is mostly a copy from gnt-instance add; import works, but it's not
optimal - device count/parametrs should be able to be reused.

Reviewed-by: ultrotter

Currently there's no way to specify no NICs for an instance, even though
this is a supported configuration.

The patch adds a --no-nics option to gnt-instance add.

Reviewed-by: amishchenko

The submit function was using a very old API. This patch brings it up to
date.

Reviewed-by: imsnah

This patch fixes the diskless and drbd/file based instances. Sorry :(

Reviewed-by: ultrotter

This big patch adds support for:
  - changing NIC/disks in the multi-device model
  - adding/removing NICs
  - adding/removing disks

The patch is big and not very nice; the error checking paths are not
very clear.

The biggest problem is that from a simple instance.ATTR=VAL change
(which didn't throw errors before) now we are creating and removing
disks in this LU.

Reviewed-by: imsnah

The mode parameter needs to be exported and shown in the info output.

Reviewed-by: imsnah

Previously we had only one nic, so the NIC information was all on one
line. This patch changes it to:

    - NICs:
      - nic/0: MAC: aa:00:00:e8:b2:ef, IP: None, bridge: None
      - nic/1: MAC: aa:00:00:53:ca:92, IP: None, bridge: xen-br0
      - nic/2: MAC: aa:00:00:ba:e5:30, IP: None, bridge: None

Reviewed-by: ultrotter

This patch enables by default the old 'human-readable' option, but in a
slightly different model.

The option is now called "units" and takes either:
 - 'h' for automatic formatting
 - 'm', 'g' or 't' for mebi/gibi/tebibytes

If 'h' is used, we add a unit suffix, otherwise nothing is added so that
parsing is easy.

The default value of this unit is:
  - 'h' if a separator is not passed
  - 'm' if a separator is passed

Reviewed-by: ultrotter

This patch adds nicer formatting for some gnt-instance list operations.

Reviewed-by: ultrotter

This changes cli.py to FieldSet usage so that gnt-instance list will
format nicely the disk.size/*, and the count of disks/nics.

Reviewed-by: ultrotter

This patch changes the instance.FindDisk method to take index arguments
(instead of iv_names), and changes GrowDisk and list instances
accordingly.

Reviewed-by: ultrotter

ganeti-masterd: Add initialization and shutdown of RPC pool. It needs
to be shutdown before forking.

ganeti.cli: Add decorator function to initialize and shutdown RPC pool.

ganeti.rpc: Add functions to initialize and shutdown RPC pool. Throw
exception when used without proper initialization.

gnt-cluster, gnt-node: Use decorator function to initialize and shutdown
RPC pool.

Reviewed-by: iustinp