The current implementation of “gnt-cluster getmaster” doesn't work on
non-master nodes, which is a regression from 1.2. This patch implements
it (again) via ssconf.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Alexander Schreiber <als@google.com>

 # gnt-cluster queue foo
 Failure: prerequisites not met for this operation:
 Command 'foo' is not valid.

Reviewed-by: iustinp

If hvparams is not set it will be [], so dict() will transform it to an
empty dict, which is safe in all cases.

Reviewed-by: iustinp

We want all the hv/be parameters to have a known type, rather than a
random mix of empty string, boolean values, and None, so we declare the
type of each variable and we enforce/convert it.

- Add some new constants for enforceable value types
- Add new constants dicts HVS_PARAMETER_TYPES and BES_PARAMETER_TYPES
  holding not only the valid parameters but also their types
- Drop the old HVS_PARAMETERS and BES_PARAMETERS constants and calculate
  the values from the type dict
- Convert all the default parameters to a valid type value
- Create a new ForceDictType utils function, to check/enforce a dict's
  element value types, with relevant unit tests
- Drop a few custom functions to check/convert the BE param types in
  utils and cli, in favor of ForceDictType
- Double-check the parameter types using ForceDictType in both scripts
  and LogicalUnits, when possible.

As a bonus:
- Remove some old commented-out code in gnt-instance
- Remove some already fixed FIXME
- Fix a bug which prevented VALUE_DEFAULT to be applied to BE parameters
  in SetInstanceParams because the value was checked for validity before
  that transformation was made
- Fix a bug which prevented initing a cluster and passing hvparams to
  work at all
- ForceDictType allows an allowed_values for exceptions, which makes us
  able to do the checking even when some values must not be
  converted/typechecked (for example the 'default' string in
  SetInstanceParameters)

Reviewed-by: iustinp

This patch converts some more jobs with only queries into cheaper luxi
queries (no job created), and fixes some fallout from the lockless
queries changes.

Reviewed-by: ultrotter

This patch adds the online node list and instance list to the ssconf
keys. In order to do distribute correctly the instance list, we need to
update the cluster serial number on instance additions and removals.

The patch also changes the permissions on the ssconf files to be 0444:
  - no write for root, in order to signal that these file should not be
    modified
  - read for everyone since the files don't contain sensitive data
    anymore (and permissions can be controlled via the parent directory
    if needed)

The patch also fixes a small typo on gnt-cluster.

Reviewed-by: ultrotter

This patch unifies the hardcoded re-encoding attempts into a single
function in utils.py. This function is used to take either an unicode or
str object and convert it to a ASCII-only str object which can be safely
displayed and transmitted.

We replace then the current manual re-encodings with this function. In
mcpu we stop re-encoding the hooks output and instead we do it right at
the hook generation in backend.py.

This passes on my 'custom' lvs output with non-ASCII chars. But there
are probably other places we will need to fix.

Reviewed-by: ultrotter

Instead of having the default live in the gnt-cluster script, we move it
to the constants file. The patch also fixes a typo on constants.py.

Reviewed-by: ultrotter

This patch makes gnt-cluster copyfile and command skip the offline
nodes.

Reviwed-by: ultrotter, imsnah

This LU can be used to force a push of the config in case it's needed,
for example after an upgrade to update the ssconf_release_version file.

Reviewed-by: imsnah

- Add a new command line option, defaulting to the constant value
- Pass the value to bootstrap.InitCluster
- Use it to init the new Cluster object

Reviewed-by: imsnah

On cluster init if the user specifies a default hypervisor (with -t)
which is not in the default list of enabled hypervisors (currently just
xen-pvm) without explicitely specifying the list we silently override
the choice.

With this patch we set the list by default to just the required one, and
we bail out should the list be hand-specified and not contain the
default one. This still has an issue when the user doesn't specify a
default hypervisor but specifies a list which doesn't include xen-pvm:
in this case though we give an error, rather than silently discarding
choices.

Reviewed-by: imsnah

This patch adds a new cluster paramater "candidate_pool_size" which
tracks the desired size of the list of nodes with the master_candidate
flag set.

Reviewed-by: imsnah

ganeti-masterd: Add initialization and shutdown of RPC pool. It needs
to be shutdown before forking.

ganeti.cli: Add decorator function to initialize and shutdown RPC pool.

ganeti.rpc: Add functions to initialize and shutdown RPC pool. Throw
exception when used without proper initialization.

gnt-cluster, gnt-node: Use decorator function to initialize and shutdown
RPC pool.

Reviewed-by: iustinp

Reviewed-by: imsnah

Reviewed-by: imsnah

Reviewed-by: imsnah

Reviewed-by: imsnah

During cluster init, set the default hypervisor to be used for instances.
Ensure that the default hypervisor belongs to the set enabled hypervisors
for this cluster. Also fix a small bug with setting the default enabled
hypervisor list.

Reviewed-by: imsnah

Some names were wrong, and similar stuff detected by pylint.
gnt-debug.GenericOpCode is still broken.

Reviewed-by: imsnah

Currently the gnt-* scripts are using a mix of print, logger.ToStd* and
sys.stderr.write. We convert them all to using cli.ToStdout/err. This
way, we can easily change the implementation for all at once.

Reviewed-by: imsnah

We no longer use a single, cluster-wide hypervisor, but configure the
actual to be used hypervisor on the instance level.

Reviewed-by: imsnah

Reviewed-by: iustinp

This patch enables the cluster modify to change:
  - enabled hypervisor list
  - hvparams (per hypervisor)
  - beparams (only the default group)

Syntax:
  gnt-cluster modify -B vcpus=3 -H xen-pvm:no_initrd_path

Validation for parameters is somewhat missing - the individual
hypervisors will be checked for syntax and validation, but beparams
doesn't have validation yes (nowhere), it should be added here once we
have a global method (will come soon).

Reviewed-by: imsnah

This is just a raw update without any special formatting.

Reviewed-by: imsnah

This adds the set/reset in the jqueue and luxi modules, and a way to
query it in OpQueryConfigValues, and also the comand line interface for
it:
$ gnt-cluster queue info
The drain flag is unset
$ gnt-cluster queue drain
$ gnt-cluster queue info
The drain flag is set
$ gnt-cluster queue undrain
$ gnt-cluster queue info
The drain flag is unset

The choice of making the setting via luxi and not an opcode is that
opcodes can't be executed when drained, but we don't query via luxi
since in the future it might become a cluster property as opposed to a
node one.

Reviewed-by: imsnah

Reviewed-by: ultrotter

Currently the SshRunner uses a SimpleConfigReader instance, however this
is not best. We change it to use the cluster name directly (and its
constructor now takes this as parameter, instead of SCR), and its
callers are change to pass the name directly.

As a consequence, we can now remove the initialization of SCR in
gnt-cluster (copyfile and command), and instead we query the master for
the cluster name).

Reviewed-by: imsnah

Replace ssconf with configuration.

Reviewed-by: iustinp

Get rid of ssconf and convert to configuration instead.

Reviewed-by: iustinp

Fixing the check in gnt-cluster, or gnt-cluster verify-disks is broken.
Since the version in 1.2 used to return a tuple we'll accept both.

Reviewed-by: iustinp

Reviewed-by: schreiberal

Reviewed-by: imsnah

With the recent startup/shutdown changes (and with the master daemon in
place), the cluster destroy needs some fixing.

This patch moves the finalization of the destroy out from cmdlib into
bootstrap, so we can nicely shutdown the rapi and master daemons.

Reviewed-by: ultrotter

This is special in the sense that it can run on any node. As such, we
just instantiate ssconf and read the data from it.

Reviewed-by: ultrotter

This (big) patch reworks the master startup/shutdown and the fixes the
master failover.

What does the patch do?

For master start/stop:
  - remove the old ganeti-master script and its associated man page
  - moves the ip start/stop directly into the backend.(Start|Stop)Master
  - adds start/stop of the master/rapi daemon into these functions,
    selectively based on the start/stop arguments
  - makes the master call via rpc StartMaster(start_daemons=False) to
    the local node so that the master IP is started
  - and finally changes the example init.d script to directly start and
    stop all three daemons, since they do the right thing (depending on
    master/not master role)

For master failover:
  - moves the code from LUMasterFailover into bootstrap.MasterFailover,
    since we need to start/stop the master during this operation and
    thus it can't be executed from the master
  - removes the LUMasterFailover and its associated opcode

Notes: ubuntu's /etc/lsb-base-logging.sh is dumb, so the messages 'not
master' are not seen during startup on non-master nodes.

Reviewed-by: ultrotter

Since the disabling of forking in the master daemon, the two ssh-based
subcommands were not working anymore. However, there is no need at all
for the commands to be run from the master daemon (permissions to read
the cluster private ssh key notwithstanding), they can be run directly
from the command line utilities.

The patch removes the two opcodes OpRunClusterCommand and
OpClusterCopyFile (and their associated LUs) and changes the code in
‘gnt-cluster’ to query the list of nodes and run directly the SshRunner
over the list. As such, all forking is done from the gnt-cluster script,
and the commands are working again.

Reviewed-by: imsnah

This allows us to initialize a new cluster. The code certainly contains
bugs and hooks aren't implemented yet.

Reviewed-by: iustinp

Author: schreiberal
Reviewed-by: iustinp

[Trunk version]

Reviwed-by: imsnah