The current fork+close fds sequence has deficiencies which are hard to
work around:
  - logging can start logging before we fork (e.g. if we need to emit
    messages related to master checking), and thus use FDs which we
    can't track nicely
  - the queue locks the queue file, and again this fd needs to be kept
    open which is hard from the main loop (and this error is currently
    hidden by the fact that we don't log it)

Given the above, it's much simpler, in case we will fork later, to close
file descriptors right at the beginning of the program, and in Daemonize
only close/reopen the stdin/out/err fds.

In addition, we also close() the handlers we remove in SetupLogging so
that the cleanup is more thorough.

Reviewed-by: imsnah

This is a forward-port of commit 1194 on the 1.2 branch:

  This call will check whether an instance is up on its primary, and that
  it has been started with symlinks. We currently have no on-secondary
  checks, nor any hypervisor specific call.

  Reviewed-by: iustinp

The difference from the original patch is that we don't include the
cmdlib changes, since those will come as a copy from the 1.2 cmdlib.py,
and not as individual patches.

Original-Author: ultrotter

This is an extract of commit 1166 on the 1.2 branch (Add a rpc call for
drbd network reconfiguration), but only the blockdev_close part.

The patch changes the blockdev_close call to take the instance so that
it can remove the symlinks of the instance.

Originally-Reviewed-by: imsnah

Two bad indentation cases and a missing variable.

Reviewed-by: imsnah

Passwords are stored in "$localstatedir/lib/ganeti/rapi_users". User
options specify the access permissions of a user (see docstring for
ganeti.http.ReadPasswordFile), for which only "write" is supported
to grant write access. Every other user has read-only access.

Reviewed-by: amishchenko

This will be used to evaluate access permissions to resources.

Reviewed-by: amishchenko

Reviewed-by: ultrotter

With a large job queue, auto-archiving jobs can take a very long time,
causing timeouts on the luxi RPC layer. With this change, auto-
archive returns after half of the RPC timeout has passed. The user
will see how many jobs are left unchecked.

Reviewed-by: ultrotter

This patch should fix all outstanding epydoc parsing errors; as such, we
switch epydoc into verbose mode so that any new errors will be visible.

Reviewed-by: imsnah

This patch adds a simple rpc which makes a backup of the config file and
then removes it. This is done so that cluster verify doesn't complain
immediately after demoting a node.

Reviewed-by: imsnah

This patch changes the LUQueryInstances to show a different state for
offline nodes and also modifies the watcher to understand the offline
state in its checks.

Reviewed-by: ultrotter

Reviewed-by: amishchenko

Reviewed-by: amishchenko

It should be consistent.

Reviewed-by: amishchenko

The ssconf files were not updated by the master failover. We need to
push them, and since we already have RPC initialized, we can use the
standard ConfigWriter to do so - this will take care of both the config
file and the ssconf files.

Reviewed-by: imsnah

Since we're not sure ganeti-noded has started yet, we need to create
RUN_GANETI_DIR before SOCKET_DIR as well, with the proper permissions.

Reviewed-by: imsnah

ganeti-noded used to create all directories under /var/run with an
hard-coded mode. convert it to a constant.

Reviewed-by: imsnah

Before it was in the abstract linux namespace, where unfortunately we
couldn't easily check from python the credentials of the connecting
clients. Now we also have to remove the file on exit and when starting.

Reviewed-by: imsnah

If SOCKET_DIR doesn't exist we create it in the master daemon, before
trying to put a socket inside it.

Reviewed-by: imsnah

Instead of parsing the configuration on the node, we pass the ssconf
values from the master.

Reviewed-by: iustinp

This patch enables SSL between masterd and noded.

Reviewed-by: iustinp

With the new SSL client certificate stuff it's no longer needed.

Reviewed-by: iustinp

Removing the PID file should be the last thing done. This patch makes
sure it's also removed when master.server_cleanup() throws an exception.

Also initialize logging only after writing the PID file.

Reviewed-by: iustinp

ganeti-masterd: Add initialization and shutdown of RPC pool. It needs
to be shutdown before forking.

ganeti.cli: Add decorator function to initialize and shutdown RPC pool.

ganeti.rpc: Add functions to initialize and shutdown RPC pool. Throw
exception when used without proper initialization.

gnt-cluster, gnt-node: Use decorator function to initialize and shutdown
RPC pool.

Reviewed-by: iustinp

Reviewed-by: iustinp

Currently the dir creation in ganeti-noded is in the main function. This
is not nice: we move it into a separate function and also add creation
of the OS_LOG_DIR (with different permissions, but in the same way).
This will permit cleanup of the creation of the OS_LOG_DIR from the
backend module (it's done multiple places currently).

Reviewed-by: imsnah

At the same time, simplify the interface a bit by not using a tuple.

Reviewed-by: killerfoxi, ultrotter

We want to export the disk index as some OSes will only want to export
the first disk (or the second one, etc.), even if we have multiple
disks.

The patch also updates the backend.ExportSnapshot docstring.

Reviewed-by: ultrotter

- Change ImportOSIntoInstance not to get any "os_disk" and "swap_disk"
  arguments but to accept multiple target images to import, and to
  return a list of booleans with the result of each import
- Change the relevant rpc call and the only caller to conform
- Pass arguments to the import script through the environment
- Run one import os script for each disk image, passing an IMPORT_DEVICE

Reviewed-by: iustinp

Reviewed-by: iustinp

The two main multi-node job queue RPC calls (jobqueue_update,
jobqueue_rename) are converted to address-based calls, in order to speed
up queue changes. For this, we need to change the _nodes attribute on
the jobqueue to be a dict {name: ip}, instead of a set.

Reviewed-by: imsnah

Since now we use only one function from the logger module
(SetupLogging), we move it to utils.py (which is already imported by all
users of this function), and we remove the module.

Reviewed-by: imsnah

- remove now unused osdev and swapdev arguments from backend, noded,
  rpc, cmdlib
- convert docstrings to epydoc

Reviewed-by: iustinp

Reviewed-by: imsnah

Requests are no longer logged to a separate file.

Reviewed-by: amishchenko

In order to account for future improvements to master failover, we move
the actual data gathering capabilities from ganeti-masterd into
bootstrap.py, and we leave only the verification into masterd.

The verification procedure is then changed to retry multiple times (up
to one minute) in case most nodes do not respond, and also the algorithm
is changed to require at least half (but not half+1) votes, since our
vote also should count (and we vote for ourselves).

Example for consistent (config-wise) cluster:
  - 5 node cluster, 2 nodes down: still start
  - 4 node cluster, 2 nodes down: retry for one minute, abort

Reviewed-by: ultrotter

This adds the set/reset in the jqueue and luxi modules, and a way to
query it in OpQueryConfigValues, and also the comand line interface for
it:
$ gnt-cluster queue info
The drain flag is unset
$ gnt-cluster queue drain
$ gnt-cluster queue info
The drain flag is set
$ gnt-cluster queue undrain
$ gnt-cluster queue info
The drain flag is unset

The choice of making the setting via luxi and not an opcode is that
opcodes can't be executed when drained, but we don't query via luxi
since in the future it might become a cluster property as opposed to a
node one.

Reviewed-by: imsnah

A new multi-node call is added that sets/resets the drain flag.

Reviewed-by: imsnah

This patch adds a generic method to identify the ganeti error given its
class name, and implements this across the luxi protocol.

Reviewed-by: imsnah

Reviewed-by: ultrotter