This function will return the first unused integer based on a list of
used integers (e.g. [0, 1, 3] will return 2).

Reviewed-by: imsnah

Now that we use the locking library to make sure running opcodes cannot
step on each other toes we can have a bigger thread size, and
potentially process many opcodes in a parallel manner.

Reviewed-by: iustinp

If a LU required the BGL (all LUs do, right now, by default) we'll
acquire it in the Processor before starting them. For LUs that don't
we'll still acquire it, but in a shared fashion, so that they cannot run
together with LUs that do.

We'll also note down whether we own the BGL exclusively, and if we don't
and we try to chain a LU that does, we'll fail.

More work will need to be done, of course, to convert LUs not to require
the BGL, but this basic infrastructure should guarantee the coexistance
of the old and new world for the time being.

Reviewed-by: iustinp

The processor used to create a new ConfigWriter when it was initialized.
We now have one in the context, so we'll just recycle it. First of all
we'll pass the context in when creating a new Processor object, then
we'll just use context.cfg, which is granted to be initialized, wherever
we used self.cfg, and stop checking whether the config is already
initialized or not.

In the future the Processor will be able to use the context also to
acquire the BGL for LUs that require it, and to push the context down to
LUs that don't in order for them to manage their own locking.

Reviewed-by: iustinp

When logical units have REQ_BGL set (it is currently the default) they
need to be the only ganeti operation run on the cluster, and we'll
guarantee it at the master daemon level. Currently only one thread is
running at a time, so this requirement is never broken.

Reviewed-by: iustinp

In 2.0 burnin submits job to the master daemon, so it doesn't need to
create an internal Processor anymore. Even if the processor is not used
anywhere in the burnin code it was still initialized as a leftover of
how burnin used to work. Fixing this.

Reviewed-by: iustinp

This adds the same “-o +...” functionality in gnt-job as in the node and
instance scripts.

Reviewed-by: imsnah

- no need to keep the sstore as an object member, remove it
- don't reinitialize sstore only if self.cfg is None
    This is not an issue, as the Processor is recycled for every opcode,
    but in general we know that (a) we might need a different type of
    sstore for different opcodes and (b) initializating them is cheap
- recreate sstore when chaining opcodes
    Without this fix chaining an opcode which requires a writable sstore
    to one which doesn't would fail. This doesn't happen today, but it's
    better to fix it anyway

These changes are possible because nowadays all opcodes already require
a working cluster/configuration.

Reviewed-by: iustinp

All the tests there used to creare a cfg, a sstore, an opcode and a LU.
Put all the duplicate code in the setUp function.

Reviewed-by: iustinp

This patch creates a new GanetiContext class, which is used to hold
context common to all ganeti worker threads. As for the
GanetiLockingManager class it is paramount that there is only one such
class throughout the execution of Ganeti, so the class checks for that,
and also forbids its own modification after it's been initialized. The
context for now contains a ConfigWriter and a GanetiLockingManager and
is created by the daemon and propagated to PoolWorker(s) and
JobRunner(s).

Reviewed-by: iustinp

From the master node we can't start ssh and connect to the remote node,
nor we can do it from ganeti-noded as this ssh section will possibly ask
for key confirmation and password. So the code to copy the ganeti-noded
password and SSL key has been moved to bootstrap.py, and it's called by
gnt-node before the AddNode opcode.

Reviewed-by: iustinp

In the "new world" we'll need to setup ganeti-noded via ssh on the node
before calling the AddNode opcode. Before doing it we'll check that the
node is not already in the cluster, if --readd was not passed. This
guarantees we're not going to restart ganeti-noded on a running node.

This patch also incidentally fixes a non-style-guide conformant
docstring.

Reviewed-by: iustinp

As we can't use ssh.VerifyNodeHostname directly, we'll set up a mini
node-verify to do checking between the master and the new node. In the
future networking checks, or more nodes, can be added as well.

Reviewed-by: iustinp

Since we're inside a LU we have access to self.sstore.
No need to use ss, which separate instantiation will disappear in a few
patches! ;)

Reviewed-by: iustinp

We used to scp all the ssconf files, and the vnc password file to the
new node. With this patch we use the upload_file rpc, specifying just
the new node as a destination. All the files previously copied by scp
are already allowed by the backend.

Reviewed-by: iustinp

What could possibly go wrong?

Reviewed-by: iustinp

Two LUs are using RunCmd to call fping, in order to check for an IP
presence on the network. Substituting it with TcpPing will get rid of
it, which makes it not break in the new world order, where the master
cannot fork.

Reviewed-by: iustinp

Reviewed-by: iustinp

- s/GanetiQuitException/QuitGanetiException/
- Look for the arguments in err.args, not err itself

Reviewed-by: iustinp

Rather than packing all the arguments in a tuple, let's pass them
plainly. The superclass won't complain.

Reviewed-by: iustinp

Having a timestamp on log messages is very useful. The default
format string doesn't include a timestamp.

Reviewed-by: ultrotter

Even in 1.2 this behaviour is broken, as the rpc call will remove the
ssh keys before we get a chance to log in. Now the rpc takes care of
shutting down the node daemon as well, so we definitely can avoid this.

This makes the LURemoveNode operation work again with the threaded
master daemon.

Reviewed-by: iustinp

Accoring to the usage documented in the QuitGanetiException docstring,
if we receive such an exception we'll set the global _EXIT_GANETI_NODED
variable to True, and then return either a valid value or an error
message to the user. This will be the last request we serve, though,
because the main loop will be interrupted and the daemon will terminate.

Reviewed-by: iustinp

This exception does not signal an error but serves the purpose of making
the ganeti daemon shut down after handling a request. Currently it will
be used by ganeti-noded but in the future ganeti-masterd might make use
of it as well. Its usage is documented in the docstring.

Reviewed-by: iustinp

Rather than calling httpd.serve_forever() in ganeti-noded we'll call
httpd.handle_request() but just while a global variable, which we'll
call _EXIT_GANETI_NODED, remains false.

Reviewed-by: iustinp

Reviewed-by: iustinp

There is no point in checking whether the cluster VNC password file
exists as a prerequisite for AddNode, considering the check happens on
the master node, not the target one. Removing this check.

Reviewed-by: iustinp

Reviewed-by: iustinp

If --no-restart is passed to devel/upload, it'll not run
"/etc/init.d/ganeti restart" (which kills processes), making
debugging on a terminal a bit easier.

Reviewed-by: iustinp, ultrotter

Apparently there were still some leftovers. While removing an instance,
I got the message "unhandled exception 'module' object has no attribute
'LD_MD_R1'".

Reviewed-by: iustinp

Currently, when seeing if a LV is degraded or not (i.e. virtual volume),
we first attach to the device (which does an lvdisplay), then do a lvs
in order to display the lv_attr. This generates two external commands to
do (almost) the same thing.

This patch changes the Attach() method for LVs to call lvs and display
both the major/minor (needed for attach) and the lv_status (needed for
GetSyncStatus). Thus, later in GetSyncStatus, we don't need to run lvs
again, and instead just return the value computed in Attach().

Reviewed-by: imsnah

This makes it easier to setup new git repositories, and makes it more
likely all people have the same ignore rules.

Reviewed-by: imsnah

Reviewed-by: iustinp

Reviewed-by: iustinp

Since the disabling of forking in the master daemon, the two ssh-based
subcommands were not working anymore. However, there is no need at all
for the commands to be run from the master daemon (permissions to read
the cluster private ssh key notwithstanding), they can be run directly
from the command line utilities.

The patch removes the two opcodes OpRunClusterCommand and
OpClusterCopyFile (and their associated LUs) and changes the code in
‘gnt-cluster’ to query the list of nodes and run directly the SshRunner
over the list. As such, all forking is done from the gnt-cluster script,
and the commands are working again.

Reviewed-by: imsnah

If an uncaught exception is thrown currently it destroys the calling
thread. This patch changes the behaviour to failing the current job,
logging a message, but trying to keep the daemon up.

Reviewed-by: imsnah

Reviewed-by: iustinp

Automake automatically appends "--" to @mkdir_p@. In case you have
a directory named "--" in your source tree, you can remove it using
the command "rm -rf -- --".

Reviewed-by: iustinp

Reviewed-by: ultrotter

Reviewed-by: iustinp