Since we share the ConfigWriter we need somehow to make sure that
accessing it is properly synchronized. We'll do it using the
locking.ssynchronized decorator and a module-private shared lock.

This patch also renames a few functions, which were called inside the
ConfigWriter, to a private version _UnlockedFunctionName, and exports
the synchronized public ones. The internal callers, which are already
synchronized, are then changed to use the _Unlocked version, to prevent
double locking.

Reviewed-by: iustinp

This patch creates a new decorator function ssynchronized in the locking
library, which takes as input a SharedLock, and synchronizes access to
the decorated functions using it. The usual SharedLock semantics apply,
so it's possible to call more than one synchronized function at the same
time, when the lock is acquired in shared mode, and still protect
against exclusive access.

The patch also adds a few unit test to check the basic decorator's
functionality, and to provide an example on how to use it.

Reviewed-by: iustinp

Remove empty function _ReleaseLock and all its calls. Since we only
have one configwriter per cluster the locking needs to cover all the
data in the object, and not just the file contents. Locking in
ConfigWriter will be handled using the ganeti locking library.

Reviewed-by: iustinp

Reviewed-by: ultrotter

By a mistake, we don't reuse the luxi client. As such, we open and close
the connection at each poll cycle and spam the server logs.

Reviewed-by: ultrotter

It's better for daemons if:
  - they log only to one log file
  - the log level is included
  - for debug runs, the filename/line number is included

This patch moves the custom formatter from the watcher to the logging
module and generalizes it; then it changes the master daemon to use this
function instead of the generic logging (which might be deprecated
anyway in the future).

Reviewed-by: imsnah

Make the GanetiLockManager instance of GanetiContext lowercase

Reviewed-by: imsnah

This function will return the first unused integer based on a list of
used integers (e.g. [0, 1, 3] will return 2).

Reviewed-by: imsnah

If a LU required the BGL (all LUs do, right now, by default) we'll
acquire it in the Processor before starting them. For LUs that don't
we'll still acquire it, but in a shared fashion, so that they cannot run
together with LUs that do.

We'll also note down whether we own the BGL exclusively, and if we don't
and we try to chain a LU that does, we'll fail.

More work will need to be done, of course, to convert LUs not to require
the BGL, but this basic infrastructure should guarantee the coexistance
of the old and new world for the time being.

Reviewed-by: iustinp

The processor used to create a new ConfigWriter when it was initialized.
We now have one in the context, so we'll just recycle it. First of all
we'll pass the context in when creating a new Processor object, then
we'll just use context.cfg, which is granted to be initialized, wherever
we used self.cfg, and stop checking whether the config is already
initialized or not.

In the future the Processor will be able to use the context also to
acquire the BGL for LUs that require it, and to push the context down to
LUs that don't in order for them to manage their own locking.

Reviewed-by: iustinp

When logical units have REQ_BGL set (it is currently the default) they
need to be the only ganeti operation run on the cluster, and we'll
guarantee it at the master daemon level. Currently only one thread is
running at a time, so this requirement is never broken.

Reviewed-by: iustinp

- no need to keep the sstore as an object member, remove it
- don't reinitialize sstore only if self.cfg is None
    This is not an issue, as the Processor is recycled for every opcode,
    but in general we know that (a) we might need a different type of
    sstore for different opcodes and (b) initializating them is cheap
- recreate sstore when chaining opcodes
    Without this fix chaining an opcode which requires a writable sstore
    to one which doesn't would fail. This doesn't happen today, but it's
    better to fix it anyway

These changes are possible because nowadays all opcodes already require
a working cluster/configuration.

Reviewed-by: iustinp

From the master node we can't start ssh and connect to the remote node,
nor we can do it from ganeti-noded as this ssh section will possibly ask
for key confirmation and password. So the code to copy the ganeti-noded
password and SSL key has been moved to bootstrap.py, and it's called by
gnt-node before the AddNode opcode.

Reviewed-by: iustinp

As we can't use ssh.VerifyNodeHostname directly, we'll set up a mini
node-verify to do checking between the master and the new node. In the
future networking checks, or more nodes, can be added as well.

Reviewed-by: iustinp

Since we're inside a LU we have access to self.sstore.
No need to use ss, which separate instantiation will disappear in a few
patches! ;)

Reviewed-by: iustinp

We used to scp all the ssconf files, and the vnc password file to the
new node. With this patch we use the upload_file rpc, specifying just
the new node as a destination. All the files previously copied by scp
are already allowed by the backend.

Reviewed-by: iustinp

What could possibly go wrong?

Reviewed-by: iustinp

Two LUs are using RunCmd to call fping, in order to check for an IP
presence on the network. Substituting it with TcpPing will get rid of
it, which makes it not break in the new world order, where the master
cannot fork.

Reviewed-by: iustinp

Reviewed-by: iustinp

Rather than packing all the arguments in a tuple, let's pass them
plainly. The superclass won't complain.

Reviewed-by: iustinp

Having a timestamp on log messages is very useful. The default
format string doesn't include a timestamp.

Reviewed-by: ultrotter

Even in 1.2 this behaviour is broken, as the rpc call will remove the
ssh keys before we get a chance to log in. Now the rpc takes care of
shutting down the node daemon as well, so we definitely can avoid this.

This makes the LURemoveNode operation work again with the threaded
master daemon.

Reviewed-by: iustinp

This exception does not signal an error but serves the purpose of making
the ganeti daemon shut down after handling a request. Currently it will
be used by ganeti-noded but in the future ganeti-masterd might make use
of it as well. Its usage is documented in the docstring.

Reviewed-by: iustinp

Reviewed-by: iustinp

There is no point in checking whether the cluster VNC password file
exists as a prerequisite for AddNode, considering the check happens on
the master node, not the target one. Removing this check.

Reviewed-by: iustinp

Reviewed-by: iustinp

Apparently there were still some leftovers. While removing an instance,
I got the message "unhandled exception 'module' object has no attribute
'LD_MD_R1'".

Reviewed-by: iustinp

Currently, when seeing if a LV is degraded or not (i.e. virtual volume),
we first attach to the device (which does an lvdisplay), then do a lvs
in order to display the lv_attr. This generates two external commands to
do (almost) the same thing.

This patch changes the Attach() method for LVs to call lvs and display
both the major/minor (needed for attach) and the lv_status (needed for
GetSyncStatus). Thus, later in GetSyncStatus, we don't need to run lvs
again, and instead just return the value computed in Attach().

Reviewed-by: imsnah

Reviewed-by: iustinp

Since the disabling of forking in the master daemon, the two ssh-based
subcommands were not working anymore. However, there is no need at all
for the commands to be run from the master daemon (permissions to read
the cluster private ssh key notwithstanding), they can be run directly
from the command line utilities.

The patch removes the two opcodes OpRunClusterCommand and
OpClusterCopyFile (and their associated LUs) and changes the code in
‘gnt-cluster’ to query the list of nodes and run directly the SshRunner
over the list. As such, all forking is done from the gnt-cluster script,
and the commands are working again.

Reviewed-by: imsnah

Reviewed-by: ultrotter

In cfgupgrade, we need to extract parts of and build new version numbers.

Reviewed-by: iustinp

cfgupgrade will not work with relative paths at all, but rather get them
from constants.py.

Reviewed-by: iustinp

Currently there isn't any easy way to list all nodes or instance and
their tags; you have to query each node in turn, or list all the tags
via something like “gnt-cluster search-tags '.*'”. Of course, this is
not optimal.

The patch adds a new fields to “gnt-instance list” and “gnt-node list”
called ‘tags’, that will list the tags of the object in comma-separated
form. This field will be empty if there are no tags (when using a
separator this output can still be parsed by other scripts).

At opcode level, there is a new fields called ‘tags’ that returns a
(python) list of the object tags.

Reviewed-by: ultrotter

Currently the generic handling of ganeti errors in cli.py (GenericMain
and FormatError) only handles the core ganeti errors, and not the client
protocol errors (which live in a separate hierarchy).

This patch adds handling of luxi errors too, and also adds another luxi
error for the case when the master is not running. This gives us a nice:

  gnta1:~# gnt-node list
  Cannot communicate with the master daemon.
  Is it running and listening on '/var/run/ganeti-master.sock'?

error message instead of a traceback.

Reviewed-by: amishchenko

This patch adds rpc layer calls (in rpc.py and the equivalent in
ganeti-noded) to close a list of block devices, and the wrapper in
backend.py that takes a list of Disk objects, identifies them and
returns correctly formatted results.

The reason why this very basic call was missing until now from the rpc
layer is that we usually don't care about device closes (though we
should, and will do so in the future) as only drbd has a meaningful
Close() operation; right now we directly do Shutdown().

The patch is clean enough that it's actually independent of the live
migration implementation.

Reviewed-by: imsnah

This change allows us to use cleaner dependencies between
directories. The build system is basically rewritten in large parts
and may contain bugs.

Reviewed-by: iustinp

Currently, compute the status of a drbd8 device in GetSyncStatus and
return only the values that we need (and fit in the framework of
GetSyncStatus). However, the full status details are useful (and needed)
in other places, so the patch attempts to improve this situation.

We abstract the status of a device outside in a separate class, that
knows how to parse contents from /proc/drbd and set easily accessible
attributes. We then simplify the GetSyncStatus to use this and return
the values that it needs, and add a separate method that returns the
full status object.

The move to a separate class cleans up a little bit the old
sync-progress computation from GetSyncStatus, but it's still many
regexes.

The patch also adds unittests for a few statuses, and modifies one
BaseDRBD call to accept a custom filename instead of '/proc/drbd' to
ease unittests.

Reviewed-by: imsnah

Reviewed-by: iustinp

- Make closing file optional: Required by ganeti-watcher to keep
  file open after writing it. Changes return value of utils.WriteFile
  if "close" parameter evaluates to True.
- Pre- and post-write functions: Can be used to lock files. This
  will be used by ganeti-watcher to lock the temporary file before
  renaming.

Reviewed-by: iustinp