With this patch we simplify usage on the SingleActionCondition (which
wasn't a condition at all) by making it a real condition. This way we
can just wait() on it, or notifyAll() as we would on a normal one. The
only catch is that notifyAll can be called only once, and wait can only
be called before notifyAll has, but luckily our PipeCondition, now quite
simplified, takes care of this, by providing a new SingleActionCondition
each time the previous one has been notified.

No Start/StopUsing function are needed anymore, and thus the condition
is a lot more robust, and there's no way file descriptors can be left
open, as they are closed in a finally block, in the same function where
they were opened, by the last thread exiting the class.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This way they can be used to test different condition classes.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

All (ok, all but one) _ThreadedTestCase users have a done Queue, so we
move its building in the _ThreadedTestCase setUp

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Each condition has an underlying lock, the acquire and release methods,
and a few helper methods to check that it's called in the proper way.

Abstract them to a separate class so we can have more than one without
duplicating this code.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

SharedLock.__acquire_unlocked uses keyword parameters. Just passing
the timeout would set the “shared” parameter.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Also remove the “blocking” parameter from LockSet.remove and
GanetiLockManager.remove. There's no point in implementing timeouts on removal
unless we need them.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Our automated test system found a few problems in the new locking
unittests. This patch should fix them, although I wasn't able to
reproduce the problem. All are race conditions.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

_PipeCondition is a condition implemented using pipe(2) and poll(2).
It allows the implementation of timeouts without using a busy-wait loop
with time.sleep.

Unlike Python's built-in threading.Condition class and to save file
descriptors and an internal queue, it can only be used to notify
all waiters. Ganeti's use case for this condition class doesn't
require the ability to notify only one waiter.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This class will be used as a basic block for pipe(2)-based
conditions. Upon initialization it creates a pipe and can be
notified once (hence the “single action” in the name). A
callable helper class is used to wait for notifications.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch greatly simplifies the SharedLock code and implements
timeouts for the acquire() and delete() functions. A wrapper around
Python's threading.Condition class must be used to ensure thread
safety when check whether there are any waiters left.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The query now accepts a link parameter.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

* origin/master:
  Fix burnin's verbose mode
  Final NEWS update and version increase for 2.0.4
  Encode the actual exception raised by LU execution
  Move the luxi error handling into errors.py
  Fix the confusing ssh/hostname message in node add
  Add man page for ganeti-cleaner
  Update NEWS file for version 2.0.4
  Automatically cleanup _temporary_ids at save
  Separate the computation of all config IDs
  Change config upgrade to be explicit
  Fix unittest breakage due to new test file
  Fix /proc/drbd parsing in presence of gaps
  repair-size: ensure child disks have sane sizes
  Fix yet another bug in LURepairDiskSizes
  Fix a bug in LURepairDiskSizes

Conflicts:
	NEWS          (trivial, the RST change)
	lib/cmdlib.py (trivial, some small change in 2.0)
	lib/config.py (due to the cherry-picks and UUID changes in 2.1)

Signed-off-by: Iustin Pop <iustin@google.com>

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Remove variables which are listed at the beginning of the section and variables
which are not declared when building hooks env.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The timestamp need special formatting, which was done for the internal
buffer storage but not for the messages logged in verbose mode. This
patch unifies the formatting for these two cases.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

QA passed successfully, let's try to have a release.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Some basic tests for the confd client library

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Until now the peers have to be the same all the time. Adding a new
function to update the list, and call it from the constructor to avoid
duplicating code.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is not needed anymore - the original change was more than a year
ago when masterd was in its incipient phase.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

It may be handy for upcalls to know which client called them, and call
it back. So we create a new "client" field in the upcall target,
containing the current client instance

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Only install ganeti specific files if they exist. This way we can call
ganeti's devel/upload in another sub-projects (eg. nbma) and have it
uploaded to a host as well, without having to create a new script there.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The HandleExpire function takes the whole "up" structure, and not just
the salt.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Many functionalities of confd have been moved to other classes/modules,
and the main confd daemon doesn't reference these modules directly
anymore.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

ganeti-confd doesn't need to log anything related to os installations.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, the actual exception raised during an LU execution (one of
OpPrereqError, OpExecError, HooksError, etc.) is lost because the
jqueue.py code simply sets that to a str(err), and the code in cli.py
simply passes that string to OpExecError.

This patch moves to encoding the errors as per errors.EncodeError and
changes the cli code to parse and raise that (if possible).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
(cherry picked from commit bcb66fca)

Currently the luxi error handling is hardcoded as special encoding on
the masterd-side and special decoding on the client side. This patch
moves it to errors.py such that other parts of the code can reuse the
same encoding.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
(cherry picked from commit 6956e9cd)

Before, it used to say:

  ssh/hostname verification failed node1.example.com -> hostname mismatch, got
  node2

Now it says for wrong hostnames (maybe too verbose):

  ssh/hostname verification failed (checking from node1.example.com): hostname
  mismatch, expected node2.example.com but got node3

And for non-FQDN hostnames:

  ssh/hostname verification failed (checking from node1.example.com): hostname
  not FQDN: expected node2.example.com but got node2

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This callback can be stacked with another one, and will filter duplicate
or old results, making handling of results easier.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Populate the docstring with documentation on the client library's usage.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

We don't bump up the version yet, pending more QA tests.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch does a little bit of cleanup first, since we want to call
GenerateUniqueID without reacquiring the lock.

Note that we don't necessarily need to do this for the cluster, since at
first startup ConfigWriter will do it anyway. But it's better to
explicitely do this instead of relying on the automated upgrade.

Additionally this patch adds ctime/mtime population at cluster init
time. mtime is not necessarily needed (master will update it
automatically, but we're doing it anyway for consistency).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This also adds the new nic.links query field.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
(cherry picked from commit 34d657ba)

We will need this in another place, so we abstract the 'compute all
current IDs' functionality into a separate function. We also change the
name of the _ComputeAllLVs to _AllLVs to match the other _All*s
functions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
(cherry picked from commit 34e54ebc)

Currently the config upgrade is done at each object instantiation, that
means that ganeti-noded will run UpgradeConfig on all objects received
remotely (instances, disks, nics). This is not so good, so this patch
changes it so that only the ConfigWriter runs this method at
configuration load time.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
(cherry picked from commit 90d726a8)

* commit 'origin/next': (74 commits)
  Fix gnt-node modify online help
  Fix gnt-job info entry in gnt-job(8)
  locking: Don't swallow exceptions
  Add check for duplicate MACs in instance add
  scripts/gnt-node: fix a help string
  Optimise multi-job submit
  Extend gnt-debug with more debugging options
  Return cluster tags from LUQueryClusterInfo
  Add script to clean archived jobs after 21 days
  rapi: export more static node information
  Pass the correct signal to handlers
  cli: Use ToStdout/ToStderr instead of print
  Fix small typo in gnt-node
  Simplify handling of boolean args in rapi
  Fix checks in LUSetNodeParms for the master node
  Improve the example startup script
  Fix insserv dependencies
  Fix a typo in InitCluster
  Ignore results from drained nodes in iallocator
  Ship the ethers hook
  ...