Currently the RAPI client uses the urllib2 and httplib modules from
Python's standard library. They're used with pyOpenSSL in a very fragile
way, and there are known issues when receiving large responses from a RAPI
server.

By switching to PycURL we leverage the power and stability of the
widely-used curl library (libcurl). This brings us much more flexibility
than before, and timeouts were easily implemented (something that would
have involved a lot of work with the built-in modules).

There's one small drawback: Programs using libcurl have to call
curl_global_init(3) (available as pycurl.global_init) while exactly one
thread is running (e.g. before other threads) and are supposed to call
curl_global_cleanup(3) (available as pycurl.global_cleanup) upon exiting.
See the manpages for details. A decorator is provided to simplify this.

Unittests for the new code are provided, increasing the test coverage of
the RAPI client from 74% to 89%.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is needed in a new test script for cfgupgrade where it's better
to only use the minimum parts necessary for testing cfgupgrade.
Including full configuration data to ensure verification doesn't fail
would make maintenance difficult.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In the condition for checking the configuration directory, one “or” should have
been an “and”. This bug was in cfgupgrade since commit 95e4a814 (June 2008).
Found thanks to a test script.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Since configuration changes are now usually done in the configuration
writer class, cfgupgrade's only task is to change the config version.
This means we can easily support upgrading from Ganeti 2.0 and 2.1 to
2.2.

This patch also brings a couple of small cleanups.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Three constants were no longer user after commit 11c31f5c, which
removed support for upgrading from Ganeti 1.2.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This makes it compatible with the python builtin, and we can even use
the builtin when running under the right version of python. The all and
any functions are renamed to _all and _any, so that they can be tested,
and (non)existing unittests are updated (translation: there are no unit
tests, so none are updated).

This patch also moves the code deciding which partial to use next to
the definition of the _partial function, rather than after TryToRoman.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is not yet complete, as it lacks proper support for instance
import.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is necessary to allow cfgupgrade to work on a non-standard directory.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Until now, move-instance used different status variables: “success”,
“abort” and “error_message”. With this patch, everything is changed
to use “error_message” only. This simplifies the code a bit.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a first version of the instance move tool and it supports moving
1..N instances from one cluster to another. When moving a single instance,
the instance can be renamed, allowing for moves within the same cluster
(not really useful in practice, but good for testing). Documentation is
updated to describe this new tool and its usage.

The “move-instance” tool uses the workerpool to support parallel moves of
instances. Supporting them was simple as threads were required anyway due
to the synchronous RAPI client.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Importing/exporting an instance to a remote machine creates X509
certificates which expire after some time. They need to be removed from
the nodes as they become useless.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Since utils.WriteFile requires an absolute path, cfgupgrade needs to
normalize the directory it is given.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

utils.py, where they were before, is already huge.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The patch adds selection of hypervisor in burnin (on multi-hypervisor
clusters, we always used the default) and optional hypervisor
parameters, and skips migration if the hypervisor doesn't support it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In master, the function GenerateSelfSignedSslCert has been moved to
utils.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Additionally also move to the RunWhileStopped method for the configuration merger.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch updates the docstrings in tools/ (mostly in lvmstrap, which
is very very old code-base) and then enabled the tools in this directory
for 'make apidoc' too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

2.1's cfgupgrade doesn't support upgrading from 1.2, only from 2.0.
However, it's trivial to forward-port the 2.0 cfgupgrade to work with
2.1, thus providing an upgrade path for 1.2 users directly to 2.1,
without the intermediate step of installing 2.0.

The new cfgupgrade12 file has just a few trivial changes from 2.0, and
skipping the load of the config file.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This can be used for two purposes:

- safety copy of the config file, with just the secrets changed
- cleanup of the config file (full randomization), so that (e.g.) users
  could send a broken config file to the devel-list

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Commit 43575108 added bootstrap.GenerateclusterCrypto and commit
7506a7f1 changed cfgupgrade to use it. However, this lost the
functionality of upgrading in non-default DATA_DIR.

To fix this, we enhance bootstrap.GenerateclusterCrypto to accept custom
file paths for the three files it modifies. If more files will be needed
in the future, we could just pass in modified DATA_DIR, but for now it
suffices.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Currently burnin, if proceding in parallel, will remove all instances
which were passed, even if they failed to add. This is bad because it
will also remove instances which existed before burnin started. By
adding the instances to the removal queue only if their creation was
successful (passing the action as a post processing action to
ExecOrQueue) we guarantee pre-existing instances are saved.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

If a post-process function is passed to ExecOrQueue it is executed if
and only if the job is successful. This happens immediately if we're
proceding iteratively, and at the end, when we collect all job results,
if we're proceding in parallel.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In order to later add an optional parameter we transform the variable
ops argument list in an explicit list.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Most creation of confd clients will do the same steps: read MC file,
parse it, read HMAC key, etc. We abstract this functionality so that
we don't duplicate the code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Abstract the growable disk types in a ganeti constants, and only run
disk grow, from burnin, on them.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Just a few queries are checked, but this should give us confidence that
at least the basic confd framework is working properly.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Information exchanged between different clusters via untrusted
third parties (e.g. for remote instance import/export) must be
signed with a secret shared between all involved clusters to
ensure the third party doesn't modify the information.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, the ganeti-confd's HMAC key is called “cluster HMAC key” or
simply “HMAC key” everywhere. With the implementation of inter-cluster
instance moves, another HMAC key will be introduced for signing critical
data. They can not be the same, so this patch clarifies the purpose of the
“cluster HMAC key” by renaming it. The actual file name is not changed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Burnin has a custom job executor, because of its need to retry some job
series.

While we cannot replace all of it, at least the execution we can switch
to cli.JobExecutor, to take advantage of the recently-introduced
out-of-order waiting.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This makes the Log function able to take multiple args for simplified
message construction, similar to the ToStdout one.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This passes a full burnin with lots of instances, and should be safe as
we mostly to join a known root (various constants) to a run-time
variable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

I was using a too old version which doesn't got all those. This
patch is fixing the new lint errors.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is the implementation of docs/design-cluster-merger.rst. It allows
the automatic merging of one or more clusters into the invoking cluster.

While this version is tested and working it still needs some tweaking
here and there for error handling and user experience.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

There are two entry points to job execution in burnin, ExecOp and
ExecOrQueue, and these are modified to call the new _SetDebug method on
the opcodes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit 154b9580 changed (correctly) the __slots__ usage, but this broke
dumpers/loaders since we relied directly on the own class __slots__
field.

To compensate, we introduce a simple function for computing the slots
across all parent classes (if any), and use this instead of __slots__
directly.

Note: the _all_slots() function is duplicated between objects.py and
opcodes.py, but the only other options is to introduce a lang.py for
such very basic language items.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds an early_release parameter in the OpReplaceDisks and
OpEvacuateNode opcodes, allowing earlier release of storage and more
importantly of internal Ganeti locks.

The behaviour of the early release is that any locks and storage on all
secondary nodes are released early. This is valid for change secondary
(where we remove the storage on the old secondary, and release the locks
on the old and new secondary) and replace on secondary (where we remove
the old storage and release the lock on the secondary node.

Using this, on a three node setup:

- instance1 on nodes A:B
- instance2 on nodes C:B

It is possible to run in parallel a replace-disks -s (on secondary) for
instances 1 and 2.

Replace on primary will remove the storage, but not the locks, as we use
the primary node later in the LU to check consistency.

It is debatable whether to also remove the locks on the primary node,
and thus making replace-disks keep zero locks during the sync. While
this would allow greatly enhanced parallelism, let's first see how
removal of secondary locks works.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>