The import/export daemon code is already large. Moving some code
to a separate module will make it smaller and easier to test.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is a first version of the instance move tool and it supports moving
1..N instances from one cluster to another. When moving a single instance,
the instance can be renamed, allowing for moves within the same cluster
(not really useful in practice, but good for testing). Documentation is
updated to describe this new tool and its usage.

The “move-instance” tool uses the workerpool to support parallel moves of
instances. Supporting them was simple as threads were required anyway due
to the synchronous RAPI client.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Importing/exporting an instance to a remote machine creates X509
certificates which expire after some time. They need to be removed from
the nodes as they become useless.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This also updates a comment in the unittest for utils.py. We unittest
the new function for two things: correct reporting on real case (for
localhost), and correct reporting with a mocked-out TcpPing that returns
false.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

As mentioned in commit d975f482, the current way of creating an
instance via RAPI is not very flexible. With this patch, a new
instance creation request data format is introduced and documented.
Support can be detected by checking the list of features returned
by the /2/features resource for the value "instance-create-reqv1".

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add Mlockall() utility to lock current process' virtual adress space into RAM.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The RAPI client module shouldn't depend on any Ganeti module, yet it's
useful to have some Ganeti-specific code, like a PollJob function for
RAPI.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: David Knowles <dknowles@google.com>

Interpreting the backend's import/export daemon status is a bit tricky.
This utility code keeps track of multiple transfers at the same time.
Users can supply callback functions to react to events.

Timeouts are currently hardcoded. Intra-cluster instance moves will likely
require other timeouts.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

utils.py, where they were before, is already huge.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

distcheck, lint and apidoc. Take three pay only one.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be used to keep masterd-specific code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 5fa16422 removed the checks at sed time, but only for %.7. We
need to do it for %.8 too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

*.py[co] were not removed from lib/confd. Removing *.py[co]
from all directories listed in $DIRS can't hurt.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This backend daemon for instance import and export will be used to
transfer instance data to other machines. It is implemented in a generic
way to support different ways of data input and output. The third-party
program “socat”, which is already used by the KVM hypervisor abstraction,
is used to connect to remote machines using SSL/TLS. After starting the
child processes in a separate process group, the import/export daemon
monitors their output and updates a status file regularily. This status
file can then be read by ganeti-noded (not in this patch).

Three I/O methods are supported: Raw disk, file and script. Each of these
can be used for import and export.

Similar to daemon-util, an incomplete set of tests written in Bash is
included.

Two future enhancements are planned:
- Run parts of the command chain as a dedicated user (privilege
  separation).
- Currently users of this daemon have to poll the status file while data
  is transferred. This is inefficient and creates unnecessary delays. By
  adding “dd” into the chain and sending it SIGUSR1 regularily, we can get
  some statistics, optimize the polling frequenc and even provide the user
  with an ETA (which isn't available with all current methods to
  import/export instance data).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

2.1's cfgupgrade doesn't support upgrading from 1.2, only from 2.0.
However, it's trivial to forward-port the 2.0 cfgupgrade to work with
2.1, thus providing an upgrade path for 1.2 users directly to 2.1,
without the intermediate step of installing 2.0.

The new cfgupgrade12 file has just a few trivial changes from 2.0, and
skipping the load of the config file.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This can be used for two purposes:

- safety copy of the config file, with just the secrets changed
- cleanup of the config file (full randomization), so that (e.g.) users
  could send a broken config file to the devel-list

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is to keep same naming across all tests (modules separate with dot,
followed by _unittest.py).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: David Knowles <dknowles@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
(modified slightly the unittest to account for
 missing httplib2 library)

Certificates and keys generated using these functions will be used for
inter-cluster instance moves. As per design, the private key should never
leave the node.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

First, we don't need to check man pages at sed time, because this means
everyone building the package will do - we only need to check at docbook
time, which is mostly at developer time.

Second, don't force LC_ALL to C, as this breaks newer man-db. I've
verified and removing LC_ALL works fine across etch, hardy, lenny aid
squeeze/sid.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This test doesn't cover everything, but it's better than nothing.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

X509 uses ASN1 GENERALIZEDTIME or UTCTIME to store certificate validity.
pyOpenSSL 0.7 and above allow us to retrieve both “notBefore” and
“notAfter” as strings. Parsing them turned out to be a challenge since
they can be in a variety of formats (YYYYMMDDhhmmssZ, YYYYMMDDhhmmss+hhmm
or YYYMMDDhhmmss-hhmm).

This will be used to verify the validity of cluster certificates in
LUVerifyCluster.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Rationale: the file-based storage backend can add/remove files under a
certain directory. However, the master node is also controlling the
setting of the file-based root directory, so basically it means we can't
prevent arbitrary modifications by the master of the node's filesystem.

In order to mitigate this for setups where the file-based storage is not
used, we introduce a new setting at ./configure time, that controls the
enable/disable of file-based storage. Since this is not modifiable by
the master (over RPC), it is now possible in this case to prevent
unintended modifications of the node's filesystem from the master.

The new setting is used in bdev.py to not expose the file-based storage
at all, and in cmdlib.py to prevent attempts at creation of such
instances.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds ability to customize ssh config directory with --with-ssh-config-dir
(instead of hardcoded /etc/ssh value). This is useful in Linux distributions with
custom ssh config directories (/etc/openssh in ALTLinux, for example).

Signed-off-by: Vitaly Kuznetsov <vitty@altlinux.ru>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is the implementation of docs/design-cluster-merger.rst. It allows
the automatic merging of one or more clusters into the invoking cluster.

While this version is tested and working it still needs some tweaking
here and there for error handling and user experience.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

According to “coverage”, this covers 99% of the code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This saves us one dependency and saves us from complicated handling of
external files if we need key and certificate separated from each other.

At the same time, the number of bits used for RSA keys is increased from
1024 to 2048.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

On fork, the tempfile module's pseudo random generator is
not reset. If several processes (e.g. two children or parent
and child) try to create a temporary file, they'll conflict.
This function can be used to reset the name generator which
contains the pseudo random generator.

A unittest is included. It is in a separate script because
it changes a variable in the tempfile module to speed up the
test.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(applied slightly modified patch)

This reverts commit 83d9f436.

man is still unable to wrap some long lines, so we simply revert this patch
(and filter out the specific message in autotools/check-man).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

A long PREFIX variable (to configure) will result in very long
LOCALSTATEDIR, which when concatenated with lib/ganeti/ (and even more
items under it) will go over the 80 char line length we enforce in the
man checker.

To workaround this, we change two things:

- use a specific REPLACE_VARS_MAN which adds breaking points after each
  slash in paths
- replace some <filename> entries with <literallayout> so that docbook
  generates a non-fill block around them (only a few cases need this
  after the breaking points are added

Note that with normal prefixes (e.g. / or /usr/local) this won't happen.

The patch also fixes a wording in the watcher man page.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

One fix is necessary in gnt-cluster.sgml. Also adding “DELETE_ON_ERROR”
target to remove output file if an error occurred while building it (in
this case the manpage).

This was reported by Iustin Pop in issue 87 and proposed check method
taken from Lintian.
http://code.google.com/p/ganeti/issues/detail?id=87



Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>