utils.py, where they were before, is already huge.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

distcheck, lint and apidoc. Take three pay only one.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be used to keep masterd-specific code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 5fa16422 removed the checks at sed time, but only for %.7. We
need to do it for %.8 too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

*.py[co] were not removed from lib/confd. Removing *.py[co]
from all directories listed in $DIRS can't hurt.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This backend daemon for instance import and export will be used to
transfer instance data to other machines. It is implemented in a generic
way to support different ways of data input and output. The third-party
program “socat”, which is already used by the KVM hypervisor abstraction,
is used to connect to remote machines using SSL/TLS. After starting the
child processes in a separate process group, the import/export daemon
monitors their output and updates a status file regularily. This status
file can then be read by ganeti-noded (not in this patch).

Three I/O methods are supported: Raw disk, file and script. Each of these
can be used for import and export.

Similar to daemon-util, an incomplete set of tests written in Bash is
included.

Two future enhancements are planned:
- Run parts of the command chain as a dedicated user (privilege
  separation).
- Currently users of this daemon have to poll the status file while data
  is transferred. This is inefficient and creates unnecessary delays. By
  adding “dd” into the chain and sending it SIGUSR1 regularily, we can get
  some statistics, optimize the polling frequenc and even provide the user
  with an ETA (which isn't available with all current methods to
  import/export instance data).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

2.1's cfgupgrade doesn't support upgrading from 1.2, only from 2.0.
However, it's trivial to forward-port the 2.0 cfgupgrade to work with
2.1, thus providing an upgrade path for 1.2 users directly to 2.1,
without the intermediate step of installing 2.0.

The new cfgupgrade12 file has just a few trivial changes from 2.0, and
skipping the load of the config file.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This can be used for two purposes:

- safety copy of the config file, with just the secrets changed
- cleanup of the config file (full randomization), so that (e.g.) users
  could send a broken config file to the devel-list

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is to keep same naming across all tests (modules separate with dot,
followed by _unittest.py).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: David Knowles <dknowles@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
(modified slightly the unittest to account for
 missing httplib2 library)

Certificates and keys generated using these functions will be used for
inter-cluster instance moves. As per design, the private key should never
leave the node.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

First, we don't need to check man pages at sed time, because this means
everyone building the package will do - we only need to check at docbook
time, which is mostly at developer time.

Second, don't force LC_ALL to C, as this breaks newer man-db. I've
verified and removing LC_ALL works fine across etch, hardy, lenny aid
squeeze/sid.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This test doesn't cover everything, but it's better than nothing.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

X509 uses ASN1 GENERALIZEDTIME or UTCTIME to store certificate validity.
pyOpenSSL 0.7 and above allow us to retrieve both “notBefore” and
“notAfter” as strings. Parsing them turned out to be a challenge since
they can be in a variety of formats (YYYYMMDDhhmmssZ, YYYYMMDDhhmmss+hhmm
or YYYMMDDhhmmss-hhmm).

This will be used to verify the validity of cluster certificates in
LUVerifyCluster.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Rationale: the file-based storage backend can add/remove files under a
certain directory. However, the master node is also controlling the
setting of the file-based root directory, so basically it means we can't
prevent arbitrary modifications by the master of the node's filesystem.

In order to mitigate this for setups where the file-based storage is not
used, we introduce a new setting at ./configure time, that controls the
enable/disable of file-based storage. Since this is not modifiable by
the master (over RPC), it is now possible in this case to prevent
unintended modifications of the node's filesystem from the master.

The new setting is used in bdev.py to not expose the file-based storage
at all, and in cmdlib.py to prevent attempts at creation of such
instances.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds ability to customize ssh config directory with --with-ssh-config-dir
(instead of hardcoded /etc/ssh value). This is useful in Linux distributions with
custom ssh config directories (/etc/openssh in ALTLinux, for example).

Signed-off-by: Vitaly Kuznetsov <vitty@altlinux.ru>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is the implementation of docs/design-cluster-merger.rst. It allows
the automatic merging of one or more clusters into the invoking cluster.

While this version is tested and working it still needs some tweaking
here and there for error handling and user experience.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

According to “coverage”, this covers 99% of the code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This saves us one dependency and saves us from complicated handling of
external files if we need key and certificate separated from each other.

At the same time, the number of bits used for RSA keys is increased from
1024 to 2048.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

On fork, the tempfile module's pseudo random generator is
not reset. If several processes (e.g. two children or parent
and child) try to create a temporary file, they'll conflict.
This function can be used to reset the name generator which
contains the pseudo random generator.

A unittest is included. It is in a separate script because
it changes a variable in the tempfile module to speed up the
test.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(applied slightly modified patch)

This reverts commit 83d9f436.

man is still unable to wrap some long lines, so we simply revert this patch
(and filter out the specific message in autotools/check-man).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

A long PREFIX variable (to configure) will result in very long
LOCALSTATEDIR, which when concatenated with lib/ganeti/ (and even more
items under it) will go over the 80 char line length we enforce in the
man checker.

To workaround this, we change two things:

- use a specific REPLACE_VARS_MAN which adds breaking points after each
  slash in paths
- replace some <filename> entries with <literallayout> so that docbook
  generates a non-fill block around them (only a few cases need this
  after the breaking points are added

Note that with normal prefixes (e.g. / or /usr/local) this won't happen.

The patch also fixes a wording in the watcher man page.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

One fix is necessary in gnt-cluster.sgml. Also adding “DELETE_ON_ERROR”
target to remove output file if an error occurred while building it (in
this case the manpage).

This was reported by Iustin Pop in issue 87 and proposed check method
taken from Lintian.
http://code.google.com/p/ganeti/issues/detail?id=87



Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Otherwise we depend on PATH at runtime.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch adds a configure-time parameter that will set the defaults
used by all programs, and command-line parameters in the daemons that
allow overriding it.

Syslog 'yes' enables syslog in addition to file-based logging, 'only'
enables syslog and disables file-based syslog.

The log entries will be of the form:
Jan 27 08:45:04 node2 ganeti-noded[14504]: INFO 172.24.227.5:50850 PUT
/jobqueue_update HTTP/1.0 200
Jan 27 08:45:05 node2 ganeti-noded[14505]: INFO 172.24.227.5:50853 PUT
/lv_list HTTP/1.0 200

and (for a multi-threaded program):
Jan 27 08:51:48 node1 ganeti-masterd[15491]: (MainThread) INFO
ganeti-masterd daemon startup
Jan 27 08:51:49 node1 ganeti-masterd[15491]: (MainThread) INFO
Inspecting job queue

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Ideally we want to/will have per-device DRBD controls of disk/metadata
flushes. In the meantime, we want at least a disable of the barrier
functionality for cases where one has battery-backed caches.

Background: DRBD has four mechanism of handling ordered disk-writes.
From the drbdsetup man-page, these are: barrier, flush, drain and none.
DRBD prior to 8.2 only has drain and none. This patch makes all 8.x
versions of DRBD disable all methods, and revert to none, in case one
fully trusts batteries (either UPS for the whole system or battery for
NVRAM).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Also fix a typo in http/__init__.py and add unittests
for the LUXI parsing and formatting functions.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Look it up at configure time instead.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This partially reverts commit e4e7c7df,
which made newer pylint versions fail.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

That is, all Python code except unittests. In particular,
autotool/build-bash-completion was missing from the
previous list.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This allows automated builders to override the format, for example.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

This runs pylint over all the python files. Yes, it takes a long while,
but it's the only way to properly analyze the source codes as only in
this way pylint can see all uses of the various modules/classes/etc.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>