I still don't know why Michael is a fan of %r, but in the meantime
this patch changes:

WARNING: import u'import-2011-07-29_01_39_33-y3gZKV' on node1 failed:
Exited with status 1

into:

WARNING: import 'import-2011-07-29_01_39_33-y3gZKV' on node1 failed:
Exited with status 1

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

During instance installations, you do not want the instance to reboot
and start again with the same parameters, as that will most likely
re-start the install process. Therefore, when the instance requests a
reboot it should instead shutdown. This flag allows this to be
controlled.

Signed-off-by: Stephen Shirley <diamond@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The OS scripts currently run with the whole noded environment; this is
different from the hooks which run with a cleared one and most likely
an oversight.

This _might_ create problems when upgrading, so it needs to be clearly
announced for the new version.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

“upfile” is a bad name.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The node maintenance class is standalone.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Sphinx 1.0.7 complains if an indented block in .warning starts with :option.
This fixes it.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.4:
  Add support for cluster/OS parameters in QA
  Add OS search path to gnt-cluster info

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This removes:

- the check in backend that such OSes have a variants file or if it
  exists that is non-empty; in order for this to work, we also rework
  the logic in backend._TryOSFromDisk to allow for optional OS files
- the check in cluster verify such OSes to have a non-empty variant
  list (the check for consistent variants is still kept)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently there is no way to QA with (for example) an initrd because
the QA only inits the cluster with the default parameters. This makes
it impossible to QA using anything but the default parameters, which
doesn't always work.

Additionally, we add OS parameters and OS hypervisor parameters, for
completeness and for testing that these commands also work.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This reverts commit 7421df8e.

The feedback_fn argument to JobExecutor is used for PollJob, and thus
has a fixed signature: a single arg, tuple of (timestamp, log type,
log message). It's use as drop-in replacement for ToStdout doesn't
work, as that function has a different signature.

For now, I propose to revert this, until we either change JobExecutor
to use the same log messages (and add an intermediate wrapper between
JobExecutor and ToStdout) or we add another parameter to
JobExecutor.__init__.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This seems to be used and reused multiple times, let's abstract it…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit aef59ae7 reworked the file verification, but forgot to take
into account offline nodes.

The fact that this was not detected yet is due to the fact that we
don't test clusters with offline nodes in QA :(

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Otherwise we get no variant checks at all, but the variant is still
recorded.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The patch changing the OS api in QA to 20 was not complete, sorry.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

v20 is (mostly) a superset of the other versions, so testing with it
should be better than with V10. This detects properly the breakage
fixed by the previous patch.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

OS parameters is a list of tuples, so we can't pass it directly to
utils.NiceSort, hence we use a sort key.

This was not detected in QA since QA only tests API v10 :(

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds a function for abstracting
“dict.fromkeys(locking.LEVELS, 1)”. It also removes a duplicate
assignment for the share_locks in LUInstanceQuerydata.

Additionally, it moves the _SupportsOob function to the helper
function list.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This adds the infrastructure necessary to check opcode results using
ht-based functions. Checks are added for two opcodes.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Until now verifying disks, which is also used by the watcher,
would lock all nodes and instances. With this patch the opcode
is changed to operate on per nodegroup, requiring fewer locks.

Both “gnt-cluster” and “ganeti-watcher” are changed for the
new interface.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be used in the watcher where we don't want to
pollute stdout unless in debug mode.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Otherwise, it's pretty hard to figure it out from the command line.

Signed-off-by: Ben Lipton <benlipton@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It doesn't make sense to have to keep them up to date twice, and --help
already lists all of them with help strings.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Right now we always restart all instances, which is not right if some
instances were already down for other reasons. Thus we add an option to
decide how to handle this. The right default should be "up" which is:
"restart all options which were switched off by the merge", but since
that's not implemented yet, the default remains the old one, for now.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, most htools code depends on Constants.hs which is generated
from constants.py and also depends on _autoconf.py. Also, _autoconf.py
depends on vcs-version, which all together means that when 'make
regen-vcs-version' is run, for example by ./devel/upload, most of the
Haskell code needs recompilation.

Since htools already has its 'optimised' vcs-version (and doesn't use
the _autoconf.VCS_VERSION constants), we can optimise this as follows:

- _autoconf.py doesn't contain the VCS_VERSION anymore, and that is
  instead moved to _vcsversion.py
- constants.py depends on and imports this new module
- _autoconf.py doesn't get regenerated at vcs-version changes, but
  only at re-running configure/changing Makefile time

The end result is that only htools/Ganeti/HTools/Version.hs is
recompiled now, which is a significant speedup (usually < 1 second
versus 10 seconds previously).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

s/'/"/ in (hopefully) the right places.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

* devel-2.4:
  Reopen daemon's stdio on SIGHUP
  Reopen log file only once after SIGHUP
  Don't leak file descriptors when setting up daemon output
  Fix aliases in bash completion

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Before this patch daemons would continue to refer to an old logfile for
their standard I/O if they had been asked to reopen the log (SIGHUP).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit b6fa9a44 added a re-openable log handler. The log file is
reopened when a daemon is sent a HUP signal. Due to a bug in the code,
fixed by this patch, the log file would be reopened for every single log
message thereafter.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

When a daemon's output is configured using “utils.SetupDaemonFDs”, the
function must use dup2(2). Unfortunately the code didn't close the
original file descriptors, leaking them in the process.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

I think I've identified the problem with the current ChangeAll
mode. The current algorithm works as follows:

- identify a new primary by choosing the node which gives best score
  as new secondary
- failover to it
- identify a new secondary by choosing the node which gives best score
  as new secondary

This means that the future primary is 'fixed' after the first
iteration, leaving to possibly suboptimal results. This patch changes
the algorithm to do what, in hindsight, seems the obvious thing to do:
- generate all pairs (primary, secondary)
- identify the pair that after the above sequence (r:np, f, r:ns)
  gives the best group score

This fixes some of the corner cases I've seen in relocation, but not
all; the remaining cases are related to multi-instance relocation and
while they can't be fixed in the current framework, the needed
rebalancing is much smaller than with the current algorithm.

The patch also fixes an issue with the docstring of another function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Before this patch “gnt-instance info” would fail with the error message
“Error checking node $node: Node is marked offline” if the instance's
primary node is marked offline and the user didn't explicitely request
static information only. With this patch the LU will automatically
return static information if the instance's primary node is marked
offline.

Some explicit loops are changed to map().

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

When the source node for a failover is marked offline, there's no need
to require the user to specify “--ignore-consistency”.

To make it work at all, a number of bugs introduced by the merge of
migration and failover are also fixed by this patch.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>