* stable-2.6: (26 commits)
  Bump version for beta 2 release
  Fix epydoc raises L{...} usage
  Fix race condition in test for *FileID functions
  Move KVM keymap from GenerateRuntime to ExecuteRuntime
  Add the keymap directory to the list of runtime KVM dirs
  Fix make -n use
  Enable hlint in lint and reorganise this target
  Improve error message for auto-promote/node modify
  Fix QA import of existing instances
  Remove one obsolete hlint override
  Fix prefix bug in Haskell ssconf implementation
  Move the ssconf file prefix constant to constants.py
  QA: Add configuration options for static MAC address
  QA: Ignore tags using regular expression
  Fix parallel build failures
  QA: Add default setting for tests
  QA: add toggle for fewer common instance tests
  QA: stop logging SSH arguments for each invocation
  Switch QA to tty-less mode
  Fix a type issue and bad logic in cluster verification
  ...

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

I've added the changes under beta1, not beta2, since it's easier to
keep it separate.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

For epydoc raise statement, it already expects an exception which will
be hyperlinked; if passing an L{…} construct, it will instead break
the formatting (the @raise will not be recognised, but merged into the
previous paragraph).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

* stable-2.5:
  Fix race condition in test for *FileID functions
  Move KVM keymap from GenerateRuntime to ExecuteRuntime
  Add the keymap directory to the list of runtime KVM dirs

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In this test the “file ID” of a temporary file is compared against the
file ID gathered via an open file descriptor to the same file. For
reasons unknown to me utime(2) is called in-between to update the
inode's a- and mtime. Depending on the file system's timestamp
resolution this can lead to a different file ID.

Found by chance during QA and reproduced by adding a delay before the
call to utime(2).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit fbd55434)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Per issue 243, "side-effects" are GenerateRuntime are bad as they
execute only on the initial node of the instance. By moving the
write-out of the keymap file to ExecuteRuntime, it will be done both
at start and at migrate time.

Furthermore, we update the docstring of GenerateKVMRuntime to explain
this, and add a fixme related to the spice per-interface binding.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 4f580fef added the keymap support, but missed that this
directory needs to be ensured/created at hypervisor init time.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Due to the way $(MAKE) behaves (it forces execution of the commands
that contain it), use of $(MAKE) in a command line together with
side-effects commands will break stuff. It's better to keep it either
on a separate line, or if not possible, move said commands to a
separate make target.

After this patch, make -n distcheck, make -n regen-vcs-version and
possibly other commands now work correctly.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the lint target does a sequential: pep8, pylint, pylint on
the QA sources. hlint is run only when explicitly requested.

This patch reorganises the lint target by:

- splitting the current lint target into separate pylint (slow),
  pylint-qa (fast), pep8 (fast) targets
- making lint itself depend on the above plus hlint
- both pep8 and hlint are optional, based on whether the said binaries
  were detected at configure time

make -j is slightly faster after this patch (1m18s → 1m7s).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This has been reported internally 3-4 times already, and the current
version (from 8b437a6e) is still not good enough, it seems.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit f346a7d9 broke the TestInstanceImport test in QA, since it
dropped the "always generate" parameter for backup import. We add an
override mac parameter so that, no matter what is defined in the
config file, imported instances (can) have a different MAC.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

During the preparations for 2.6, we cleaned up the codebase to use
newer exceptions (e.g. commits 30d25dd8, 79ac58fa), so we can now drop
this override.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch fixes a (brown-bag) bug in the ssconf implementation where
the looked-for filenames were not prefixed with the ssconf file
prefix.

Additionally, a test for this is added (which fails without the fix).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This way, it can be reused in the Haskell code too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In some QA environments static MAC addresses must be used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In some QA environments external scripts may add tags. With this patch a
regular expression can be used to ignore them during QA runs.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

* stable-2.5:
  Fix parallel build failures
  QA: Add default setting for tests

Conflicts:
        Makefile.am (expected, from the parallel build fixes)
        qa/qa_config.py (due to cherry-pick, ignored, simply checkout out with --ours)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is the 2.5 version of the "fix build failures":

- man/%.gen could be left over even in case of failure, due to
  automake bug
- make man/%.gen runs RUN_IN_TEMPDIR, so let's depend on it, since
  that target has the proper dependencies (create needed dirs)
- man/%.gen depends on a number of built sources, but the dependency
  was not declared

Furthermore, wraps a long comment.

Tested with -j4/-j16, after `make maintainer-clean'.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit 1010ec70 enabled all tests by default. In some scenarios where
the configuration is static one may not want newly added tests to be
run.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit 59a8fe48)

Currently, the "common" instance tests (shutdown, rapi stopped
console, stopped modify, startup, rapi shutdown, rapi startup, list,
info, modify, rapi modify , console, rapi console, reinstall, rename,
rapi rename, rename and back, rapi rename and back, grow disk, reboot,
tags, cluster verify, rapi test instance, node list, job list) are run
4 times:

- one time for plain instance created via RAPI, using client
- one time for plain instance created via RAPI, without client
- one time for plain instance created via gnt-instance
- one time for DRBD instance created via gnt-instance

This makes the QA long, and is over-doing it (for non-full QAs): an
instance created via RAPI (either client) and via gnt-instance should
have the same parameters, but our duplicate tests do not ensure that,
just that instances behave OK.

This patch adds a toggle so that we can skip the common tests for
RAPI-created instances; the creation/deletion is still performed, but
all the other operations are not. This reduces the time of a "quick"
QA by ~20% (1h:15m → 1h).

The common tests will still be run (unconditionally) for
gnt-instance-created instances.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Currently, this is how the QA log looks like:

Command: ssh -oEscapeChar=none -oBatchMode=yes -l root -t -oStrictHostKeyChecking=yes -oClearAllForwardings=yes -oForwardAgent=yes -oControlPath=/tmp/ganeti-qa-multiplexer.DMzkuH -oControlMaster=no node14 exit
Command: ssh -oEscapeChar=none -oBatchMode=yes -l root -t -oStrictHostKeyChecking=yes -oClearAllForwardings=yes -oForwardAgent=yes node8 exit
Command: ssh -oEscapeChar=none -oBatchMode=yes -l root -t -oStrictHostKeyChecking=yes -oClearAllForwardings=yes -oForwardAgent=yes node18 exit

And these arguments are repeated over and over. This patch proposes to
log once the arguments, at the beginning of the QA (so that the SSH
commands can be reproduced, if needed) and then drop them from the
log. The new output looks like:

SSH command for primary node: ssh -oEscapeChar=none -oBatchMode=yes -lroot -oStrictHostKeyChecking=yes -oClearAllForwardings=yes -oForwardAgent=yes -oControlPath=/tmp/ganeti-qa-multiplexer.24lgrK -oControlMaster=no node14
SSH command for other nodes: ssh -oEscapeChar=none -oBatchMode=yes -lroot -oStrictHostKeyChecking=yes -oClearAllForwardings=yes -oForwardAgent=yes NODE
(the above are the informational messages about parameters, then)
Command: ssh node14 exit
Command: ssh node8 exit
Command: ssh node18 exit

This makes the QA log much more readable, by dropping unneeded clutter
(look how long the original lines were), and hopefully also a bit
smaller (current QA log is > 100MB of text output).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This makes ssh generate much less noise when QA is run itself without
a tty, as otherwise we get thousands of lines of:

  Pseudo-terminal will not be allocated because stdin is not a terminal.

in the QA logs (which are already big enough).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Commit 2e04d454 introduced the new offline state for the instance
state, but being a big monolithic patch it sneaked in something that
doesn't make sense.

The checks for extra instances (either wrongly up or just unknown) are
done purely on a name-basis, not on objects, so the types there are
wrong. Furthermore, they have no relation to the admin state of the
instance, so we just drop the entire if block. We keep the increment
of the offline instance count, but move it to a different loop over
instances.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

I was not able to find where the “.d” suffix is documented, so this
clarification should make it more obvious.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

xm and xl are mostly compatible, there is just a change in migrate,
which for now we simple bridge by comparing the command string.

Also XEN_CMD is now set during configure time. This will then be fixed
in Ganeti 2.7 when we make it an hvparams.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is needed so we can specify the xen command upon configure time

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Redhat links its pycurl against NSS/ so this check fails and therefore
makes ganeti-watcher unusable.

Even though this seems to fix the problem and works, further testing on
this might be needed.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Reported by user ewoud, thanks!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

* stable-2.6:
  Document that the disk/hypervisor state is not used
  Correct the watcher state files in the man page
  Re-wrap some lines in watcher code
  Add support for SIGHUP handling in Haskell daemons
  Rework logging setup for Haskell daemons
  Adapt example init.d script to work on Fedora
  Update the INSTALL doc for Fedora users
  kvm-ifup: Use /bin/bash

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This didn't get completed in time for 2.6, so we document that it's
not working as is. Well, it's not broken, just not used by anything.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

The move to per-group state files forgot to update the man page of
ganeti-watcher; let's fix this and add some more details on what
removal means and when it's needed.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

These were using exactly 80 chars, and I like them smaller.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch adds support for handling SIGHUP and reopening the
stdout/stderr logfile in daemon mode (but not in foreground mode).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This fixes a missing functionality: closing of stdout/stderr when
forking daemons. Without this, starting the daemons from the command
line seems to work, but doing it from utils.RunCmd breaks, since
without closing the standard descriptors, RunCmd never finishes.

The code is not very nice; basically what we need is an actual type
for the LogMode desired (LogConsole, LogStderr, LogFile FilePath), but
the syslog (yes/no and *only*) complicates this even more. I'll think
more on this and probably fix it in master.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* stable-2.5:
  kvm-ifup: Use /bin/bash

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

As reported by Ben Beuchler, kvm-ifup uses bash-specific constructs and
doesn't work with Ubuntu's default shell. We use bash all over the place
anyway, so changing the shebang line is the easiest solution.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

* stable-2.6:
  Version bump for 2.6.0~beta1
  Adjust cfgupgrade for new minor version
  Add support to daemon-util for distributions without start-stop-daemon
  Improve the check-news script
  Fix NEWS file
  _MaybeParseUnit: The elements are dicts not arrays
  Make it possible to reset vcpu/spindle ratio to default
  Add man page documentation for cpu_mask hv parameter
  Add checks from wrong hard-coded paths in manpages
  Fix hardcoded paths in gnt-cluster.rst man page
  First round of NEWS file updates for 2.6
  gnt-group: Adding vcpu/spindle ratio to ipolicy flags
  gnt-cluster modify: Add ipolicy for spindle ratio opt
  cli: Add ipolicy_spindle_ratio to CreateIPolicyFromOpts
  gnt-instance info: Rework beparams listings
  Fix a tiny typo
  constants: Add FIXME regarding CPU pinning bitfield

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>