There are two bugs in the current resize code, affecting mostly DRBD.

First, due to bugs in old DRBD versions (pre 8.0.14), the code currently
calls `drbdsetup resize' on both the primary or secondary. However,
this is actually wrong per current DRBD (from drbdsetup(8)):

     resize
       This causes DRBD to reexamine the size of the device's backing
       storage device. To actually do online growing you need to
       extend the backing storages on both devices and call the resize
       command on one of your nodes.

So calling it just on the primary node should be enough. However, we
can't simply remove the calls to the secondary nodes, since that would
break the growth of the underlying storage (LVM) on the
secondary. Which leads to the second existing bug: we call resize on
each node, even before finish the growth of the underlying
storage. This can leads to all kind of issues if DRDB is not well
behaved.

So to fix both these bugs, we have to extend the current RPC call with
another parameter, which denotes whether to extend the actual backing
storage or just the "logical" one (DRBD being the only one; MD would
be another, if implemented). This allows us to do the growth in two
steps, first the backing store on all nodes, then the logical storage
on just the primary node.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Problem introduced by commit 87f0aa48.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Remove a useless space at the end of the line in a config file.

Signed-off-by: Jack Sitnikov <sitnikove@gmail.com>
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This hv parameter allows passing PCI devices to virtual machines.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Two blank lines are needed between functions.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

The test is disabled by default, as it takes a long time.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is useful to simulate HW problems during QA.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

To change the cluster from single homed to multi homed or vice versa one
must target the master node first, and pass the --force option. All
other nodes then will work as long as they are reachable by the master.

Note that this will also prevent a node to be set to single-homed if the
master is multi-homed, which wasn't disallowed before, and warn if a
single-homed <-> multi-homed transition happens.

Also note that it's still theoretically possible to flip a cluster
inadvertently by changing the master node this way, and then doing a
master failover before fixing the other nodes.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add unittests for FormatJobID, GetArchiveDirectory, ParseJobId.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch adds the necessary conversions for comparing split timestamps
(such as used by jobs) and job IDs (by defining custom converters).
Unittests are included.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

These don't really need to be in jqueue, and a new function will
be added to convert job IDs to an integer for queries.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

* stable-2.6:
  Fix cfgupgrade unittests
  Upgrade wrong iv_names at cfgupgrade time
  query2: Add <, >, <=, >= comparison operators
  Fix creation of plain instances with --no-wait-for-sync
  Fixed typos, formatting, etc and added a comment
  Fix daemon-util with non-root user models

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Sorry, I broke the cfgupgrade unittests via 904910c4, since that
commit added the requirement for the "instances" dict in the
configuration.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Since Ganeti 2.6 does warn about disks with wrong iv_names, it would
be better to fix them when upgrading to 2.6.

Not sure if this is too brittle and whether we want to add more
safeguards on the contents of the config/instance/disks before
touching them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

These were recently added to the python version.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

These can be used, for example, to get jobs submitted after a certain
timestamp.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

As reported on the devel mailing list by Christos Stavrakakis,
creation of plain instances is broken when the --no-wait-for-sync flag
is passed, because in that case WaitForSync is not called, hence
SetDiskID is not called at all, resulting in a None physical_id being
passed to backend.

We fix that by explicitly calling SetDiskID, which will cover the
pause/resume and os_add RPC calls.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This contains the types to express query2 queries, filters and results.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch only touches comments and man pages.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit 4b42c3d6 broke non-root user mode since, while trying to do a
cleanup a move all local variable definitions to the start of the
function; however, the plain_name var is only defined later, so this
actually doesn't work.

Note that enabling set -u doesn't _directly_ work, since we rely on
some variables which are usually undefined (EXTRA_*_ARGS, etc.). It
can be fixed, but in a later patch.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

* stable-2.6: (26 commits)
  Bump version for beta 2 release
  Fix epydoc raises L{...} usage
  Fix race condition in test for *FileID functions
  Move KVM keymap from GenerateRuntime to ExecuteRuntime
  Add the keymap directory to the list of runtime KVM dirs
  Fix make -n use
  Enable hlint in lint and reorganise this target
  Improve error message for auto-promote/node modify
  Fix QA import of existing instances
  Remove one obsolete hlint override
  Fix prefix bug in Haskell ssconf implementation
  Move the ssconf file prefix constant to constants.py
  QA: Add configuration options for static MAC address
  QA: Ignore tags using regular expression
  Fix parallel build failures
  QA: Add default setting for tests
  QA: add toggle for fewer common instance tests
  QA: stop logging SSH arguments for each invocation
  Switch QA to tty-less mode
  Fix a type issue and bad logic in cluster verification
  ...

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

I've added the changes under beta1, not beta2, since it's easier to
keep it separate.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

For epydoc raise statement, it already expects an exception which will
be hyperlinked; if passing an L{…} construct, it will instead break
the formatting (the @raise will not be recognised, but merged into the
previous paragraph).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

* stable-2.5:
  Fix race condition in test for *FileID functions
  Move KVM keymap from GenerateRuntime to ExecuteRuntime
  Add the keymap directory to the list of runtime KVM dirs

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In this test the “file ID” of a temporary file is compared against the
file ID gathered via an open file descriptor to the same file. For
reasons unknown to me utime(2) is called in-between to update the
inode's a- and mtime. Depending on the file system's timestamp
resolution this can lead to a different file ID.

Found by chance during QA and reproduced by adding a delay before the
call to utime(2).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit fbd55434)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Per issue 243, "side-effects" are GenerateRuntime are bad as they
execute only on the initial node of the instance. By moving the
write-out of the keymap file to ExecuteRuntime, it will be done both
at start and at migrate time.

Furthermore, we update the docstring of GenerateKVMRuntime to explain
this, and add a fixme related to the spice per-interface binding.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 4f580fef added the keymap support, but missed that this
directory needs to be ensured/created at hypervisor init time.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Due to the way $(MAKE) behaves (it forces execution of the commands
that contain it), use of $(MAKE) in a command line together with
side-effects commands will break stuff. It's better to keep it either
on a separate line, or if not possible, move said commands to a
separate make target.

After this patch, make -n distcheck, make -n regen-vcs-version and
possibly other commands now work correctly.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the lint target does a sequential: pep8, pylint, pylint on
the QA sources. hlint is run only when explicitly requested.

This patch reorganises the lint target by:

- splitting the current lint target into separate pylint (slow),
  pylint-qa (fast), pep8 (fast) targets
- making lint itself depend on the above plus hlint
- both pep8 and hlint are optional, based on whether the said binaries
  were detected at configure time

make -j is slightly faster after this patch (1m18s → 1m7s).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This has been reported internally 3-4 times already, and the current
version (from 8b437a6e) is still not good enough, it seems.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit f346a7d9 broke the TestInstanceImport test in QA, since it
dropped the "always generate" parameter for backup import. We add an
override mac parameter so that, no matter what is defined in the
config file, imported instances (can) have a different MAC.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

During the preparations for 2.6, we cleaned up the codebase to use
newer exceptions (e.g. commits 30d25dd8, 79ac58fa), so we can now drop
this override.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch fixes a (brown-bag) bug in the ssconf implementation where
the looked-for filenames were not prefixed with the ssconf file
prefix.

Additionally, a test for this is added (which fails without the fix).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This way, it can be reused in the Haskell code too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In some QA environments static MAC addresses must be used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In some QA environments external scripts may add tags. With this patch a
regular expression can be used to ignore them during QA runs.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

* stable-2.5:
  Fix parallel build failures
  QA: Add default setting for tests

Conflicts:
        Makefile.am (expected, from the parallel build fixes)
        qa/qa_config.py (due to cherry-pick, ignored, simply checkout out with --ours)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is the 2.5 version of the "fix build failures":

- man/%.gen could be left over even in case of failure, due to
  automake bug
- make man/%.gen runs RUN_IN_TEMPDIR, so let's depend on it, since
  that target has the proper dependencies (create needed dirs)
- man/%.gen depends on a number of built sources, but the dependency
  was not declared

Furthermore, wraps a long comment.

Tested with -j4/-j16, after `make maintainer-clean'.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit 1010ec70 enabled all tests by default. In some scenarios where
the configuration is static one may not want newly added tests to be
run.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit 59a8fe48)