This is a follow-up for “qa_config: Remove exclusive storage flag from
config”. Instead of storing the flag in a module-level variable it is
now stored within the new QA configuration class and unit tests are
provided. Wrappers in “qa_config” maintain the existing interface.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Ever since its introduction (sometime before commit cec9845c in September
2007), the QA configuration was stored in a dictionary at module-level
in “qa/qa_config.py”. The configuration was loaded, verified and
evaluated using module-level functions. Since then the configuration has
become more complicated and more functionality has been added. This
patch refactors handling the configuration to use a class and provides
unittests.

- The configuration is loaded through a class method which also verifies
  it for consistency
- Wrapper methods are provided in “qa_config” to not change the
  interface
- Unit tests are provided for the new configuration class
- The configuration object is still stored in a module-level variable
  and can be retrieved using “GetConfig” (direct access should be
  avoided so an uninitialized configuration can be detected)

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Commit 6a0f22e1 added the ability to track the status of the cluster-wide
flag for exclusive storage. It did so in a way which modifies the
in-memory configuration dictionary. This patch changes the code to use a
plain global variable as a temporary solution until a configuration
object is added.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The QA code will also have to generate virtual paths and this function
comes in handy.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

- Capitalize acronyms such as “QMP” or “SPICE”
- Remove punctuation from error messages
- Improve error reporting when sending a monitor command failed
- Put “or” at end of line
- Fix indentation for a concatenated string

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Monitor calls should be replaced with QMP once KVM >= 0.14 is required.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Change the docstring of chompPrefix to prevent the error
"doc comment parse failed" that was raised by some version of
haddock while generating the documentation for this function.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

* devel-2.7:
  Redirect output for gnt-* list/list-fields commands
  Add function to execute QA commands with redirected output
  Implement option to skip logging of QA commands
  Fix typo in a comment
  Add some more Haskell/Python equivalence tests
  Fix Haskell log file naming after virtual cluster changes
  QA: Clean up after "instance-remove-drbd-offline" test
  Switch KVM to multi-error verify results
  Make LXC fail verification if cgroup is not mounted
  Add a helper function for hypervisor verification
  hv_lxc: fix whitespace errors
  s/nic/NIC/ in manpages
  LXC: add support for the memory controller
  LXC: adapt hv for newer lxc userspace tools

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This enabled the redirection; QA output after this patch should become
much much smaller (on a 'quick' QA, we are left with ~0.5MB output on
stdout and 64MB redirected output).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Some QA commands generate non-trivial output: for a full QA, the gnt-*
list commands generate more than 300MB of data. To make the QA logs
more parseable by humans, we need support for redirecting output of
commands to a different place.

This patch adds a 'AssertRedirectedCommand' that does this,
redirecting the output to a (fixed) log file in the Ganeti log
directory.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds an optional kwarg that allows skipping the "Command:
…" line in QA output. This is helpful if we only need to run some
maintenance actions, that are QA-related and not Ganeti-related.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is yet another fix for type confusion between python
and haskell. ;) The network field of PartialNic should be
a string and not of type Network. This makes it necessary
to add a helper function to look up a network by name
and not by UUID.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This would have caught the log file problem fixed in the previous
patch (9411474b), for example. Also we test user/group equivalence,
name only.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 3329f4de changed the Haskell log file from constants to
functions, but introduced a bug: it uses now the daemon name instead
of the correct log file, which means "ganeti-confd.log" instead of
"conf-daemon.log".

In order to fix this, we need to abstract the log file base (in
constants.py) into a separate set of constants, so that we can reuse
it in the Haskell code.

This fixes issue 343.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This test used to leave a node with orphan logical volumes and unused DRBD
minors. Now, the expected leftovers are cleaned up after the test, and
cluster-verify is run.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

It is only used in one place and not necessary.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

“objects.ConfigObject” contains two useful functions for working with
containers of serialized objects, “_ContainerToDicts” and
“_ContainerFromDicts”. This patch separates those functions and moves
them into “objectutils” as they'll be useful for converting parts of the
QA configuration to objects. Unittests are added to cover the parts not
already tested through other code paths.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

“utils.RunCmd” is re-exported from “utils.process.RunCmd”. Epydoc
doesn't fully understand this, so we have to refer to the original.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

- Handle de-serialization correctly when pool is not defined
- Serialize to empty list when the attribute is None (this should never
  happen in reality as the attribute is always set when de-serializing)
- Add tests

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This uses the new _FormatVerifyResults helper function to return
multiple errors.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Since LXC now relies on cgroup memory limits to enforce memory, let's
make hypervisor verification (and thus cluster-verify) return errors
when the cgroup filesystem is not mounted.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This will allow easier multi-error results from hypervisors; right
now, we only report the first error, which is not nice.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The latest lxc patches included a few whitespace style errors, that make
lint fail. This patch fixes those.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

For consistency, let's make sure NICs are capitalized always in the same
way.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Add support for the memory resource controller, useful to actually place
memory limits on instances.

Support is still optional, in case the kernel doesn't have it compiled
in, or -in the case of Debian- has it dependent on a kernel command-line
switch (cgroup_enable=memory specifically).

Signed-off-by: Faidon Liambotis <paravoid@debian.org>
Reviewed-by: Guido Trotter <ultrotter@google.com>

* devel-2.7: (24 commits)
  Fix typo in ganeti-noded man page
  QA: Run instance tests with different cluster configurations
  QA: Run cluster-verify after instance tests
  QA: Refactored launching of the various instance tests
  QA: Use lists of nodes as argument to instance tests
  QA: Instance tests check the configuration before running
  QA: Added functions to allocate/free N nodes at once
  QA: Added exclusive_storage to qa_config
  QA: Better initialization of a global dictionary
  QA: Added disk template to config
  QA cleanup: Removed instance-disk-failure test
  QA fix: TestInstanceExportWithRemove() updates configuration
  QA: Cluster-verify reports shared PVs with exclusive storage
  QA: Added constants for LVM volumes
  QA: Templates incompatible with exclusive storage reported
  QA: Uniformity check for exclusive_storage in cluster-verify
  QA: Test for basic features of exclusive storage
  QA: Added function to read cluster configuration fields
  QA: Added function to check cluster-verify result
  Document better gnt-network
  ...

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently hv_lxc depends on the behavior and output of older LXC tools,
which have since changed, making it unable to function in current
distributions (e.g. Debian wheezy).

Adapt the tools and expectation for the output and make it into a
working state again, at least for LXC 0.8.0.

LXC tools seem to be split into two categories: the actual, binary lxc
tools, plus a set of shell scripts to provide a higher-level interface.
Ganeti doesn't need the latter and the shell scripts make some
assumptions (like /var/lib/lxc) so remove the only reference to such a
shell script (lxc-ls).

While at it, remove a few comments about "current (Spring 2010)
distributions" that do not apply anymore.

This is a backwards-incompatible change, meaning it won't work on
systems with older LXC userspace. However, hv_lxc is experimental anyway
and there's virtually no documentation about it in the tree.

Signed-off-by: Faidon Liambotis <paravoid@debian.org>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add a missing "in".

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Instance tests are run with or without the exclusive-storage flag set. More
configurations can be added easily.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

We want to make sure that instance operations don't leave the cluster in a
bad state. The "instance-remove-drbd-offline" test leaves some debris
behind, so it's been moved to the last position.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The code launching instance tests for "plain" and "drbd" templates has been
unified. In this way it's easier to add more templates, and to re-use the same
tests with different configurations.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Some instance test functions took two node arguments, some took one, and
some took two but the second argument could be None. This patch makes such
functions uniform by using a list of nodes as an argument.  This simplifies
the following refactoring.

No test logic has been changed.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Instance test functions check the current configuration, and they will run
the actual tests only if the configuration supports them.  This will be
used for refactoring in following patches.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

These make possible to simplify the logic of tests requiring more
nodes. Used in the following patches.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

QA configuration now tracks the state of the exclusive_storage flag at
cluster level. This will be used to selectively enable tests according to
the configuration.

Also, it's now possible to specify the initial value of the
exclusive_storage flag in the configuration file.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The global 'cfg' variable is initialized to an empty dictionary, so there
is no more need to disable some pylint checks.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The disk templates used by the instances are tracked in the QA
configuration.  This will be used to selectively enable tests according to
the configuration.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The test was broken, out of sync with the rest of the code, and prone to
crashes. Until someone does a better job, it's better to remove the
test. This simplifies further refactoring.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>