It is only used in one place and not necessary.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

“objects.ConfigObject” contains two useful functions for working with
containers of serialized objects, “_ContainerToDicts” and
“_ContainerFromDicts”. This patch separates those functions and moves
them into “objectutils” as they'll be useful for converting parts of the
QA configuration to objects. Unittests are added to cover the parts not
already tested through other code paths.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

“utils.RunCmd” is re-exported from “utils.process.RunCmd”. Epydoc
doesn't fully understand this, so we have to refer to the original.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

- Handle de-serialization correctly when pool is not defined
- Serialize to empty list when the attribute is None (this should never
  happen in reality as the attribute is always set when de-serializing)
- Add tests

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.7: (24 commits)
  Fix typo in ganeti-noded man page
  QA: Run instance tests with different cluster configurations
  QA: Run cluster-verify after instance tests
  QA: Refactored launching of the various instance tests
  QA: Use lists of nodes as argument to instance tests
  QA: Instance tests check the configuration before running
  QA: Added functions to allocate/free N nodes at once
  QA: Added exclusive_storage to qa_config
  QA: Better initialization of a global dictionary
  QA: Added disk template to config
  QA cleanup: Removed instance-disk-failure test
  QA fix: TestInstanceExportWithRemove() updates configuration
  QA: Cluster-verify reports shared PVs with exclusive storage
  QA: Added constants for LVM volumes
  QA: Templates incompatible with exclusive storage reported
  QA: Uniformity check for exclusive_storage in cluster-verify
  QA: Test for basic features of exclusive storage
  QA: Added function to read cluster configuration fields
  QA: Added function to check cluster-verify result
  Document better gnt-network
  ...

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add a missing "in".

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Instance tests are run with or without the exclusive-storage flag set. More
configurations can be added easily.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

We want to make sure that instance operations don't leave the cluster in a
bad state. The "instance-remove-drbd-offline" test leaves some debris
behind, so it's been moved to the last position.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The code launching instance tests for "plain" and "drbd" templates has been
unified. In this way it's easier to add more templates, and to re-use the same
tests with different configurations.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Some instance test functions took two node arguments, some took one, and
some took two but the second argument could be None. This patch makes such
functions uniform by using a list of nodes as an argument.  This simplifies
the following refactoring.

No test logic has been changed.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Instance test functions check the current configuration, and they will run
the actual tests only if the configuration supports them.  This will be
used for refactoring in following patches.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

These make possible to simplify the logic of tests requiring more
nodes. Used in the following patches.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

QA configuration now tracks the state of the exclusive_storage flag at
cluster level. This will be used to selectively enable tests according to
the configuration.

Also, it's now possible to specify the initial value of the
exclusive_storage flag in the configuration file.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The global 'cfg' variable is initialized to an empty dictionary, so there
is no more need to disable some pylint checks.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The disk templates used by the instances are tracked in the QA
configuration.  This will be used to selectively enable tests according to
the configuration.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The test was broken, out of sync with the rest of the code, and prone to
crashes. Until someone does a better job, it's better to remove the
test. This simplifies further refactoring.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

When the instance is removed from the cluster by this test, it's also freed
in the QA configuration.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Exclusive storage forbids sharing PVs between unrelated LVs. This is a test
that cluster-verify correctly report such cases.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The LVM volume group becomes a configurable parameter, and a prefix for
logical volume names used in tests is introduced as a constant.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This fixes: “ganeti/hypervisor/hv_kvm.py:93: [W1201, _GetTunFeatures]
Specify string format arguments as logging function parameters”, and
also updates the copyright years.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Cluster-verify should report instances whose disk template is not
compatible with the exlusive storage setting.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Cluster-verify should report nodes in the same nodegroup with different
values of the exclusive_storage flag.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Check that creation of instances doesn't trigger any error.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This function will be used in following patches.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Now it's possible to check that cluster-verify reports all the expected
errors, instead of simply checking that it fails.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Specify clearly that the nic parameters depend on the
network-to-nodegroup connection.

Clarify in gnt-network how instance nics are connected to a network.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Include a handful of important changes that weren't mentioned.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Move two designs that have been implemented in 2.7 to design-2.7 and out
of drafts.

Other designs were partly implemented: explain what part was, and leave
them as drafts pending their completion.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, the node listing RPC is very slow due to missing
parallelisation. For the 2.7 release, we reset these back to masterd,
hoping to revert them by the time 2.8 is ready.

There are a number of queries that I've left pointing to confd, as
they are non-RPC:

- node fields query
- node static fields query (and added a note in one case where it
  could become a problem)

All the other converted queries (job listing, group queries, tags,
cluster config) remain pointing to confd, as they don't use RPC.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The harep tool prints messages for every action that it performs (or that
it doesn't perform). In case nothing is to be done at all, always print
some statistics of the current state of the cluster.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Because of instance locks, after submitting a repair job we weren't able to
set the "pending" tag until at least the first opcode of the job finished.
Introduce a small delay in the repair job so as to allow the subsequent
TAGS_SET job to finish immediately, resulting in faster operation from the
user's point of view.

Make the duration of the delay configurable with --job-delay; if zero,
avoid inserting the TestDelay opcode entirely.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Also, fix @rtype and @return elements of utils.TestDelay(), which now
returns a tuple but this wasn't being indicated.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Implement 'doRepair' to create a repair job from a list of opcodes if
the instance's policy allows it (otherwise set an ENOPERM result label),
_and_ the instance was previously healthy (i.e. not in ArFailed or
ArPendingRepair).

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add a 'detectBroken' function that determines whether an instance is in an
unhealthy state, and what's needed to repair it. The repair is specified as
an AutoRepairType constant, and a list of opcodes. The opcodes will only be
executed (in following patches) if the repair type is allowed by the policy.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

As a first step before detecting any brokeness with instances, see if any
of our previous repairs have completed, and move instances to ArFailed or
ArHealthy accordingly. Do nothing if there are still running jobs for the
repair.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Parse auto-repair tags to set each instance in one of ArHealthy, ArFailed,
or ArPendingRepair. The implementation tries to be well behaved when old
tags have been left behind, which future patches will still try _not_ to do.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

harep(1) detects certain kind of problems with instances and applies the
allowed set of solutions. See doc/design-autorepair.rst.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In particular:

  - make ArHealthy take an optional AutoRepairData; this allows to
    represent the situation where a repair completed successfully, and
    hence there's an associated tag we might want to know about.

  - add an 'arTag' member to AutoRepairData to store the exact tag this
    data was derived from; this allows for easier removal of the tag when
    its associated state is superseded. It also stores the tag that will
    be written to the cluster when a new AutoRepairData record is created.

  - change the default sort order of AutoRepairResult to reflect the
    precedence used when reading tags from an object (a failure, if
    present, is always the resulting state; a success overrides an
    ENOPERM).

  - (trivial) make AutoRepairStatus and AutoRepairData derive Eq and Show.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Instead of waiting 15 seconds every iteration of the waiting loop, start
with 0.5 seconds and increment exponentially until a certain maximum is
reached. Keep the maximum at 15 seconds.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>