GitLab

The previous patch loading utilities omitted empty patches, as they
were thought to be of no significance, and when no patches were used,
the import and therefore dependency should not be used. If a user has
added an empty patch file, and made an entry in the order file, the QA
would treat this as an error as it had no means of differentiating
between a patch not present and an empty patch.

This patch fixes the solution by better handling empty patches, and
logging warnings when they are encountered.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

To explicitly specify the order of patches executed, the QA provides an
"order" file. It can contain names of patches that will be executed
first, and in the order listed, before all the other patches that still
follow an alphabetical order.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This patch allows support for multiple patches placed in the "patch"
directory, which are executed in alphabetical order.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* Refactors the current patch code to allow for multiple patches that
  can be applied, yet leaves only one for now.

* Rewords the message shown to the user in case the modules needed for
  patching are missing.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

In order to work, it is necessary:
* Write a JSON Patch to file "qa/qa-patch.json" in a source repository.
  (And be careful not to commit it!) The file must conform to RFC6902
  and describes what changes should be done to the QA JSON
  configuration. This is simple as
   [
     { "op": "add",
       "path": "default",
       "value": false },
     { "op": "add",
       "path": "/tests/instance-add-file",
       "value": true }
   ]

* Install "jsonpatch" and "jsonpointer" modules on the machines QA is
  running on. Either using "easy_install" or as Debian packages using
  "pypi-install".

QA only loads the modules if the patch file is present and non-empty.
Therefore no changes to buildbots are needed, if the feature is not
used.

It's advisable to add a git pre-commit hook for the main repository to
prevent changes to "qa/patch.json", keeping it to "[]". Suggestion for
such a hook:

  # Exit code 1 if the patch file isn't just "[]":
  python -c 'import json, sys; \
    exit(0 if json.load(open("qa/qa-patch.json", "r")) == [] else 1)'

Locally it's possible to tell git not to track changes to the file at
all (see https://help.github.com/articles/ignoring-files):

  git update-index --assume-unchanged qa/qa-patch.json

Cherry-pick of e5398c3a

 from stable-2.11.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

Newer pylint versions complain that QaConfig is not a proper container because
it does not implement the __len__(), __delitem__() and __setitem__() methods.
This commit implements them.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

QA tests should make sure not to be run if the necessary
environment for them is not present. This patch fixes that
for a couple of tests that fail to check that before being
executed.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch renames the mapping of disk templates to storage
types to MAP_DISK_TEMPLATE_STORAGE_TYPE, which is a more
expressive name since it indicates that it is a mapping and
does not have the inconsistent usage of singular and plural
anymore.

Additionally, this patch sorts the disk templates
alphabetically in the constants file.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This improves the QA code wrt to file storage in the
following ways:
- It fixes a FIXME to make the file storage directory
  customizable in the QA config. Fallback is the default
  value.
- It makes QA derive the enabled storage types from
  the enabled disk templates and runs the 'gnt-node
  storage tests' only on the enabled ones.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

"gnt-instance add" and "gnt-instance recreate-disks" are tested giving the
number of spindles, when supported. Also, QA for "gnt-instance
recreate-disks" now covers the case where disks are resized.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch makes the constant DEFAULT_ENABLED_DISK_TEMPLATES
a list instead of a set, because the order of the elements actually matters
as the first one is planned to be the default disk template for instance
creation. Also, this constant is always used like a list and thus providing it
as a set complicates the code that uses it.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Modify existing tests to use the name option for instance disks. The
configuration options 'disk' and 'disk-growth' are merged into a single
'disks' option, which is a list of dictionaries with 'size', 'growth' and
'name' items.

Also, add 'instance-device-names' QA test, which tests addition, renaming
and removal of named devices.
Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>

qa_config.IsTemplateSupported() is already used to decide which disk
templates are supported by QA. Enabled templates now are part of that
decision.

Unit tests are updated.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Fixes the QA of 'gnt-cluster init' and 'gnt-cluster modify' with respect
to enabling disk templates.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michele Tartara <mtarara@google.com>

This extends the QA scripts to test the changes in 'gnt-cluster init',
'gnt-cluster info', and 'gnt-cluster modify' with respect to enabled
storage types.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Until now all tests depended on simple AND/OR conditions. With the
introduction of virtual cluster support, some tests can only run if
virtual clusters are not in use (e.g. not yet supported or not
possible). Instead of introducing more logic for enabling/disabling
tests, callable values are supported instead. “ht.WithDesc” should be
used on them to continue to provide a readable description.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

If these new options are set, a virtual cluster is created and used.
Virtual clusters consist of 1..N virtual nodes on a single physical (or
virtual) node. The design of virtual clusters is described in
“doc/design-virtual-clusters.rst”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Use a sort key function instead of using a comparing function (“cmp=…”).
The latter is not supported in Python 3 and using a sort key function is
easier.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Having a readable representation makes debugging easier.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Also make “used” a property.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Use a method on the instance object instead of going through
“qa_config.GetInstanceTemplate”. This provides for better encapsulation.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Call the wrapped instance method directly.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

These are no longer used.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Instead of having a module-level function in “qa_config” to release a
node, a new method on node objects is used.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Up until now nodes were stored as a dictionary. The keys were hardcoded
in a lot of places and entries modified directly.

This patch introduces a new class for nodes in QA named “_QaNode”. It
still supports accessing details via dictionary syntax, but that will be
removed after a couple of patches changing all users. Unit tests for
“qa_config.AcquireNode” are also included.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Instead of having a module-level function in “qa_config” to release an
instance, a new method on instance objects is used.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Up until now instances were stored as a dictionary. The keys were
hardcoded in a lot of places and entries modified directly.

This patch introduces a new class for instances in QA named
“_QaInstance”. It still supports accessing details via dictionary
syntax, but that will be removed after a couple of patches changing all
users. Unit tests for “qa_config.AcquireInstance” are also included.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

The same code can be written in a single expression. The earlier patch
named “Refactor storage of runtime exclusive storage flag in QA”
provided the unit test.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This is a follow-up for “qa_config: Remove exclusive storage flag from
config”. Instead of storing the flag in a module-level variable it is
now stored within the new QA configuration class and unit tests are
provided. Wrappers in “qa_config” maintain the existing interface.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Ever since its introduction (sometime before commit cec9845c

 in September
2007), the QA configuration was stored in a dictionary at module-level
in “qa/qa_config.py”. The configuration was loaded, verified and
evaluated using module-level functions. Since then the configuration has
become more complicated and more functionality has been added. This
patch refactors handling the configuration to use a class and provides
unittests.

- The configuration is loaded through a class method which also verifies
  it for consistency
- Wrapper methods are provided in “qa_config” to not change the
  interface
- Unit tests are provided for the new configuration class
- The configuration object is still stored in a module-level variable
  and can be retrieved using “GetConfig” (direct access should be
  avoided so an uninitialized configuration can be detected)
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Commit 6a0f22e1

 added the ability to track the status of the cluster-wide
flag for exclusive storage. It did so in a way which modifies the
in-memory configuration dictionary. This patch changes the code to use a
plain global variable as a temporary solution until a configuration
object is added.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

It is only used in one place and not necessary.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The code launching instance tests for "plain" and "drbd" templates has been
unified. In this way it's easier to add more templates, and to re-use the same
tests with different configurations.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

These make possible to simplify the logic of tests requiring more
nodes. Used in the following patches.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

QA configuration now tracks the state of the exclusive_storage flag at
cluster level. This will be used to selectively enable tests according to
the configuration.

Also, it's now possible to specify the initial value of the
exclusive_storage flag in the configuration file.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The global 'cfg' variable is initialized to an empty dictionary, so there
is no more need to disable some pylint checks.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The disk templates used by the instances are tracked in the QA
configuration.  This will be used to selectively enable tests according to
the configuration.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

To help developing Ganeti on newer distributions, let's try to fix
pylint warnings/errors. I'm using pylint from current Debian wheezy:
pylint 0.25.1, astng 0.23.1, common 0.58.0, and we have 3 things that
needs fixing.

First, a really wide "except", with the silencing in the wrong
place. I'm not sure why this doesn't have "except Exception", so let's
add it. However, pylint still complains about "Catching too general
exception", even though we do want to catch both system and our
exception, so let's add a silence for W0703. It's true that we
shouldn't catch KeyboardInterrupt and friends, but that should be
cleaned up on the master branch.

Second, pylint complains about "redefining name builtin tuple",
because we do some pattern matching in the except blocks in
netutils. This seems to be a false positive, but let's clean the code
around this.

And finally, type inference again goes bad, so let's silence E1103
with its "boolean doesn't have 'get' method".

After this, I can run "make lint", and by extension "make
commit-check" on Debian Wheezy, yay! We might be able to bump our
required pylint versions to something not ancient…
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Some parts of QA should only be run with certain hypervisors (e.g.
testing instance HV parameters). To make that happen we always define
the cluster's hypervisors and provide a function to get the default
hypervisor.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>