GitLab

This is the first performance related test. It creates as many instances
as available in the QA config in parallel and removes them (again in
parallel) immediately after the creation succeeded.

In order to ease writing of additional tests, a lot of the logic is kept
in helper classes.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Calm a few lint errors introduced during cherry-picking code in
qa_job_utils.py. The fixes were intentionally made in a way which should
produce merge conflicts later on, so it's not forgotten to undo them.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The cherry-picked function _GetOutputFromMaster() calls
GetCommandOutput() with parameters only present in newer Ganeti
branches. Remove those parameters as they cause errors in stable-2.10.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This patch cleans RunWithLocks up a little bit by reducing the number
of delay function terminations, and using the QAThread class to ensure
exceptions are thrown at the right time and in the right place.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

(cherry picked from commit 9558c2a4)

To have better control over threads, this patch adds a helper Thread
subclass which captures any exceptions occurring for later use.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

(cherry picked from commit e677d5e8)

The RunWithLocks test assumed that gnt-debug delay would have the info
needed for interruption ready immediately after being run, and in some
situations this is not the case. This patch makes the test more patient
and forgiving by retrying a number of times.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

(cherry picked from commit fbab1c76)

To compensate for the cases where a QA test is supposed to block when
a lock is present, add an additional option showing whether blocking is
supposed to happen or not.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

(cherry picked from commit 57efdaf5)

Conflicts:
	qa/qa_rapi.py (didn't backport changes to this file)

This patch adds threading to the RunWithTests function, allowing one
thread to execute the QA test, and the other to monitor if it is being
blocked by locks set up during the test. If it is, terminate the
blocking job, and let the QA continue, reporting the test failure at
the very end.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

(cherry picked from commit 34c5ec6c)

This patch adds a QA utility function that acquires a set of locks, and
attempts to run a given function with the locks in place. Should the
given function block, this function does not detect this - later
patches will address the issue.

An example of its use is provided by having the move-instance test
modified to use it.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

(cherry picked from commit 343c9ed2)

Conflicts:
	qa/qa_rapi.py (didn't backport changes to this file)

The list-drbd QA test assumed falsely that no instance has more than
one disk. This can happen, and this patch accounts for it.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

When disk templates were ignored in the QA, no messages were emitted,
in contrast to pretty much all tests. To make it easier to see why
a test was not executed, this patch performs a minor refactoring and
adds some output for the templates.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This patch adds a tiny helper function that allows error messages
output by the QA asserts to be prefixed with whatever the user
supplies.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The UUID/name switch avoided this particular bit of code, and as a
result the list-drbd command failed as it tried to compare UUIDs and
names. This patch fixes the retrieval, converts the newly returned
UUIDs to names, and modifies the QA to the results and not only the
invocation are checked.
Signed-off-by: Gerard Oskamp <gjo@google.com>
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch invokes the RAPI symmetry tests for groups.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch allows users to specify specific parameter aliases enabled
by RAPI, and test whether their value can be retrieved and set.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch adds the RAPI symmetry test for nodes.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

The original design of the RAPI symmetry test assumed that all RAPI
calls that modify parameters use the PUT method. This is not true in
the case of the node parameter modification method, and while that
itself should be addressed or at least examined, the test method has
to change to accomodate both PUT and POST requests.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch adds the test for RAPI symmetry at the cluster level. There
are some fields that have not been exposed yet, and their absence was
noted.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

The RAPI should allow all the parameters of objects to be gotten and
set under the same names. This patch adds a test that checks if this is
the case by using the underlying opcode arguments.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch does some housecleaning and sorts imports.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

The previous patch loading utilities omitted empty patches, as they
were thought to be of no significance, and when no patches were used,
the import and therefore dependency should not be used. If a user has
added an empty patch file, and made an entry in the order file, the QA
would treat this as an error as it had no means of differentiating
between a patch not present and an empty patch.

This patch fixes the solution by better handling empty patches, and
logging warnings when they are encountered.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

The formatting functions in qa_utils.py cannot be used by modules
imported there, such as qa_config. This patch factors the function
calls into a separate file. Also reorders imports in touched files.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

To explicitly specify the order of patches executed, the QA provides an
"order" file. It can contain names of patches that will be executed
first, and in the order listed, before all the other patches that still
follow an alphabetical order.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This patch allows support for multiple patches placed in the "patch"
directory, which are executed in alphabetical order.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* Refactors the current patch code to allow for multiple patches that
  can be applied, yet leaves only one for now.

* Rewords the message shown to the user in case the modules needed for
  patching are missing.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

In order to work, it is necessary:
* Write a JSON Patch to file "qa/qa-patch.json" in a source repository.
  (And be careful not to commit it!) The file must conform to RFC6902
  and describes what changes should be done to the QA JSON
  configuration. This is simple as
   [
     { "op": "add",
       "path": "default",
       "value": false },
     { "op": "add",
       "path": "/tests/instance-add-file",
       "value": true }
   ]

* Install "jsonpatch" and "jsonpointer" modules on the machines QA is
  running on. Either using "easy_install" or as Debian packages using
  "pypi-install".

QA only loads the modules if the patch file is present and non-empty.
Therefore no changes to buildbots are needed, if the feature is not
used.

It's advisable to add a git pre-commit hook for the main repository to
prevent changes to "qa/patch.json", keeping it to "[]". Suggestion for
such a hook:

  # Exit code 1 if the patch file isn't just "[]":
  python -c 'import json, sys; \
    exit(0 if json.load(open("qa/qa-patch.json", "r")) == [] else 1)'

Locally it's possible to tell git not to track changes to the file at
all (see https://help.github.com/articles/ignoring-files):

  git update-index --assume-unchanged qa/qa-patch.json

Cherry-pick of e5398c3a

 from stable-2.11.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

The instance multi-allocation had no tests to detect its breakage, and
this patch fixes that.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The newly added job QA has some flakiness with respect to its use of
gnt-job watch. Fix this by waiting until the canceling status is
replaced with the canceled status, or a timeout is reached.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This patch introduces a QA test in which a job is cancelled while
waiting.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The QA did not have a test for network tags until now, and this patch
remedies the situation.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* Mention restrictions in NEWS
  - RBD userspace access mode
  - In case of a downgrade instances should suffer a reboot to be migrateable
* Bypass interactive verification in NIC modifications via --force option
  - Mention it in man page
* Print "modifications take place after restart" message no matter if
  --hotplug option is passed. Change cmdlib to append hotplug status info
  (if any) to the final result printed out to the user
* Change hotplug option description
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Only if default hypervisor is KVM try all hotplug related actions.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch adds additional 'gnt-node list' commands to
the testing of 'gnt-cluster master-failover' in order to
test if ConfD (or LuxiD) is still running after a
master-failover.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This patch removes a QA test that was supposed to test
that unused disk templates can be disabled safely. The
test is now subsumed by unit tests and thus we remove it
here. Besides that, it was actually testing the wrong
thing, as it tested that a _used_ disk template could
be disabled.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This patch replaces all usages of the utility function
'GetLvmDiskTemplate' by the new 'DTS_LVM' constant
to make it consistant with the usage of other DTS_*
constants.

Additionally, it provides a unit tests to ensure
consistancy between DTS_LVM and the mapping of disk
templates and storage types.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Newer versions of pylint are more strict regarding indentation rules. This
commit fixes a couple of indentations that they consider wrong.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

Update references to constants that were previously in the generated
'lib/_autoconf.py' and since the previous patch have been included in
the Hs2Py constant generation.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Fix all instances of pep8's error: "E128 continuation line under-indented for
visual indent".
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Mark a string representing a "sed" command including a regular expression as a
raw string, so that the entire escaping is performed by "sed" itself.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>