GitLab

The formatting functions in qa_utils.py cannot be used by modules
imported there, such as qa_config. This patch factors the function
calls into a separate file. Also reorders imports in touched files.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Mark a string representing a "sed" command including a regular expression as a
raw string, so that the entire escaping is performed by "sed" itself.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This reverts commit
bfbef99b

. The problem is that
in some cases it makes sense for '?' to be displayed. This
is the case for storage reporting on clusters that have only
the 'diskless' disk template enabled. It would be nicer,
if a '-' would be displayed in those cases, but this is a
bigger change in the haskell query infrastructure. Thus,
I am reverting this for now and send a follow up patch in
the next days.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

When something goes wrong in the Haskell implementation of
queries, the 'gnt-* list' command does not actually fail,
but it just displays '?' as a sign of failure. We so far
do not test for that, which makes errors in the haskell
queries go undetected. This patch fixes that.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The tempfile command only exists in Debian. In order to run the VCluster
QA also on Fedora, replace its use by mktemp.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Existing tests are updated to cope with the new instance specs format.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The format used to store instance specs in QA is changed to better handle
multiple instance specs, a feature contained in the following patches.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The main parts, moved to qa_util, will be used for node groups too.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The code is more modular, and the output is YAML-compliant. QA has been
updated to reflect the new format.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The code is more modular and the output is YAML-compliant. QA updated to
take advantage of it.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The code is more modular and the output is YAML-compliant.

Added function in QA that uses PyYAML to parse the command output, and QA
is updated to take advantage of it (instead of using lots of complicated
REs).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

“qa_utils.BackupFile” is always called with the path for a normal node.
In virtual clusters the node prefix path must be added.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

In virtual clusters, all “gnt-*” and “ganeti-*” commands require
environment variables to be set for the virtual cluster root directory
and node name. The “cmd” script generated by “vcluster-setup” is a
wrapper setting those variables.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is no longer needed now that instance/node objects in QA have
attributes. Use a helper function instead to retrieve an entity's
identifying attribute.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

The new objects support attributes, which are cleaner than
dictionary-style access. Item access (like a dictionary) is going away
in a couple of patches. Also, pylint is better at checking attributes
than dictionary entries.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This is a temporary solution until nodes and instances are converted to
objects with attributes.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This enabled the redirection; QA output after this patch should become
much much smaller (on a 'quick' QA, we are left with ~0.5MB output on
stdout and 64MB redirected output).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Some QA commands generate non-trivial output: for a full QA, the gnt-*
list commands generate more than 300MB of data. To make the QA logs
more parseable by humans, we need support for redirecting output of
commands to a different place.

This patch adds a 'AssertRedirectedCommand' that does this,
redirecting the output to a (fixed) log file in the Ganeti log
directory.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds an optional kwarg that allows skipping the "Command:
…" line in QA output. This is helpful if we only need to run some
maintenance actions, that are QA-related and not Ganeti-related.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Now it's possible to check that cluster-verify reports all the expected
errors, instead of simply checking that it fails.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This adds a script for the QA of 'gnt-network'. So far it
covers adding/removing and connecting/disconnecting networks.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Also add a check to make sure there are enough groups defined.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch removes code which is no longer used due to refactoring:

- http.InitSsl, last usage removed in commit 33231500 (“Convert RPC
  client to PycURL”)
- rapi.baserlib.MakeParamsDict, last usage remove in commit 4e5a68f8
  (“RAPI: rlib1 removal”)
- rapi.baserlib.ExtractField, last usage removed in commit 028c6b76
  (“RAPI:Fix root list and unittest for it.”)
- qa.qa_utils.AssertNotEqual, last real usage removed in commit
  2f4b4f78 (“Simplify QA commands”) (but it was still imported for a
  while, until we extended pylint to run over the QA code as well)
- qlang._IsHostname, added in commit f8638e28 (“Detect globbing
  patterns as query arguments”) but never used
- cmdlib._BuildNetworkHookEnvByObject, last usage removed in commit
  f22433c0

 (“Locking related fixes for networks”)
- NLD constants, which are obsolete—the nbma project is no longer
  maintained and it's not working with current ganeti
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Some fixes, added more information in a few points, removed a stale (5+
year old) TODO comment.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

- Simplified shell command for adding hosts, too
- The exception handling is also improved by re-raising any exceptions
  instead of swallowing qa_error.Error
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, this is how the QA log looks like:

Command: ssh -oEscapeChar=none -oBatchMode=yes -l root -t -oStrictHostKeyChecking=yes -oClearAllForwardings=yes -oForwardAgent=yes -oControlPath=/tmp/ganeti-qa-multiplexer.DMzkuH -oControlMaster=no node14 exit
Command: ssh -oEscapeChar=none -oBatchMode=yes -l root -t -oStrictHostKeyChecking=yes -oClearAllForwardings=yes -oForwardAgent=yes node8 exit
Command: ssh -oEscapeChar=none -oBatchMode=yes -l root -t -oStrictHostKeyChecking=yes -oClearAllForwardings=yes -oForwardAgent=yes node18 exit

And these arguments are repeated over and over. This patch proposes to
log once the arguments, at the beginning of the QA (so that the SSH
commands can be reproduced, if needed) and then drop them from the
log. The new output looks like:

SSH command for primary node: ssh -oEscapeChar=none -oBatchMode=yes -lroot -oStrictHostKeyChecking=yes -oClearAllForwardings=yes -oForwardAgent=yes -oControlPath=/tmp/ganeti-qa-multiplexer.24lgrK -oControlMaster=no node14
SSH command for other nodes: ssh -oEscapeChar=none -oBatchMode=yes -lroot -oStrictHostKeyChecking=yes -oClearAllForwardings=yes -oForwardAgent=yes NODE
(the above are the informational messages about parameters, then)
Command: ssh node14 exit
Command: ssh node8 exit
Command: ssh node18 exit

This makes the QA log much more readable, by dropping unneeded clutter
(look how long the original lines were), and hopefully also a bit
smaller (current QA log is > 100MB of text output).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This makes ssh generate much less noise when QA is run itself without
a tty, as otherwise we get thousands of lines of:

  Pseudo-terminal will not be allocated because stdin is not a terminal.

in the QA logs (which are already big enough).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This reduces code duplication and provides better error messages if
something is incorrect while getting a node or instance name.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Some tests don't have access to an instance dictionary.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This script can be used to check if an instance is running or stopped at
various points during a QA run. Environment variables are used to pass
the most essential information.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This solves one case where locks are acquired during LUXI queries.
Pretty late into the transition I noticed that OpBackupQuery had a
“use_locking” parameter for a long time, but didn't use it. Since
most of the other changes were already and this allows exports to
be listed via RAPI (/2/query) I decided to finish.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Jobs have no “name” field.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In version 0.21, pylint unified all the disable-* (and enable-*)
directives to disable (resp. enable). This leads to a lot of
DeprecationWarning being emitted even if one uses the recommended
version of pylint (0.21.1, as stated in devnotes.rst).

This commit changes all the disable-msg directives to disable.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

s/'/"/ in (hopefully) the right places.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

There's a problem if you run gnt-instance rename with a non FQDN and the
renamed LU tries to resolve the hostname to make it FQDN. It could be
that this resolved hostname was just a CNAME to another name which leads
to wrongly renamed instances name.

Given a name of "foobar" is a CNAME to "moohost" now if you gonna rewrite an
instance named "foobar1" to "foobar" it will pickup the host "moohost".
This is not obvious behaviour. If we detect this case we abort with a
prereq error. The user then however can continue with a FQDN and
--no-name-check if he intends to force the operation.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The tty can have strange side effect as it seems to replace \n with
\r\n. In case of simply catting a file and compare it's output this
will break. This patch adds the possibility to disable tty if needed.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Right now, the QA code is not covered by pylint, and this shows at
least one low-impact bug.

This patch does the necessary changes to make QA pylint-clean, and the
changes the makefile to run pylint for it.

Notable changes:

- qa_utils.GenericQueryTest: randfields was not used at all, and my
  belief is that it was indented to be used in order not to modify the
  input list; so I replaced randfields with fields, so we only shuffle
  the our local copy
- qa_node.TestOutOfBand was using it's own copy of AcquireNode(), so I
  replaced it with the existing version
- qa_os: was using 'dir' in a couple of places, replaced with dirname
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>