Determine the job file path with qa_utils.MakeNodePath, so that we
get the correct path, even for vcluster.
Signed-off-by: Lisa Velden <velden@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Our tests running via RunWithLocks strictly depend on no
watcher jobs interfering. Therefore they pause the watcher;
unfortunately, there still is a race: the watcher only checks
the pause status upon its invocation, but submits jobs later
in its run time. Therefore not only pause it (doesn't hurt),
but also add a filter to reject all its jobs, and then wait
for all running jobs to terminate.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...so that it can be used outside the filter test as well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This test is done with the newly introduced opcode 'gnt-debug
test-osparams'. Redaction of secret parameters within the job file is
also tested again with this opcode.
Signed-off-by: Lisa Velden <velden@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Test if secret parameter values for instance create jobs are
redacted in job files.
Signed-off-by: Lisa Velden <velden@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

When running the QA, we copy the rapi certficate to the
machine which steers the QA to use it later in the QA
for testing RAPI calls. However, before we get to that
part of the QA, the rapi certificate is replaced when
'gnt-renew crypto' is called.

This patch makes sure that the new rapi certificate is
copied to the steering machine so that later RAPI calls
do not fail. It remains mysterious how this worked before.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Avoiding lint errors on the onehand, and code complexity
on the other.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

This patch fixes a bug in the comparison of the
'ssconf_master_candidate_certs' and disables the test
for vcluster.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Currently, there is not test for just running renew-crypto
to only renew the cluster certificate (and not the node
certificates). This patches adds the test.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In order to hunt down some flakiness, this patch adds
and additional check to the QA of renew-crypto to ensure
the state of ssconf_master_candidate_certs.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Currently the cluster is only verified after a series of
renew-crypto operations is carried out. This makes it hard
to trace errors back to originating call. This patch adds
a verifcation step after each renew-crypto call.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In tests, give Ganeti enough time to actually start up jobs before
asserting that they succeed. While normally forking and executing
a job is finished in less than a second in some circumstances it
can take longer; so give enough time to avoid flaky tests.

While there, also pause the watcher, so that it doesn't submit
jobs inbetween that cause our 0.01 second delay jobs to take
longer while waiting for locks taken by the watcher's jobs.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This test verifies a global limit on the number of jobs running. This
requires knowledge of all jobs submitted between the addition of the
filter and the submission of our last test job. While we send these
commands directly one after the other, this still takes a second or
two, thus giving the watcher slightly less than a 1% change of
interfering. Avoid this by pausing the watcher during this test.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The test submits 3 jobs in an environment rate-limited to two, and verifies
that the first 2 jobs are running whereas the third one remains queued. However,
it takes for Ganeti some time to get a submitted job to actually being confirmed
waiting (in some circumstances over a second). So wait a bit longer to avoid
losing the race. While there, increase the time of the delay jobs to avoid
them finishing during that test in an extremely slowly running QA. (Note that
the jobs are killed forcefully at the end of the test, so that time is never
really waited for.)
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The test submits 3 jobs in an environment rate-limited to two, and verifies
that the first 2 jobs are running whereas the third one remains queued. However,
it takes for Ganeti some time to get a submitted job to actually being confirmed
waiting (in some circumstances over a second). So wait a bit longer to avoid
losing the race. While there, increase the time of the delay jobs to avoid
them finishing during that test in an extremely slowly running QA. (Note that
the jobs are killed forcefully at the end of the test, so that time is never
really waited for.)
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The QA had the annoying bug that if you run burnin, it
would try to upload the burnin file from the tools/
directory relative to the working directory and not
the location of the qa_cluster.py script. If you
accidentally start the QA from a differen directory
than qa/ it would run for a long while and only
crash when it comes to the burnin part. This patch
fixes the glitch by taking the location of the
QA scripts as a reference rather than the working
directory.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Cluster verification must not do any changes to the configuration.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The function checks the SHA1 sum of a given file before and after a
block of code is executed, verifying that the content of the file
remains the same.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Unlike almost all other modification commands, hotplugging a device and
failing does not result in an error code showing that the operation was
unsuccessful. Because of this, the QA ignored hotplugging failures. To
fix this, this patch makes the QA examine the output of the command for
signs of failure.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

A previous patch suppressed command output if the command succeded,
which reduces the amount of information we have in the QA, especially
warnings or the like. This patch restores the output, while still
ignoring the use cases in which we really do not care whether the
command succeeds or not.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

If a command fails/succeeds that should not do so,
show stdout and stderr. This information can be
relevant for debugging.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Cherry-picked-from: 7f7c9c2cSigned-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This is for cases where we don't care about the exit code.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Cherry-picked-from: 226455dfSigned-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This is usefull if the run commands output something that's needed afterwards.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Cherry-picked-from: b47587b0Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

While previous patches enabled QA tests of exports in instance moves,
this patch does the same for local exports.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This removes a restriction on a QA that file-based exports are not even
tried. Since we can do them, we should.
Signed-off-by: Aaron Karper <akarper@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Cherry-picked-from: 0389f215Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This removes a restriction on a QA that file-based exports are not even
tried. Since we can do them, we should.
Signed-off-by: Aaron Karper <akarper@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

The disk template in the QA should not be extracted from the
aggregate information shown, but the disks.
Signed-off-by: Aaron Karper <akarper@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

As noted in the previous commit, the lxc_ prefix is bad and it's good
to be rid of it, and we can take advantage of this change to rename the
variable.
Its previous name was meant to reflect the lxc config option being set
(lxc.tty), but we should use a more descriptive name.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The rapi-workload test suite was introduced in 2.10 to try and provide
a chunk of workload which could be used to detect regressions in the
RAPI functionality and format. It served its purpose, helping detect a
few important regressions in time, but with the assistance of manual
tools which were never automated. Automating them proved difficult, and
probably infeasible.

The flaws in the QA that caused the regressions were removed over time,
and with 2.13 around the corner, it's time to shoot this horse. This
patch removes the file, and a sole mention of it in the Makefile.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

It is hard to debug the upgrade QA if the output is not
shown in the logs. Redirecting the output of cluster
verify.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This adds a few 'gnt-cluster verify' to QA tests that are
prone to mess up the SSH key files.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This was disabled due to Issue 885. This issue states that NIC
hotplugging sometimes makes an instance unresponsive. This cannot be
the case, since NIC hotplug takes place on an extra NIC and does not
mess up with instance's primary NIC. Still this sometimes might be
related with --online fix (see commit b0a383aa).
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Since the qa can take a lot of connection, it might happen that one of
them times out for reasons unrelated to ganeti. This patch allows the qa
to retry the command in that case.
Signed-off-by: Aaron Karper <akarper@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

For the oStrictHostKeyChecking we use the variable tmp, which is yes or
no, depending on the strict.
Signed-off-by: Aaron Karper <akarper@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

If a command fails/succeeds that should not do so,
show stdout and stderr. This information can be
relevant for debugging.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Otherwise the watcher might restart instances just after the EPO,
failing the test.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The context is responsible for stopping and re-starting the watcher for
the corresponding block of code.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Even starting a (delay 0.01) job without filters does not happen
within the 0.5 seconds we allowed if run in a big cluster.
Use retry here as well.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>