The previous patch series made it simpler to deal with ERROR_up offline
instances, but didn't fix qa, that had the old expectations. This fixes
that, and adds more testing about the expected behavior.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is not a trivial s/frozenset/compat.UniqueFrozenset/, but rather
only replaces “frozenset” where appropriate. Most of the places are
“static” information that doesn't change after the module has been
loaded.

Some docstrings and code formatting (e.g. empty lines) issues are
addressed as well. Some lines got too long and were wrapped.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch removes code which is no longer used due to refactoring:

- http.InitSsl, last usage removed in commit 33231500 (“Convert RPC
  client to PycURL”)
- rapi.baserlib.MakeParamsDict, last usage remove in commit 4e5a68f8
  (“RAPI: rlib1 removal”)
- rapi.baserlib.ExtractField, last usage removed in commit 028c6b76
  (“RAPI:Fix root list and unittest for it.”)
- qa.qa_utils.AssertNotEqual, last real usage removed in commit
  2f4b4f78 (“Simplify QA commands”) (but it was still imported for a
  while, until we extended pylint to run over the QA code as well)
- qlang._IsHostname, added in commit f8638e28 (“Detect globbing
  patterns as query arguments”) but never used
- cmdlib._BuildNetworkHookEnvByObject, last usage removed in commit
  f22433c0 (“Locking related fixes for networks”)
- NLD constants, which are obsolete—the nbma project is no longer
  maintained and it's not working with current ganeti

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Some fixes, added more information in a few points, removed a stale (5+
year old) TODO comment.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Iustin Pop <iustin@google.com>

We also need to change the signature for _GetInstanceInfo, since we
don't have access to the instance dictionary in the rename instance
test.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The test can now be enabled/disabled with the "group-rwops" flag.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Before, it could happen that the master node was chosen to be set offline
in the "instance-remove-drbd-offline" test. Also changed a finally clause
to make sure that an exception while making the node online doesn't break
undoing other things.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

By default some QA tests use iallocators. It is now possible to disable
iallocators in QA, so that QA doesn't force having iallocators set up.
The affected tests have now alternatives pathways.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

- Simplified shell command for adding hosts, too
- The exception handling is also improved by re-raising any exceptions
  instead of swallowing qa_error.Error

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Bug introduced in commit cc4b14f0: Part of a statement was missing. This
makes pylint-qa fail.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

QA for recreate-disk and replace-disks now run also with the default
iallocator.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

If one or more jobs failed the exit status should be set accordingly.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

If an administrator sets always_failover, it means that there is no need
for another explicit approval to failover instead of migrating.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
(cherry picked from commit b5f0b5cc)

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

File system paths moved from constants to pathutils.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Also, the value of always_failover is not hard-coded any more, and it's
possible to disable changing the always_failover parameter, if needed.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

If an administrator sets always_failover, it means that there is no need
for another explicit approval to failover instead of migrating.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

To help developing Ganeti on newer distributions, let's try to fix
pylint warnings/errors. I'm using pylint from current Debian wheezy:
pylint 0.25.1, astng 0.23.1, common 0.58.0, and we have 3 things that
needs fixing.

First, a really wide "except", with the silencing in the wrong
place. I'm not sure why this doesn't have "except Exception", so let's
add it. However, pylint still complains about "Catching too general
exception", even though we do want to catch both system and our
exception, so let's add a silence for W0703. It's true that we
shouldn't catch KeyboardInterrupt and friends, but that should be
cleaned up on the master branch.

Second, pylint complains about "redefining name builtin tuple",
because we do some pattern matching in the except blocks in
netutils. This seems to be a false positive, but let's clean the code
around this.

And finally, type inference again goes bad, so let's silence E1103
with its "boolean doesn't have 'get' method".

After this, I can run "make lint", and by extension "make
commit-check" on Debian Wheezy, yay! We might be able to bump our
required pylint versions to something not ancient…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Pylint version:

  pylint 0.25.1,
  astng 0.23.1, common 0.58.0

crashes when passing the fully-qualified decorator name with:

  File "/usr/lib/pymodules/python2.7/pylint/checkers/base.py", line 161, in visit_function
    if not redefined_by_decorator(node):
  File "/usr/lib/pymodules/python2.7/pylint/checkers/base.py", line 116, in redefined_by_decorator
    decorator.expr.name == node.name):
AttributeError: 'Getattr' object has no attribute 'name'

I found out that simply using a shortened name will 'fix' this issue,
so let's do this to allow running newer pylint versions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:

- bump version in the docs
- silence some new checks that are wrong due to our indent=2 instead of 4
- fix lots of errors in the code where the indentation was wrong by 1
  or 2 spaces
- fix a few cases of == True, False, None and replace with 'is'
- re-indent some cases where the code is OK, but pep8 complains

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This has been a long-standing cleanup item, which we've always
refrained from doing due to the high estimated effort needed.

In reality, it turned out that after some infrastructure improvements
(the previous patches), the actual job queue-related changes are quite
small.

We will need to update the NEWS file later, but so far the RAPI
documentation doesn't mention that the job ID is a string (it only
says it is "a number"), so it doesn't look like it needs update.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Additionally, we run hbal if the htools tests are enabled (new
key). Note that hbal will directly exit, since there are no instances,
but at least it will load and parse the group details.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

QA for recreate-disks called it twice with a list of node; the first call
now uses an allocator.

Signed-off-by: Adam Ingrassia <api@google.com>
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be run only if 'haskell-confd' QA configuration has been
enabled.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Added test checking that it is possible to remove an instance with drbd
disk when non-primary node is offline.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

We don't have a specific test for activate disks, so let's add it in
the cases where we run (incidentally) activate-disks.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Two blank lines are needed between functions.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

The test is disabled by default, as it takes a long time.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is useful to simulate HW problems during QA.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is basically another case of commit d488adb6.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The hypervisors are a list and need to be comma-separated.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

For example, the “initrd_path” parameter is only available with xen-pvm,
whereas “boot_order” is only available with xen-hvm.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Some parts of QA should only be run with certain hypervisors (e.g.
testing instance HV parameters). To make that happen we always define
the cluster's hypervisors and provide a function to get the default
hypervisor.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch only touches comments and man pages.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit f346a7d9 broke the TestInstanceImport test in QA, since it
dropped the "always generate" parameter for backup import. We add an
override mac parameter so that, no matter what is defined in the
config file, imported instances (can) have a different MAC.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

In some QA environments static MAC addresses must be used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In some QA environments external scripts may add tags. With this patch a
regular expression can be used to ignore them during QA runs.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Commit 1010ec70 enabled all tests by default. In some scenarios where
the configuration is static one may not want newly added tests to be
run.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit 59a8fe48)

Currently, the "common" instance tests (shutdown, rapi stopped
console, stopped modify, startup, rapi shutdown, rapi startup, list,
info, modify, rapi modify , console, rapi console, reinstall, rename,
rapi rename, rename and back, rapi rename and back, grow disk, reboot,
tags, cluster verify, rapi test instance, node list, job list) are run
4 times:

- one time for plain instance created via RAPI, using client
- one time for plain instance created via RAPI, without client
- one time for plain instance created via gnt-instance
- one time for DRBD instance created via gnt-instance

This makes the QA long, and is over-doing it (for non-full QAs): an
instance created via RAPI (either client) and via gnt-instance should
have the same parameters, but our duplicate tests do not ensure that,
just that instances behave OK.

This patch adds a toggle so that we can skip the common tests for
RAPI-created instances; the creation/deletion is still performed, but
all the other operations are not. This reduces the time of a "quick"
QA by ~20% (1h:15m → 1h).

The common tests will still be run (unconditionally) for
gnt-instance-created instances.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>