This add support for exporting constants from multiple modules (as
opposed to hard-coding constants), and also makes the output more
readable by skipping things we know for sure we don't want to convert
(as opposed to things we would like to but don't know _how_ to
convert).

Additionally, we export the constants from the luxi module too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The file was there and listed in Makefile.am, but I forgot to add it to
commit a0c3e726.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

The “RunTestIf” uses an AND association between the passed test names,
so the instance would be be shut down if both rename and reinstall were
enabled. Since this is not necessarily the case, OR must be used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Until now “TestRunIf” and “TestEnabled” could only handle AND. With this
patch a new class named “Either” is added to “qa_config” and allows OR.
The name “Either” was chosen instead of “Or” as the latter is very close
to the reserved keyword “or”.

Examples:
  ["rapi", Either(["instance-rename", "instance-reboot"])]

  Either(["node-list", "instance-list", "job-list"])

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit 30d25dd8 moved the htools code to new-style exception handling,
but the hconfd code hasn't been, which fails when compiling on newer
GHC versions.

This patch does the rest of the move; however, the situation is not
very nice, we should have a better way to handle this (introduce
catchIO? or something like that).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

The reason why grow-disk was doing:

$ gnt-instance grow-disk instance3 0 -64
Unhandled Ganeti error: Invalid format

Is because it does it's own ParseUnit call, and doesn't transform that
into a nicer message.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Since we use a testutils.InputTestClient(), then the actual error
expected is VerificationError, and not GanetiApiError (which is used
at real run-time).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

So while testing my previous patch I run setup-ssh manually without
any arguments, and it did nothing (no complains, no messages,
anything). That was very surprising, so let's make it behave better if
no nodes are passed.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This works around a problem in Paramiko, but which looks like it's a
Ganeti problem:

[cluster] root@node4:~# gnt-node add --readd node3
/usr/lib/python2.6/dist-packages/Crypto/Util/randpool.py:40:
RandomPool_DeprecationWarning: This application uses RandomPool, which
is BROKEN in older releases.  See http://www.pycrypto.org/randpool-broken


  RandomPool_DeprecationWarning)

To silence this, we do an ugly thing (since Python 2.4 doesn't support
context managers): manually reset the warnings filter, which resets
any -W command line flags (hopefully noone calls this tool manually so
often).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This script shows a few examples on how to use the RAPI input test
client. It is also run at “make check” time to ensure it's not
completely broken.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch adds a mock RAPI client to test input values to methods. All
methods either raise an exception if there was a problem or return None.
Third-party code can use this to test their input values to the RAPI
client.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

With this patch headers are constructed from the PycURL options
and passed to the mock implementation.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is preparation for the mock system, where we need the same cURL
mock.

Signed-off-by: René Nussbaumer <rn@google.com>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This exception is raised to abort before actually sending a LUXI call
(there is no LUXI server involved in the test). The testing utilities
catch the exception to report a success (i.e. the code didn't throw
an exception before due to invalid types, etc.). To allow the exception
to be thrown all the way to the test utilities, the HTTP server library
must ignore it. Also some overly generic exception handling is removed
from the RAPI request handler.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be used in a unittest for RAPI testing utilities.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

By moving the code loading RAPI users from a file into a standalone
function it becomes easier to use the RAPI request handler in
tests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This will be used for providing a type-checking RAPI client for tests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This also improves the log messages for the (default) relative mode
("by %s to %s").

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The CLI currently doesn't permit this, but a direct job submission
would allow it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This logs all external commands in normal (non-debug) mode. This will
leak the DRBD secrets, but in any case we do log failed commands, so
it's not a significant change.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit 1010ec70 enabled all tests by default. In some scenarios where
the configuration is static one may not want newly added tests to be
run.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Chris Schrier <schrierc@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Chris Schrier <schrierc@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Fixes QA after commit cfdf561d.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Chris Schrier <schrierc@google.com>

Short description: This fixes an issue whereby masterd would become
unresponsive on the LUXI socket, leading to client timeouts. While made
worse in 2.5, the underlying issue was already present in 2.4.

Longer description: Until now all LUXI queries would acquire the BGL
(big Ganeti lock) in shared mode. With the exception of OpNodeAdd and
OpNodeRemove, this was also the case for all opcodes before version 2.5.
In 2.5 we split OpClusterVerify into multiple opcodes, one of which
(OpClusterVerifyConfig) now acquires the BGL in exclusive mode. Whether
or not doing so is good is a separate discussion: OpNodeAdd and
OpNodeRemove, as of this writing, still require an exclusive BGL.
OpClusterVerifyConfig is run more often than OpNodeAdd or OpNodeRemove
in normal clusters, which is why we only recognized this issue in 2.5.

What would happen is that once OpClusterVerifyConfig tried to acquire
its exclusive BGL while it was actually held by other opcodes (e.g.
OpInstanceReplaceDisks), the locking code would not grant shared
acquires for the BGL, even when the exclusive acquire is removed from
the queue for a short amount of time after a timeout. This is necessary
to prevent lock starvation.

In this situation further LUXI queries requiring the BGL in shared mode,
e.g. OpClusterQuery, would block and the client eventually time out.
Over time they fill the client request workerpool's queue and at that
point even requests not requiring the BGL stop working. Once the
long-running operation(s) holding the BGL in shared mode finished,
OpClusterVerifyConfig gets it in exclusive mode and everything returns
to normal. LUXI recovers very soon too.

I'd like to thank Bernardo Dal Seno for his contribution to this bugfix.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit 0fa753ba)

Also mark it as deprecated in NEWS as normal queries can be used
instead.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The main intention of this patch is to make it possible to retrieve
cluster tags via query2. While at it I decided to convert
LUClusterConfigQuery right away. Some of the values returned by
LUClusterQuery are also included, but the conversion of LUClusterQuery
is not yet complete.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

These are not used anywhere in Python or Haskell.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This solves one case where locks are acquired during LUXI queries.
Pretty late into the transition I noticed that OpBackupQuery had a
“use_locking” parameter for a long time, but didn't use it. Since
most of the other changes were already and this allows exports to
be listed via RAPI (/2/query) I decided to finish.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

str.isdigit is about 4x faster than using a regular expression ("\d+").
This is in the inner sorting code so speed matters.

$ python -m timeit -s 'import re; s = re.compile("^\d+$")' \
's.match(""); s.match("Hello World"); s.match("1234")'
1000000 loops, best of 3: 0.937 usec per loop

$ python -m timeit '"".isdigit(); "Hello World".isdigit(); "1234".isdigit()'
1000000 loops, best of 3: 0.218 usec per loop

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Chris Schrier <schrierc@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It was only used in one place and wasn't really necessary.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>