GitLab

To compensate for the cases where a QA test is supposed to block when
a lock is present, add an additional option showing whether blocking is
supposed to happen or not.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch adds threading to the RunWithTests function, allowing one
thread to execute the QA test, and the other to monitor if it is being
blocked by locks set up during the test. If it is, terminate the
blocking job, and let the QA continue, reporting the test failure at
the very end.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The command logging performed by GetCommandOutput can be distracting
and unwelcome. This patch adds a switch allowing the command output to
be muted as needed.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The SSH multiplexer is used by default in every connection made, and
when multiple threads are used, this causes strange issues when running
the QA locally. This patch allows the multiplexer to be bypassed,
making a new connection every time.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch adds a QA utility function that acquires a set of locks, and
attempts to run a given function with the locks in place. Should the
given function block, this function does not detect this - later
patches will address the issue.

An example of its use is provided by having the move-instance test
modified to use it.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch sorts the imports of qa_rapi.py to make things a little bit
more orderly.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch aligns the timestamps output as a part of gnt-job info, and
performs minor refactorings in the process.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Aligning dictionary entries makes no difference to a YAML parser, but
makes the output much easier to read and compare. This patch adds the
possibility of specifying alignment groups to ordered dictionary
entries.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch changes gnt-job info to use standard functions defined in
cli.py, and output valid YAML.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The PrintGenericInfo function in cli.py did not handle tuples as
containers of items, making it impossible for these to be deserialized
automatically when a YAML parser is used. This patch adds separate
handling of tuples, including inlining them for readability when
possible.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The move-instance QA test will have to be changed in the following
patches to allow testing opportunistic locking.

This patch retains the same functionality as before, but allows an
iallocator to be used, and splits the move back and forth into two
invocations, the first of which will be wrapped later on.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The gnt-debug delay command could be useful as a means of acquiring
locks for testing purposes. In practice, to be useful it should be
interruptible, otherwise we risk race conditions or long delays.

This patch follows the examples of the move-instance command and the
gnt-debug test-jobqueue commands, and introduces a mechanism for
communicating with the delay command, allowing it to be interrupted
early.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Following the example of the job-queue test, the delay also defines a
log message type of its own.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch allows the opcode option to be used through the gnt-debug
client.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

To prepare for the use of sockets to interrupt a running delay opcode,
this patch adds an option to the opcode for doing so.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

As the delay class will also have to start using domain sockets,
extract the functionality into a helper class.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The previously introduced changes caused some tests to fail. This patch
alters the tests to expect different values, while still ensuring
correct behavior.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The previous version of the LUTestDelay opcode relied on the utility
function complaining about the negative duration. As this function has
been removed for now, do the check ourselves, and issue a more
appropriate exception.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The gnt-debug delay command executes the delay first on the master, and
only then on all the other nodes, causing a significant delay. This
patch makes the command treat the master as it would all other nodes.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The RAPI client is meant to be independent from the wider Ganeti
codebase, but it still needs to use Ganeti constants. To enforce this,
a unit test is used. This patch extends the unit test with error code
messages, as these will be used in the commits to come.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Attempting to acquire node locks with a certain fixed delay could lead
to unwanted synchronization with other processes competing for the same
locks. This patch adds some random variation to the attempts.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch allows opportunistic instance creations to be attempted,
with a delay between them giving nodes the opportunity to become
available. After the given number of opportunistic tries has been
exhausted, a normal and blocking instance creation request is issued.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

To start off the introduction of oppportunistic locking during instance
creation, this patch adds the options allowing the locking to be
invoked.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

It seems '_CheckOSVariant' was moved from 'ganeti.cmdlib.instance' to
'ganeti.cmdlib.instance_utils' but the source was never deleted.  This
patch deletes the source copy if this function.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Before most of the calls were implemented manually, ignoring any
generated code for maniupulating JSON. Using 'ArrayObject' we can
automate the process.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. which is one of the few ones that use extra fields.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

It checks if 'fromJSArray . toJSArray' is equivalent to 'Ok'.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The type classes are more closely related to JSON and for using them
it should not be necessary to import Ganeti.THH.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This is in particular needed for RPC requests which are serialized this
way and therefore using `showJSON` on the generated objects doesn't
work.

The fiels are serialized in the order they're declared, together with
any extra fields they declare.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Since thre already is `DictObject` class with `toDict`, there is no need
for them, just polluting the name space.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Since there is already `DictObject` type class which could share code
with generated instances of `showJSON`, the generated code is split into
two parts: One that converts between `[(String, JSValue)]` and the data
type and another that uses `DictObject` instance to implement `showJSON`
and `readJSON`.

This removes code duplication between the two and naturally adds
`fromDict` to `DictObject`.

Also let loading functions be of type `Q Exp` instead of less flexible
`Q (Name, Stmt)`.

While at it, fix also `map (uncurry ..) (zip ..)` into `zipWith` at
several places.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Make use of asynchronous worker to save to log status after every
update. The answer to the query is only returned, once the state
is saved on disk. Note that replication of the lock status is not
necessary, as all jobs will be gone if the master node goes away.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

It will be used to persist the state of the lock allocation on
disk, allowing locks to survive reboots of WConfD.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

According to the specification in our design, if A is a group
lock and B belogs to this group, then all locks between A and B
in the lock order also must belong to A. Verify this.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

As specified in our design, if A is a group lock and B
belongs to A, then A must be earlier in the lock order
than B. Verify this property.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

With the extension of Ganeti.Locking.Locks.GanetiLocks to the
full lock hierarchy, also extend the instantiation of Arbitrary.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Make Ganeti.Locking.Locks.GanetiLocks present the full Ganeti
lock hierarchy. This is in accordance with the current Python
implementation.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>