Just using ht.TListOf as a type check doesn't work correctly. The
function must be called with the expected item type. In this specific
case TListOf was always called with the filter as a value, and the
result of that call evaluated to truth. Since filters can be quite
complex there's no check yet, and therefore just “TList” is used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

- Commit b7a1c816 changed the LU to generate jobs
- Mention documented results in NEWS

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

If an instance had actually a missing disk, the type check would fail.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In version 0.21, pylint unified all the disable-* (and enable-*)
directives to disable (resp. enable). This leads to a lot of
DeprecationWarning being emitted even if one uses the recommended
version of pylint (0.21.1, as stated in devnotes.rst).

This commit changes all the disable-msg directives to disable.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Identified using the “pep8” utility.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The original design for query2 specifically excluded locking, but now
it's turned out that it would be a good thing to have in watcher. This
patch adds a new parameter to OpQuery and enables its use in LUQuery. A
missing function is added to LUGroupQuery, a comment clarified in
_NodeQuery and all locks declared as shared acquires in the same LU.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Some of these will be used by the RAPI documentation.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Instead of a rather complicated expression only “JobId” is output. Job
ID lists (like generated by “SubmitManyJobs”) are limited to two-item
lists. Unittests are added.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch removes the list of node groups (not used anymore since
commit fcad7225) from OpClusterVerifyConfig's result and adds result
verification to all OpClusterVerify* opcodes.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch moves the logic for verifying the various node groups in a
cluster into the master daemon. Job dependencies are used to ensure the
configuration, which requires the BGL, is verified first.

With this change it will be possible to expose whole-cluster
verification through the remote API without requiring additional client
logic on top of standard features like LU-generated jobs and job
dependencies.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Just some cleanup before the 2.5 release.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This gives a hint as to what's queried. “QUERY(instance)” or
“QUERY(node)” are way better than just “QUERY”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

It is no longer used and has been deprecated in 2.5.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is quite similar to evacuating a group, but the locking
is different.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It turns out that the only use of the operator module was for
itemgetter, so patch eb62069e should have removed that import too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

operator.itemgetter(0) → fst
operator.itemgetter(1) → snd

snd is not used yet, but it makes sense to add both.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This adds the infrastructure necessary to check opcode results using
ht-based functions. Checks are added for two opcodes.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Until now verifying disks, which is also used by the watcher,
would lock all nodes and instances. With this patch the opcode
is changed to operate on per nodegroup, requiring fewer locks.

Both “gnt-cluster” and “ganeti-watcher” are changed for the
new interface.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This attribute allows programmatic submitters of jobs (e.g. iallocator)
to add a comment to each opcode, describing its purpose. Example:

$ gnt-job info 123
Job ID: 123
  …
  Opcodes:
    OP_INSTANCE_REPLACE_DISKS
      …
      Input fields:
        comment: Replaces disks on inst1.example.com
      …

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

With this change users of the “SubmitManyJobs” interface can use
relative job dependencies. Relative job IDs in dependencies are resolved
before handing the job off to the workerpool.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

An overview is available in the design document for this change,
doc/design-chained-jobs.rst.

When a job enters the job processor, the current opcode's dependencies
are evaluated. If a referenced job has not yet reached the desired
status, the current job is registered as a dependant. The job processor
will continue to work on other pending tasks. When a job finishes it
notifies any pending dependants by re-adding them to the workerpool.

A per-job processor lock is necessary for rare cases where the same job
can be re-added twice.

There is no way to view waiting jobs at the moment, but I plan to
export this information to “gnt-debug locks”.

A so-called dependency manager takes care of managing waiting jobs and
keeping track of their status.

Unittests are included.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Places which receive floats can usually also deal with integers, e.g.
OpTestDelay. Tests are added and the new check function is used for the
aforementioned opcode and verifying query results.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Creates the instance, but pauses execution before booting. This combined
with 'gnt-instance console' unpausing instances means that the entire
boot process can be viewed and monitored.

Signed-off-by: Stephen Shirley <diamond@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

LUNodeEvacStrategy has been replaced with LUNodeEvacuate.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This new opcode will replace LUNodeEvacStrategy, which used to return a
list of instances and new secondary nodes. With the new opcode the
iallocator (if available) is tasked to generate the necessary operations
in the form of opcodes. This moves some logic from the client to the
master daemon.

At the same time support is added to evacuate primary instances, which
are also evacuated by default.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch removes all occurrences of the “multi-relocate” iallocator
mode. Commit 25ee7fd8 updated the design document and introduced
separate modes, “change-group” and “node-evacuate”. The constants aren't
removed yet as they're still used by htools.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Add a tag slot to opcodes.OpInstanceCreate. We do not reuse _PTags, as this is
intended for OpTagsSet and thus:

  a) is not documented
  b) does not carry a default value, making it mandatory

Also pass the tags to the iallocator during instance creation.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, the iallocator in 'allocate' requires mem_size on input
but serialises that as 'memory'. This inconsistency makes it hard to
automatically validate the parameters, hence this patch renames
mem_size.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit aac4511a added CheckArguments to LUNodeMigrate with a call to
_CheckIAllocatorOrNode. When no default iallocator is defined,
evacuating a node would always fail:

$ gnt-node migrate node123
Migrate instance(s) '...'?
y/[n]/?: y
Failure: prerequisites not met for this operation:
No iallocator or node given and no cluster-wide default iallocator
found; please specify either an iallocator or a node, or set a
cluster-wide default iallocator

This patch adds a new parameter to specify a target node. This doesn't
solve all issues, but will make the most important cases work again in
the meantime. This opcode will receive more work for node group support.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

With this change, LUClusterVerifyConfig becomes a "light" LU that only
verifies the global config and other, master-only settings, and the bulk of
node/instance verification is done by LUClusterVerifyGroup, which only acts
on nodes and instances of a given group.

To ensure that `gnt-cluster verify` continues to operate on the whole
cluster, the client creates an OpClusterVerifyGroup job per node group; for
convenience, the list of node groups is returned by LUClusterVerifyConfig.

Signed-off-by: Adeodato Simo <dato@google.com>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will allow stopping or starting an instance without changing the
remembered state. While this seems counter-intuitive at first (it will
create cluster verify errors), it can help in a few corner cases:

- shutting down an entire cluster for maintenance but without having
  to remember state
- doing testing of Ganeti itself

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The opcode parameter ignore_consistency was used in the LU, but not
actually declared in the OpCode. The patch adds it in the opcode and
the command line client.

ObQuote — Please, please, can I have static typing?

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Two opcodes already use it and we need it for a third, time to add a
constant for it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Depending on the opcode and its parameters, the existing “Summary”
function can give a rater long summary. For displaying the summary in
logs and in the lock monitor, it should be shorter. Hence this new
function is added to just use the opcode ID with common prefixes
replaced (e.g. “INSTANCE_” becomes “I_”). Opcode values are not used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, when converting an instance from plain to DRBD, the
instance is blocked during the entire resync period. This patch adds
the --no-wait-for-sync so that the operation finishes as soon as the
DRBD sync has started, without waiting for the entire sync. This makes
the instance available much faster.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch introduces the option of changing an instance's nodes when
doing the disk recreation. The rationale is that currently if an
instance lives on a node that has gone down and is marked offline,
it's not possible to re-create the disks and reinstall the instance on
a different node without hacking the config file.

Additionally, the LU now locks the instance's nodes (which was not
done before), as we most likely allocate new resources on them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>