Burnin has a custom job executor, because of its need to retry some job
series.

While we cannot replace all of it, at least the execution we can switch
to cli.JobExecutor, to take advantage of the recently-introduced
out-of-order waiting.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This makes the Log function able to take multiple args for simplified
message construction, similar to the ToStdout one.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This passes a full burnin with lots of instances, and should be safe as
we mostly to join a known root (various constants) to a run-time
variable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

I was using a too old version which doesn't got all those. This
patch is fixing the new lint errors.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is the implementation of docs/design-cluster-merger.rst. It allows
the automatic merging of one or more clusters into the invoking cluster.

While this version is tested and working it still needs some tweaking
here and there for error handling and user experience.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

There are two entry points to job execution in burnin, ExecOp and
ExecOrQueue, and these are modified to call the new _SetDebug method on
the opcodes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit 154b9580 changed (correctly) the __slots__ usage, but this broke
dumpers/loaders since we relied directly on the own class __slots__
field.

To compensate, we introduce a simple function for computing the slots
across all parent classes (if any), and use this instead of __slots__
directly.

Note: the _all_slots() function is duplicated between objects.py and
opcodes.py, but the only other options is to introduce a lang.py for
such very basic language items.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds an early_release parameter in the OpReplaceDisks and
OpEvacuateNode opcodes, allowing earlier release of storage and more
importantly of internal Ganeti locks.

The behaviour of the early release is that any locks and storage on all
secondary nodes are released early. This is valid for change secondary
(where we remove the storage on the old secondary, and release the locks
on the old and new secondary) and replace on secondary (where we remove
the old storage and release the lock on the secondary node.

Using this, on a three node setup:

- instance1 on nodes A:B
- instance2 on nodes C:B

It is possible to run in parallel a replace-disks -s (on secondary) for
instances 1 and 2.

Replace on primary will remove the storage, but not the locks, as we use
the primary node later in the LU to check consistency.

It is debatable whether to also remove the locks on the primary node,
and thus making replace-disks keep zero locks during the sync. While
this would allow greatly enhanced parallelism, let's first see how
removal of secondary locks works.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In some cases pylint doesn't parse the import correctly, so we add
silences; but there are also many cases of unused imports, which we
simply remove.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Many of our functions have to follow a given API, and thus we have to
keep a given signature, but pylint doesn't understand this. Therefore,
we silence this warning.

The patch does a few other cleanups.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Many methods are simple pure functions, and not depending on the object
state. We convert these to staticmethods.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

This removes unused variables in the rest of the code (outside lib/).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

This patch should have only:

- pylint disables
- docstring changes
- whitespace changes

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Similar to commit c881c5, we move the decorators out of classes, such
that they become simple functions instead of methods. This more clean,
since only the wrapped functions need to be methods/have access to
‘self’.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

The logging functions expand the arguments themselves, thus it's safer
to let them do it rather than manual string formatting.

Also re-wraps one comment.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

This patch modifies burnin to accept the --no-name-check option and also
adds --no-ip-check (which was always set to True before).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch removes the quotes from CommaJoin and converts most of the
callers (that I could find) to it. Since CommaJoin does str(i) for i in
param, we can remove these, thus simplifying slightly a few calls.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This also fixes a few typos.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

If we have only one node, instance moves fail, because it tries to move
the instance to itself. Skipping the operation, because in that case it
doesn't make sense.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

List of OSes are displayed by gnt-os list, rapi, and gnt-instance
reinstall --select-os, and checked by burnin. In all of these show the
list with name+variant, if the os has variants.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

The timestamp need special formatting, which was done for the internal
buffer storage but not for the messages logged in verbose mode. This
patch unifies the formatting for these two cases.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently multiple files define a 'verbose' option. This patch moves all
these definitions to a single VERBOSE_OPT option which is exported from
cli.py.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

This patch adds support for instance moves in burnin. This means that
non-drbd instances finally get support for being moved.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Replacing the secondary node always replaces it for all disks. Commit
2945fd2d added an error message for this case.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This way the produced table is formatted nicely.

Signed-off-by: Stephen Shirley <diamond@google.com>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Many burnin steps initialize the batch queue at the beginning and commit
it at the end of their operation. This patch moves this code to a
decorator, in order to reduce redundant code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Many burn steps to a manual check of instance aliveness, via duplicate
code. This patch moves this code to a decorator.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Some burnin steps are idempotent: e.g. reinstalling an instance (from
burning p.o.v.) can be done multiple times without any side-effects that
would affect later burnin steps. As such, failing the whole burnin
process due a reinstall failure is undesirable.

This patch modifies burnin by marking each opcode (in case of individual
execution) and job set retryable or not. Retryable actions will be
retried up to a number of times, after which we give up and return
failure.

One side-effect is that in case of full-failure in retryable job sets we
lose the original exception (but we do log its string format), so we
have a little bit less information in this case.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A long-standing bug in burnin makes errors during the removal phase
(e.g. because an import has failed, or because the initial creation has
failed) hide the original error.

This patch suppresses removal errors if we are already in ‘has_err’
mode, and otherwise it displays them normally.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

A few places in the code open a file "manually" rather than using our
wrapper function, because they need an array with the lines. Combining
the result of utils.ReadFile with splitlines() we get rid of the
exceptions.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This way the same constant can represent the default profile also for
nic, disk and OS parameters.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The instance objects did not get a serial_no field. This patch adds a
new constants for the field name and uses it for all three cases
(cluster, nodes, instances).

Reviewed-by: imsnah

This patch makes the cfgupgrade script to handle:
  - instance changes
  - disk changes
  - further cluster fixes
  - adds configuration checks at the end, in non-dry-run mode

Reviewed-by: ultrotter

This patch makes cfgupgrade work on empty cluster (i.e. no instances),
up to a point that the config file can be converted from 1.2 to 2.0.
This is not yet complete, though.

Reviewed-by: ultrotter