Currently, the ChangeAll mode of nodeEvac computes the primary group
of the instance and then uses the resulting group index for computing
the group score. However, during the change-group operation (which
also uses ChangeAll), the group of the instance at the beginning and
end of the operation differs, so we can't do that and we must instead
pass the final group to the function.

This patch does that, by passing the group from the caller of
nodeEvac.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

As the IAllocator backend is using a different data path than the
others, it doesn't get the full functionality that loadExternalData
does. This results in the current situation where checkData is not run
on the input cluster state, which means the node memory properties are
not correctly set, leading to no Xmem being set correctly.

This patch fixes this bug by adding a manual call to checkData.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This will make it possible to reuse this in IAllocator too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This eliminates duplication of codes (and was forgotten back when
maybePrintNodes was added).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This add support for saving the cluster state (both pre- and
post-iallocator run) to a text file such that it can be fed back into
any of the htools commands.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Similar to the previous patch, this returns the final instance map
from the iallocator run, which will allow saving the cluster state for
further examination/post-processing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch changes the IAllocator result formatting workflow to return
the final node list, which can be then used to display the final node
status too—currently only the initial status can be shown, which is
only half useful.

Note that as the FIXME in the code says, doing this right for the
evacuate mode is hard; however, as that mode is deprecated, we can
live it for the moment.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

While working on a function to submit jobsets, I realized that we
actually don't need them anymore. With the new job dependencies, the
iallocator plugin can just generate the right dependencies and gets the
same effect as with jobsets. This saves us from having to implement
jobsets in Ganeti core.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit 66bd7445 added an assertion to ensure a finalized job has its
“end_timestamp” attribute set. Unfortunately it didn't cover a case when
the queue is recovering from an unclean master shutdown.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be useful for assertions. GanetiLockManager._is_owned is
exported, too.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The old modes already return the node list (as part of AllocSolution),
this patch makes the new modes provide this new information.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

… per the new design changes. This also requires changes to
updateEvacSolution, as passing in the original instance is not
helpful; we just pass the instance index now and look up the actual
instance in the modified instance map.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This builds a lot upon the node-evacuation mode, and its
implementation is therefore quite easy (once node-evac was added).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

While a normal multi-group allocation doesn't need to restrict the
list of target groups beyond the unallocable status of some groups,
when we relocate instances from one group to the other we need to
limit the subset of searched groups.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently this function works in the context of an instance, computing
its local group nodes. We can use generalise it to take a group index
argument, so that it's not bound to an instance, and we rename it
accordingly.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch updates the types and functions in IAllocator.hs and
Loader.hs to the new design changes (elimination of multi-relocate
mode and replacement with change-group).

It also removes an extra re-export of EvacMode from Loader.hs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, this function does both identify the best group for an
instance, and format the results (partially). The former functionality
is useful in other contexts too, so let's split the function in two.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is much more complicated than either change primary (which is
trivial) and change secondary (which is also simple).

The main strategy is:

- choose a new secondary (that will become the final primary) and
  replace-disks to it
- failover
- choose a new secondary again (this will be the final secondary)

However, if the original primary was offline, then we first have to
failover to the secondary of the instance.

Note: currently this evacuation mode, while producing correct
evacuation results, does not result in optimal placement of the
instance; it will need attention and possibly an algorithm change.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This makes fewer temporary data structures, and thus brings a bit of
improvement in total runtime.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the tieredAlloc/iterateAlloc functions will not return
until the allocation fails; this means unit-testing their
functionality (e.g. that an instance can be allocated) is slow, since
they will allocate all possible instances.

This patch adds an optional limit that allows allocation to return
early; this makes the cluster unittests twice as fast.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In some cases, we have a [Node.Node] list, instead of a Node.List
structure. Instead of converting [Node.Node] to the latter, let's
rework compCV and add a compCVNodes such that we skip this
(potentially expensive) step.

This already allows a simplification in a unittest, and will simplify
consumers of Node.computeGroups.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The wrapper will connect to the console, and check in the background if
the instance is paused, unpausing it as necessary.

Signed-off-by: Stephen Shirley <diamond@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The wrapper will connect to the console, and check in the background if
the instance is paused, unpausing it as necessary.

Signed-off-by: Stephen Shirley <diamond@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Similar to the previous hspace fix.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

And point to the main INSTALL file for up-to-date information.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, the test prop_Node_rMem does some reserved memory checks on
nodes, and for the test to work correctly (can add more instances) we
require that the node free memory is greater than zero via a
prerequisite.

However, if the node free memory is exactly 1, the call to
setInstanceSmallerThanNode will set the instance memory to "free
memory `div` 2", and since this is integer division, the instance
memory will get set to 0. Which means that the node reserved memory
will not increase when we add this instance as secondary.

To work around, we simply require that the node memory is high enough
(we use our 'base' unit of memory).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

These move the 'not supported' message deeper into the workflow;
assuming that the workflow as added by this patch is correct, all that
remains is:

- for disk template DRBD8, plug in the currently existing mechanisms
  to implement relocation
- for other disk templates, first modify the Node function to handle
  node changes correctly for each disk template, and then add support
  for them in nodeEvacInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

For monadic operations, these function will make it easier to convert
other monads (or monad-like structure) to our 'Result' type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The new IAllocator interface is based on instances, not nodes, so we
need to backtrack and compute on which nodes we can't allocate
instances during the current operation. This patch adds a function for
computing this node set.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

'emptySolution' is fine while we have only one Solution type
(AllocSolution), but in the future we'll introduce another one and
thus we should rename it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In the future, this 'split/associated instance idx list' functionality
will be used by other code, so let's move it to its own code.

The new method of computing the split also requires exporting one more
function from IntMap in Container.hs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

These are useful for checking what instance specifications we start
with (as they are not necessarily the first entry in the TSPEC map).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch converts hspace to allow units in all disk and memory
specifications, and updates the man page accordingly. It also correct
a mistake in the man page for the --max-cpu option.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is similar, but not identical, to Utils.ParseUnit. The biggest
difference is that we don't round up/down; as we only use integral
types, the result will always be rounded down.

Moreover, since (real-world) disk sizes come in SI units, the function
differentiates between SI and binary prefixes, using lower-case for
binary and upper-case for SI (similar to lvm usage). This distinction
should be ported to the Python code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Rather than running all possible tests in one go, split them out in
three cases: "normal", undefined default value, undefined list values
(both of which require a valid response before them).

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>