While diagnosing some (unrelated) memory usage in htools, I've
stumbled upon some very bad behaviour in checkData: mapAccum is
non-strict, and the tuple we use also, so that results in the list of
list of messages being very bad space-wise (hundreds of MB of memory
for a simulated cluster with thousands of nodes, all with errors).

The new, explicit reuse of the old message list has a linear memory
behaviour. The only downside is that messages are listed in the
reverse order (which I'll fix on master).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch changes an internal assert (which can only be triggered
when a node group is empty) into properly handling this case (and
returning empty node/instance lists).

While we could handle this in the backend (Cluster.splitNodeGroup)
this would actually mean than we change the behaviour for a cluster
with just two node groups, once of which is empty (where today we
don't require a node group argument).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This just adds the primary node of the instance as 'non-allocable'
during the choosing of the new secondary.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
(cherry picked from commit 7073b3a8)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

If we select the primary as new secondary, better to fail than return
wrong data to Ganeti.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
(cherry picked from commit f25508be)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

According to the iallocator documentation the “node-evacuate” call needs
to return a list of jobs, not a list of lists of jobs.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

As discussed offline, the new node-change mode could be used for
evacuation, but it's not directly useful as it returns a list of
opcodes; therefore, we need to partially revert commits fbe5fcf6 and
5b53ca79 that removed it (and multi-evacuate, which remains removed).

The new version of relocate is actually just a wrapper over the
tryNodeEvac (which does the node evacuate); we run that and then we do
some extra checks that the nodes we got from that function are
consistent with the instance's new state.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

I think I've identified the problem with the current ChangeAll
mode. The current algorithm works as follows:

- identify a new primary by choosing the node which gives best score
  as new secondary
- failover to it
- identify a new secondary by choosing the node which gives best score
  as new secondary

This means that the future primary is 'fixed' after the first
iteration, leaving to possibly suboptimal results. This patch changes
the algorithm to do what, in hindsight, seems the obvious thing to do:
- generate all pairs (primary, secondary)
- identify the pair that after the above sequence (r:np, f, r:ns)
  gives the best group score

This fixes some of the corner cases I've seen in relocation, but not
all; the remaining cases are related to multi-instance relocation and
while they can't be fixed in the current framework, the needed
rebalancing is much smaller than with the current algorithm.

The patch also fixes an issue with the docstring of another function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

These two cases use explicit uses of primary and secondary nodes with
Instance.allNodes, which means the code is more flexible if the
internal layout of the instance changes.

I've verified that the output of involvedNodes  is not required to be
4-element long, and as such the function docstring has been updated.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

… and failover too. Not many changes otherwise except for
serialisation and unittests.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

These will be used in Node.hs for proper add/remove instance
code. Furthermore, we restrict the movable status to the right disk
templates only, so that we don't attempt to move the 'wrong' instance
types.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This adds tests for the opToResult and eitherToResult functions from
Types.hs, and changes two other tests for the same module to test JSON
serialisation (which automatically also tests the lower-level to/from
string conversion functions).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Tested only on GHC 7.x, will test on 6.1x too before commit.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This adds parameter documentation for Cluster.iMoveToJob (I think it
was not clear if the new or old node list is needed) and fixes other
docstring style issues.

After this patch, all modules except for CLI.hs (which has many
obvious declarations for command-line options) and QC.hs (unittests)
have 100% doc-strings.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This abstracts the JSON parsing of the type EvacMode near its
definition, and simplifies its conversion in IAlloc.parseData.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, hspace can only output a machine-readable format that
(while detailed) is hard to parse quickly by people. This patch adds
(and enables by default) a human-readable output that shows the most
important metrics in a simple format.

Most of the work of the patch is in moving the display of various
metrics from the 'main' function to separate functions, each of which
can output either a machine or human intended format.

The patch also corrects a bug in the CPU efficiency display: before,
the efficiency was computed as instance virtual CPUs divided by total
physical CPUs, which is almost always supra-unitary. More correct is
to divide by the total virtual CPUs, which shows a more meaningful
number (when the p-to-v CPU ratio has been defined correctly).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit 56c094b4 added use of job constants, but I didn't pay
attention and ended up mixing things: job constants were used for
opcode ones, and the job ones didn't get converted.

This patch corrects it and uses only C.* constants throughout the Jobs
module.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch renames the {JOB,OP}_STATUS_WAITLOCK constants to
{JOB,OP}_STATUS_WAITING, as per design document for chained jobs.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

hspace and hbal treat -O differently, and use aliases for short names
(although hbal succeeds in that, and hspace doesn't). Uniform this with
a name lookup, using the same functions we used for instance
selection/exclusion.

Some of the code is by the way a bit repetitive, and could probably be
merged in a single function. That needs to be a monadic one, though, so
I promise to do it as soon as I realize how to write them! ;)

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be used in hspace to toggle between "human" readable
and machine readable output formats.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is used just in hspace, so let's help in making Cluster.hs
smaller. We also split the function in two, as computing the spec map
and formatting it are two different tasks.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This adds the binaries code to the coverage, and thus the coverage
finally shows the real coverage over all logic code (except for the
htools.hs code, which is not logic code related to the algorithms, so
it doesn't matter — plus it's also very small).

Next steps will be to actually add coverage for this code, especially
for hbal and hspace, which are relatively big compared to hail and
hscan (around 800 expressions versus 200-300 expressions).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is the last patch of the binaries conversion.

As information, we now have a single binary that is approx. 5.4MiB in
size, compared to 4 binaries that were approx. 5.1-5.2MiB in size;
this will result in a smaller package and install size, and the single
compilation phase should also help.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In addition, the patch adds a separate Makefile variable for holding
the binary roles to make it more clear what we symlink.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This converts the first binary to the generic 'htools' binary.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is the start of a series of patches that will unify all the
binaries currently in use in a single one, which can perform different
roles based on the name it is installed as.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

When compiling with the parallel-3.x library, we get a deprecation
warning, which makes understanding any other error messages
harder. This patch adds a compatibility module that will hold such
code for transitioning libraries.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

… which was deprecated by the previous patch.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Since the new node-evacuate mode does both their work and also
supports better multi-group clusters (including handling split
instances).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the ChangeAll mode of nodeEvac computes the primary group
of the instance and then uses the resulting group index for computing
the group score. However, during the change-group operation (which
also uses ChangeAll), the group of the instance at the beginning and
end of the operation differs, so we can't do that and we must instead
pass the final group to the function.

This patch does that, by passing the group from the caller of
nodeEvac.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

As the IAllocator backend is using a different data path than the
others, it doesn't get the full functionality that loadExternalData
does. This results in the current situation where checkData is not run
on the input cluster state, which means the node memory properties are
not correctly set, leading to no Xmem being set correctly.

This patch fixes this bug by adding a manual call to checkData.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This will make it possible to reuse this in IAllocator too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This eliminates duplication of codes (and was forgotten back when
maybePrintNodes was added).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This add support for saving the cluster state (both pre- and
post-iallocator run) to a text file such that it can be fed back into
any of the htools commands.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Similar to the previous patch, this returns the final instance map
from the iallocator run, which will allow saving the cluster state for
further examination/post-processing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch changes the IAllocator result formatting workflow to return
the final node list, which can be then used to display the final node
status too—currently only the initial status can be shown, which is
only half useful.

Note that as the FIXME in the code says, doing this right for the
evacuate mode is hard; however, as that mode is deprecated, we can
live it for the moment.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The old modes already return the node list (as part of AllocSolution),
this patch makes the new modes provide this new information.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

… per the new design changes. This also requires changes to
updateEvacSolution, as passing in the original instance is not
helpful; we just pass the instance index now and look up the actual
instance in the modified instance map.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>