Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This allows a cluster structure to be easily serialized via "read";
together with the already existing instances of Show, this gives a
poor man's serialization/deserialization implementation.

The patch also exports the compDetailedCV function from Cluster.hs, so
that it can be used by other modules too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Currently the balancing function is a modified version of the standard
deviation (stddev divided by list length), due to historical reasons.

While this works fine for small clusters, for big clusters it makes
the balancing effect too "weak", and in some cases it refuses to
balance correctly some clusters. It also makes the balancing behaviour
dependant on the cluster size, which is a big no-no.

Therefore we revert to the normal version of standard deviation, and
we also rename the function to reflect what it does. The new version
correctly balances some corner cases that the previous version didn't,
and passes the current balancing unittests.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

This splits out a bit of code from hspace.hs and moves it into its own
function in Cluster.hs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This patch changes the allocate mode to respect the alloc_policy for
groups. It does this by changing the sort key from simply the solution
score, to a tuple with two elements: the alloc policy (which is now an
Ord instance) and the solution score. Also, the unallocable groups are
filtered out in the filterMGResults phase.

The patch also slightly enhances the informational message by
including the policy in the group information, to help debugging.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This patch switches from the group index to the group name for the
informational messages in the hail results.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Currently, the Node.group attribute is the UUID of the group, as until
recently Ganeti didn't export the node group properties. Since it does
so now, we make the following changes (again apologies for a big
patch):

- we change the group attribute to be an index, similar to the way an
  Instance.pnode and snode attributes point to the parent node(s)
- on load, we read the group.uuid attribute and we use that to lookup
  the actual group index, from previously-loaded groups info
- this means that we now first read groups, then read nodes using the
  group info, and then read instances using the node info

This patch leaves a few functions showing the group index (ugly since
it's htools internal), will be converted later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

When doing a two-node allocation on a cluster/group in which only one
node is online, or a one-node allocation without any online nodes, we
didn't show a valid error mesage. The patch changes tryAlloc to "fail
hard" in this case, to make the failure explicit.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This is a bit hackish. We add a new function that takes the input data,
splits it into groups, runs the original tryAlloc for each group, and
then chooses the best solution, but adds the log messages from all the
groups, as to give better debugging information. In hail, we just point
to this new function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

And also a couple of functions for describing a given solution; these
will be used in the future instead of the ones currently in hail.

The patch also enhances the description of failure messages.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Tuples are good for two, three, at most four elements. Beyond that, the
continuous pattern matching and construction/deconstruction becomes
tedious.

Since in the future we'll probably keep more information in the
AllocSolution type, we change it now from a triple to a "real" data
type. We also do some cleanups: adding a real emptyAlloc value, instead
of the previous hardcoded ones, and add some more comments on how we do
the multi-evacuation.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Since we added the score to AllocElement, we don't need to wrap
AllocElement in yet another tuple, just to attach the cluster score. So
we simplify the AllocSolution type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

AllocElement, a type used as a result of allocations, holds the status
of the nodes after the allocation. In most cases, we'll compare this
allocation result with others, to see which allocation decision makes
the most sense. This comparison is done via the cluster score.

However, if we later need to redo this computation, as part of other
comparisons, we'd need to evaluate it again, etc. So it's easier to just
compute the score at the place where we compute the node list in the
initial step.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Actually, this just moves the functions from the QC module to Types, and
removes a duplicate entry from Cluster.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This splits a top-level cluster information into the component node
groups. Instance go to the group of their primary node, but otherwise we
don't disallow split instances.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

For now, we don't support instances allocated across two groups, and we
will reject such clusters. The isClusterConsistent function will return
a list of inconsistent instances, potentially allowing operation without
touch them (but only the rest).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Currently, the tag exclusion metric has a weight of one, which means
there might be cases where we won't move instances around because it
upsets the cluster metrics. However, we do want to make a higher effort
for cleaning up tag collisions, so we increase the weight to an
empirically-determined value of 2.

Also adds them in hbal.

The Cluster.iterateAlloc and tieredAlloc functions are changed to also
return the updated instance list, since it is needed to have a “full”
cluster view.

Currently we show the instance index, but this makes no sense outside
the current running program. Instead, we show the instance name.

Currently, this metric tracks the nodes failing the N+1 check. While
this helps (in some cases) to evacuate such nodes, it's not a good
metric since rarely it will change during a step (only at the last
instance moving away). Therefore we replace it with the count of
instances living on such nodes, which is much better because:
- moving an instance away while the node is still N+1 failing will still
  reflect in the score as an optimization
- moving the last instance causing an N+1 failure will result in a heavy
  decrease of this score, thus giving the right bonus to clear this
  status

Currently all metrics have the same weight (we just sum them together).
However, for the hard constraints (N+1 failures, offline nodes, etc.)
we should handle the metrics differently based on their meaning. For
example, an instance living on a primary offline node is worse than an
instance having its secondary node offline, which in turn is worse than
an instance having its secondary node failing N+1.

To express this case in our code, we introduce a table of weights for
the metrics, with which we can influence their relative importance.

This patch switches the applyMove function to the extended versions of
Node.addPri and addSec, and passes the override flag based on the state
of the node that we're moving away from.

This was a regression from the name handling changes, as we started
using the original names for the solution list (which is not designed
for parsing/feeding back into ganeti).

printSolution is no longer used, as we print the solution iteratively
now.

When the field list is prefixed with a plus sign, this will extend the
default field list, instead of replacing it entirely.

The patch adds some simple unit-tests for both the allocation function
(we can allocate small instances on an empty cluster, we can allocate in
tiered more starting from any size) and the balancing functions (one
single instance is placed optimally, a full cluster plus an empty node
can be rebalanced). The coverage has increased greatly, since this is
the bulk of the algorithm/code.

Also, the cluster tests are now being run with different options, since
they are much slower.

This is done so we can test a longer pipeline.

This helps debugging via ghci.

Currently the name used internally is modified and holds the shortened
name of the nodes/instances. This has caused issues before, since we
always have to strip the suffix from input data and reapply it if we
need to send data back to Ganeti.

This patch changes the code such that the names are never modified, only
the alias, and all the internal computations can forget about the common
suffix addition/removal.

This patch moves from allowing no-limits for disk/cpu ratios, and always
use a real limit. For disk, it's simple since we use 0, which means no
reservations for disks. For CPU, we set an (arbitrary) limit of 64 v/p,
which should be reasonable as a default limit (it can be changed via the
command line).

We returned the KM_POOL_* metrics as the final state, not as the delta
between the final and the initial state.

Given two cluster states, the new function can answer the following
questions:

- how much resources currently allocated
- how much resources finally allocated (delta from above is how much we
  can actually allocate on the cluster)
- unallocable resources (whatever is left free after the previous step)

We add a new field that tracks the available virtual cpus (expressed as
node cpus times the vcpu ratio).

Instead of deciding based on secondary node, use the new flag.

This will be used by the node evacuate IAllocator request type.

Signed-off-by: Iustin Pop <iustin@google.com>

We'll need AllocElement in both Cluster and IAlloc in the future, so we
move it to Node.hs which is imported by both.

Signed-off-by: Iustin Pop <iustin@google.com>

In preparation for multiple responses, we change from Maybe to List
(both used in the container sense).

This allows us to keep the same workflow for all kind of requests.

Signed-off-by: Iustin Pop <iustin@google.com>

This mode restricts the list of instances to be moved to the instances
living on the offline (and drained) nodes.

Signed-off-by: Iustin Pop <iustin@google.com>