This will be used to hold all the disparate uses of the cluster data:
we have either tuples with these four elements, or functions taking
these four arguments, etc.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This patch moves the allocation policy from hardcoded to be read from
the given specification, and extends the error message for invalid
specifications.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This patch changes the behaviour of the --simulation option to be an
incremental option, where each new use defines a new node group. This
allows simulation of more complex clusters.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

* stable-0.2:
  Move man files to man/ subdirectory

Conflicts (all removed):
        man/hail.1
        man/hbal.1
        man/hscan.1
        man/hspace.1

This is just change on the 0.2 branch to synchronize with the master
branch. It allows automated builds to work better across the two
versions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

* devel-0.2:
  Update NEWS file for 0.2.8 release
  hbal: return meaningful exit code for job failures
  Change the balancing function

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

Currently, LUXI job failures only display a warning message, while
still returning a success exit code. We change hbal to return
true/false from within execJobSet/runJobSet, and add a wrapper for
simpler code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

Currently the balancing function is a modified version of the standard
deviation (stddev divided by list length), due to historical reasons.

While this works fine for small clusters, for big clusters it makes
the balancing effect too "weak", and in some cases it refuses to
balance correctly some clusters. It also makes the balancing behaviour
dependant on the cluster size, which is a big no-no.

Therefore we revert to the normal version of standard deviation, and
we also rename the function to reflect what it does. The new version
correctly balances some corner cases that the previous version didn't,
and passes the current balancing unittests.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

This splits out a bit of code from hspace.hs and moves it into its own
function in Cluster.hs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This patch changes the allocate mode to respect the alloc_policy for
groups. It does this by changing the sort key from simply the solution
score, to a tuple with two elements: the alloc policy (which is now an
Ord instance) and the solution score. Also, the unallocable groups are
filtered out in the filterMGResults phase.

The patch also slightly enhances the informational message by
including the policy in the group information, to help debugging.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Currently, it's not easy to generate “fake” IAllocator request files
for hail. As such, testing on simulated clusters is hard to do.

To workaround this, we change hail to also take the ‘-t’ and
‘--simulate’ options, so that we can override the cluster data read
from the request. Note that this will not change the request itself
(so for example an evacuation will need to make sure uses the correct
node names), but it's a step forward in testing hail. The other tools
already can use text files which allow for better flexibility.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This will allow reading this attribute via the Rapi/Luxi backends.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Since currently hbal can only work on single groups at a time, we need
to be able to specify the target group when running the live test.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This is the complement to the reading part. Now the live-test works
correctly against clusters with configured exclusion tags.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This means that a file with the correct information is as accurate as
the other backends (Luxi, Rapi). Serialization of tags is in the next
patch.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

The new sepSplit function can split based on empty lines, so we remove
the hackish text splitting from before and simply use sepSplit. This
is needed as the addition of extra sections would have increased the
code linearly, which we don't want :)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Currently it works on splitting strings by individual chars, but we
can generalise it to split lists by list elements, which means we can
reuse it later in the Text module for splitting both lists of chars by
'|' or lists of lines by empty newlines. The change also makes the
code cleaner (uses “null xs” instead of string-specific “xs == ""”).

Note: I tried to rewrite this in a nicer, functional style using
unfolds, but I failed to account for the final terminator case
(e.g. ab|cd|) resulting in a valid but empty element.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This patch switches from the group index to the group name for the
informational messages in the hail results.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This should have been in the previous patches, but sent separate for
clarity.

The live-test script is updated to read the first node from the
cluster, now that the text files don't start anymore with the node
data.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Currently, the Node.group attribute is the UUID of the group, as until
recently Ganeti didn't export the node group properties. Since it does
so now, we make the following changes (again apologies for a big
patch):

- we change the group attribute to be an index, similar to the way an
  Instance.pnode and snode attributes point to the parent node(s)
- on load, we read the group.uuid attribute and we use that to lookup
  the actual group index, from previously-loaded groups info
- this means that we now first read groups, then read nodes using the
  group info, and then read instances using the node info

This patch leaves a few functions showing the group index (ugly since
it's htools internal), will be converted later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This (invasive) patch changes all the loader pipelines to read the node
groups data from the cluster, via the various backends. It is invasive
as it needs coordinated changes across all the loaders.

Note that the new group data is not used, just returned.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This will be used in the various backends similar to the lookupNode
function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This is not yet used by the rest of the code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This patch fixes two issues related to coloured sources generation.
First, recent hscolour has changed the css file (and we need to update
it), but it also can output it at runtime, so there's no need to store
it anymore in the source tree.

Second, the current source generation predates the addition of sources
in Ganeti/ (as opposed to just Ganeti/HTools), and thus we were missing
the sources in that directory. We replace the target file name to
account for different base directories.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

When doing a two-node allocation on a cluster/group in which only one
node is online, or a one-node allocation without any online nodes, we
didn't show a valid error mesage. The patch changes tryAlloc to "fail
hard" in this case, to make the failure explicit.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This is a bit hackish. We add a new function that takes the input data,
splits it into groups, runs the original tryAlloc for each group, and
then chooses the best solution, but adds the log messages from all the
groups, as to give better debugging information. In hail, we just point
to this new function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Since the solutions are "self-annotated", we can remove the custom code
from hail, and just keep a very small processResults function.

After this change, allocation/failure shows the new detailed
information.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

And also a couple of functions for describing a given solution; these
will be used in the future instead of the ones currently in hail.

The patch also enhances the description of failure messages.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Tuples are good for two, three, at most four elements. Beyond that, the
continuous pattern matching and construction/deconstruction becomes
tedious.

Since in the future we'll probably keep more information in the
AllocSolution type, we change it now from a triple to a "real" data
type. We also do some cleanups: adding a real emptyAlloc value, instead
of the previous hardcoded ones, and add some more comments on how we do
the multi-evacuation.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Since we added the score to AllocElement, we don't need to wrap
AllocElement in yet another tuple, just to attach the cluster score. So
we simplify the AllocSolution type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

AllocElement, a type used as a result of allocations, holds the status
of the nodes after the allocation. In most cases, we'll compare this
allocation result with others, to see which allocation decision makes
the most sense. This comparison is done via the cluster score.

However, if we later need to redo this computation, as part of other
comparisons, we'd need to evaluate it again, etc. So it's easier to just
compute the score at the place where we compute the node list in the
initial step.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Actually, this just moves the functions from the QC module to Types, and
removes a duplicate entry from Cluster.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This improves on the previous change. Currently, the node and instance
lists shipped around during data loading are (again) association lists.
For instances it's not a big issue, but the node list is rewritten
continuously while we assign instances to nodes, and that is very slow.
The code was originally written for small (10-20 node) clusters, and
today with multinodes… :)

Rewriting to use Node.List/Instance.List makes a bit of a messy patch,
but it allows to remove some custom functions for assoc. list
processing, and also some custom unittests.

At the end, the load time is almost halved, and we spend time now just
in the string parsing code (which is, as we know, slow…).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>