Currently, this backend just gives:

  Error: failed to load data. Details:
  Node 'node2': Cannot convert value JSNull, error: Unable to read Double

That is because for Luxi we get an array back from the query, as
opposed to RAPI where we get a dict (and thus have the key at
hand). This patch moves the hardcoded 'convert' functions to a
genericConvert, that also takes the attribute name, with the result
that the error message is now:

  Error: failed to load data. Details:
  Node 'node2', attribute 'mtotal': Cannot convert value 'JSNull', error: Unable to read Double

which is better for debugging.

The patch also adds quotes around the value in the Utils.fromJVal
function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

Also fixes a small typo.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

The file apidoc/Ganeti/HTools/hscolour.css was a broken symlink in the
tarball. This was happening because `tar --transform` was being allowed to
act in symlink targets.

Pass the flag "S" to tar --transform, to keep symlink targets intact (and
hence valid).

Signed-off-by: Adeodato Simo <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit 6c7448bb changed the syntax for the --simulation option, but
didn't update live-test for it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

This patch changes genAllocNodes so that it also takes the group list,
using it to filter out any nodes which live on unallocable groups.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

The backend didn't properly set the group index, leaving to a
"corrupted" group list.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of re-generating the allocation nodes at each step, we only do
it once before calling the allocation routines, and then reuse
them. While this provides just a small speedup (~3%), it's worth doing
as it also reduces the GC cost.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the node pairs or singles that we generate are actually
node objects, as node objects do change over time during recursive
allocations. However, if we switch to node indices, which are stable,
they can be precomputed once and then reused during recursive
allocations.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, tryAlloc generates the list of allocation nodes or node
pairs (depending on how many nodes are requested) internally. This
patch moves the generation into a new function genAllocNodes, for both
code cleanup and future performance improvements.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a cheap way to make capacity calculation work well with
multi-group clusters.

There are two alternatives in implementing this:

- we can split the cluster into groups, run individual group
  allocation, and then try to recombine the groups; but this doesn't
  represent the allocation as it would happen in real life, when
  groups are not filled in succession, and would complicate TSPECS
  ordering
- or add this "hack" which does the allocation look at all node pairs
  (per group) at each step, which makes it correct from the point of
  view of ordering of allocations w.r.t. groups

For now, we take the second choice, with the future possibility of
revisiting the first one.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Just a bit of small cleanup, since we might want to use more functions
with this signature in the future.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Similar to the IAllocator change, this patch reads and uses the
vm_capable flag in Rapi. Furthermore, it changes the group UUID
reading to the same maybeFromObj infrastructure.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This allows non-vm_capable nodes, which don't export runtime data, to
not break the IAllocator message parsing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This small patch actually reads the allocation policy from the
IAllocator message.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Currently, we list the entire object in error messages. But for large
objects (e.g. an IAllocator message), this makes the output
unreadable, as the elements are containers themselves.

To simplify the reporting, we only list the keys, as this is more
relevent than the values.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Currently, fromObj/maybeFromObj take first the key, and then the
object. This is suboptimal, as this form is not easy to use with
partial functional application.

To make it easier to switch between tryFromObj, fromObj and
maybeFromObj, we unify the latter two functions to the same order
(object and then key) like the first one. The code churn in the other
modules is due to this, but the main change in this patch is in
Utils.hs.

Furthermore, since we change anyway the calls, we do replace fromObj
with tryFromObj in IAllocator.hs so that we get better error messages.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This patch does the necessary changes to make the new tryMGEvac work
correctly: each instance remains inside its primary node's group when
it is evacuated.

This is done by splitting up the to-be-evacuated instance list per
group, together with the group node and instance list, and then
running the old tryEvac for each group.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This makes the code more readable, which will help with the
multi-group evacuation.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Currently, the Rapi.getUrl function returns the body without
evaluating it, and the other functions (loadData, parseData) do the
same. In effect, the top-level structure returned from loadData can be
a thunk which depends on the curl operation, thus keeping the curl
resources alive even after return from the Rapi module.

While this is not a problem in single-threaded programs, it can become
an issue with the threaded runtime. So it's better to fully evaluate
the HTTP body before returning from the getUrl function, so that at
least the curl resources are released.

Either "return $! body" or "(code, !body) <- …" works (i.e. avoids the
segfaults in my tests), but I think it's cleaner to force full
evaluation of the body when returning from the curlGetString
function. For this, we also enable the BangPatterns language
extension.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

A small cleanup, this just moves the options to a separate list to
avoid instantiation at every call.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

There is a long-standing bug in hscan, due to the serialization
workflow. Basically, hscan loads data from the remote cluster(s),
processes it via Loader.mergeData and Loader.checkData, then
serializes it. The Loader function will update the node memory values
as if the down instances were running, and then hscan saves the
_modified_ values.

When another program (e.g. hbal) loads the text files, it will
re-apply the Loader functions, in effect subtracting the instance
memory twice, which is course offers a wrong cluster view.

To fix this bug, we change a bit the internal workflow in hscan, so
that we serialize the unmodified cluster data, and only use the
modified one for the display of cluster state (and verification that
yes, we can load the data).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This is similar to iMem.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This exports the iMem function as a standalone function, instead of
being hardcoded in showField.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This patch adds a new top-level relocation function that restricts the
relocation to the instance's group, and switches hail to it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Container.fromAssocList is just a re-export of IntMap.fromList; it
makes sense to remove it and simply export the original name, as it
needs just a bit of renaming in the rest of the code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This small patch changes the balancing computation to work in
parallel, if possible.

While the normal linking is against the single-threaded runtime, if
the code is linked against the multi-threaded one, the balancing will
get a significant speedup (80% efficiency at 4 cores, 60% at 12 cores,
and with GC tweaks it can reach 70%+).

On the single-threaded runtime, due to the fact that we only use the
weak head normal form, it doesn't introduce any extra penalties,
neither in space nor in CPU time (or if there are, they are too small
to detect easily).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Currently, the allocation routines (iterateAlloc and tieredAlloc)
return only the final state of the cluster and the list of allocated
instances. For better visibility in how the cluster resources change,
we compute and return a list of cluster resources at each step, which
should be a good indicator of the cluster state. The cost of the
computation for the new resource when not used is undetectable (< 1%).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This is a cheap way to get back compatibility with Ganeti 2.3 (and
lower) in the RAPI backend. It is however not very safe (the /2/groups
resource could fail due to other reasons), so it is added only
temporarily.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This allows a cluster structure to be easily serialized via "read";
together with the already existing instances of Show, this gives a
poor man's serialization/deserialization implementation.

The patch also exports the compDetailedCV function from Cluster.hs, so
that it can be used by other modules too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Just another cleanup of duplicate code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This again abstracts a bit the instance listing. Due to the fact that
I don't want to import Cluster.hs in CLI.hs, we pass the already
generated output. It also moves the instance display to stderr.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Since this bit of code (including the “when (isJust …)” is used in
multiple places, let's abstract it in a function that is used
consistently. One (bad?) side-effect is that all node lists are done
to stderr, including the ones from hbal where it was previously done
to stdout (is that fine?).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This functionality was replicated in multiple places (hbal & hspace),
so we abstract it for better clarity.

Additionally, in hbal we now save the state both before and after
balancing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

We change processData to just load the data, and return a simpler
type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This patch converts the backends and mergeData to the new ClusterData
type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>