The function `allocateOnSingle' has a bug in the calculation of the
cluster score used for deciding which of the many target nodes to use
in placing the instance: it uses the original node list for the score
calculation.

Due to this, since the original node list is the same for all target
nodes, it means that basically `allocateOnSingle' returns the same
score, no matter the target node, and hence the choosing of the node
is arbitrary, instead of being done on the basis of the algorithm.

This has gone uncaught until reported because the unittests only test
1 allocation at a time on an empty cluster, and do not check the
consistency of the score. I'll send separate patches on the master
branch for adding more checks to prevent this in the future.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

It seems that 'explicitely' is wrong, and that the right form is
'explicitly'. This is just fixing the typo plus adjusting affected
paragraphs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This can be queried remotely since it's a pure configuration query.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This fixes an old FIXME. Since we now how actual DRBD configuration
data, we can finally compute the instance's secondary nodes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is a complex field, so we have to do a lot of manual work for now.

The complexity arises from the fact that the contents of the field,
and the way to parse it, depends on the disk type field, so we don't
have a single, static way of parsing it. Hence we needed the
extensions to the Template Haskell code.

Since we now can both load and save the disk type, we can remove the
in-memory (duplicate) disk type from the disk objects, relying only on
the logical ID to hold the type information.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Currently, we only allow field-by-field de-serialisation. Since we
have cases where information about how to un-serialise a field is
split across two JSON fields (e.g. disk type and disk logical_id,
hypervisor and hvparams, etc.), we need to pass the entire object to
custom read functions.

Furthermore, since we will have to generate two actual fields from the
single in-memory field, we need to extend the custom save function so
that they can generate additional fields beyond the "main" field value
they currently generate.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Instead of passing an expression (which cannot come from the current
module), we pass a name, which is allowed to reference functions from
the module we're in. Since we currently don't have custom fields, we
don't need to modify any callers.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This uses the recently-moved functions to implement partial lookup of
names on getNode and getInstance, similar to the Python codebase.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Currently, the LookupResult, MatchPriority and related functions are
locate in Loader.hs, since (so far) only hbal needs them in the
selection of instances. However, with the new functionality on confd
side, we need these functions there too, but we don't want to import
Loader.hs (which pulls in lots of balancing-related code). So we move
all these function to BasicTypes.hs, since that module is a leaf one,
with no other dependencies.

Unittests are slightly adjusted (but they are still tested under the
'Loader' group).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is not used, as we need a more complex serialisation, which is
done in the saveObjectField function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

The same types are reused a couple of times, so let's add a couple of
type aliases for easier change later and readability.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Looking at the output of hcheck, in human readable mode, it looks like
it always starts with a blank line. This is not nice, so I wanted to
redo this to start cleanly.

However, looking at the code, I realised that we need some internal
cleanup; 'perGroupChecks' was doing both pure computation (computing
the score/stats) and I/O work (printing the stats), which is ugly, so
I split that in two; this allows perGroupChecks to be simpler, and we
can therefore first compute the group stats, and start the output with
the mode in which we run.

So right now, hcheck will first output whether we need to run a
rebalance, or if no-simulation is selected, etc., and only then
display the actual stats, making the output a bit more consistent.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

While removing split instances check from hcheck, I saw a wrong use of
'length': this should only be used when one needs the actual length of
the list, and not when one wants to check whether the list is empty or
not (in that case, 'null' should be used).

To prevent this in the future, let's add a hlint hint for it, and fix
the warning it generates in hbal.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Since hbal can now run rebalances even in the presence of split
instances, we can remove this check and always run the rebalance.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This was not update when the alloc policy was added to the parameter.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

The recent patch series on hcheck made me realise how much pain we
have due to hbal not being able to work (at all) with split instances.

This is sub-optimal; ideally, hbal would degrade its behaviour, but
not completely refuse to work. This patch does this, by marking any
split instances as not movable (since such instances most likely mean
that some manual intervention is done on then), but otherwise
proceeding with the rebalance of the selected node group.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This fixes a couple of issue I've seen while fixing the List import:

- removes over-use of printf with putStr/show
- wraps some lines
- removes superfluous parentheses in constructs like 'IO (a)'
- makes sure that in type signatures, the '->' are placed at the
  beginning, not end of the line

We probably need to update the style guide for some of these…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 1213f9d6 (re)added an old-style import, which fails with recent
compilers.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

PrintGroupStats and printClusterStats had very similar code
structure. This patch abstracts those two into printStats, making it
a little more generic. Also, printStats has its arguments documented
as requested in previous patch series.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This replaces the 'if .. then .. else ..' with a large block of code in
'else' branch by separating this code into a separate function.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Reordered functions and shortened one function name.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As suggested by Rene, we should keep the check separate from printing.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As discussed in cover letter of patch series introducing hcheck, in
machine readable version we should always print values of all variables.
When hbal simulation was not run, FINAL values are equal to INIT values.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As per Iustin Pop's suggestion, we are printing group mappings
from group idx (known only to htools) to group uuid, which identifies
the group uniquely. This is required because group uuid and name
may contain characters such as '-' which are not valid parts of
bash variable name.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Instead of printing group id in human readable version of hcheck,
we now print group name.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As suggested during offline discussion, this patch adds a new bash
variable and description field in hcheck, which informs if the
rebalance was possible. This is to determine if there are split
instances on the cluster, in which case rebalance is impossible.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As suggested by Rene, I added comments to types for printKeys
and printFinal. Also, realigned some lines in Hspace.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Machine readable output for hcheck.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Full hcheck functionality is implemented in this patch. Hcheck runs
per-group checks, provides summary for the entire cluster and simulates
rebalance if required, the printing the modified metrics.
In this patch, only the human-readable output is available.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Part of hcheck's simulation required us to silently run hbal. For that
purpose we expose iterateDepth and allow it to run silently.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As hspace and hcheck both use machine readable options, they require
similar functions - which are now generalized and moved to CLI and
Utils.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Introduce infrastructure required to add Hcheck and build it
successfuly.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Thanks to Agata Murawska, a random seed that generates bad test cases
was found ("seed was 1768143307 2118231514, test size 74"). This is
due to the fact that some nodes were generated with tMem=0, fMem=0,
which means that pMem was NaN and thus the node was not comparing
equal to itself.

Worked around by limiting the test case generation to nodes with
positive values; a better fix would be to compare nodes differently
(subset of properties or ignore NaN values).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch fixes a (brown-bag) bug in the ssconf implementation where
the looked-for filenames were not prefixed with the ssconf file
prefix.

Additionally, a test for this is added (which fails without the fix).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds support for handling SIGHUP and reopening the
stdout/stderr logfile in daemon mode (but not in foreground mode).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This fixes a missing functionality: closing of stdout/stderr when
forking daemons. Without this, starting the daemons from the command
line seems to work, but doing it from utils.RunCmd breaks, since
without closing the standard descriptors, RunCmd never finishes.

The code is not very nice; basically what we need is an actual type
for the LogMode desired (LogConsole, LogStderr, LogFile FilePath), but
the syslog (yes/no and *only*) complicates this even more. I'll think
more on this and probably fix it in master.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch adds a hand-coded decoder for LuxiCall arguments, as the
data-structure is not uniform enough for automated generation (even
for the serialisation, we had to add hints for some fields,
de-serialisation is even harder).

It also fixes a tiny issue with WaitForJobChange job ID encoding, and
adds unittests for the encoding/decoding of LuxiOp structures.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

In preparation for unit-testing the Luxi definitions, this improves
the auto-generation of OpCodes: currently this uses unbounded plain
Strings for names and fields, and this creates too big test sizes
(e.g. when automatically generating job sets).

Furthermore, there are some more cleanups: switching from liftM to
applicative style for easier definition, and some improvements in
generators.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>