PrintGroupStats and printClusterStats had very similar code
structure. This patch abstracts those two into printStats, making it
a little more generic. Also, printStats has its arguments documented
as requested in previous patch series.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This replaces the 'if .. then .. else ..' with a large block of code in
'else' branch by separating this code into a separate function.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Reordered functions and shortened one function name.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As suggested by Rene, we should keep the check separate from printing.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As discussed in cover letter of patch series introducing hcheck, in
machine readable version we should always print values of all variables.
When hbal simulation was not run, FINAL values are equal to INIT values.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As per Iustin Pop's suggestion, we are printing group mappings
from group idx (known only to htools) to group uuid, which identifies
the group uniquely. This is required because group uuid and name
may contain characters such as '-' which are not valid parts of
bash variable name.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Instead of printing group id in human readable version of hcheck,
we now print group name.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As suggested during offline discussion, this patch adds a new bash
variable and description field in hcheck, which informs if the
rebalance was possible. This is to determine if there are split
instances on the cluster, in which case rebalance is impossible.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As suggested by Rene, I added comments to types for printKeys
and printFinal. Also, realigned some lines in Hspace.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Machine readable output for hcheck.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Full hcheck functionality is implemented in this patch. Hcheck runs
per-group checks, provides summary for the entire cluster and simulates
rebalance if required, the printing the modified metrics.
In this patch, only the human-readable output is available.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Part of hcheck's simulation required us to silently run hbal. For that
purpose we expose iterateDepth and allow it to run silently.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As hspace and hcheck both use machine readable options, they require
similar functions - which are now generalized and moved to CLI and
Utils.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Introduce infrastructure required to add Hcheck and build it
successfuly.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Thanks to Agata Murawska, a random seed that generates bad test cases
was found ("seed was 1768143307 2118231514, test size 74"). This is
due to the fact that some nodes were generated with tMem=0, fMem=0,
which means that pMem was NaN and thus the node was not comparing
equal to itself.

Worked around by limiting the test case generation to nodes with
positive values; a better fix would be to compare nodes differently
(subset of properties or ignore NaN values).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch fixes a (brown-bag) bug in the ssconf implementation where
the looked-for filenames were not prefixed with the ssconf file
prefix.

Additionally, a test for this is added (which fails without the fix).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds support for handling SIGHUP and reopening the
stdout/stderr logfile in daemon mode (but not in foreground mode).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This fixes a missing functionality: closing of stdout/stderr when
forking daemons. Without this, starting the daemons from the command
line seems to work, but doing it from utils.RunCmd breaks, since
without closing the standard descriptors, RunCmd never finishes.

The code is not very nice; basically what we need is an actual type
for the LogMode desired (LogConsole, LogStderr, LogFile FilePath), but
the syslog (yes/no and *only*) complicates this even more. I'll think
more on this and probably fix it in master.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch adds a hand-coded decoder for LuxiCall arguments, as the
data-structure is not uniform enough for automated generation (even
for the serialisation, we had to add hints for some fields,
de-serialisation is even harder).

It also fixes a tiny issue with WaitForJobChange job ID encoding, and
adds unittests for the encoding/decoding of LuxiOp structures.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

In preparation for unit-testing the Luxi definitions, this improves
the auto-generation of OpCodes: currently this uses unbounded plain
Strings for names and fields, and this creates too big test sizes
(e.g. when automatically generating job sets).

Furthermore, there are some more cleanups: switching from liftM to
applicative style for easier definition, and some improvements in
generators.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

We currently auto-generate a LuxiOp data type, which holds the entire
operation (including parameters). However, having a data type just for
the method call would be useful, so let's change THH to also
defineSADT for the Luxi constructors.

Currently I don't know how to match automatically a LuxiReq to its
LuxiOp counterpart (not even sure we need that), so any matching will
remain manual.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Now that luxi.py constants are exported, we can use them for more
consistency.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

And its associated unittests.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Commit 30d25dd8 moved the htools code to new-style exception handling,
but the hconfd code hasn't been, which fails when compiling on newer
GHC versions.

This patch does the rest of the move; however, the situation is not
very nice, we should have a better way to handle this (introduce
catchIO? or something like that).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, we're using Prelude.catch to handle I/O errors in
htools. This style of error handling has been deprecated for a while,
but it still used to work without warnings.

However, the GHC release 7.4 has started to emit deprecation warnings
for it, so we change to the Control.Exception module; the code is a
bit less clean since we only care about I/O errors (but
Control.Exception deals with other error types too), so we have to
filter the exceptions.

Note that the new style exception handling is not really "new"; it has
existed since at least GHC 6.12, which is our oldest supported
compiler.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is the last warning related to TemplateHaskell that was 6.12
specific; for some reason, it doesn't "see" that traw/tname were used.

The patch just replaces the quoting syntax with an explicit type
declaration; while less readable, it also has less warnings.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

GHC 7.4 has updated the TemplateHaskell library, and it turns out that
the way we built the JSON instance implementation for showJSON was not
good (probably this is why GHC 6.12 was generating some warnings).

The patch changes the build of showJSON to be the same as readJSON,
which was working fine. As a bonus, this fixes both the 7.4 issue and
the 6.12 one.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

While testing some other stuff, I realised that the gnt-* commands
could be broken (as in, the script fails with syntax errors), but make
check doesn't detect it. Since we have shelltest, we can now add
trivial tests for this case.

One downside is that starting the scripts seems to be much slower
than the htools binaries, so we can't add as many tests.

The other downside is that shelltest is now required for all
development work, but I think this is a small disadvantage compared to
the increased testing possibilities.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

While updating the confd code, I realised that we have _lots_ of
duplication in the exit model for the various programs.

So this patch attempts to abstract all the exits via a couple of new
functions; sorry for the somewhat big patch, but I hope the payoff is
worth the change: the actual exit conditions are much clearer.

Note that the patch (also) moves the exitIfBad function to Utils.hs,
since that is more logical.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The new shell tests do not succeed out-of-tree, due to static paths
and other issues. This trivial patch fixes these issue, make distcheck
now passes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is more of a RFC… Basically most of the shell-based tests are
converted from exec+grep to shelltestrunner.

Things are not all fine and nice though:

- we have dependencies between tests, as some generate some data files
  needed later; this is not nice, and we depend on serial execution in
  testrunner
- we can still fail with no so nice messages in the offline-test
  script (when we generate most of the data)

But overall, I think the tests are much nicer to
define/read/debug:

- each test is standalone, with the only dependency being an optional
  input data file; this is much better than a single monolithic shell
  script
- in case of failures, the failure is clearly shown by shell test,
  both for exit code and stdout/stderr
- shelltest can run in --debug mode, where the exact details are shown
  much better than the alternative of "set -x" for the shell script

Comments welcome!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

… and enable it in hconfd.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Currently this is initialised to no from Daemon.hs, but will in the
future allow command-line options for controlling it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch makes hail treat '-' as denoting stdin, per the usual Unix
convention. This will help with testing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Instead of hardcoded IPv4 INADDR_ANY, this patch changes hconfd to use
either the any network for the configured cluster address family
(ipv4/ipv6), or whatever the user passes in via the --bind option.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This implements the same logic as the Python code: if the option is
not used, use the default appropriate for the cluster, otherwise try
to parse and use whatever was passed in.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

These both are work with/on the Result type, so we add them to
BasicTypes. The functions will be used as more generic versions of
some more specialised functions that are right now spread across the
modules.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This currently has only one export function in it, which will be used
for future bind address functionality in daemons.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a stupid/trivial patch. I didn't realise when adding
htools-hpc that this will "break" tab completion on htools; as in, I
will have to add / manually always when completing files on the
command line.

So to fix my laziness, let's rename htools-hpc to hpc-htools,
eliminating this completion conflict.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

The dashes do not need to be in, if they are then the resulting option
is:

  -p PORT  ----port=PORT     Network port (default: 1814)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>