Currently, we're using Prelude.catch to handle I/O errors in
htools. This style of error handling has been deprecated for a while,
but it still used to work without warnings.

However, the GHC release 7.4 has started to emit deprecation warnings
for it, so we change to the Control.Exception module; the code is a
bit less clean since we only care about I/O errors (but
Control.Exception deals with other error types too), so we have to
filter the exceptions.

Note that the new style exception handling is not really "new"; it has
existed since at least GHC 6.12, which is our oldest supported
compiler.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is the last warning related to TemplateHaskell that was 6.12
specific; for some reason, it doesn't "see" that traw/tname were used.

The patch just replaces the quoting syntax with an explicit type
declaration; while less readable, it also has less warnings.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

GHC 7.4 has updated the TemplateHaskell library, and it turns out that
the way we built the JSON instance implementation for showJSON was not
good (probably this is why GHC 6.12 was generating some warnings).

The patch changes the build of showJSON to be the same as readJSON,
which was working fine. As a bonus, this fixes both the 7.4 issue and
the 6.12 one.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

While testing some other stuff, I realised that the gnt-* commands
could be broken (as in, the script fails with syntax errors), but make
check doesn't detect it. Since we have shelltest, we can now add
trivial tests for this case.

One downside is that starting the scripts seems to be much slower
than the htools binaries, so we can't add as many tests.

The other downside is that shelltest is now required for all
development work, but I think this is a small disadvantage compared to
the increased testing possibilities.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

While updating the confd code, I realised that we have _lots_ of
duplication in the exit model for the various programs.

So this patch attempts to abstract all the exits via a couple of new
functions; sorry for the somewhat big patch, but I hope the payoff is
worth the change: the actual exit conditions are much clearer.

Note that the patch (also) moves the exitIfBad function to Utils.hs,
since that is more logical.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The new shell tests do not succeed out-of-tree, due to static paths
and other issues. This trivial patch fixes these issue, make distcheck
now passes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is more of a RFC… Basically most of the shell-based tests are
converted from exec+grep to shelltestrunner.

Things are not all fine and nice though:

- we have dependencies between tests, as some generate some data files
  needed later; this is not nice, and we depend on serial execution in
  testrunner
- we can still fail with no so nice messages in the offline-test
  script (when we generate most of the data)

But overall, I think the tests are much nicer to
define/read/debug:

- each test is standalone, with the only dependency being an optional
  input data file; this is much better than a single monolithic shell
  script
- in case of failures, the failure is clearly shown by shell test,
  both for exit code and stdout/stderr
- shelltest can run in --debug mode, where the exact details are shown
  much better than the alternative of "set -x" for the shell script

Comments welcome!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

… and enable it in hconfd.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Currently this is initialised to no from Daemon.hs, but will in the
future allow command-line options for controlling it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch makes hail treat '-' as denoting stdin, per the usual Unix
convention. This will help with testing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Instead of hardcoded IPv4 INADDR_ANY, this patch changes hconfd to use
either the any network for the configured cluster address family
(ipv4/ipv6), or whatever the user passes in via the --bind option.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This implements the same logic as the Python code: if the option is
not used, use the default appropriate for the cluster, otherwise try
to parse and use whatever was passed in.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

These both are work with/on the Result type, so we add them to
BasicTypes. The functions will be used as more generic versions of
some more specialised functions that are right now spread across the
modules.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This currently has only one export function in it, which will be used
for future bind address functionality in daemons.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a stupid/trivial patch. I didn't realise when adding
htools-hpc that this will "break" tab completion on htools; as in, I
will have to add / manually always when completing files on the
command line.

So to fix my laziness, let's rename htools-hpc to hpc-htools,
eliminating this completion conflict.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

The dashes do not need to be in, if they are then the resulting option
is:

  -p PORT  ----port=PORT     Network port (default: 1814)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

These allow more accurate checks.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This will allow offline testing of this backend (except for the actual
curl part), also in the case where we didn't compile against curl.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Using some hand-crafted files (manually modified from a real cluster
running master branch), we test that hail behaves as we expect:
doesn't load obvious wrong data, can do relocations/allocations, etc.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

We most always use `templateMirrorType . diskTemplate`, so let's add a
helper function for this.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

After the many patches implementing the shared storage functionality
in iallocator, balances, etc., and after the patches preparing the
unit-tests to do the right thing for shared storage, we can finally
flip the bits that make instances have "random" disk templates as
opposed to just DRBD. Yes, the unit-tests still pass ☺️



We also need to enable all disk templates in the IPolicy (for
allocation/relocation).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch toggles the final bit and enables nodeEvacInstance to work
on shared storage instances. Diskless instances are handled the same
(as in, None is the same everywhere :).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of manually case-ing on the Either contents, let's just use
either with const functions (not really readable, but…).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The function is now generic, so let's rename it and cleanup/improve
the docstrings.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is generic enough (no longer depending on the replace type) that
it will be used for other replace modes too. So all we need to do is
to abstract it to a separate function (that can be called from
multiple branches) and to make the imove op_fn computation more
generic, instead of hard-coding it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, this test assumes localStorageTemplates, so let's make sure
we enforce that on the given instance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This brings together all previous pieces and allows balancing to work
(with the known caveats related to disk space calculation) for shared
storage disk templates.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Also add the spindle usage to QC

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In the same transition also do a simple refactor of the caculation

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The old text files remain compatible with this change.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

For now we just pass in "1" as spindle usage and adapt the backend
later.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Due to how conversions were implemented previously, 1TB failed to
parse on 32-bit, as we were overflowing during computation, even
though the final result would fit easily.

This patch moves the parsing of the scaling factor to a separate
function, and all the conversions are done via the Rational type
(which has unlimited arbitrary precision), and conversion to the
desired type only happens at the last step.

The unit-tests are adjusted too, unfortunately they use the same
algorithm as the code… suggestions on how to improve things are
welcome.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

These test that we can actually rebalance (and that it results in
replace-disks and failovers/migrations), and that we don't
double-rebalance, etc.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>