Part of the code used for running “prepare-node-join” can be re-used for
running a tool to configure the node daemon.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A new utility for configuring the node daemon will support a dry-run
mode. This patch adds the necessary functionality to
“ssconf.SimpleStore” and provides comprehensive tests for
“SimpleStore.WriteFiles”. To enable the latter, a testing-only parameter
specifying the lockfile was added to the class.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The new utility for configuring the node daemon will have to check
whether it received valid ssconf names.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add the design document for opportunistic locking.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This function can be applied to any type of class "Element" but its
internal variables used to refer to a list of tuples containing these
elements and their names as "nodes". Moreover it built a list of name to
indices tuples referring to it as "na" (which is not very clear).

Thus nodes becomes name_element, and na name_idx.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In case conflicts are checked, log warnings if nodes' IPs cannot
be reserved.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Ensure that locks are held only if needed.

Add conflicts_check in OpNetworkAdd. This is needed if we want to
check whether nodes/master IPs are included in network.

Depending on conflicts_check value, we have to hold node/instance locks
during OpNetworkAdd/OpNetworkConnect or not.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The serialization itself is done by Text.JSON, so the tests deal with checking
that Text.JSON objects are created correctly from the DRBD parser data
structures.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The serialized JSON is not a 1:1 dump of the data structures
populated by the parser. This is done intentionally, with the
aim of producing a more stable and more meaningful output to
be used by the (future) monitoring agent and stand-alone data
collectors.

Also:
* shorten the names of some data types.
* change data type for the fields of the Time data structure

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The new support functions in module Ganeti.JSON will be used by the next
commits to build JSON objects with optional (Maybe) fields, excluding
them when they contain Nothing.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This improves Issue 325 - new runtime and memory consumption is about
1/10 compared to before.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Given a node list in input, we get an instance that had nodes in it.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This takes an instance generator and produces a possibly empty list of
instances.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This generates a minimum of one node, because legal clusters never have
zero nodes.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

hbal doesn't have any "-o" option, but on the other hand supports -S
(described below, but not present in the short summary)

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is used only once when testing Cluster.hs, but having it abstracted
clarifies there what that call is about, makes that test shorter, and
allows us to better do refactoring of the main genInstanceSmallerThan
generator.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The new one is more consistent with the rest of the nodegroup
terminology and more generic for use in different htools.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It turns out that in newer GHC versions, the suffix shouldn't be
prefixed with a dot; otherwise name mangling is broken.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, the And and Or filters use very nice code, e.g. in case of
OrFilter:

   any id <$> mapM evaluateFilter flts

However, looking at the memory profiles shows that application of
any/id to monadic values via '<$>' does not work nicely, losing the
'early' success property. This results in too much memory being used
for thunks in monadic sequencing.

Rather than trying to add more strictness (not sure exactly how, TBH),
switching these to explicit recursion solves the problem, since we
take the 'early' exit problem in our hands and we are explicit about
it.

Memory usage in case of big (e.g. 1000 'Or' elements) is reduced
significantly, and thus also the runtime.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, the query operators (binop, etc.) create thunks, instead of
forcing the evaluation of the simple boolean results. This results in
higher than needed memory use.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Since this program is not built normally, a few recent commits broke
it:

- 707cd3d7 (“Use exitErr instead of explicit error message and
  exitWith”) introduced wrong syntax (missing '$')
- 29a30533 (“Remove unsafePerformIO usage”) changed the signature of
  G.Path.clusterConfFile
- 5183e8be (“Convert query path from string errors to GanetiException”)
  changed the signature of 'getNode'

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

I've been bitten a couple of times with arbitrary opcodes working on
UTF-8 locale, but failing on buildbot (ASCII). So let's add an
explicit test that checks always (even with UTF-8) for correct
arbitrary values, showing explicitly which opcodes fail.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

It turns out that optimising 'read' derived instances (via -O) for
complex data types (like OpCode, or the various objects) can be slow
to very slow. Disabling such instances results in (time make
$all_our_haskell_binaries) large compile-time savings and also smaller
(unstripped) binaries (by a significant amount):

ghc 6.12:        time  htools sz  hconfd sz
  with read:    4m50s 12,244,694 14,927,928
  no read:      3m30s 10,234,305 12,536,745
ghc 7.6:
  with read:   14m45s 13,694,761 15,741,755
  no read:      3m40s 11,631,373 13,245,134

So let's remove these instances, since we never use read in production
for our custom types, and even when debugging in GHCI, we can simply
use the 'show' representation to create the types, without needing to
actually parse from strings.

Note: for the very slow ghc 7.6 compilation time, I filled a ticket
(ghc #7450), since it is surprising(ly slow).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This is the last inline-written field; I've done it separately from
the last patch due to changes being needed in HTools (field changed
type).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

… except one, and replace them with separately-defined ones in
OpParams.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

It would be even better if the opcodes would actually have all the
same definitions, until then we have two sets of definitions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Due to lack of attention, we have two styles for generators of
arbitrary values: get* and gen* (e.g. getFQDN and genDiskIndices). In
order to make this more obvious that we deal with a function in the
Gen monad, let's rename all get* functions to gen*.

A few other simplifications were done on existing tests, switching to
helpers that were introduced later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This patch enhances the opcode list checks - instead of spawning a
Python interpreter to display the opcode list, we export it statically
in Constants.hs via a slight convert-constants change.

Furthermore, since we now have opcode parity, we enable full opcode
list checks.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

OpInstanceQuery was missing accidentally, whereas OpRestrictedCommand
was just recently added without Haskell definitions.

The patch also slightly improves the OpNodeQuery arbitrary generation.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This completes the last missing opcode group. The only difficulty was
with the ip addresses, where we used simple strings to represent them
and (for IPv4) a few helpers to generate arbitrary instances;
otherwise, the patch is trivial.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Also add some unittests for this type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This adds the OpTestAllocator, OpTestJqueue and OpTestDummy
opcodes. The OpTestAllocator seems to need some cleanup (on the Python
side), for now we implement it as is. As for the other two, while not
used in production, we should have full coverage for them as well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This makes the OpCode more consistent with the other opcodes. The
downside is incompatibility when upgrading from 2.6, but since this is
a test opcode it shouldn't be problematic.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This field is used with just changed description in about 10 opcodes,
so unifying it makes things simpler for future potential changes to
the field type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The regexp in OpTagsSearch is loaded as is, without testing for
validity; the rest of the patch is trivial.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This also corrects a docstring in OpBackupExport on the Python side.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Only the original instance opcodes (used in htools) are left
non-converted to only parameter style; they'll be cleaned up later,
once the htools codebase itself migrates to safer types.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This is not comprehensive, since in Python one can't determine what is
and what is not mutable; but I've added a few base cases (list, dict,
set).

The patch also improves (makes more uniform) the error messages in the
parameter definitions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit 12e62af5 (“Adding the new opcode for multi-allocation”)
introduced a "bad" default value; while porting this to Haskell, I
realised this is wrong.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>