This function can be applied to any type of class "Element" but its
internal variables used to refer to a list of tuples containing these
elements and their names as "nodes". Moreover it built a list of name to
indices tuples referring to it as "na" (which is not very clear).

Thus nodes becomes name_element, and na name_idx.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The serialization itself is done by Text.JSON, so the tests deal with checking
that Text.JSON objects are created correctly from the DRBD parser data
structures.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The serialized JSON is not a 1:1 dump of the data structures
populated by the parser. This is done intentionally, with the
aim of producing a more stable and more meaningful output to
be used by the (future) monitoring agent and stand-alone data
collectors.

Also:
* shorten the names of some data types.
* change data type for the fields of the Time data structure

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The new support functions in module Ganeti.JSON will be used by the next
commits to build JSON objects with optional (Maybe) fields, excluding
them when they contain Nothing.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The new one is more consistent with the rest of the nodegroup
terminology and more generic for use in different htools.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, the And and Or filters use very nice code, e.g. in case of
OrFilter:

   any id <$> mapM evaluateFilter flts

However, looking at the memory profiles shows that application of
any/id to monadic values via '<$>' does not work nicely, losing the
'early' success property. This results in too much memory being used
for thunks in monadic sequencing.

Rather than trying to add more strictness (not sure exactly how, TBH),
switching these to explicit recursion solves the problem, since we
take the 'early' exit problem in our hands and we are explicit about
it.

Memory usage in case of big (e.g. 1000 'Or' elements) is reduced
significantly, and thus also the runtime.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, the query operators (binop, etc.) create thunks, instead of
forcing the evaluation of the simple boolean results. This results in
higher than needed memory use.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Since this program is not built normally, a few recent commits broke
it:

- 707cd3d7 (“Use exitErr instead of explicit error message and
  exitWith”) introduced wrong syntax (missing '$')
- 29a30533 (“Remove unsafePerformIO usage”) changed the signature of
  G.Path.clusterConfFile
- 5183e8be (“Convert query path from string errors to GanetiException”)
  changed the signature of 'getNode'

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

It turns out that optimising 'read' derived instances (via -O) for
complex data types (like OpCode, or the various objects) can be slow
to very slow. Disabling such instances results in (time make
$all_our_haskell_binaries) large compile-time savings and also smaller
(unstripped) binaries (by a significant amount):

ghc 6.12:        time  htools sz  hconfd sz
  with read:    4m50s 12,244,694 14,927,928
  no read:      3m30s 10,234,305 12,536,745
ghc 7.6:
  with read:   14m45s 13,694,761 15,741,755
  no read:      3m40s 11,631,373 13,245,134

So let's remove these instances, since we never use read in production
for our custom types, and even when debugging in GHCI, we can simply
use the 'show' representation to create the types, without needing to
actually parse from strings.

Note: for the very slow ghc 7.6 compilation time, I filled a ticket
(ghc #7450), since it is surprising(ly slow).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This is the last inline-written field; I've done it separately from
the last patch due to changes being needed in HTools (field changed
type).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

… except one, and replace them with separately-defined ones in
OpParams.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

It would be even better if the opcodes would actually have all the
same definitions, until then we have two sets of definitions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

OpInstanceQuery was missing accidentally, whereas OpRestrictedCommand
was just recently added without Haskell definitions.

The patch also slightly improves the OpNodeQuery arbitrary generation.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This completes the last missing opcode group. The only difficulty was
with the ip addresses, where we used simple strings to represent them
and (for IPv4) a few helpers to generate arbitrary instances;
otherwise, the patch is trivial.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Also add some unittests for this type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This adds the OpTestAllocator, OpTestJqueue and OpTestDummy
opcodes. The OpTestAllocator seems to need some cleanup (on the Python
side), for now we implement it as is. As for the other two, while not
used in production, we should have full coverage for them as well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The regexp in OpTagsSearch is loaded as is, without testing for
validity; the rest of the patch is trivial.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This also corrects a docstring in OpBackupExport on the Python side.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Only the original instance opcodes (used in htools) are left
non-converted to only parameter style; they'll be cleaned up later,
once the htools codebase itself migrates to safer types.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This is a "big" opcode, so sending it separately.

A few types needed changing, and a few parameters were renamed to make
it more clear which are cluster-level and which are instance-level
parameters.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Another base type that we need in opcodes as well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

* Now the parser completely consumes the input, up to the end of the text.
* Name of the test suite module changed to adhere to naming conventions.
* Some reformatting of the source code.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

commaInt now recognizes only the first 3 digits for numbers without commas.

It was erroneously recognizing numbers of any size before the first comma.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

I'm ignoring the use of Control.Arrow.*** since it's a non-obvious
function, and we're not widely using Arrows in the code base, so I
think switching to it would make the code less readable to most
people.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

These are tested for equivalence with the Python code using existing
tests. These tests are not perfect - for many "TDict" we simply send
empty dicts, as we don't have a way to generate proper data, but the
tests did catch many trivial errors during writing of this patch
series.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This does a partial conversion of existing opcodes to the new
non-empty string type, which is needed so that we don't add even more
opcodes without this safety net (and to have consistent "target_node",
etc. in opcodes).

By partial I mean that only the opcode are changed, and not the entire
htools types/workflow, so we use error to abort if, when submitting
the opcodes, we detect an empty node name. FIXME added.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a bulk add of all node parameters needed for node and cluster
opcodes. The parameters are defined with a few helper functions in
this module, and, opposite from opcodes.py, there won't be any (new)
opcode attributes declared in-line (without a separate field
definition).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

These are needed for completing the node opcode parameters.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This fixes a FIXME actually recommending this move :)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The QR_VIA_OP/QR_VIA_LUXI types in Python are using yet another
validation mode: QR_VIA_OP is the base type, and QR_VIA_LUXI extends
it (when doing luxi queries). But on the wire they have the same
representation.

To accommodate this properly, we split the ItemType in two: a
QueryTypeOp and a QueryTypeLuxi, joining them back together in
ItemType itself. This requires custom serialisation/deserialisation,
but allows us to express correctly that at opcode level, we only
accept a QueryTypeOp, but at Luxi query level, we accept either of
them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a very basic type and 'Objects' is a heavy-weight module. By
moving it to 'types' we simplify (in the future) the import chains.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

Currently, we have some types that we kept as dictionaries in the
objects representation (due to inconsistent behaviour, or other
technical reasons). This should be improved in the future, but in the
meantime we can improve the Container type by allowing its keys to be
non-strings; this is needed, for example, for better safe in
DiskParams type, where we don't want arbitrary strings as keys, but
only the actually defined types.

To implement this change, we generalise the type (Container →
GenericContainer), and in the process we introduce a type class for
"things that can be represented as strings". This is very similar to a
combination of Read and Show, but with custom string
representation. The new type class is needed because in JSON
representation, object keys must be strings, so we need to be able to
serialised/deserialise the generic keys to/from plain strings.

We also add the instance for DiskTemplate at the same time.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

These are needed both in opcodes and (potentially) outside, so we add
them in the main Types module.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This mirrors the ht.py types PositiveInt, NonNegative, etc., except
that they work at a more generic level (any numeric type, respectively
any non-empty list).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

A new directory for haskell modules about block devices has been created
The parser is divided in two modules:
* one exports the data types describing the DRBD status
* one exports the parser itself

Signed-off-by: Michele Tartara <mtartara@google.com>
[iustin@google.com: indentation/alignment fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

While writing the pipe-based reporting and trying various ways to
break the startup, I fought for a while trying to understand why error
reporting was _different_ when running the daemon as a user (with no
rights). It turns out that setupDaemonFDs wants to open the log file
in append mode way before, so we are not protected by the 'prepare'
phase.

This patch explicitly runs the 'setupDaemonFDs' function under the
same handler as the prepare phase, with the only change that here we
instruct handlePrepErr to not log the message via log*, since logging
is not yet set up.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, the OpCode definitions are using normal constructors:

  data OpCode = OpTestDelay Double Bool [String]
              | OpInstanceFailover String Bool (Maybe String)
              …

While this works for a few opcodes, it becomes unwieldy when dealing
with a bigger number of opcode definitions and/or with opcodes having
many fields.

This patch changes the opcodes to record-based constructors, so that
we get for free accessor functions:

  data OpCode
    = OpTestDelay {
        opDuration :: Double,
        opOnMaster :: Bool,
        opOnNodes :: [String]
      }
      | OpInstanceFailover {
         opInstanceName :: String,
         opIgnoreConsistency :: Bool,
         opTargetNode :: Maybe String
      }
      …

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

While looking at the opcode docs and clicking accidentally on the
filter type, I saw that the haddock formatting is broken due to
non-escaped use of special chars. Let's convert the ascii-like
formatting to haddock, and have nicer apidoc.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Due to TemplateHaskell stage restrictions, we can't define parameters
in the same module as we're using them for TH, so we have to define
all module parameters in a separate module.

This patch therefore splits OpCodes.hs in two, adding that module and
moves most code there (types, parameters, etc.). The remaining parts
in OpCodes.hs, the actual opcode definitions, now use more parameters
instead of direct field definitions (more will come later)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

There are already three cases where we copied type definitions between
the htools-specific types into the main ganeti code. Let's stop doing
this ☺️

 and create a common types module that holds these.

Note that there already exists BasicTypes.hs, but that refers to very
low-level types, and can't use TH derivation itself.

A side effect of this unification is that there is a small conflict
between AdminStatus/AdminOffline and InstanceStatus/AdminOffline. As
such, I renamed AdminOffline and AdminDown to StatusOffline/StatusDown
in the InstanceStatus type.

The patch also moves the tests related to these types to a new test
module.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>