Due to lack of attention, we have two styles for generators of
arbitrary values: get* and gen* (e.g. getFQDN and genDiskIndices). In
order to make this more obvious that we deal with a function in the
Gen monad, let's rename all get* functions to gen*.

A few other simplifications were done on existing tests, switching to
helpers that were introduced later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This patch enhances the opcode list checks - instead of spawning a
Python interpreter to display the opcode list, we export it statically
in Constants.hs via a slight convert-constants change.

Furthermore, since we now have opcode parity, we enable full opcode
list checks.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

OpInstanceQuery was missing accidentally, whereas OpRestrictedCommand
was just recently added without Haskell definitions.

The patch also slightly improves the OpNodeQuery arbitrary generation.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This completes the last missing opcode group. The only difficulty was
with the ip addresses, where we used simple strings to represent them
and (for IPv4) a few helpers to generate arbitrary instances;
otherwise, the patch is trivial.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Also add some unittests for this type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This adds the OpTestAllocator, OpTestJqueue and OpTestDummy
opcodes. The OpTestAllocator seems to need some cleanup (on the Python
side), for now we implement it as is. As for the other two, while not
used in production, we should have full coverage for them as well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The regexp in OpTagsSearch is loaded as is, without testing for
validity; the rest of the patch is trivial.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This also corrects a docstring in OpBackupExport on the Python side.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Only the original instance opcodes (used in htools) are left
non-converted to only parameter style; they'll be cleaned up later,
once the htools codebase itself migrates to safer types.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This is a "big" opcode, so sending it separately.

A few types needed changing, and a few parameters were renamed to make
it more clear which are cluster-level and which are instance-level
parameters.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Another base type that we need in opcodes as well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

* Now the parser completely consumes the input, up to the end of the text.
* Name of the test suite module changed to adhere to naming conventions.
* Some reformatting of the source code.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The bug was in the test itself, not in the tested code.

Also, fixed a line longer than 80 characters in the same file.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

These new tests substitute the HTests, as by the suggestion
received after submitting the previous patch.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

I'm ignoring the use of Control.Arrow.*** since it's a non-obvious
function, and we're not widely using Arrows in the code base, so I
think switching to it would make the code less readable to most
people.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Since we generate arbitrary strings and then we send them over a pipe
to Python, we can (and will) generate non-ascii chars, which means
when run in a non-UTF8 locale, this will break with ghc 6.12 and
later. It didn't break on my workstation, since I have a "nice"
locale.

So to fix this, we replace all name generations with explicit
genName/genNameNE and similar, to ensure we only send "nice" strings.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

These are tested for equivalence with the Python code using existing
tests. These tests are not perfect - for many "TDict" we simply send
empty dicts, as we don't have a way to generate proper data, but the
tests did catch many trivial errors during writing of this patch
series.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This does a partial conversion of existing opcodes to the new
non-empty string type, which is needed so that we don't add even more
opcodes without this safety net (and to have consistent "target_node",
etc. in opcodes).

By partial I mean that only the opcode are changed, and not the entire
htools types/workflow, so we use error to abort if, when submitting
the opcodes, we detect an empty node name. FIXME added.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

These are needed for completing the node opcode parameters.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This makes it easy to generate sets of "things" that don't even need
to have arbitrary instances, but which have bounded/enum instances.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

This fixes a FIXME actually recommending this move :)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The QR_VIA_OP/QR_VIA_LUXI types in Python are using yet another
validation mode: QR_VIA_OP is the base type, and QR_VIA_LUXI extends
it (when doing luxi queries). But on the wire they have the same
representation.

To accommodate this properly, we split the ItemType in two: a
QueryTypeOp and a QueryTypeLuxi, joining them back together in
ItemType itself. This requires custom serialisation/deserialisation,
but allows us to express correctly that at opcode level, we only
accept a QueryTypeOp, but at Luxi query level, we accept either of
them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a very basic type and 'Objects' is a heavy-weight module. By
moving it to 'types' we simplify (in the future) the import chains.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

Currently, we have some types that we kept as dictionaries in the
objects representation (due to inconsistent behaviour, or other
technical reasons). This should be improved in the future, but in the
meantime we can improve the Container type by allowing its keys to be
non-strings; this is needed, for example, for better safe in
DiskParams type, where we don't want arbitrary strings as keys, but
only the actually defined types.

To implement this change, we generalise the type (Container →
GenericContainer), and in the process we introduce a type class for
"things that can be represented as strings". This is very similar to a
combination of Read and Show, but with custom string
representation. The new type class is needed because in JSON
representation, object keys must be strings, so we need to be able to
serialised/deserialise the generic keys to/from plain strings.

We also add the instance for DiskTemplate at the same time.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

These are needed both in opcodes and (potentially) outside, so we add
them in the main Types module.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Sorry…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This mirrors the ht.py types PositiveInt, NonNegative, etc., except
that they work at a more generic level (any numeric type, respectively
any non-empty list).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

This is yet-another version-specific warning (fixed in later hlint
versions), but it's better to keep it clean.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This adds tests that existing test files can be parsed by the Haskell
parser as well, plus one new test file.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Use more monadic combinators instead of explicit code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

There are already three cases where we copied type definitions between
the htools-specific types into the main ganeti code. Let's stop doing
this ☺️

 and create a common types module that holds these.

Note that there already exists BasicTypes.hs, but that refers to very
low-level types, and can't use TH derivation itself.

A side effect of this unification is that there is a small conflict
between AdminStatus/AdminOffline and InstanceStatus/AdminOffline. As
such, I renamed AdminOffline and AdminDown to StatusOffline/StatusDown
in the InstanceStatus type.

The patch also moves the tests related to these types to a new test
module.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

They mimic their python counterparts.

Added functions:
* getSourceDir
* testDataFilename
* readTestData
* readPythonTestData

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, THH.hs "injects" into the built code names of library
functions like Text.JSON.makeObj, Ganeti.JSON.fromObj, etc. built
directly from strings, via (e.g.)

  varE (mkName "makeObj")

This means that the "makeObj" name must exist in the target module,
i.o.w. must be imported there. This leads to the strange case of
having to have imports that do not appear at all in the used
(template) code, but are needed to satisfy this "hidden" dependency;
look at Ganeti/Jobs.hs before this patch, for example.

This is also not very obvious, because we usually import Text.JSON
anyway; I only stumbled upon it while doing some cleanup work.

So to clean this up, the current patch changes the THH.hs to use not
string-derived, but identifier-derived names («'identifier» versus
«mkName "identifier"»); this is better, as the names must be
resolvable when compiling THH itself (once), and not when compiling
the multiple derived modules. As you can see, this allows removal of
extraneous imports from various modules.

Background information: an `mkName "foo"` results in a name of flavour
NameS (“An unqualified name; dynamically bound”) or alternatively to a
qualified name, but still dynamically bound. Whereas what we want is a
statically bound name: `'foo` results in a NameG flavour, “Global name
bound outside of the TH AST: An original name”.

One more explanation: the change is similar to going from 'x = eval
"map"' to 'x = map'; the name is no longer dynamically evaluated, but
statically when the module is compiled. In our case, previously names
were bound at target module compile time, now they are bound at THH.hs
compile time.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is not a quickcheck property, since it doesn't have any
variable/arbitrary inputs. So let's make it a test case, and shorten a
bit the name.

Also, sorry for not catching this in the review.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

A new LUXI request is added, in both Python and Haskell.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Some GHC flags are very useful, but only appear in more recent GHC
versions. To support the use of such flags while still supporting
older compilers, let's add conditional checks and enabling based on
the results.

Currently only `-fwarn-incomplete-uni-patterns` is enabled, which
detects refutable patterns in lambda expressions and pattern bindings
(e.g. "let (Just x) = y"); such constructs are bad as they can lead to
runtime exceptions.

Additionally, fix an existing such bad construct in a test case; we
workaround it by using error, since that should never fail.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Attoparsec is known to have had issues with parsing non-ASCII strings.
This test makes sure that parsing of Unicode characters works fine.

Signed-off-by: Michele Tartara <mtartara@google.com>
[iustin: small doc string fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

Sorry, I broke lint again :), by introducing two sub-standard changes.

Additionally, this silences an older existing warning that only
triggers with some versions of hlint (e.g. 1.8.28 which is present in
Wheezy).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

We need to change a few things, most importantly CLI options defaults,
but otherwise we already used the path to files from functions which
were already in the I/O monad, so we don't have to change much of the
code flow.

Additionally, Path.hs now has an explicit export list, to stop
accidental leakage of symbols.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>