This is the first commit of a series that will attempt to cleanup the
test code organisation, which evolved somewhat organically from the
initial pure htools functionality.

The proposed organisation of the tree will be as follows:

- htools (or maybe renamed to haskell or hs): only production code
- htest: top-level test directory, containing test.hs, static helper
  scripts, etc.
- htest/Ganeti/*.hs: modules implementing the actual test properties
  and test cases for the correspondingly-named production code modules
- htest/data: containing test data files for the test cases

This particular patch moves all the test code (test.hs, hpc-htools.hs
symlink) and helper scripts (offline-test.sh, etc.) from htools/ to
htest/, while updating the files themselves (if they had paths
mentioning htools/), .gitignore and the Makefile.

The only special mention is that in Makefile, we used to have a BINARY
shell variable in binary build rule; that was computed via stripping
`htools/' prefix; I've cleaned that and replaced with $(notdir $@);
even though it's duplicated a few times, it leads to more readable
make output (and easier to copy-paste).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This means that the verification of the correctness of the regex is
done once, at the deserialisation/creation time, as in the Python
code. To do this, we have to change the FilterRegex type from an alias
to String to a more complex data type, and we have to create manual
read/show/eq instance (phew!).

Unittests are added which test these instances, since it's the first time I do
this manually.

An additional improvement is that we now check that regex-pcre has
been compiled correctly, per the documentation (otherwise we get
runtime errors).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Per the new query module hierarchy, rename Qlang to Query/Language and
Queryd to Query/Server. This way, all query-related functionality is
now "contained" in the Query/ directory.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This adds basic infrastructure for filtering (fully functional except,
as usual, for runtime data), and then uses it for node queries.

Since the filtering exports regex matching as an external
functionality, we have to use a regex library. There are many flavours
of these in Haskell (see
http://www.haskell.org/haskellwiki/Regular_expressions

), but since we
want to be as compatible as we can with Python's, we use the
regex-pcre one, which is a wrapper to the widely used 'pcre' library.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This will allow us to run generic computations in an Applicative
context, if a monadic one is not needed (or not applicable due to
class constraints).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

In preparation for introducing filtering functionality, we convert the
'Filter' type from a '*' kind to a '* -> *' kind.

This allows us to define some general properties for the filter, and
for example introduce later an easy filter compilation, etc.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Since we have all the definitions already, we can easily enable
this. Manual testing shows no difference between the Python and the
Haskell versions of node list-fields.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is the final part:

* Parsing the new request type
* Feed it to allocList
* Format the result

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This function iterates over the tryMGAlloc function, updates the node
list and instance list, and refeeds it recursively until no new
instances are left.

This allows us then to allocate multiple instances and see if they would
fit together.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This was missed; we add the definition and the de-serialisation
support.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is the initial support for Query2: basic infrastructure (except
filtering) and node query support (without RPC).

It implements all the fields (tests by comparison with list-fields on
the Python side), except that:

- filter is not done
- since RPC is not integrated yet, the runtime gathering/computing is
  simply stubbed out

However, the infrastructure seems pretty reasonable, so I'm sending as
is.

Note that I've split the functions/declarations into multiple files,
to keep each file clean and readable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This adds the VTYPE_* conversion to VType, used for QFT equivalence
later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This adds two related improvements to THH:

- for parameters, we declare a list with all their fields, so that
  Query2 can build the list of fields (e.g. for hvparams, or ndparams)
  automatically

- we declare a new type class for "DictObjects", i.e. objects which
  can be converted to a [(String, JSValue)] list of pairs; while this
  applies to all our objects, it will be used first for filled
  parameters, so that we can implement the lookup functions
  generically

Note that we re-export the new class from Objects.hs, so that other
modules don't have to import THH.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is a simple type declaration for NodeRole, a NdParamObject type
class, and a few related helper functions for nodes and node groups.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

These mirror the TaggableObject in Python, in the sense that we will
be able to define generic functions for querying such fields.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch corrects the definitions in Qlang.hs to match what Python
expects on the wire; this means replacing some manual data type
definitions with 'buildObject' so that we get serialisation (and field
names) for free, adding (manually) JSON instances for types which are
not represented as objects in JSON, etc. Due to more TH usage, I had
to shift some definitions around, since TH breaks the "define in any
order" property (☹️

).

After that, we simply add a call into the stub Query/Query.hs module
which, for all queries, responds with "query not supported". The
reason for the deep directory structure is because I want to separate
the functionality into multiple submodules, for smaller module size
and readability.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Currently, the query daemon would simply close the connection to the
client without issuing a response, if parsing the arguments failed;
the error was just logged.

Since this is very ugly from the client's point of view, we change it
so that the error is both logged and sent back to the
client. Furthermore, the execution block reporting is also cleaned up
a bit, to make the code more uniform.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Reporting things such as the following in our error messages
(indentation added by me, not originally present, so it's even worse):

  JSArray [JSArray [JSRational False (1 % 1),JSString
                    (JSONString {fromJSString = "a"})],
           JSArray [JSRational False (2 % 1),
                    JSString (JSONString {fromJSString = "b"})]]

Is not really nice. This patch adds more 'pp_value' conversions so
that the above becomes:

  [[1, "a"], [2, "b"]]

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch adds a simple function to ensure a socket doesn't exist
before binding to it. Additionally, we cleanup the exports of
Queryd.hs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Since we have now access to all cluster parameters, we can "fill" the
parameters, and can finally enable it in CLI/RAPI.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch adds the missing parameters in the cluster/group objects,
for now as simple maps (dictionaries), without type safety. The
rationale for adding them as such is:

- we need something to enable query functionality
- since we don't modify the values, we don't risk introducing bugs
- we can improve the types later, once we find a good way to declare
  them

Also, I renamed a few of the parameters, dropping capitalisation of
acronyms (NIC → Nic, etc.).

At this point, I believe that the definitions are complete, with any
missing items being just oversight, and not intentionally removed (due
to lack of types, etc.).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is similar to the Python version, objects.py:FillDict.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

As opposed to all other places in the code, we imported this as
Data.Map, leaving to longer names. Let's make it similar to the rest
of the code (import qualified Data.Map as Map).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

These test that we encode requests correctly and decode good messages,
that a message with a too old/new timestamp is never accepted, and
that a message signed with a different key is never accepted.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

The Qlang module defines ResultStatus, but it was already defined in
Ganeti/Luxi.hs; let's remove the duplicate definition from there since
the proper place is in the newer module.

Also, in order to ease testing, we split some confd functions into a
separate module; this can be imported easily into QC for testing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is a very big hack for testing the equivalence of Python and
Haskell opcode definitions. See the docstring for details; I'm not
very happy with the solution but it does the job.

An alternate option would be to launch the Python code when
initialising the tests, pass (somehow) the resource through all of the
test suite to this function, and then use normal QuickCheck
properties. Or maybe we find a better solution later; for now, this
does the job.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

While (again) trying to test Python/Haskell encoding interoperability,
I found another bug: the disk index is declared in Python as
ht.TPositiveInt, but in Haskell just as Int, so it can take negative
values too. Clearly we can do better, so let's add a wrapper type that
protects creation of invalid indices via a smart constructor
(http://www.haskell.org/haskellwiki/Smart_constructors

, the runtime
checking variant).

This means that outside of OpCodes.hs, it's not possible to load
invalid values, and this also applies to de-serialisation from JSON
data.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

While testing Haskell⇔Python interoperability for opcode
serialisation, I found this bug: the Haskell code doesn't treat
optional fields with 'null' values as missing, which the Python code
does, leading to differences.

Investigating all uses of 'maybeFromObj' and the single use of
'fromObjWithDefault' shows that these are only used in cases where we
the rules are indeed "null == missing", so let's update the functions
and their docstrings accordingly.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

For now, we only test that we don't define extra opcodes in the
Haskell code; once we have parity, we can enable the (for now)
commented-out lines that check the reverse.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This will be used in the future to make tests/assertions for
Python/Haskell codebase equivalence (e.g. checking that the same
opcodes are defined, etc.).

(Side-note: this might not work in VPATH builds; I'll see how it goes)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This will allow us to use exactly the same method as for defining and
using QuickCheck properties. The differentiation is based on name,
which is not very nice but is the same method used in
test-framework-th, for example (so we will be able to switch
trivially).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

With the change to test-framework, where individual tests are listed,
QCHelper tries to remove a "prop_$group_" prefix from the function
names, for shorter names. However, some test groups (LUXI) and test
functions are not uniform, leading to long names, so we should fix
them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch replaces our home-grown, and quite limited, test runner
infrastructure with test-framework
(http://batterseapower.github.com/test-framework/

). The rationale for
doing so is as follows:

- we will need to add support for HUnit tests, so either we add more
  custom code or we switch to an existing library
- test-framework is mature and already packaged, at least in
  Debian/Ubuntu
- it supports more features: parallel test running, better test
  selection, etc.

As you can see, the changes are trivial, and don't touch the tests at
all; if/when we split the QC.hs file into per-module files, then we
could drop QCHelper too, and replace it with test-framework-th, which
does the same, but even more automated (auto-discovery, without having
to list the tests at all).

Dependencies are updated in devnotes.rst; note that I've already added
the hunit dependencies since we're going to use that soon.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Since now we have the recvMsgExt, we can remove the handleEOF
function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

We didn't close the server correctly, so for test sizes bigger than
the max-open-files ulimit, the tests were failing in accept().

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

We were missing many docstrings in THH.hs, so let's add at least some
of them, and fix some unquoted '/'. Additionally one style change has
been done.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This can be used for cross-checking with the Python code for
consistency on defined opcodes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

We'll need this in another place shortly, so let's abstract it and add
proper verification of whether we were passed a type name correctly;
the previous version would have failed with a pattern match failure,
instead of an explicit message.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch adds the tags field to the objects which were missing it
(all except Cluster), implements handling the LuxiCall QueryTags, and
then enables the use of the query socket in cli.ListTags, used by all
commands, and in the RAPI client.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Like in the python code, this is a bit more complex since groups are
indexed by UUID, so we must fallback to lookup via the name (slow; but
the number of groups should be small).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>