GitLab

This will be used in the future to make tests/assertions for
Python/Haskell codebase equivalence (e.g. checking that the same
opcodes are defined, etc.).

(Side-note: this might not work in VPATH builds; I'll see how it goes)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

With the change to test-framework, where individual tests are listed,
QCHelper tries to remove a "prop_$group_" prefix from the function
names, for shorter names. However, some test groups (LUXI) and test
functions are not uniform, leading to long names, so we should fix
them.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Since now we have the recvMsgExt, we can remove the handleEOF
function.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

We didn't close the server correctly, so for test sizes bigger than
the max-open-files ulimit, the tests were failing in accept().
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This can be used for cross-checking with the Python code for
consistency on defined opcodes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch adds the tags field to the objects which were missing it
(all except Cluster), implements handling the LuxiCall QueryTags, and
then enables the use of the query socket in cli.ListTags, used by all
commands, and in the RAPI client.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This will allow safer code when we implement the tags query.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

The actual query definitions are now in Qlang.hs, so let's use the
ItemType from there instead of luxi-defined type (which is also
incomplete).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

While starting to use the new filter types, I realised that what is
currently implemented is the equivalent of `lib/qlang.py', not
`lib/query.py', since we only deal with data types for now and not the
actual query runtime functionality (RPC, config, etc.).

Let's rename the file to be more consistent with the Python code base.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Until now, since we didn't have a proper type for the encoded query
filters, we were ignoring the filters and handled them as null values
(JSNull and respectively ()). With the current Query2 implementation,
we can actually use correct filters (we can send filters the client
side, and on server side we can decode them).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This adds support for encoding/decoding Query2 filters to/from JSON,
in (hopefully) the same format as the Python code generates.

It also adds a simple unit-test to check that this conversion is
idempotent. Of note here is that, since the Filter data type is
recursive, we have to manually ensure that the generator for it
correctly "shrinks" at each step (first version crashed hard my
workstation after eating ~8GB of ram :).

Compared to the current Query2 implementation, the following changes
were done:

- style: shortened some names to match the Python ones (LessEqualThan
  → LE, etc.)
- changed FilterValue from string to an ADT that can encode both
  quoted strings and numeric values, since this is actually what
  qlang.py generates
- added support for EmptyField, which in hindsight it's an obvious
  missing part :)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Commit 5a1e31b4

 (Add infrastructure for, and two extra hlint rules)
was intended to add two *extra* hlint rules, but I didn't realise at
that time that "--hint" when first used overrides the built-in
lints. As such, since then we were basically running with just those
two rules, which resulted in many uncaught warnings/errors.

This patch fixes that (by importing the standard lint rules in our
custom hints file), and then goes to fix all the warnings that a
current hlint gives me. Compared to our current style, we have just a
few additions:

- zipWithM instead of map foo . zip …
- 'exitSuccess' instead of 'exitWith ExitSuccess'
- more uses of '.'

Additionally, we have to silence a case where hlint doesn't realise
why we are using '\e -> const (return False (e :: IOError)' instead of
just '\e -> return False' or even 'const (return False').

One warning that is generated by hlint ("Use void") can't be fixed
until we deprecate GHC 6.x, as only GHC 7 has the 'void' function in
Control.Monad.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Note that since we don't have yet a way to nicely handle two-level
optional parameters, the Filled/Partial types and filling function are
all manually built.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This adds a few missing/incomplete definitions. We're still missing
the special parameters (disk params, hvparams, os_hvp).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Right now we're only able to test if when a node is offline, the call
fails with an appropriate errror.
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

We introduce typeclasses for RPC call and result and create a typeclass
that binds the two together. For that we need to use
MultiParamTypeClasses and FunctionalDependencies language pragmas, which
allow us to ensure that RPC result type can be deduced based on the
RPC call type.
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is a trivial code change, but it allows us to finally test the
send-receive code on both client and server sides via a simple
in-process server.

The unittest works, but it won't handle timeouts very nicely; it will
wait until the actual Luxi timeout expires, instead of using much
shorter timeouts as we could in the same process.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the LookupResult, MatchPriority and related functions are
locate in Loader.hs, since (so far) only hbal needs them in the
selection of instances. However, with the new functionality on confd
side, we need these functions there too, but we don't want to import
Loader.hs (which pulls in lots of balancing-related code). So we move
all these function to BasicTypes.hs, since that module is a leaf one,
with no other dependencies.

Unittests are slightly adjusted (but they are still tested under the
'Loader' group).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

There are two current issues with the coverage values:

- we don't import all modules, thus leading to incomplete
  coverage results (too optimistic);
- we use hpc in its default mode (intersection), which means that even
  modules which do have coverage results but are not used in all
  binaries we test will be dropped from the results; thanks to Agata,
  passing --union to hpc is enough to have better results (don't
  remember why this wasn't there in the first place…)

After adding more modules to the import list and fixing the combining
mode, we now have a complete list of modules in coverage results, many
with zero coverage, so our overall coverage has dropped to about 60%.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, we build the Haskell unittests with custom GHC flags,
because we (I) were quite lazy when initially writing the
unittests. This is not a problem for the tests themselves, but it
creates problem when (for example) one would want to pass all
$(HS_LIB_SRCS) to the compilers; this is not doable unless we
"degrade" the flags used for all modules, instead of just for QC.hs.

So we do two things to fix this:

- first, we go and add type declaration to all functions that were
  missing them (in QC), and fix the couple of cases of monomorphism
  restrictions; this gets us rid of -fno-warn-missing-signature and
  -fno-warn-monomorphism-restriction
- then, we move the actually important remaining options
  (-fno-warn-orphans and -fno-warn-unused-imports; see the explanation
  for the latter in the newly added FIXME) to a compiler pragma in the
  file, so that when building the unittests only this file is using
  the extra options

And finally we can then drop the other unused options
(-fno-warn-missing-methods anmd -Wwarn), leaving htools/test use
simply -fhpc. This is more in-line with the other files, and thus we
can handle all of HS_LIB_SRCS the same.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Thanks to Agata Murawska, a random seed that generates bad test cases
was found ("seed was 1768143307 2118231514, test size 74"). This is
due to the fact that some nodes were generated with tMem=0, fMem=0,
which means that pMem was NaN and thus the node was not comparing
equal to itself.

Worked around by limiting the test case generation to nodes with
positive values; a better fix would be to compare nodes differently
(subset of properties or ignore NaN values).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch fixes a (brown-bag) bug in the ssconf implementation where
the looked-for filenames were not prefixed with the ssconf file
prefix.

Additionally, a test for this is added (which fails without the fix).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds a hand-coded decoder for LuxiCall arguments, as the
data-structure is not uniform enough for automated generation (even
for the serialisation, we had to add hints for some fields,
de-serialisation is even harder).

It also fixes a tiny issue with WaitForJobChange job ID encoding, and
adds unittests for the encoding/decoding of LuxiOp structures.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

In preparation for unit-testing the Luxi definitions, this improves
the auto-generation of OpCodes: currently this uses unbounded plain
Strings for names and fields, and this creates too big test sizes
(e.g. when automatically generating job sets).

Furthermore, there are some more cleanups: switching from liftM to
applicative style for easier definition, and some improvements in
generators.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

And its associated unittests.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

We most always use `templateMirrorType . diskTemplate`, so let's add a
helper function for this.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

After the many patches implementing the shared storage functionality
in iallocator, balances, etc., and after the patches preparing the
unit-tests to do the right thing for shared storage, we can finally
flip the bits that make instances have "random" disk templates as
opposed to just DRBD. Yes, the unit-tests still pass ☺



We also need to enable all disk templates in the IPolicy (for
allocation/relocation).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, this test assumes localStorageTemplates, so let's make sure
we enforce that on the given instance
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Also add the spindle usage to QC
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The old text files remain compatible with this change.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

For now we just pass in "1" as spindle usage and adapt the backend
later.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Due to how conversions were implemented previously, 1TB failed to
parse on 32-bit, as we were overflowing during computation, even
though the final result would fit easily.

This patch moves the parsing of the scaling factor to a separate
function, and all the conversions are done via the Rational type
(which has unlimited arbitrary precision), and conversion to the
desired type only happens at the last step.

The unit-tests are adjusted too, unfortunately they use the same
algorithm as the code… suggestions on how to improve things are
welcome.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

As for the evacuate tests, we require a mirrored disk template, but
otherwise the test should work for both mirror types.

Additionally, we perform a simplification that was left as TODO.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, these presume that the instance being generated always is
DRBD/relocatable (but won't be in the future), and also presume that
all relocation modes can be used (hint: not ☺

).

So we add a couple of helper functions to restrict the used disk
templates/relocation modes, and these two tests are ready for shared
storage.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of randomly generating the number of required nodes or
hard-coding it, compute it from the disk template of the
instance. Otherwise tests will fail when the disk template is not
DRBD8 and we ask for allocation on 2 nodes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Another test that only works for DRBD8 instances.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

… since memory reservation only is used for this template.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, this (single) test checks whether we can add an instance as
primary and secondary to a given node. This won't work for instance
templates other than DRBD8, so we split it into:

- an add to primary test, which should work for all disk templates
- an add to secondary test, for which we set the disk template to DRBD
  explicitly
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the code in Loader.hs marks as un-movable any instances
which don't have a secondary. This is not valid for any shared storage
templates, so let's remove that check completely. The initial state of
the instance will reflect correctly the movable status, based on the
disk template of the instance, so we don't need to re-do this check.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>