This adds support for encoding/decoding Query2 filters to/from JSON,
in (hopefully) the same format as the Python code generates.

It also adds a simple unit-test to check that this conversion is
idempotent. Of note here is that, since the Filter data type is
recursive, we have to manually ensure that the generator for it
correctly "shrinks" at each step (first version crashed hard my
workstation after eating ~8GB of ram :).

Compared to the current Query2 implementation, the following changes
were done:

- style: shortened some names to match the Python ones (LessEqualThan
  → LE, etc.)
- changed FilterValue from string to an ADT that can encode both
  quoted strings and numeric values, since this is actually what
  qlang.py generates
- added support for EmptyField, which in hindsight it's an obvious
  missing part :)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Commit 5a1e31b4 (Add infrastructure for, and two extra hlint rules)
was intended to add two *extra* hlint rules, but I didn't realise at
that time that "--hint" when first used overrides the built-in
lints. As such, since then we were basically running with just those
two rules, which resulted in many uncaught warnings/errors.

This patch fixes that (by importing the standard lint rules in our
custom hints file), and then goes to fix all the warnings that a
current hlint gives me. Compared to our current style, we have just a
few additions:

- zipWithM instead of map foo . zip …
- 'exitSuccess' instead of 'exitWith ExitSuccess'
- more uses of '.'

Additionally, we have to silence a case where hlint doesn't realise
why we are using '\e -> const (return False (e :: IOError)' instead of
just '\e -> return False' or even 'const (return False').

One warning that is generated by hlint ("Use void") can't be fixed
until we deprecate GHC 6.x, as only GHC 7 has the 'void' function in
Control.Monad.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This enables the query functionality in confd.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is just a new module that exports a runQueryD function, that can
be imported to run a separate thread handling the luxi requests.

Currently it needs access just to the configuration, in the future it
will need access to an RPC runner too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This makes the implementation a bit nicer for both for server and
client side: we add a wrapper function with a better result type, and
a few extra functions for building the response.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Note that since we don't have yet a way to nicely handle two-level
optional parameters, the Filled/Partial types and filling function are
all manually built.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This adds a few missing/incomplete definitions. We're still missing
the special parameters (disk params, hvparams, os_hvp).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Since we now handle Containers uniformly, we can remove all traces of
the special handling for this field type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is the first part of the changes related to the 'Container' type.

We currently handle this type as follows: it's a simple type alias
over the Data.Map type, which means:

- it's easy to use the Data.Map functions to change the type
- however, since Data.Map already has a JSON instance, we have to very
  carefully always use custom show/read routines to handle this type

The second point leads to potential bugs which are not caught by the
type system, so let's improve the situation by making it a proper
newtype, which can have its own JSON instance (with our desired
behaviour). Once we do this change, accessing the type requires an
extra function call, but it's as safe as before. On the positive side,
we can use the implicit read/show JSON, which means we can remove (in
the next patch) the "container" special casing.

The patch also moves the type to outside of THH, since not all users
of this will want to import that (as opposed to JSON.hs, which is
smaller).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch addresses two issues with our TH code:

- using non-unique names (e.g. "std" for a local name, instead of
  "std_XXXX" random names), which can leads to conflicts; on the other
  hand, this makes the generated code a bit harder to parse
- since only a few Python/JSON names have dashes in them, we didn't
  handle those, resulting in variables named like "disk-templates",
  which is not good; we now handle it the same as '_', i.e. we use it
  as a breaker for camel-casing

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Also add this new field and the other generic fields to the cluster
object.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Right now we're only able to test if when a node is offline, the call
fails with an appropriate errror.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

node_info call takes hypervisors and vgs to ask for information about
node and returns bootid and results from hypervisors and volume groups.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

instance_list call takes a list of hypervisors and returns a list
of running instances.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

all_instances_info call takes a list of hypervisors and returns a list
of (name, memory, state, vcpus, time) - one element for each
instance.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Prepare and execute RPC call, prepare result for the call. These
procedures are generic in the sense that they only require the
Call and Result types to be connected.
We use curl library for sending http requests for RPC; as the library's
usage can be disabled, we need to use CPP preprocessor macros for some parts
of the code.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

We introduce typeclasses for RPC call and result and create a typeclass
that binds the two together. For that we need to use
MultiParamTypeClasses and FunctionalDependencies language pragmas, which
allow us to ensure that RPC result type can be deduced based on the
RPC call type.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Very simple Hypervisor object, that we want to pass in some of RPC
calls is added. We also export AdminState data type, as it is used
in one of the calls that will be implemented in this patch series.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is a trivial code change, but it allows us to finally test the
send-receive code on both client and server sides via a simple
in-process server.

The unittest works, but it won't handle timeouts very nicely; it will
wait until the actual Luxi timeout expires, instead of using much
shorter timeouts as we could in the same process.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

I'm doing this change for future performance optimisations. Currently
we use the Luxi interface just as a client, so not in the hot path,
but when we'll use this as a server interface, we're interested to
both reduce the space and time consumption of the interface.

We have to simultaneous changes here:

- switch from using socket-related function (sendto, recv, etc.) to
  handle-based functions, since the standard network library doesn't
  work with sockets
- switch from using Strings for the internal buffer to strict
  ByteStrings; the only downside is that we now have the issue of
  decoding/encoding from binary to UTF-8 strings, a fact which brings
  its own issues into the mix (we have to check for failed decodings,
  etc.); but this is similar to what we'll have to handle on the
  Python side when moving to Python 3.x

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This has been a long-standing cleanup item, which we've always
refrained from doing due to the high estimated effort needed.

In reality, it turned out that after some infrastructure improvements
(the previous patches), the actual job queue-related changes are quite
small.

We will need to update the NEWS file later, but so far the RAPI
documentation doesn't mention that the job ID is a string (it only
says it is "a number"), so it doesn't look like it needs update.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Both the job id and submit job result parsing are abstracted into
separate functions, so that later changes are more localised.

Also, this makes submitManyJobs itself easier to read.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This will be used for easier change later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

It seems that 'explicitely' is wrong, and that the right form is
'explicitly'. This is just fixing the typo plus adjusting affected
paragraphs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This can be queried remotely since it's a pure configuration query.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This fixes an old FIXME. Since we now how actual DRBD configuration
data, we can finally compute the instance's secondary nodes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is a complex field, so we have to do a lot of manual work for now.

The complexity arises from the fact that the contents of the field,
and the way to parse it, depends on the disk type field, so we don't
have a single, static way of parsing it. Hence we needed the
extensions to the Template Haskell code.

Since we now can both load and save the disk type, we can remove the
in-memory (duplicate) disk type from the disk objects, relying only on
the logical ID to hold the type information.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Currently, we only allow field-by-field de-serialisation. Since we
have cases where information about how to un-serialise a field is
split across two JSON fields (e.g. disk type and disk logical_id,
hypervisor and hvparams, etc.), we need to pass the entire object to
custom read functions.

Furthermore, since we will have to generate two actual fields from the
single in-memory field, we need to extend the custom save function so
that they can generate additional fields beyond the "main" field value
they currently generate.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Instead of passing an expression (which cannot come from the current
module), we pass a name, which is allowed to reference functions from
the module we're in. Since we currently don't have custom fields, we
don't need to modify any callers.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This uses the recently-moved functions to implement partial lookup of
names on getNode and getInstance, similar to the Python codebase.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Currently, the LookupResult, MatchPriority and related functions are
locate in Loader.hs, since (so far) only hbal needs them in the
selection of instances. However, with the new functionality on confd
side, we need these functions there too, but we don't want to import
Loader.hs (which pulls in lots of balancing-related code). So we move
all these function to BasicTypes.hs, since that module is a leaf one,
with no other dependencies.

Unittests are slightly adjusted (but they are still tested under the
'Loader' group).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is not used, as we need a more complex serialisation, which is
done in the saveObjectField function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Since we're just talking about converting Maybe into another monad, we
can do that via the maybe function, instead of explicit casing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

There are two current issues with the coverage values:

- we don't import all modules, thus leading to incomplete
  coverage results (too optimistic);
- we use hpc in its default mode (intersection), which means that even
  modules which do have coverage results but are not used in all
  binaries we test will be dropped from the results; thanks to Agata,
  passing --union to hpc is enough to have better results (don't
  remember why this wasn't there in the first place…)

After adding more modules to the import list and fixing the combining
mode, we now have a complete list of modules in coverage results, many
with zero coverage, so our overall coverage has dropped to about 60%.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, we build the Haskell unittests with custom GHC flags,
because we (I) were quite lazy when initially writing the
unittests. This is not a problem for the tests themselves, but it
creates problem when (for example) one would want to pass all
$(HS_LIB_SRCS) to the compilers; this is not doable unless we
"degrade" the flags used for all modules, instead of just for QC.hs.

So we do two things to fix this:

- first, we go and add type declaration to all functions that were
  missing them (in QC), and fix the couple of cases of monomorphism
  restrictions; this gets us rid of -fno-warn-missing-signature and
  -fno-warn-monomorphism-restriction
- then, we move the actually important remaining options
  (-fno-warn-orphans and -fno-warn-unused-imports; see the explanation
  for the latter in the newly added FIXME) to a compiler pragma in the
  file, so that when building the unittests only this file is using
  the extra options

And finally we can then drop the other unused options
(-fno-warn-missing-methods anmd -Wwarn), leaving htools/test use
simply -fhpc. This is more in-line with the other files, and thus we
can handle all of HS_LIB_SRCS the same.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Since this is not yet used by any targets, we didn't detect yet the
compilation warnings. Just trivial exports/imports cleanup.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This derives from an internal bug, but the story is consistent across
both internal and external usage of hbal.

Basically right now, hbal returns exit code 1 if requested to exit
early, even if all jobs are successful. This is counter-intuitive due
to two reasons:

- hbal did what it was requested (exit early), so it shouldn't return error
- there were no job failures, so there's nothing to "cleanup" or
  investigate on the Ganeti cluster, so again it shouldn't return
  error

Therefore the new behaviour is as follows:

- for cases where all jobs were successful, even if terminated early
  via SIGINT or via --limit, we exit with code 0
- for cases where jobs have failed or there were other errors in
  running hbal, the exit code is 1
- for cases were hbal is requested an immediate termination (SIGTERM),
  exit code is 2, denoting "unknown whether the Ganeti cluster is
  consistent or not"

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

The same types are reused a couple of times, so let's add a couple of
type aliases for easier change later and readability.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Looking at the output of hcheck, in human readable mode, it looks like
it always starts with a blank line. This is not nice, so I wanted to
redo this to start cleanly.

However, looking at the code, I realised that we need some internal
cleanup; 'perGroupChecks' was doing both pure computation (computing
the score/stats) and I/O work (printing the stats), which is ugly, so
I split that in two; this allows perGroupChecks to be simpler, and we
can therefore first compute the group stats, and start the output with
the mode in which we run.

So right now, hcheck will first output whether we need to run a
rebalance, or if no-simulation is selected, etc., and only then
display the actual stats, making the output a bit more consistent.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>