While starting to use the new filter types, I realised that what is
currently implemented is the equivalent of `lib/qlang.py', not
`lib/query.py', since we only deal with data types for now and not the
actual query runtime functionality (RPC, config, etc.).

Let's rename the file to be more consistent with the Python code base.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Until now, since we didn't have a proper type for the encoded query
filters, we were ignoring the filters and handled them as null values
(JSNull and respectively ()). With the current Query2 implementation,
we can actually use correct filters (we can send filters the client
side, and on server side we can decode them).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This adds support for encoding/decoding Query2 filters to/from JSON,
in (hopefully) the same format as the Python code generates.

It also adds a simple unit-test to check that this conversion is
idempotent. Of note here is that, since the Filter data type is
recursive, we have to manually ensure that the generator for it
correctly "shrinks" at each step (first version crashed hard my
workstation after eating ~8GB of ram :).

Compared to the current Query2 implementation, the following changes
were done:

- style: shortened some names to match the Python ones (LessEqualThan
  → LE, etc.)
- changed FilterValue from string to an ADT that can encode both
  quoted strings and numeric values, since this is actually what
  qlang.py generates
- added support for EmptyField, which in hindsight it's an obvious
  missing part :)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Since these will be needed for Haskell's implementation of query2.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Commit 5a1e31b4 (Add infrastructure for, and two extra hlint rules)
was intended to add two *extra* hlint rules, but I didn't realise at
that time that "--hint" when first used overrides the built-in
lints. As such, since then we were basically running with just those
two rules, which resulted in many uncaught warnings/errors.

This patch fixes that (by importing the standard lint rules in our
custom hints file), and then goes to fix all the warnings that a
current hlint gives me. Compared to our current style, we have just a
few additions:

- zipWithM instead of map foo . zip …
- 'exitSuccess' instead of 'exitWith ExitSuccess'
- more uses of '.'

Additionally, we have to silence a case where hlint doesn't realise
why we are using '\e -> const (return False (e :: IOError)' instead of
just '\e -> return False' or even 'const (return False').

One warning that is generated by hlint ("Use void") can't be fixed
until we deprecate GHC 6.x, as only GHC 7 has the 'void' function in
Control.Monad.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This enables the query functionality in confd.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is just a new module that exports a runQueryD function, that can
be imported to run a separate thread handling the luxi requests.

Currently it needs access just to the configuration, in the future it
will need access to an RPC runner too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This makes the implementation a bit nicer for both for server and
client side: we add a wrapper function with a better result type, and
a few extra functions for building the response.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Note that since we don't have yet a way to nicely handle two-level
optional parameters, the Filled/Partial types and filling function are
all manually built.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This adds a few missing/incomplete definitions. We're still missing
the special parameters (disk params, hvparams, os_hvp).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Since we now handle Containers uniformly, we can remove all traces of
the special handling for this field type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is the first part of the changes related to the 'Container' type.

We currently handle this type as follows: it's a simple type alias
over the Data.Map type, which means:

- it's easy to use the Data.Map functions to change the type
- however, since Data.Map already has a JSON instance, we have to very
  carefully always use custom show/read routines to handle this type

The second point leads to potential bugs which are not caught by the
type system, so let's improve the situation by making it a proper
newtype, which can have its own JSON instance (with our desired
behaviour). Once we do this change, accessing the type requires an
extra function call, but it's as safe as before. On the positive side,
we can use the implicit read/show JSON, which means we can remove (in
the next patch) the "container" special casing.

The patch also moves the type to outside of THH, since not all users
of this will want to import that (as opposed to JSON.hs, which is
smaller).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch addresses two issues with our TH code:

- using non-unique names (e.g. "std" for a local name, instead of
  "std_XXXX" random names), which can leads to conflicts; on the other
  hand, this makes the generated code a bit harder to parse
- since only a few Python/JSON names have dashes in them, we didn't
  handle those, resulting in variables named like "disk-templates",
  which is not good; we now handle it the same as '_', i.e. we use it
  as a breaker for camel-casing

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Also add this new field and the other generic fields to the cluster
object.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is not used yet, but is helpful when developing the Luxi endpoint
for hconfd.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Right now we're only able to test if when a node is offline, the call
fails with an appropriate errror.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

node_info call takes hypervisors and vgs to ask for information about
node and returns bootid and results from hypervisors and volume groups.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

instance_list call takes a list of hypervisors and returns a list
of running instances.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

all_instances_info call takes a list of hypervisors and returns a list
of (name, memory, state, vcpus, time) - one element for each
instance.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Prepare and execute RPC call, prepare result for the call. These
procedures are generic in the sense that they only require the
Call and Result types to be connected.
We use curl library for sending http requests for RPC; as the library's
usage can be disabled, we need to use CPP preprocessor macros for some parts
of the code.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

We introduce typeclasses for RPC call and result and create a typeclass
that binds the two together. For that we need to use
MultiParamTypeClasses and FunctionalDependencies language pragmas, which
allow us to ensure that RPC result type can be deduced based on the
RPC call type.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Very simple Hypervisor object, that we want to pass in some of RPC
calls is added. We also export AdminState data type, as it is used
in one of the calls that will be implemented in this patch series.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

As timeout constants will now be used also on Haskell side, they have
been moved from rpc_defs and rpc to constants. The same is true for
connection timeout for connecting to nodes.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.6:
  Make stable-2.6 compatible with newer pep8
  Fix computation of disk sizes in _ComputeDiskSize
  Add verification of RPC results in _WipeDisks
  Add test for checking that all gnt-* subcommands run OK
  Fix double use of PRIORITY_OPT in gnt-node migrate
  Add new Makefile target to rebuild the whole dist
  rapi client: accept arbitrary shutdown arguments
  Handle offline nodes for "instance down" checks
  Add missing rst files to Makefile.am
  Release version 2.6.0 (final)
  Fix 'explicitely' common typo
  Fix issue in LUClusterVerifyGroup with multi-group clusters
  Add QA test for node group modification of ndparams
  Fix node group modification of node parameters
  Fix RST formatting in NEWS file
  Update NEWS and bump version for release 2.5.2
  Fix boot=on flag for CDROMs
  KVM: only pass boot flag once
  Ensure a stable content of the bash completion file

Conflicts (all trivial):
        Makefile.am  (design drafts on both sides, pep8 changes)
        autotools/build-bash-completion (copyright years)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

* stable-2.6:
  Make stable-2.6 compatible with newer pep8
  Fix computation of disk sizes in _ComputeDiskSize
  Add verification of RPC results in _WipeDisks
  Add test for checking that all gnt-* subcommands run OK
  Fix double use of PRIORITY_OPT in gnt-node migrate
  Add new Makefile target to rebuild the whole dist

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is done so that all current branches can run with newer pep8;
note that instead of fixing the problems (like I did on master), I've
just silenced more. These should *not* be merged onto master!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:

- bump version in the docs
- silence some new checks that are wrong due to our indent=2 instead of 4
- fix lots of errors in the code where the indentation was wrong by 1
  or 2 spaces
- fix a few cases of == True, False, None and replace with 'is'
- re-indent some cases where the code is OK, but pep8 complains

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Since this is an inheritable parameter, having it as a MABYE_STRING
creates only problems (per our derivation rules). We change it to
STRING, with the default "", meaning no program. Note that most of the
code already accepts this as valid for "no program", and some comments
even say that this is the expected value.

We have some other parameters like this, I'll have to investigate
whether they need to be changed too. But right now I need this for the
hconfd changes (it's a prerequisite for them, I forgot to send it in
that patch series).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Currently, hail fails with FailDisk when trying to add an instance
of type: 'file', 'sharedfile' and 'rbd'.

This is due to a "0" or None value in the corresponding dict inside
_ComputeDiskSize, which results in a "O" or non Int value of the
exported 'disk_space_total' parameter. This in turn makes hail fail,
when trying to process the value:

 - with "Unable to read Int" if value is None (file)
 - with FailDisk if value is 0 (sharedfile, rbd)

The latter happens because the 0 value doesn't match the instance's
IPolicy, since it is lower than the minimum disk size.

The second problem still exists when using adoption with 'plain'
and 'blockdev' template and will be addressed in another commit.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Due to an oversight, the pause/resume sync RPC calls in _WipeDisks
lack the verification of the overall RPC status, and directly iterate
over the payload. The code actually doing the wipe does verify
correctly the results. This can result in jobs failing with a hard to
diagnose:

OpExecError ['NoneType' object is not iterable]

instead of proper "RPC failed" message.

This patch adds a hard check on the pause call, but for the resume
call it just logs a warning if the RPC failed; the rationale being
that if we can't contact the node for pausing the sync, it's likely
wiping will fail too, but after the wipe has been done, we can
continue.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is the design doc for the bulk instance creation. You can more
details in the doc itself.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is a trivial code change, but it allows us to finally test the
send-receive code on both client and server sides via a simple
in-process server.

The unittest works, but it won't handle timeouts very nicely; it will
wait until the actual Luxi timeout expires, instead of using much
shorter timeouts as we could in the same process.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

I'm doing this change for future performance optimisations. Currently
we use the Luxi interface just as a client, so not in the hot path,
but when we'll use this as a server interface, we're interested to
both reduce the space and time consumption of the interface.

We have to simultaneous changes here:

- switch from using socket-related function (sendto, recv, etc.) to
  handle-based functions, since the standard network library doesn't
  work with sockets
- switch from using Strings for the internal buffer to strict
  ByteStrings; the only downside is that we now have the issue of
  decoding/encoding from binary to UTF-8 strings, a fact which brings
  its own issues into the mix (we have to check for failed decodings,
  etc.); but this is similar to what we'll have to handle on the
  Python side when moving to Python 3.x

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a bit of a shell munging trickery, but works for now. Making
it more generic can be done later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This breaks the command, as optparse considers that an error.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Due to the fact how the automake system works it doesn't rebuild already
prebuild files in distcheck. This lead to a bug, where a rebuild of the
documentation was failing because we missed the fact that the files were
missing from the archive.

By adding distrebuildcheck we workaround that issue by running a
maintainer-clean which also removes prebuild files.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The "ignore_offline_nodes" parameter is unsupported. Rather than
explicitely adding it, just pass all keyword arguments in the body of
the query, and rapi on the other side will do the right thing.

Supports for old arguments that were passed via the query is unchanged.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

When offlining an instance because its primary node is down, we must be
able to cope with the situation.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>