GitLab

Except for Ganeti.HTools.JSON, which needs rename, we split all the
other test suites into separate files.

We have to add another common test helper, due to import dependencies
(sigh), but otherwise this split is pretty straightforward.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

… from QC.hs into their own files, again mirroring the production code
source tree.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This required lots of other code moves, so I created it as a
standalone patch.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This splits the confd/utils tests, and adds the TestCommon module for
shared test code.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is the first file split out from QC.hs - an easy one, since it
has just one test.

The patch changes the way we build hpc excludes, since now we'll have
many modules that need to be excluded, and hpc doesn't seem to be able
to do wildcards (it can do packages, but all our code is
not-package-based).

Further splitting will be done in bigger batches.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch starts the move of the test haskell code from
`htools/Ganeti/HTools/' to its more proper place of
`htest/Test/Ganeti'.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This moves the last (I think) htools-related bits out of test/ under
htest/.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is another rather trivial patch, moving all the (htools)
shelltests to their own directory.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Following up on the program moves, we now move the test data files.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is the first commit of a series that will attempt to cleanup the
test code organisation, which evolved somewhat organically from the
initial pure htools functionality.

The proposed organisation of the tree will be as follows:

- htools (or maybe renamed to haskell or hs): only production code
- htest: top-level test directory, containing test.hs, static helper
  scripts, etc.
- htest/Ganeti/*.hs: modules implementing the actual test properties
  and test cases for the correspondingly-named production code modules
- htest/data: containing test data files for the test cases

This particular patch moves all the test code (test.hs, hpc-htools.hs
symlink) and helper scripts (offline-test.sh, etc.) from htools/ to
htest/, while updating the files themselves (if they had paths
mentioning htools/), .gitignore and the Makefile.

The only special mention is that in Makefile, we used to have a BINARY
shell variable in binary build rule; that was computed via stripping
`htools/' prefix; I've cleaned that and replaced with $(notdir $@);
even though it's duplicated a few times, it leads to more readable
make output (and easier to copy-paste).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Per the new query module hierarchy, rename Qlang to Query/Language and
Queryd to Query/Server. This way, all query-related functionality is
now "contained" in the Query/ directory.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This adds basic infrastructure for filtering (fully functional except,
as usual, for runtime data), and then uses it for node queries.

Since the filtering exports regex matching as an external
functionality, we have to use a regex library. There are many flavours
of these in Haskell (see
http://www.haskell.org/haskellwiki/Regular_expressions

), but since we
want to be as compatible as we can with Python's, we use the
regex-pcre one, which is a wrapper to the widely used 'pcre' library.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This design describes a tool that will perform automatic repairs on
instances when they are detected to be unhealthy (living on offline or
drained nodes, at the moment). These repairs can be scheduled
automatically or requested as a one-off by a tool or person.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is the initial support for Query2: basic infrastructure (except
filtering) and node query support (without RPC).

It implements all the fields (tests by comparison with list-fields on
the Python side), except that:

- filter is not done
- since RPC is not integrated yet, the runtime gathering/computing is
  simply stubbed out

However, the infrastructure seems pretty reasonable, so I'm sending as
is.

Note that I've split the functions/declarations into multiple files,
to keep each file clean and readable.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch corrects the definitions in Qlang.hs to match what Python
expects on the wire; this means replacing some manual data type
definitions with 'buildObject' so that we get serialisation (and field
names) for free, adding (manually) JSON instances for types which are
not represented as objects in JSON, etc. Due to more TH usage, I had
to shift some definitions around, since TH breaks the "define in any
order" property (☹

).

After that, we simply add a call into the stub Query/Query.hs module
which, for all queries, responds with "query not supported". The
reason for the deep directory structure is because I want to separate
the functionality into multiple submodules, for smaller module size
and readability.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

The Qlang module defines ResultStatus, but it was already defined in
Ganeti/Luxi.hs; let's remove the duplicate definition from there since
the proper place is in the newer module.

Also, in order to ease testing, we split some confd functions into a
separate module; this can be imported easily into QC for testing.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

And expand cli.GetClient() to allow opening the query socket, instead
of the main master socket.

Finally, enable the query socket use in gnt-cluster version, since
that is already implemented fully in Queryd.hs/hconfd.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

While starting to use the new filter types, I realised that what is
currently implemented is the equivalent of `lib/qlang.py', not
`lib/query.py', since we only deal with data types for now and not the
actual query runtime functionality (RPC, config, etc.).

Let's rename the file to be more consistent with the Python code base.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Commit 5a1e31b4

 (Add infrastructure for, and two extra hlint rules)
was intended to add two *extra* hlint rules, but I didn't realise at
that time that "--hint" when first used overrides the built-in
lints. As such, since then we were basically running with just those
two rules, which resulted in many uncaught warnings/errors.

This patch fixes that (by importing the standard lint rules in our
custom hints file), and then goes to fix all the warnings that a
current hlint gives me. Compared to our current style, we have just a
few additions:

- zipWithM instead of map foo . zip …
- 'exitSuccess' instead of 'exitWith ExitSuccess'
- more uses of '.'

Additionally, we have to silence a case where hlint doesn't realise
why we are using '\e -> const (return False (e :: IOError)' instead of
just '\e -> return False' or even 'const (return False').

One warning that is generated by hlint ("Use void") can't be fixed
until we deprecate GHC 6.x, as only GHC 7 has the 'void' function in
Control.Monad.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is just a new module that exports a runQueryD function, that can
be imported to run a separate thread handling the luxi requests.

Currently it needs access just to the configuration, in the future it
will need access to an RPC runner too.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

We introduce typeclasses for RPC call and result and create a typeclass
that binds the two together. For that we need to use
MultiParamTypeClasses and FunctionalDependencies language pragmas, which
allow us to ensure that RPC result type can be deduced based on the
RPC call type.
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is done so that all current branches can run with newer pep8;
note that instead of fixing the problems (like I did on master), I've
just silenced more. These should *not* be merged onto master!
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:

- bump version in the docs
- silence some new checks that are wrong due to our indent=2 instead of 4
- fix lots of errors in the code where the indentation was wrong by 1
  or 2 spaces
- fix a few cases of == True, False, None and replace with 'is'
- re-indent some cases where the code is OK, but pep8 complains
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is the design doc for the bulk instance creation. You can more
details in the doc itself.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Due to the fact how the automake system works it doesn't rebuild already
prebuild files in distcheck. This lead to a bug, where a rebuild of the
documentation was failing because we missed the fact that the files were
missing from the archive.

By adding distrebuildcheck we workaround that issue by running a
maintainer-clean which also removes prebuild files.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is just begging to be converted to a standard replace_vars_sed
rule, instead of custom sed calls.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A rule of type "a/%: a/%.in" will also match "a/b/%: a/b/%.in", so no
need for the explicit examples/hooks rule. As for the man rules, they
are identical and thus can be collapsed.

We still have the problem that globally, not all our %.in to %
transformations are identical; this is suboptimal and should be
cleaned sometime…
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Generic rules like:

  %: %.in

have the downside that the source (%.in) itself matches the target
(via %.in: %.in.in). This leads to things like:

  Looking for a rule with intermediate file `doc/examples/hooks/ipsec.in.in'.
   Trying pattern rule with stem `ipsec.in.in'.
   Trying implicit prerequisite `doc/examples/hooks/ipsec.in.in.in'.
   Trying pattern rule with stem `ipsec.in.in'.
   Trying implicit prerequisite `doc/examples/hooks/ipsec.in.in.in'.
   Looking for a rule with intermediate file `doc/examples/hooks/ipsec.in.in.in'.
  Rejecting impossible implicit prerequisite `doc/examples/hooks/ipsec.in.in'.

To fix this, we need to tell make that such rules are terminating, so
that it doesn't recurse into them.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

GNU Make contains some (ancient) implicit rules, that try to
_automatically_ extract source files from RCS/SCCS version control
systems. This is unneeded, and it pollutes the make -d output
significantly: after removing these rules (by defining empty targets
for their patterns), make -d line count goes from 5.3K to 3.3K.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit dc7d2c49 introduced the use of BUILT_SOURCES to work around
missing dependencies. However, on closer reading of the gmake manual,
BUILT_SOURCES is mainly used in cases where the dependencies are not
know before the build starts (e.g. with auto-generated C header
files). Additionally, there are a number of drawbacks to
BUILT_SOURCES, which we already had to work around (e.g. in commit
eb732fb5

).

Since we know all our dependencies statically, there's no need to use
this special variable. Let's rename it to GENERATED_FILES, which
doesn't have a special meaning to make, and add a few missing
dependencies (one of which was already broken by make -j on a clean
tree).

After this change, running make in a fully built tree is finally "clean":

  $ make
  make: Nothing to be done for `all'.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, if one runs 'make' in an already fully-built tree, this is
the result:

  cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; }
  make  all-am
  make[1]: Entering directory `/tmp/test'
  cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; }
  make[1]: Leaving directory `/tmp/test'

This is because commit dc7d2c49

 added 'ganeti' (which is a PHONY
target) to BUILT_SOURCES, and since that is a dependency of other,
real targets, it means the ganeti target is always remade.

To fix this, we keep ganeti as a PHONY target, but we remove it from
the 'built_base_sources' target, and instead we only remake it
manually in the stamp-directories target. A make run now is just:

  make  all-am
  make[1]: Entering directory `/tmp/test'
  make[1]: Nothing to be done for `all-am'.
  make[1]: Leaving directory `/tmp/test'

Note that we can't get rid of the all-am since we use BUILT_SOURCES.

We also remove the comment of BUILT_SOURCES since it no longer depends
on PHONY targets.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit c964d962

 changed the way we create directories, by two things:

- unifying all dependencies and ad-hoc directory creation into a
  single target (all_dirfiles)
- changing how directories are created from a stamp file to .dir files
  in each directory

The first item is a very good one, but the second item is debatable:
there's no per-se advantage of .dir files versus a single one,
top-level, since both the .dir file and stamp-directories creation are
depending on Makefile, which is the only one which can introduce new
directories.

On the other hand, moving back from .dir files to stamp-directories
has an advantage: "make -d | wc -l" does from ~8.7K lines to ~5.3K
lines, because we eliminate the many .dir files and their multiple
implicit and explicit dependencies (the %/.dir files fall under
multiple patterns).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Seen while debugging make rules.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Reading the automake documentation, it seems that explicit calls to
$(MAKE) need $(AM_MAKEFLAGS) passed, so let's do that.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

There are two current issues with the coverage values:

- we don't import all modules, thus leading to incomplete
  coverage results (too optimistic);
- we use hpc in its default mode (intersection), which means that even
  modules which do have coverage results but are not used in all
  binaries we test will be dropped from the results; thanks to Agata,
  passing --union to hpc is enough to have better results (don't
  remember why this wasn't there in the first place…)

After adding more modules to the import list and fixing the combining
mode, we now have a complete list of modules in coverage results, many
with zero coverage, so our overall coverage has dropped to about 60%.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Note however that this won't work correctly with older compilers, due
to ghc issue #4256; however, it's a handy way to build complete a TAGS
file for Emacs.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, we build the Haskell unittests with custom GHC flags,
because we (I) were quite lazy when initially writing the
unittests. This is not a problem for the tests themselves, but it
creates problem when (for example) one would want to pass all
$(HS_LIB_SRCS) to the compilers; this is not doable unless we
"degrade" the flags used for all modules, instead of just for QC.hs.

So we do two things to fix this:

- first, we go and add type declaration to all functions that were
  missing them (in QC), and fix the couple of cases of monomorphism
  restrictions; this gets us rid of -fno-warn-missing-signature and
  -fno-warn-monomorphism-restriction
- then, we move the actually important remaining options
  (-fno-warn-orphans and -fno-warn-unused-imports; see the explanation
  for the latter in the newly added FIXME) to a compiler pragma in the
  file, so that when building the unittests only this file is using
  the extra options

And finally we can then drop the other unused options
(-fno-warn-missing-methods anmd -Wwarn), leaving htools/test use
simply -fhpc. This is more in-line with the other files, and thus we
can handle all of HS_LIB_SRCS the same.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is trivial, but can be used easily from automated builds to check
that the git tree is clean: files are not modified and untracked files
are not present.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This patch works around an issue in our build system. Since we don't
use cabal or 'ghc -M', we don't track actual dependencies in our
Makefile; this in turn means that editing a file that only is used in
the main 'htools' binary will keep triggering 'ghc --make' for the
hconfd binary; and this actually takes a bit of time to look at all
the dependencies.

So to work around this, we add a touch after each ghc --make, such
that we trigger the unneeded build at most once.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>