GitLab

This is the initial support for Query2: basic infrastructure (except
filtering) and node query support (without RPC).

It implements all the fields (tests by comparison with list-fields on
the Python side), except that:

- filter is not done
- since RPC is not integrated yet, the runtime gathering/computing is
  simply stubbed out

However, the infrastructure seems pretty reasonable, so I'm sending as
is.

Note that I've split the functions/declarations into multiple files,
to keep each file clean and readable.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch corrects the definitions in Qlang.hs to match what Python
expects on the wire; this means replacing some manual data type
definitions with 'buildObject' so that we get serialisation (and field
names) for free, adding (manually) JSON instances for types which are
not represented as objects in JSON, etc. Due to more TH usage, I had
to shift some definitions around, since TH breaks the "define in any
order" property (☹

).

After that, we simply add a call into the stub Query/Query.hs module
which, for all queries, responds with "query not supported". The
reason for the deep directory structure is because I want to separate
the functionality into multiple submodules, for smaller module size
and readability.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

The Qlang module defines ResultStatus, but it was already defined in
Ganeti/Luxi.hs; let's remove the duplicate definition from there since
the proper place is in the newer module.

Also, in order to ease testing, we split some confd functions into a
separate module; this can be imported easily into QC for testing.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

And expand cli.GetClient() to allow opening the query socket, instead
of the main master socket.

Finally, enable the query socket use in gnt-cluster version, since
that is already implemented fully in Queryd.hs/hconfd.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

While starting to use the new filter types, I realised that what is
currently implemented is the equivalent of `lib/qlang.py', not
`lib/query.py', since we only deal with data types for now and not the
actual query runtime functionality (RPC, config, etc.).

Let's rename the file to be more consistent with the Python code base.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Commit 5a1e31b4

 (Add infrastructure for, and two extra hlint rules)
was intended to add two *extra* hlint rules, but I didn't realise at
that time that "--hint" when first used overrides the built-in
lints. As such, since then we were basically running with just those
two rules, which resulted in many uncaught warnings/errors.

This patch fixes that (by importing the standard lint rules in our
custom hints file), and then goes to fix all the warnings that a
current hlint gives me. Compared to our current style, we have just a
few additions:

- zipWithM instead of map foo . zip …
- 'exitSuccess' instead of 'exitWith ExitSuccess'
- more uses of '.'

Additionally, we have to silence a case where hlint doesn't realise
why we are using '\e -> const (return False (e :: IOError)' instead of
just '\e -> return False' or even 'const (return False').

One warning that is generated by hlint ("Use void") can't be fixed
until we deprecate GHC 6.x, as only GHC 7 has the 'void' function in
Control.Monad.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is just a new module that exports a runQueryD function, that can
be imported to run a separate thread handling the luxi requests.

Currently it needs access just to the configuration, in the future it
will need access to an RPC runner too.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

We introduce typeclasses for RPC call and result and create a typeclass
that binds the two together. For that we need to use
MultiParamTypeClasses and FunctionalDependencies language pragmas, which
allow us to ensure that RPC result type can be deduced based on the
RPC call type.
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is done so that all current branches can run with newer pep8;
note that instead of fixing the problems (like I did on master), I've
just silenced more. These should *not* be merged onto master!
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:

- bump version in the docs
- silence some new checks that are wrong due to our indent=2 instead of 4
- fix lots of errors in the code where the indentation was wrong by 1
  or 2 spaces
- fix a few cases of == True, False, None and replace with 'is'
- re-indent some cases where the code is OK, but pep8 complains
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is the design doc for the bulk instance creation. You can more
details in the doc itself.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Due to the fact how the automake system works it doesn't rebuild already
prebuild files in distcheck. This lead to a bug, where a rebuild of the
documentation was failing because we missed the fact that the files were
missing from the archive.

By adding distrebuildcheck we workaround that issue by running a
maintainer-clean which also removes prebuild files.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is just begging to be converted to a standard replace_vars_sed
rule, instead of custom sed calls.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A rule of type "a/%: a/%.in" will also match "a/b/%: a/b/%.in", so no
need for the explicit examples/hooks rule. As for the man rules, they
are identical and thus can be collapsed.

We still have the problem that globally, not all our %.in to %
transformations are identical; this is suboptimal and should be
cleaned sometime…
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Generic rules like:

  %: %.in

have the downside that the source (%.in) itself matches the target
(via %.in: %.in.in). This leads to things like:

  Looking for a rule with intermediate file `doc/examples/hooks/ipsec.in.in'.
   Trying pattern rule with stem `ipsec.in.in'.
   Trying implicit prerequisite `doc/examples/hooks/ipsec.in.in.in'.
   Trying pattern rule with stem `ipsec.in.in'.
   Trying implicit prerequisite `doc/examples/hooks/ipsec.in.in.in'.
   Looking for a rule with intermediate file `doc/examples/hooks/ipsec.in.in.in'.
  Rejecting impossible implicit prerequisite `doc/examples/hooks/ipsec.in.in'.

To fix this, we need to tell make that such rules are terminating, so
that it doesn't recurse into them.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

GNU Make contains some (ancient) implicit rules, that try to
_automatically_ extract source files from RCS/SCCS version control
systems. This is unneeded, and it pollutes the make -d output
significantly: after removing these rules (by defining empty targets
for their patterns), make -d line count goes from 5.3K to 3.3K.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit dc7d2c49 introduced the use of BUILT_SOURCES to work around
missing dependencies. However, on closer reading of the gmake manual,
BUILT_SOURCES is mainly used in cases where the dependencies are not
know before the build starts (e.g. with auto-generated C header
files). Additionally, there are a number of drawbacks to
BUILT_SOURCES, which we already had to work around (e.g. in commit
eb732fb5

).

Since we know all our dependencies statically, there's no need to use
this special variable. Let's rename it to GENERATED_FILES, which
doesn't have a special meaning to make, and add a few missing
dependencies (one of which was already broken by make -j on a clean
tree).

After this change, running make in a fully built tree is finally "clean":

  $ make
  make: Nothing to be done for `all'.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, if one runs 'make' in an already fully-built tree, this is
the result:

  cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; }
  make  all-am
  make[1]: Entering directory `/tmp/test'
  cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; }
  make[1]: Leaving directory `/tmp/test'

This is because commit dc7d2c49

 added 'ganeti' (which is a PHONY
target) to BUILT_SOURCES, and since that is a dependency of other,
real targets, it means the ganeti target is always remade.

To fix this, we keep ganeti as a PHONY target, but we remove it from
the 'built_base_sources' target, and instead we only remake it
manually in the stamp-directories target. A make run now is just:

  make  all-am
  make[1]: Entering directory `/tmp/test'
  make[1]: Nothing to be done for `all-am'.
  make[1]: Leaving directory `/tmp/test'

Note that we can't get rid of the all-am since we use BUILT_SOURCES.

We also remove the comment of BUILT_SOURCES since it no longer depends
on PHONY targets.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit c964d962

 changed the way we create directories, by two things:

- unifying all dependencies and ad-hoc directory creation into a
  single target (all_dirfiles)
- changing how directories are created from a stamp file to .dir files
  in each directory

The first item is a very good one, but the second item is debatable:
there's no per-se advantage of .dir files versus a single one,
top-level, since both the .dir file and stamp-directories creation are
depending on Makefile, which is the only one which can introduce new
directories.

On the other hand, moving back from .dir files to stamp-directories
has an advantage: "make -d | wc -l" does from ~8.7K lines to ~5.3K
lines, because we eliminate the many .dir files and their multiple
implicit and explicit dependencies (the %/.dir files fall under
multiple patterns).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Seen while debugging make rules.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Reading the automake documentation, it seems that explicit calls to
$(MAKE) need $(AM_MAKEFLAGS) passed, so let's do that.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

There are two current issues with the coverage values:

- we don't import all modules, thus leading to incomplete
  coverage results (too optimistic);
- we use hpc in its default mode (intersection), which means that even
  modules which do have coverage results but are not used in all
  binaries we test will be dropped from the results; thanks to Agata,
  passing --union to hpc is enough to have better results (don't
  remember why this wasn't there in the first place…)

After adding more modules to the import list and fixing the combining
mode, we now have a complete list of modules in coverage results, many
with zero coverage, so our overall coverage has dropped to about 60%.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Note however that this won't work correctly with older compilers, due
to ghc issue #4256; however, it's a handy way to build complete a TAGS
file for Emacs.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, we build the Haskell unittests with custom GHC flags,
because we (I) were quite lazy when initially writing the
unittests. This is not a problem for the tests themselves, but it
creates problem when (for example) one would want to pass all
$(HS_LIB_SRCS) to the compilers; this is not doable unless we
"degrade" the flags used for all modules, instead of just for QC.hs.

So we do two things to fix this:

- first, we go and add type declaration to all functions that were
  missing them (in QC), and fix the couple of cases of monomorphism
  restrictions; this gets us rid of -fno-warn-missing-signature and
  -fno-warn-monomorphism-restriction
- then, we move the actually important remaining options
  (-fno-warn-orphans and -fno-warn-unused-imports; see the explanation
  for the latter in the newly added FIXME) to a compiler pragma in the
  file, so that when building the unittests only this file is using
  the extra options

And finally we can then drop the other unused options
(-fno-warn-missing-methods anmd -Wwarn), leaving htools/test use
simply -fhpc. This is more in-line with the other files, and thus we
can handle all of HS_LIB_SRCS the same.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is trivial, but can be used easily from automated builds to check
that the git tree is clean: files are not modified and untracked files
are not present.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This patch works around an issue in our build system. Since we don't
use cabal or 'ghc -M', we don't track actual dependencies in our
Makefile; this in turn means that editing a file that only is used in
the main 'htools' binary will keep triggering 'ghc --make' for the
hconfd binary; and this actually takes a bit of time to look at all
the dependencies.

So to work around this, we add a touch after each ghc --make, such
that we trigger the unneeded build at most once.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

The code used a space, not a tab, in Makefile.am, which means it
actually broke the variable declaring the man pages.

Furthermore, we add some blank lines in the man page, otherwise the
paragraphs will be run-in together.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is not perfect, as we only test that hbal completes successfully
and that it show a score improvement, but it's better than nothing.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Introduce infrastructure required to add Hcheck and build it
successfuly.
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Add unittests for FormatJobID, GetArchiveDirectory, ParseJobId.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This contains the types to express query2 queries, filters and results.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Due to the way $(MAKE) behaves (it forces execution of the commands
that contain it), use of $(MAKE) in a command line together with
side-effects commands will break stuff. It's better to keep it either
on a separate line, or if not possible, move said commands to a
separate make target.

After this patch, make -n distcheck, make -n regen-vcs-version and
possibly other commands now work correctly.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the lint target does a sequential: pep8, pylint, pylint on
the QA sources. hlint is run only when explicitly requested.

This patch reorganises the lint target by:

- splitting the current lint target into separate pylint (slow),
  pylint-qa (fast), pep8 (fast) targets
- making lint itself depend on the above plus hlint
- both pep8 and hlint are optional, based on whether the said binaries
  were detected at configure time

make -j is slightly faster after this patch (1m18s → 1m7s).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

During the preparations for 2.6, we cleaned up the codebase to use
newer exceptions (e.g. commits 30d25dd8, 79ac58fa

), so we can now drop
this override.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is the 2.5 version of the "fix build failures":

- man/%.gen could be left over even in case of failure, due to
  automake bug
- make man/%.gen runs RUN_IN_TEMPDIR, so let's depend on it, since
  that target has the proper dependencies (create needed dirs)
- man/%.gen depends on a number of built sources, but the dependency
  was not declared

Furthermore, wraps a long comment.

Tested with -j4/-j16, after `make maintainer-clean'.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is needed so we can specify the xen command upon configure time
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is trivial, but should prevent such things in the future.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Sorry, forgot this in previous commit.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
(cherry picked from commit 1a1e7ab3

)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The vgreduce command has changed behaviour from when we initially
wrote the code (2.02.02 versus 2.02.66, 4 years delta):

- if there are LVs which will be impacted, it requires --force
- otherwise refuses to proceed, but it still returns exit code 0

We handle this by looking to see if it returns "Wrote out consistent
volume group" (behaviour unchanged), or if it complains about
"--force"; in the case it didn't complete, we retry the operation.

We improve a bit the checking of "vgs", as it uses to fail silently
and we didn't detect it.

New tests for this function should test, I believe, all the expected
variations; at the least we now have data files with the expected
output.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
(cherry picked from commit 048eeb2b

)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>