GitLab

This commit adds the design document for introducing "reason trails",
tracing the reason why opcodes are executed, step by step.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This commit adds the user and group variables for the monitoring agent in all
the configuration files where they are required.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently the man page build rule is run without ‘-e’, so the actual
checks in there are just warnings (check-man-warnings,
check-man-dashes).

The patch fixes this and also fixes the man pages which made me see the
problem.

Additionally, check-man-dashes is now verbose, otherwise it's hard to
find out actually where in the page the error is.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently we only symlink this into two hard-coded directories, which
breaks the source code viewing for all the others.

The patch replaces this with automatic building; as playing with
relative symlinks is non-trivial, I just generate it many times (the
file is small enough (<250) that it shouldn't matter, size-wise, and
it takes 2ms to do it).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This patch does multiple enhancements to the way we build the Haddock
docs, motivated by the fact that if we don't run Haddock over test
files as well, bad formatting can be submitted and will accumulate
over time (without any checks).

The patch does:

- replace manual built ALL_APIDOC_HS_DIRS with automatically built one
  (from HS_DIRS)
- change Haddock so that it runs from the top directory (instead of
  from src/)
- change HsColour target file to be built via bash parameter
  substitution, rather than sed (I don't know how to do it in one go,
  so I use 2 intermediate variables)
- change 'hs-apidoc' target so that it depends on the real target
  file; in case no source file has been modified, running 'make
  hs-apidoc' twice will not result in two runs
- run HsColour/Haddock under en_US.UTF-8 locale, otherwise they can't
  parse correctly the Unicode chars in the test files

Additionally, wrong formatting (oops) in a test file has been
corrected.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This is a simple query as it has only two fields, however it's the
first query that doesn't have a clear 'base' object and 1:1
correspondence between such objects and the results (groups, nodes and
networks do so).

We keep nodes as the 'base' object, since that's what we want to
filter on for RPC selection, and we have a very simple type as the
runtime type, since we only have one other field. The
'collectLiveData' function is the one that does the expansion from
[node] to [(node, path)], with the help of 'rpcExtractor'.

Also, this patch introduces a slightly different naming structure for
the exported names out of the module, to reduce naming conflicts.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

As we want to be able to run queries against multiple nodes in
parallel, and furthermore in parallel with other work, we need to
implement the Curl Multi interface (see libcurl-multi(3)).

This patch adds a Ganeti-specific such implementation, to be used
until upstream Curl bindings provide it. The implemented interface
(there are two 'multi' interfaces) is the older curl_multi_perform(3).

It has one downside (which is also, somewhat, and advantage): we use
polling instead of more properly select() or poll. This is due to how
waiting for FDs is implemented in Haskell: currently, it's not
possible to wait for multiple FDs at once nicely, so we'd have to fork
many threads for each FD to be watched, or alternatively one could use
FFI select, but that would block the entire runtime.

With the current poll method, the implementation achieves consistent
~100 RPC/s per second (with 10 multi interfaces running in parallel,
each with 10 easy handles), and over ~1 hour of runtime the memory
usage is stable, so memory allocation/deallocation (manual when
dealing with FFI) _seems_ to be well handled.

Future optimisations could be to move to curl_multi_socket_action(3),
which might allow better integration with the Haskell runtime.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, we support curl being optional via some sporting exercises:
ifdefs in the code, data types that represent 'Curl is disabled'
state, etc. However, with the future work on RPC, we would have to
even make the dependencies list conditional on it, etc. This is too
much work, when the curl library is included even in stable
distributions.

This patch changes curl from option to required, the same as the other
base libraries.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

The RAPI documentation now depends on “connector.py” (handler
registration) and “rlib2.py” (access permissions).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds a couple of unit tests for Query/Network.hs.
Note that they'll need to be adapted, once issue 362 is addressed.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In order to fetch precise information about the uptime of the VMs
running in Xen, we need to analyze the output of the "xm uptime" command.

This commit adds the parser to do that, and its tests.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In order to fetch precise information about the status of the VMs running in
Xen, we need to analyze the output of the "xm list --long" command.

This commit adds the parser to do that, and its tests.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is a long standing bug in Ganeti. Add a small design on how we plan
to fix this for Ganeti 2.7.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Back when this was introduced, I mentioned that it breaks heavily tab
completion (ob<TAB> doesn't work anymore), but at that moment I didn't
have a suggestion what to name it. I think outils is good and short
enough, and doesn't conflict with anything else, so here it goes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The documentation still points to /etc/ganeti/remote-commands,
although the code is already using restricted-command. Update the
documentation and a few docstrings accordingly.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add unit tests for the RADOSBlockDevice JSON output parsing function,
and modify the unit tests for the plain output parsing function to
work with the new code.
Signed-off-by: Stratos Psomadakis <psomas@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a follow-up for “qa_config: Remove exclusive storage flag from
config”. Instead of storing the flag in a module-level variable it is
now stored within the new QA configuration class and unit tests are
provided. Wrappers in “qa_config” maintain the existing interface.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

When hlint sees a perfectly nice code tree, it will skip generating
the reports (both stdout and html). This means that usually the old
report will remain in place, so it makes development harder, as the
html file is out of sync with the tree.

To workaround this, we generate an empty/minimal file in this case.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This would have caught the log file problem fixed in the previous
patch (9411474b), for example. Also we test user/group equivalence,
name only.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

harep(1) detects certain kind of problems with instances and applies the
allowed set of solutions. See doc/design-autorepair.rst.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is the beginning of the implementation of network
queries. This includes establishing all infrastructure
to run the network queries and implement querying of
some simpler fields and the node group listing.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Refactor and add tests for getting node (Domain-0) information.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This patch refactors “_RunXmList” and adds some tests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Stop hardcoding the path in “hv_xen.py”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

If the placeholder has exactly the same name as the variable used in
autoconf it would be replaced as well. In the case of
“@ENABLE_SPLIT_QUERY@” this led to the sed command “s#False#False#g”.
Inserting additional quotes after the first @ sign prevents this.

Since ENABLE_SPLIT_QUERY was not actually used it is removed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Documentation with included man pages should not be distributed as it
contains fixed paths. Since the documentation with man pages is actually
written to a separate directory, tarball creation is still possible.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

To preserve disk space on copying all files to a temporary directory,
the “doc” directory is only copied when building documentation. A
comment describing the need is also added to Makefile.am, as well as a
superfluous direct invocation of Bash removed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This commit adds the part of the JSON report generation code that will be
common to all the data collectors, according to the format specified in the
design document.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

When new a new disk template is added, burnin should be updated.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Similar to other scripts, “tools/burnin” just becomes a wrapper script.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch depends on “Option to include man pages in documentation”. In
the documentation build including man pages, all “:manpage:`…`”
references are converted to links. For man pages not provided by Ganeti,
Sphinx' standard formatting is used.

A small dance is necessary to hook into Sphinx' processing of man page
roles and to generate automatically resolved links. The code converts
“:manpage:`…`” for known man pages to the data structure equivalent of
“:doc:`$name($section) <man-$name>`”. Additionally it checks the section
numbers and formatting of references (in all builds).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Before this patch, HTML versions of man pages (man/*.rst) were already
built. However, since they are separate from the normal documentation,
their content is not indexed for Sphinx' search functionality.
Additionally it would simply be nice to have everything in one place.

To this end a new configure-time option is added to enable the inclusion
of man pages into the documentation. A dedicated option is necessary to
still be able to provide a static documentation build in the tarball
(not including man pages) as man pages contain build-specific paths and
values. The documentation with man pages is written to the directory
“doc/man-html”.

A future patch will extend Sphinx to link occurences of “:manpage:`…`”
to these man pages.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

One is the --help output for kvm 1.1.2 and the second one is the same as
0.9.1 but with a fake option added between -drive and its boot=on|off
option: this tests that if boot=on|off appears in another option after
-drive it is not accepted by our regexp.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

When man pages should be included they need to be copied from man/*.rst.
This means documentation can no longer be built from the static reST
files alone (which are referenced using “abs_top_srcdir”). Similar to
Python files, automake doesn't copy or link the input files for
documentation into the build tree.

This patch adds all files required to build the documentation to
“srclink_files”, renames the “docrst" variable to “docinput”, and then
references the files in the build tree using “abs_top_builddir”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This script checks for some of the most obvious mistakes when formatting
man page references (which should have the form “**ganeti**\(7)”). While
this works now, it is very hard to avoid ambiguities (e.g. references
within verbatim blocks) when using regular expressions.

Also fixes a typo in Makefile.am by replacing “harcoded” with
“hardcoded”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit dbee5c92 had a wrong path.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Based on MediaWiki's monobook skin.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Also amended a comment of a tested method.
The tests are for methods introduced in commit 63c73073 (LVM disk creation
uses dedicated PVs).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This commits enables the DRBD data collector to use the Confd client to
gather information about the pairing between DRBD minors and instances.

For testing purposes, the DRBD data collector now requires either zero
or two parameters: one is the DRBD file, one is the pairings file.
When no parameter is passed, the collector is in "production mode" and takes
the data from /proc/drbd and from the Confd client.

The shell tests of mon-collector are updated accordingly.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>