Common functionality between the DRBD 8.3 and DRBD 8.4 parser has been
extracted into BaseShowInfo. A test which verifies the behaviour is
included, but the DRBD84ShowInfo class is not yet used within the DRBD8
class.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Parsing the status (/proc/drbd) and `drbdsetup show` output takes quite
a bit of code, which has to be augmented by even more with the DRBD 8.4
support. So extract all the related classes into their own file.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

The current parser already supports the output of DRBD 8.4 /proc/drbd,
so add tests which actually verify this behaviour.
Note that the empty lines in proc_drbd84.txt are actually present in the
output of DRBD 8.4, they always appear when minors are not contiguous.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

The previous patch removed a (rather implicit) test for /proc/drbd not
containing a version line. This adds an explicit test for this case.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This patch add a couple of utility functions dealing with
storage types and disk templates.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is in preparation to introducing support for DRBD8.4.
base.py had to be extracted as well in order to avoid cylic imports
between bdev.py and drbd.py. It now contains the BlockDev class and
utility functions needed by bdev.py and drbd.py.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

That's in preparation of extracting DRBD related code from bdev.py. As
bdev.py is already rather long, new features will require to split it
into more manageable pieces. That's why it's moved in an own directory.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Add a unit test for hroller in a situation where the order
is completely determined. There are 3 nodes where node 2 has,
for each of the other nodes, an instanced shared with it. So
node 2 has to be a reboot group of its own. There are no more
instances, hence, my minimality, the other two nodes have to
form a single group. Node 1 is the master, and hence has to
be rebooted last; this determines the order.

Also add a unit test verifying that hroller rejects configurations
with multiple master nodes, even if the option --force is given.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This was renamed in b284f504 but the old name was still in use in the
makefile and draft designs list.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This commit adds the design document for adding UUID and name slots to NIC and
Disk objects and the ability to refer to them by these slots.

Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>

Due to a bug in 8e55e20f burnin's installation path got changed.
Moving tools/burnin to the right makefile target reverts this (while
preserving the value of the original commit).

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

We test on two minimal examples, a positive and a negative one, where
the possibility to allocate a node depends on whether the disk policy
is checked on a per-disk level or on the total amount of disk space
requested.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

When executed in split user mode, Ganeti requires a specific set of users
and groups to be created in the appropriate way.

This commit introduces a tool that takes care of that automatically.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Setting up a proper environment for building a Ganeti can be quite difficult.

This patch adds a script for building a compressed chroot with all the
required libraries and tools.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This commit adds the core infrastructure of the monitoring daemon,
and integrates it in the build and test systems.

The actual functionality of the monitoring daemon is still completely
missing.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This commit adds the design document for introducing "reason trails",
tracing the reason why opcodes are executed, step by step.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This commit adds the user and group variables for the monitoring agent in all
the configuration files where they are required.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently the man page build rule is run without ‘-e’, so the actual
checks in there are just warnings (check-man-warnings,
check-man-dashes).

The patch fixes this and also fixes the man pages which made me see the
problem.

Additionally, check-man-dashes is now verbose, otherwise it's hard to
find out actually where in the page the error is.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently we only symlink this into two hard-coded directories, which
breaks the source code viewing for all the others.

The patch replaces this with automatic building; as playing with
relative symlinks is non-trivial, I just generate it many times (the
file is small enough (<250) that it shouldn't matter, size-wise, and
it takes 2ms to do it).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This patch does multiple enhancements to the way we build the Haddock
docs, motivated by the fact that if we don't run Haddock over test
files as well, bad formatting can be submitted and will accumulate
over time (without any checks).

The patch does:

- replace manual built ALL_APIDOC_HS_DIRS with automatically built one
  (from HS_DIRS)
- change Haddock so that it runs from the top directory (instead of
  from src/)
- change HsColour target file to be built via bash parameter
  substitution, rather than sed (I don't know how to do it in one go,
  so I use 2 intermediate variables)
- change 'hs-apidoc' target so that it depends on the real target
  file; in case no source file has been modified, running 'make
  hs-apidoc' twice will not result in two runs
- run HsColour/Haddock under en_US.UTF-8 locale, otherwise they can't
  parse correctly the Unicode chars in the test files

Additionally, wrong formatting (oops) in a test file has been
corrected.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This is a simple query as it has only two fields, however it's the
first query that doesn't have a clear 'base' object and 1:1
correspondence between such objects and the results (groups, nodes and
networks do so).

We keep nodes as the 'base' object, since that's what we want to
filter on for RPC selection, and we have a very simple type as the
runtime type, since we only have one other field. The
'collectLiveData' function is the one that does the expansion from
[node] to [(node, path)], with the help of 'rpcExtractor'.

Also, this patch introduces a slightly different naming structure for
the exported names out of the module, to reduce naming conflicts.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

As we want to be able to run queries against multiple nodes in
parallel, and furthermore in parallel with other work, we need to
implement the Curl Multi interface (see libcurl-multi(3)).

This patch adds a Ganeti-specific such implementation, to be used
until upstream Curl bindings provide it. The implemented interface
(there are two 'multi' interfaces) is the older curl_multi_perform(3).

It has one downside (which is also, somewhat, and advantage): we use
polling instead of more properly select() or poll. This is due to how
waiting for FDs is implemented in Haskell: currently, it's not
possible to wait for multiple FDs at once nicely, so we'd have to fork
many threads for each FD to be watched, or alternatively one could use
FFI select, but that would block the entire runtime.

With the current poll method, the implementation achieves consistent
~100 RPC/s per second (with 10 multi interfaces running in parallel,
each with 10 easy handles), and over ~1 hour of runtime the memory
usage is stable, so memory allocation/deallocation (manual when
dealing with FFI) _seems_ to be well handled.

Future optimisations could be to move to curl_multi_socket_action(3),
which might allow better integration with the Haskell runtime.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, we support curl being optional via some sporting exercises:
ifdefs in the code, data types that represent 'Curl is disabled'
state, etc. However, with the future work on RPC, we would have to
even make the dependencies list conditional on it, etc. This is too
much work, when the curl library is included even in stable
distributions.

This patch changes curl from option to required, the same as the other
base libraries.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

The RAPI documentation now depends on “connector.py” (handler
registration) and “rlib2.py” (access permissions).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds a couple of unit tests for Query/Network.hs.
Note that they'll need to be adapted, once issue 362 is addressed.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In order to fetch precise information about the uptime of the VMs
running in Xen, we need to analyze the output of the "xm uptime" command.

This commit adds the parser to do that, and its tests.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In order to fetch precise information about the status of the VMs running in
Xen, we need to analyze the output of the "xm list --long" command.

This commit adds the parser to do that, and its tests.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is a long standing bug in Ganeti. Add a small design on how we plan
to fix this for Ganeti 2.7.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Back when this was introduced, I mentioned that it breaks heavily tab
completion (ob<TAB> doesn't work anymore), but at that moment I didn't
have a suggestion what to name it. I think outils is good and short
enough, and doesn't conflict with anything else, so here it goes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The documentation still points to /etc/ganeti/remote-commands,
although the code is already using restricted-command. Update the
documentation and a few docstrings accordingly.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add unit tests for the RADOSBlockDevice JSON output parsing function,
and modify the unit tests for the plain output parsing function to
work with the new code.

Signed-off-by: Stratos Psomadakis <psomas@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a follow-up for “qa_config: Remove exclusive storage flag from
config”. Instead of storing the flag in a module-level variable it is
now stored within the new QA configuration class and unit tests are
provided. Wrappers in “qa_config” maintain the existing interface.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

When hlint sees a perfectly nice code tree, it will skip generating
the reports (both stdout and html). This means that usually the old
report will remain in place, so it makes development harder, as the
html file is out of sync with the tree.

To workaround this, we generate an empty/minimal file in this case.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This would have caught the log file problem fixed in the previous
patch (9411474b), for example. Also we test user/group equivalence,
name only.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

harep(1) detects certain kind of problems with instances and applies the
allowed set of solutions. See doc/design-autorepair.rst.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is the beginning of the implementation of network
queries. This includes establishing all infrastructure
to run the network queries and implement querying of
some simpler fields and the node group listing.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Refactor and add tests for getting node (Domain-0) information.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This patch refactors “_RunXmList” and adds some tests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Stop hardcoding the path in “hv_xen.py”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>