GitLab

One is the --help output for kvm 1.1.2 and the second one is the same as
0.9.1 but with a fake option added between -drive and its boot=on|off
option: this tests that if boot=on|off appears in another option after
-drive it is not accepted by our regexp.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

When man pages should be included they need to be copied from man/*.rst.
This means documentation can no longer be built from the static reST
files alone (which are referenced using “abs_top_srcdir”). Similar to
Python files, automake doesn't copy or link the input files for
documentation into the build tree.

This patch adds all files required to build the documentation to
“srclink_files”, renames the “docrst" variable to “docinput”, and then
references the files in the build tree using “abs_top_builddir”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This script checks for some of the most obvious mistakes when formatting
man page references (which should have the form “**ganeti**\(7)”). While
this works now, it is very hard to avoid ambiguities (e.g. references
within verbatim blocks) when using regular expressions.

Also fixes a typo in Makefile.am by replacing “harcoded” with
“hardcoded”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit dbee5c92 had a wrong path.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Based on MediaWiki's monobook skin.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Also amended a comment of a tested method.
The tests are for methods introduced in commit 63c73073 (LVM disk creation
uses dedicated PVs).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This commits enables the DRBD data collector to use the Confd client to
gather information about the pairing between DRBD minors and instances.

For testing purposes, the DRBD data collector now requires either zero
or two parameters: one is the DRBD file, one is the pairings file.
When no parameter is passed, the collector is in "production mode" and takes
the data from /proc/drbd and from the Confd client.

The shell tests of mon-collector are updated accordingly.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit 6aff0514 removed an intermediate step, but forgot to add
“footer.rst” as a direct dependency.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This removes entirely the standalone dot files by inlining them in the
RST documents - as they weren't referenced from more than one document
either way.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The first dot files we added way back in the 2.0 release are managed
manually: we build the .pngs ourselves, track them in .gitingore,
etc. Since then, we have switched to allow sphinx to build/manage them
itself, so it makes sense to cleanup the build system and covert
everything to this simpler method.

This patch simply converts from "image: x.png" to "graphviz: x.dot",
as an externally-referenced dot file, and removes all the custom
makefile rules for png conversion. The next patch will bring the
conversion further (by inlining then).

Additionally, while checking that the patch doesn't change the html
output, I saw that the use of "shape=diamond" with non-trivial labels
results in very ugly diagrams; so we split the question in such
decision boxes to make them more reasonably looking.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This makes the (very dull) html version of the man pages a bit more
readable, until such a time that we can switch over to sphinx fully.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

I don't remember why I introduced these when switching to pandoc, but
they are not absolutely needed. Instead of pre-generating them (which
might save some trivial cpu time), we can simply pass footer.rst as an
additional input file to pandoc.

The advantage in doing so is that semantic information is preserved;
when pre-generating and including them as-is, the entire structure is
lost, as pandoc won't reinterpret them, so things like a potential
table of contents will simply ignore the footer.

The patch also adds a new target to rebuild all man pages (both groff
and html formats), for testing changes to the pages easier.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

When working on documentation it can be helpful to use a browser. In
some environments it's not possible to access the files directly from a
graphical browser. This trivial webserver exports all files in the
current directory and unlike alternatives such as thttpd, doesn't check
for world-readable permissions.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit acaf3ca7 missed this space before a backslash.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The mon-collector binary was erroneously installed twice: once in
/usr/bin/ and once in /usr/lib/ganeti/.
With this modification, it is now installed in /usr/lib/ganeti/ only,
as intended.

In order to do this in a proper way, some parts of the Makefile.am where
refactored.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This removes one more tab conflict; this is the last module in our
code where we have both x.hs and x/.

Furthermore, we collapse all actual code into the new Main.hs module,
leaving the htools.hs basically empty (will allow better testing in
the future).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

- gen-coverage → gen-py-coverage
- coverage-htools.tix → coverage-hs.tix
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, both the unit test and the shell tests are run (manually)
from this rule. This has two downsides: you can't (compile and) run
just one of them easily, and make can't run them in parallel.

By splitting the rules, we can both of the above easily.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The current base name of 'test' means that this binary will generate
profiling, coverage, etc. files with the name 'test.*', result in
tab-completion conflicts with the test directory. Let's slightly
change its base name for typing easiness.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

htest/data becomes test/data/htools (basically reverting commit
8feabc89), and htest/* becomes test/hs/*.

Most changes beside the rename are trivial s/…/…, with the exception
of autotools/run-in-tempdir, which needed some more changes now that
test/ is not just Python files:

- test/py is still being copied
- test/hs moves from individual symlinks to entire dir symlink
- test/data is symlinked in its entirety

Checked with make distcheck, pep8 and pylint, so at least VPATH builds
are OK.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is the first step of the test files reorganisation: moving test/*
(except test/data) to new directory test/py/.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This continues the cleanup of the 'htools' name, by replacing 'htools'
with 'hs' in configure and makefile rules.

There is still 'WANT_HTOOLS' left as it is correct, but some of the
conditionals in Makefile.am will need some cleanup (in a different
patch).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Since we call into Python for opcodes and other type compatibility
checks, we need to ensure that the python sources are
built. Otherwise, full parallel make might or might not break,
depending on timing.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Per offline discussions, this is the first patch of the
renames. Tested with "make distcheck", seems to work fine.

The only change outside of the renaming is a bit of simplification in
the .gitignore rules; otherwise, simply s/htools/src/.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Unless this is run under run-in-tempdir, it will create as usual the
.pyc files.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

 * ganeti-extstorage-interface man page
 * gnt-storage man page
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add a new client called 'gnt-storage'.
The client interacts with the ExtStorage interface, similarly to
the way gnt-os interacts with the OS interface.

For now, only two commands are supported: 'info' and 'diagnose'.

'diagnose' calculates the node status of each provider on each node,
similarly to gnt-os diagnose. Furthermore, for every provider, it
calculates it's nodegroup validity for each nodegroup. This is done
inside the LU and not the client (marked as 'TODO' for the  global
validity of gnt-os diagnose).

In the future, gnt-storage can be used to manage storage pools,
or even be extended to diagnose other storage types supported by
Ganeti, such as lvm, drbd (INT_MIRROR) or rbd (EXT_MIRROR).
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: fixed Haskell compatibility and style fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

This adds a few new hcheck tests, to extend the coverage.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

After this patch, almost all of hbal except the actual Luxi execution
is being covered by unit- or shelltests.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Of course I forgot to add this to TEST_FILES… and only distcheck
catches it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This makes the code more readable and easier to upgrade.
bdev.LogicalVolume.GetPVInfo and the code that depends on it have been
refactored to use the new class.

utils.CheckVolumeGroupSize() has been moved to lib/utils/lvm.py, where more
functions will be added.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The client queries all the master candidates in parallel, until the minimum
number of replies, defined in the constant file, is received.
A timeout prevents the waiting from being of indefinite length.

The reply to be returned to the function that made the query is decided
according to the Confd design document.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

With this commit we introduce the External Storage Interface
to Ganeti, abbreviated: ExtStorage Interface.

The ExtStorage Interface provides Ganeti with the ability to interact
with externally connected shared storage pools, visible by all
VM-capable nodes. This means that Ganeti is able to handle VM disks
that reside inside a NAS/SAN or any distributed block storage provider.

The ExtStorage Interface provides a clear API, heavily inspired by the
gnt-os-interface API, that can be used by storage vendors or sysadmins
to write simple ExtStorage Providers (correlated to gnt-os-interface's
OS Definitions). Those Providers will glue externally attached shared
storage with Ganeti, without the need of preprovisioned block devices
on Ganeti VM-capable nodes as confined be the current `blockdev' disk
template.

To do so, we implement a new disk template called `ext' (of type
DTS_EXT_MIRROR) that passes control to externally provided scripts
(the ExtStorage Provider) for the template's basic functions:

 create / attach / detach / remove / grow

The scripts reside under ES_SEARCH_PATH (correlated to OS_SEARCH_PATH)
and only one ExtStorage Provider is supported called `ext'.

The disk's logical id is the tuple ('ext', UUID.ext.diskX), where UUID
is generated as in disk template `plain' and X is the disk's index.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: small simplification in bdev code, pylint fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

This is a new personality that for the moment doesn't do anything.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This adds support for job queries, including (basic) unit-tests.

I've tested this for memory and cpu usage as follows:

- 3600 jobs (live queue):
  - via masterd, default: ~1.1s (masterd: ~60MB ram)
  - via confd,   default: ~1.1s (hconfd:  ~25MB ram)
  - via masterd, id only: ~1.0s (masterd: ~57MB ram)
  - via confd,   id only: ~0.2s (hconfd:  ~15MB ram)

- all jobs (128K in total, around 570MB size on disk):
  - via masterd, default: 1m22s (masterd cpu: 48s), masterd: 1.4G ram
  - via confd,   default: 1m16s (hconfd  cpu: 51s), hconfd:  570MB ram peak
    (peak only right before starting luxi send, hconfd decreases in RSS as
    results are streamed over luxi, back to ~18MB after the send)
  - via masterd, id only:  ~49s (masterd cpu: 45s), masterd: 1.3G ram
  - via confd,   id only:  ~39s (hconfd cpu: 35s),  hconfd:  110MB ram peak
    (right before luxi send, decreasing as results are sent, back to ~14MB
    after the send)

Given this, and that in production it's not likely to have hundreds of
thousand of job files, I believe the implementation is safe from this
point of view.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This patch adds implementation for a read-only job queue module,
together with "full" test (as full as can be in a lazy language…).

One note about the behaviour of the job queue is the handling of
opcodes that fail validation: the 'input' opcode actually is a
meta-type, which can hold either a real opcode or a plain JSValue, so
that we can still load jobs with invalid opcodes for querying. The
only downside of this is that, as opposed to Python code, we can't
show the correct summary for such an opcode - we try to parse the
OP_ID but not the extended OP_DSC_FIELD-equivalent.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

"Use comparing" was not needed, and "Use on" can be solved by actually
using on in the single case we have (the patch also reorders imports
in that file).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

These test the newly-changed behaviour, and fix an inconsistency in
the hs-check target (versus hs-coverage).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This just ties in all the pieces so far and enables bash completion
for it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, the mon-collector binary uses the HTools/CLI module, which
is OK but mean it links in lots of htools code. By copying that module
to DataCollectors/CLI and removing the unneeded code, we reduce the
number of modules it depends on fro 20 to 12, meaning both a shorter
compiler time (24s to 9s) and a smaller binary (~9.6MB to ~7MB).

Also fixes a typo in the original HTools/CLI module, thanks Michele!
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>