When executed in split user mode, Ganeti requires a specific set of users
and groups to be created in the appropriate way.

This commit introduces a tool that takes care of that automatically.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Setting up a proper environment for building a Ganeti can be quite difficult.

This patch adds a script for building a compressed chroot with all the
required libraries and tools.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This commit adds the core infrastructure of the monitoring daemon,
and integrates it in the build and test systems.

The actual functionality of the monitoring daemon is still completely
missing.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Dato Simó <dato@google.com>

This is an autogenerated file, so it has to go into gitignore, thanks
Michael!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Similar to other scripts, “tools/burnin” just becomes a wrapper script.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Before this patch, HTML versions of man pages (man/*.rst) were already
built. However, since they are separate from the normal documentation,
their content is not indexed for Sphinx' search functionality.
Additionally it would simply be nice to have everything in one place.

To this end a new configure-time option is added to enable the inclusion
of man pages into the documentation. A dedicated option is necessary to
still be able to provide a static documentation build in the tarball
(not including man pages) as man pages contain build-specific paths and
values. The documentation with man pages is written to the directory
“doc/man-html”.

A future patch will extend Sphinx to link occurences of “:manpage:`…`”
to these man pages.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Found while doing the previous .dot file changes and wondering what's
this "doc/*.in" stuff… it looks like these were obsoleted over the
years by not paying attention on cleanup patches:

- doc/build: obsoleted by commit 13de7d59
- doc/*.in: obsoleted by commit 2ab2b9f5
- doc/html: was always a directory since its introduction in commit
  d17e74b4, mark it as so
- devel/clean-cluster: was introduced with initial .gitignore addition
  (commit 7a47c582), but we don't actually generate it

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The first dot files we added way back in the 2.0 release are managed
manually: we build the .pngs ourselves, track them in .gitingore,
etc. Since then, we have switched to allow sphinx to build/manage them
itself, so it makes sense to cleanup the build system and covert
everything to this simpler method.

This patch simply converts from "image: x.png" to "graphviz: x.dot",
as an externally-referenced dot file, and removes all the custom
makefile rules for png conversion. The next patch will bring the
conversion further (by inlining then).

Additionally, while checking that the patch doesn't change the html
output, I saw that the use of "shape=diamond" with non-trivial labels
results in very ugly diagrams; so we split the question in such
decision boxes to make them more reasonably looking.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

I don't remember why I introduced these when switching to pandoc, but
they are not absolutely needed. Instead of pre-generating them (which
might save some trivial cpu time), we can simply pass footer.rst as an
additional input file to pandoc.

The advantage in doing so is that semantic information is preserved;
when pre-generating and including them as-is, the entire structure is
lost, as pandoc won't reinterpret them, so things like a potential
table of contents will simply ignore the footer.

The patch also adds a new target to rebuild all man pages (both groff
and html formats), for testing changes to the pages easier.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The current base name of 'test' means that this binary will generate
profiling, coverage, etc. files with the name 'test.*', result in
tab-completion conflicts with the test directory. Let's slightly
change its base name for typing easiness.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

htest/data becomes test/data/htools (basically reverting commit
8feabc89), and htest/* becomes test/hs/*.

Most changes beside the rename are trivial s/…/…, with the exception
of autotools/run-in-tempdir, which needed some more changes now that
test/ is not just Python files:

- test/py is still being copied
- test/hs moves from individual symlinks to entire dir symlink
- test/data is symlinked in its entirety

Checked with make distcheck, pep8 and pylint, so at least VPATH builds
are OK.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Per offline discussions, this is the first patch of the
renames. Tested with "make distcheck", seems to work fine.

The only change outside of the renaming is a bit of simplification in
the .gitignore rules; otherwise, simply s/htools/src/.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

 * ganeti-extstorage-interface man page
 * gnt-storage man page

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

It is a binary file and it should never be added to the git repository.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It is implemented as a single binary receiving, as its first parameter,
the name of the actual data collector to be run.
This way, its structure can be used for all the future data collectors.

Also, factored out of bdev.py into constants.py the location of the DRBD
status file.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Sometimes a node is not removed properly from a cluster (especially
during development). This new tool stops all daemons and removes (after
making copies) the most critical files.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The design for this is in “doc/design-node-add.rst”. The tool receives a
JSON data structure on stdin and configures the node's daemon after
verifying the received values.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This allows offlining nodes that don't respond if they are part of a
linux-HA cluster.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This allows controlling the cluster master role if the nodes are part of
a linux-HA cluster.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

We had twice in the past days questions about devel/upload being
"broken", since bash re-reads shell scripts during their run and this
file can get regenerated due to Makefile changes.

Since we only need this to be dynamically built for 3 variables, let's
make the file static and read those three variables when it is run,
instead of when it is built, which allows us to re-read the "latest
version" of these vars as well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is a new tool as per the design document “design-ssh-setup”. It
receives a JSON data structure on its standard input and configures the
SSH daemon and root's SSH keys accordingly. Unit tests are included.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This only supports test delay for now, is not built by default (only
on demand), and is also not installed anywhere.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Sorry, this should have went in b0dcdc10, but I forgot it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

As I wrote during/after the review on commit 2958c56e, “ganeti-cleaner:
Separate queue cleaning code”, while I appreciated the permission
separation, I didn't like too much the file-based approach:

- it is a very simple script, and lots of the code is duplicated
  between the two; I wouldn't like to see "ganeti-vmcapable-cleaner",
  "ganeti-master-candidate-cleaner", etc. in the future
- ganeti-master-cleaner "pollutes" the namespace, creating
  tab-completion conflicts with ganeti-masterd

This patch simply merges the master-cleaner back into cleaner, while
keeping the separate user permissions scheme, separate log files, etc.

Additionally, it fixes two bugs in the unit-test (not run with set -u
and wrong path in the master-cleaner log files test; yay for even
worse safety than Python?).

And finally, since we have now support for --help-completion, it adds
bash completion support for this script :) (needs to be applied on top
of my argument support patch series).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In 2.7 we will only support the Haskell version, if enabled.

Since the original hconfd enabling was a bit hack-ish (copying over
the actual installed ganeti-confd, Python version), the Makefile.am
changes are a bit more involved than just the removal of the Python
code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This code does not need to run as root, therefore it's better to split
it out. It is now run with the same permissions as the master daemon.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Instead of having a different set in (almost) every shell script, this
inserts the most commonly used variables at build time. This way the
code for injecting a root directory for virtual clusters also is just
needed once.

I chose inserting the variables using “sed” at build time over a runtime
script as the latter would need to be located from within the script and
the path would be different depending on the environment (unittest vs.
actual installation).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This utility can be used to configure an environment for a virtual
cluster. It sets up entries in /etc/hosts, creates the necessary
directory structure, and generates helper scripts.

Documentation for virtual clusters will come in a later patch.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

First the numbers:
$ stat --format '%s %n' doc/examples/bash_completion*
77847 doc/examples/bash_completion
86492 doc/examples/bash_completion-debug

The non-debug version doesn't use indentation and does not have some
(primitive) facilities for debugging. The savings are about 8.5 kB or
10 %. The “-debug” version is used by “devel/upload”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is another module that is generic, and not htools-specific.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

In order to have correct code coverage results, we must somehow import
all production modules into the test runner. Until now, this was done
manually (when we didn't forget) in QC.hs.

To improve the situation, we remove QC.hs and replace it with an
auto-generated file which imports all modules. This reduces the
maintenance burden and ensures we'll always have correct coverage.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This moves the last (I think) htools-related bits out of test/ under
htest/.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is the first commit of a series that will attempt to cleanup the
test code organisation, which evolved somewhat organically from the
initial pure htools functionality.

The proposed organisation of the tree will be as follows:

- htools (or maybe renamed to haskell or hs): only production code
- htest: top-level test directory, containing test.hs, static helper
  scripts, etc.
- htest/Ganeti/*.hs: modules implementing the actual test properties
  and test cases for the correspondingly-named production code modules
- htest/data: containing test data files for the test cases

This particular patch moves all the test code (test.hs, hpc-htools.hs
symlink) and helper scripts (offline-test.sh, etc.) from htools/ to
htest/, while updating the files themselves (if they had paths
mentioning htools/), .gitignore and the Makefile.

The only special mention is that in Makefile, we used to have a BINARY
shell variable in binary build rule; that was computed via stripping
`htools/' prefix; I've cleaned that and replaced with $(notdir $@);
even though it's duplicated a few times, it leads to more readable
make output (and easier to copy-paste).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Commit c964d962 changed the way we create directories, by two things:

- unifying all dependencies and ad-hoc directory creation into a
  single target (all_dirfiles)
- changing how directories are created from a stamp file to .dir files
  in each directory

The first item is a very good one, but the second item is debatable:
there's no per-se advantage of .dir files versus a single one,
top-level, since both the .dir file and stamp-directories creation are
depending on Makefile, which is the only one which can introduce new
directories.

On the other hand, moving back from .dir files to stamp-directories
has an advantage: "make -d | wc -l" does from ~8.7K lines to ~5.3K
lines, because we eliminate the many .dir files and their multiple
implicit and explicit dependencies (the %/.dir files fall under
multiple patterns).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 3096524e enabled building of test helpers, but forgot to add
them to .gitignore, resulting in "unclean" git status output.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This is a stupid/trivial patch. I didn't realise when adding
htools-hpc that this will "break" tab completion on htools; as in, I
will have to add / manually always when completing files on the
command line.

So to fix my laziness, let's rename htools-hpc to hpc-htools,
eliminating this completion conflict.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This allows the offline tests to also generate coverage data, and the
hs-coverage target is changed to show both unit-test and offline-tests
coverage.

The downside is that now we build yet-another-binary, which makes the
unit-test time slower…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>