It is a binary file and it should never be added to the git repository.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It is implemented as a single binary receiving, as its first parameter,
the name of the actual data collector to be run.
This way, its structure can be used for all the future data collectors.

Also, factored out of bdev.py into constants.py the location of the DRBD
status file.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Sometimes a node is not removed properly from a cluster (especially
during development). This new tool stops all daemons and removes (after
making copies) the most critical files.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The design for this is in “doc/design-node-add.rst”. The tool receives a
JSON data structure on stdin and configures the node's daemon after
verifying the received values.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This allows offlining nodes that don't respond if they are part of a
linux-HA cluster.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This allows controlling the cluster master role if the nodes are part of
a linux-HA cluster.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

We had twice in the past days questions about devel/upload being
"broken", since bash re-reads shell scripts during their run and this
file can get regenerated due to Makefile changes.

Since we only need this to be dynamically built for 3 variables, let's
make the file static and read those three variables when it is run,
instead of when it is built, which allows us to re-read the "latest
version" of these vars as well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is a new tool as per the design document “design-ssh-setup”. It
receives a JSON data structure on its standard input and configures the
SSH daemon and root's SSH keys accordingly. Unit tests are included.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This only supports test delay for now, is not built by default (only
on demand), and is also not installed anywhere.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Sorry, this should have went in b0dcdc10, but I forgot it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

As I wrote during/after the review on commit 2958c56e, “ganeti-cleaner:
Separate queue cleaning code”, while I appreciated the permission
separation, I didn't like too much the file-based approach:

- it is a very simple script, and lots of the code is duplicated
  between the two; I wouldn't like to see "ganeti-vmcapable-cleaner",
  "ganeti-master-candidate-cleaner", etc. in the future
- ganeti-master-cleaner "pollutes" the namespace, creating
  tab-completion conflicts with ganeti-masterd

This patch simply merges the master-cleaner back into cleaner, while
keeping the separate user permissions scheme, separate log files, etc.

Additionally, it fixes two bugs in the unit-test (not run with set -u
and wrong path in the master-cleaner log files test; yay for even
worse safety than Python?).

And finally, since we have now support for --help-completion, it adds
bash completion support for this script :) (needs to be applied on top
of my argument support patch series).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In 2.7 we will only support the Haskell version, if enabled.

Since the original hconfd enabling was a bit hack-ish (copying over
the actual installed ganeti-confd, Python version), the Makefile.am
changes are a bit more involved than just the removal of the Python
code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This code does not need to run as root, therefore it's better to split
it out. It is now run with the same permissions as the master daemon.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Instead of having a different set in (almost) every shell script, this
inserts the most commonly used variables at build time. This way the
code for injecting a root directory for virtual clusters also is just
needed once.

I chose inserting the variables using “sed” at build time over a runtime
script as the latter would need to be located from within the script and
the path would be different depending on the environment (unittest vs.
actual installation).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This utility can be used to configure an environment for a virtual
cluster. It sets up entries in /etc/hosts, creates the necessary
directory structure, and generates helper scripts.

Documentation for virtual clusters will come in a later patch.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

First the numbers:
$ stat --format '%s %n' doc/examples/bash_completion*
77847 doc/examples/bash_completion
86492 doc/examples/bash_completion-debug

The non-debug version doesn't use indentation and does not have some
(primitive) facilities for debugging. The savings are about 8.5 kB or
10 %. The “-debug” version is used by “devel/upload”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is another module that is generic, and not htools-specific.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

In order to have correct code coverage results, we must somehow import
all production modules into the test runner. Until now, this was done
manually (when we didn't forget) in QC.hs.

To improve the situation, we remove QC.hs and replace it with an
auto-generated file which imports all modules. This reduces the
maintenance burden and ensures we'll always have correct coverage.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This moves the last (I think) htools-related bits out of test/ under
htest/.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is the first commit of a series that will attempt to cleanup the
test code organisation, which evolved somewhat organically from the
initial pure htools functionality.

The proposed organisation of the tree will be as follows:

- htools (or maybe renamed to haskell or hs): only production code
- htest: top-level test directory, containing test.hs, static helper
  scripts, etc.
- htest/Ganeti/*.hs: modules implementing the actual test properties
  and test cases for the correspondingly-named production code modules
- htest/data: containing test data files for the test cases

This particular patch moves all the test code (test.hs, hpc-htools.hs
symlink) and helper scripts (offline-test.sh, etc.) from htools/ to
htest/, while updating the files themselves (if they had paths
mentioning htools/), .gitignore and the Makefile.

The only special mention is that in Makefile, we used to have a BINARY
shell variable in binary build rule; that was computed via stripping
`htools/' prefix; I've cleaned that and replaced with $(notdir $@);
even though it's duplicated a few times, it leads to more readable
make output (and easier to copy-paste).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Commit c964d962 changed the way we create directories, by two things:

- unifying all dependencies and ad-hoc directory creation into a
  single target (all_dirfiles)
- changing how directories are created from a stamp file to .dir files
  in each directory

The first item is a very good one, but the second item is debatable:
there's no per-se advantage of .dir files versus a single one,
top-level, since both the .dir file and stamp-directories creation are
depending on Makefile, which is the only one which can introduce new
directories.

On the other hand, moving back from .dir files to stamp-directories
has an advantage: "make -d | wc -l" does from ~8.7K lines to ~5.3K
lines, because we eliminate the many .dir files and their multiple
implicit and explicit dependencies (the %/.dir files fall under
multiple patterns).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 3096524e enabled building of test helpers, but forgot to add
them to .gitignore, resulting in "unclean" git status output.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This is a stupid/trivial patch. I didn't realise when adding
htools-hpc that this will "break" tab completion on htools; as in, I
will have to add / manually always when completing files on the
command line.

So to fix my laziness, let's rename htools-hpc to hpc-htools,
eliminating this completion conflict.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This allows the offline tests to also generate coverage data, and the
hs-coverage target is changed to show both unit-test and offline-tests
coverage.

The downside is that now we build yet-another-binary, which makes the
unit-test time slower…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is currently not installed anywhere, just built.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

autotools/testrunner was missing in distcheck.  Now some changes from
70041061 have been reverted, and testrunner
is not built anymore.  coverage now uses fakeroot when needed.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Tests ending in "-runasroot" or "-runasroot.py" are run through fakeroot,
so activity that needs root privileges can be tested.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

A new script, “autotools/build-rpc”, will be used to generate code for
RPC client wrappers. This is done because “lib/rpc.py” contains lots and
lots of boilerplate code. Forthcoming patches will start converting
RPC wrappers.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, most htools code depends on Constants.hs which is generated
from constants.py and also depends on _autoconf.py. Also, _autoconf.py
depends on vcs-version, which all together means that when 'make
regen-vcs-version' is run, for example by ./devel/upload, most of the
Haskell code needs recompilation.

Since htools already has its 'optimised' vcs-version (and doesn't use
the _autoconf.VCS_VERSION constants), we can optimise this as follows:

- _autoconf.py doesn't contain the VCS_VERSION anymore, and that is
  instead moved to _vcsversion.py
- constants.py depends on and imports this new module
- _autoconf.py doesn't get regenerated at vcs-version changes, but
  only at re-running configure/changing Makefile time

The end result is that only htools/Ganeti/HTools/Version.hs is
recompiled now, which is a significant speedup (usually < 1 second
versus 10 seconds previously).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is the last patch of the binaries conversion.

As information, we now have a single binary that is approx. 5.4MiB in
size, compared to 4 binaries that were approx. 5.1-5.2MiB in size;
this will result in a smaller package and install size, and the single
compilation phase should also help.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In addition, the patch adds a separate Makefile variable for holding
the binary roles to make it more clear what we symlink.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This converts the first binary to the generic 'htools' binary.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is the start of a series of patches that will unify all the
binaries currently in use in a single one, which can perform different
roles based on the name it is installed as.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

With the merge of the repositories, we can now auto-generate the code
for Haskell constants from the Python code.

Currently this only handles the basic types (strings and
integers). Handling containers such as lists and dictionaries is only
possible if we would use a parser such that we recognise the element
names. We could extend the convert-constants script if that becomes
necessary, right now I'm looking at just the simple constants such as
Iallocator modes, instance states, etc.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

I provided unittest to test the important pieces of the infrastructure.
The one remaining function (ResuriveEnsure) is not easy to unittest
but also not critical if it fails to operate correctly.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Based on Michael's suggestion, this patch partially reverts my
changes. The new directories are:

- doc/api/py
- doc/api/hs
- doc/coverage/py
- doc/coverage/hs

Basically the Python-specific output moves one level down (into py/)
compared to the original location, and the Haskell stuff goes into
hs/.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Since we do have errors currently, this is not enabled from the main
'make lint' rule. That will get cleaned up later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>