Currently this is checked only when distcheck-release is called.
Check it in check-news instead, for all versions before the "current"
one (as defined by configure.ac)

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Before:
$ ./autotools/check-news < ./NEWS
Traceback (most recent call last):
  File "./autotools/check-news", line 150, in <module>
    main()
  File "./autotools/check-news", line 120, in main
    parsed_ts = time.mktime(time.strptime(m.group("date"), "%d %b %Y"))
AttributeError: 'NoneType' object has no attribute 'group'

After:
$ ./autotools/check-news < ./NEWS
Line 8: Invalid release line

This happened because the error function just updates an error log,
rather than bailing out immediately.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This commit adds the user and group variables for the monitoring agent in all
the configuration files where they are required.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently the man page build rule is run without ‘-e’, so the actual
checks in there are just warnings (check-man-warnings,
check-man-dashes).

The patch fixes this and also fixes the man pages which made me see the
problem.

Additionally, check-man-dashes is now verbose, otherwise it's hard to
find out actually where in the page the error is.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently the Python source constants are used as-is, which is not
nice on the eyes (especially for unqualified names, it was hard to
separate them visually).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

It turns out that, in Python, booleans are also integers. So they fall
under the “isinstance(value, int)” case, resulting in all enable*
constants being integers in Haskell, which is not nice, even though
we're not using them directly today.

Patch simply adds a special casing for booleans, before integers.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit b6ad806f contained a date a month in the future. With this patch
all release dates in NEWS may at most be three days into the future
(assuming the build machine's clock is correct).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This reverts commit dd60179d as it broke coverage for Python.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

By creating symlinks instead of actually creating files we still get the
benefits of using a temporary directory, but the preparation is about
three times as fast. Test used:

$ time bash -c 'for ((i=0; i < 100; ++i)); do
    autotools/run-in-tempdir true; done'

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

To preserve disk space on copying all files to a temporary directory,
the “doc” directory is only copied when building documentation. A
comment describing the need is also added to Makefile.am, as well as a
superfluous direct invocation of Bash removed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Similar to other scripts, “tools/burnin” just becomes a wrapper script.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Before this patch, HTML versions of man pages (man/*.rst) were already
built. However, since they are separate from the normal documentation,
their content is not indexed for Sphinx' search functionality.
Additionally it would simply be nice to have everything in one place.

To this end a new configure-time option is added to enable the inclusion
of man pages into the documentation. A dedicated option is necessary to
still be able to provide a static documentation build in the tarball
(not including man pages) as man pages contain build-specific paths and
values. The documentation with man pages is written to the directory
“doc/man-html”.

A future patch will extend Sphinx to link occurences of “:manpage:`…`”
to these man pages.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This script checks for some of the most obvious mistakes when formatting
man page references (which should have the form “**ganeti**\(7)”). While
this works now, it is very hard to avoid ambiguities (e.g. references
within verbatim blocks) when using regular expressions.

Also fixes a typo in Makefile.am by replacing “harcoded” with
“hardcoded”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

- gen-coverage → gen-py-coverage
- coverage-htools.tix → coverage-hs.tix

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

htest/data becomes test/data/htools (basically reverting commit
8feabc89), and htest/* becomes test/hs/*.

Most changes beside the rename are trivial s/…/…, with the exception
of autotools/run-in-tempdir, which needed some more changes now that
test/ is not just Python files:

- test/py is still being copied
- test/hs moves from individual symlinks to entire dir symlink
- test/data is symlinked in its entirety

Checked with make distcheck, pep8 and pylint, so at least VPATH builds
are OK.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is the first step of the test files reorganisation: moving test/*
(except test/data) to new directory test/py/.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Per offline discussions, this is the first patch of the
renames. Tested with "make distcheck", seems to work fine.

The only change outside of the renaming is a bit of simplification in
the .gitignore rules; otherwise, simply s/htools/src/.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

I made a mistake when adding support for htools to run-in-tempdir, by
needlessly copying the htools binaries, when it's enough to link
them. The copying is only really needed for python code, so we change
the Haskell binaries to linking and also a few other directories.

Difference:
- before: 0.8s, 95MB copied
- after: 0.6s, 6.2MB copied

And, by the way, can't wait for Python 2.6 as minimum supported
version…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add a new client called 'gnt-storage'.
The client interacts with the ExtStorage interface, similarly to
the way gnt-os interacts with the OS interface.

For now, only two commands are supported: 'info' and 'diagnose'.

'diagnose' calculates the node status of each provider on each node,
similarly to gnt-os diagnose. Furthermore, for every provider, it
calculates it's nodegroup validity for each nodegroup. This is done
inside the LU and not the client (marked as 'TODO' for the  global
validity of gnt-os diagnose).

In the future, gnt-storage can be used to manage storage pools,
or even be extended to diagnose other storage types supported by
Ganeti, such as lvm, drbd (INT_MIRROR) or rbd (EXT_MIRROR).

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: fixed Haskell compatibility and style fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

This is a new personality that for the moment doesn't do anything.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Accidentally stumbled upon this while testing unrelated code on a
machine with ~3K active jobs - the bash completion unittest was
hanging.

Upon investigation, it turns out that bash's ${var//pattern/repl/} is
probably quadratic in the size of input (or worse, even):

  $ touch job-{1..500}
  $ time ( a=$(echo job-*); echo ${a//job-/}| wc -c; )
  1892

  real    0m0.597s
  user    0m0.590s

  $ touch job-{1..1000}
  $ time ( a=$(echo job-*); echo ${a//job-/}| wc -c; )
  3893

  real    0m4.654s
  user    0m4.580s

We can easily fix this if we change to array-based substitution (once
per element):

  $ time ( a=($(echo job-*)); echo ${a[*]/job-/} |wc -c; )
  3893

  real    0m0.028s
  user    0m0.010s

  $ touch job-{1..10000}
  $ time ( a=($(echo job-*)); echo ${a[*]/job-/} |wc -c; )
  48894

  real    0m0.233s
  user    0m0.220s

This means that exactly when the master node is busy processing many
jobs, we could accidentally start consuming lots of CPU in the bash
completion, which is not good.

Note: the code might have problems with filenames containing spaces (I
didn't reset the IFS, etc.), but the original code had the same issue,
I think.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This just ties in all the pieces so far and enables bash completion
for it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds support for parsing the command list out of a binary
(very strict format), and then iterating over that and building the
sub-commands options/arguments.

It also does a bit of general cleanup in the script.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This single constant could be as well moved to constants.py. For now I
choose to just export the module as well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

There is no obvious need for this space--the line would wrap at the next
space, which comes soon after.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This commit adds shelltests for the mon-collector binary and for the DRBD
data collector.

Also, it fixes a small bug in the DRBD parser found thanks to the tests.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch enhances the opcode list checks - instead of spawning a
Python interpreter to display the opcode list, we export it statically
in Constants.hs via a slight convert-constants change.

Furthermore, since we now have opcode parity, we enable full opcode
list checks.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, the cluster verify errors are defined as follows:

CV_ECLUSTER_FOO = (TCLUSTER, "ECLUSTER_FOO", "description")

This means there's no standalone name for the string "ECLUSTER_FOO",
which makes it hard to derive automatically a type for this union in
Haskell.

There are three possible fixes:

- manually separate the CV_ECLUSTER_FOO_STR = "ECLUSTER_FOO" in
  constants.py
- manually extract the strings in Haskell code
- change convert-constants to automatically export virtual constants
  for those

After discussion on IRC, I've taken the latter approach; even though a
bit hack-ish, it avoids manual work and potential errors.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is similar to commit f5ce7613. A stable sort order makes changes
between versions easier to find.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

- Debian Squeeze and up have a package named “python-coverage”, but it
  doesn't use the same binary name as upstream (“coverage”).
- Said package includes a patch to use symlinks instead of file copies
  for jQuery. If files from previous runs are around, an exception is
  raised. This is fixed by removing all regular files and symlinks
  before generating a report.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

These will be needed for the serialisation of errors.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This part of the code was duplicated by accident.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch cleans up  the Haskell library detection and defaults.

First, it makes the base compiler/libraries required, per the email discussion.

It then adds two new small autoconf macros, on to check for a required
Haskell library and one to do custom action based on test results. We
use these macros to cleanup and simplify a bit the module detection:

- rapi, confd, and split query are auto detected and enabled if _all_
  required libraries are present
- unittests are enabled if _all_ required libraries are present

The patch also updates the documentation regarding required libraries.

After this patch, base Ganeti fully buildable on Debian Squeeze/Ubuntu
Lucid.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

As I wrote during/after the review on commit 2958c56e, “ganeti-cleaner:
Separate queue cleaning code”, while I appreciated the permission
separation, I didn't like too much the file-based approach:

- it is a very simple script, and lots of the code is duplicated
  between the two; I wouldn't like to see "ganeti-vmcapable-cleaner",
  "ganeti-master-candidate-cleaner", etc. in the future
- ganeti-master-cleaner "pollutes" the namespace, creating
  tab-completion conflicts with ganeti-masterd

This patch simply merges the master-cleaner back into cleaner, while
keeping the separate user permissions scheme, separate log files, etc.

Additionally, it fixes two bugs in the unit-test (not run with set -u
and wrong path in the master-cleaner log files test; yay for even
worse safety than Python?).

And finally, since we have now support for --help-completion, it adds
bash completion support for this script :) (needs to be applied on top
of my argument support patch series).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This allows the command completion to treat them as separate option
types; this is not useful in build-bash-completion right now, but
might become so in the future.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This requires a few tiny changes to build-bash-completion and to
Makefile rules, but is straightforward.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Beside parsing the arguments as well, we also change the serialisation
format for choices/suggests, to make the Python code simpler.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Tests that depend on fakeroot were not distributed in the tarball created
by "make dist" when fakeroot was missing. Now all tests are distributed,
and a warning is printed when a test requires fakeroot but fakeroot is
missing.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In older versions of GNU Bash extended patterns, such as “@(…)”, are only
available with the “extglob” shell option. As pointed out in [1] and [2],
“extglob” must be enabled while parsing the code. Therefore the flag must be
enabled at the beginning of the script and be reset to its original value at
the end as to not interfere with other code on shell initialization.

[1] http://unix.stackexchange.com/questions/45957
[2] http://mywiki.wooledge.org/glob



Reported by Sascha Lucas.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit 893ad76d)