With this commit we introduce the External Storage Interface
to Ganeti, abbreviated: ExtStorage Interface.

The ExtStorage Interface provides Ganeti with the ability to interact
with externally connected shared storage pools, visible by all
VM-capable nodes. This means that Ganeti is able to handle VM disks
that reside inside a NAS/SAN or any distributed block storage provider.

The ExtStorage Interface provides a clear API, heavily inspired by the
gnt-os-interface API, that can be used by storage vendors or sysadmins
to write simple ExtStorage Providers (correlated to gnt-os-interface's
OS Definitions). Those Providers will glue externally attached shared
storage with Ganeti, without the need of preprovisioned block devices
on Ganeti VM-capable nodes as confined be the current `blockdev' disk
template.

To do so, we implement a new disk template called `ext' (of type
DTS_EXT_MIRROR) that passes control to externally provided scripts
(the ExtStorage Provider) for the template's basic functions:

 create / attach / detach / remove / grow

The scripts reside under ES_SEARCH_PATH (correlated to OS_SEARCH_PATH)
and only one ExtStorage Provider is supported called `ext'.

The disk's logical id is the tuple ('ext', UUID.ext.diskX), where UUID
is generated as in disk template `plain' and X is the disk's index.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: small simplification in bdev code, pylint fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

This is a new personality that for the moment doesn't do anything.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This adds support for job queries, including (basic) unit-tests.

I've tested this for memory and cpu usage as follows:

- 3600 jobs (live queue):
  - via masterd, default: ~1.1s (masterd: ~60MB ram)
  - via confd,   default: ~1.1s (hconfd:  ~25MB ram)
  - via masterd, id only: ~1.0s (masterd: ~57MB ram)
  - via confd,   id only: ~0.2s (hconfd:  ~15MB ram)

- all jobs (128K in total, around 570MB size on disk):
  - via masterd, default: 1m22s (masterd cpu: 48s), masterd: 1.4G ram
  - via confd,   default: 1m16s (hconfd  cpu: 51s), hconfd:  570MB ram peak
    (peak only right before starting luxi send, hconfd decreases in RSS as
    results are streamed over luxi, back to ~18MB after the send)
  - via masterd, id only:  ~49s (masterd cpu: 45s), masterd: 1.3G ram
  - via confd,   id only:  ~39s (hconfd cpu: 35s),  hconfd:  110MB ram peak
    (right before luxi send, decreasing as results are sent, back to ~14MB
    after the send)

Given this, and that in production it's not likely to have hundreds of
thousand of job files, I believe the implementation is safe from this
point of view.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This patch adds implementation for a read-only job queue module,
together with "full" test (as full as can be in a lazy language…).

One note about the behaviour of the job queue is the handling of
opcodes that fail validation: the 'input' opcode actually is a
meta-type, which can hold either a real opcode or a plain JSValue, so
that we can still load jobs with invalid opcodes for querying. The
only downside of this is that, as opposed to Python code, we can't
show the correct summary for such an opcode - we try to parse the
OP_ID but not the extended OP_DSC_FIELD-equivalent.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

"Use comparing" was not needed, and "Use on" can be solved by actually
using on in the single case we have (the patch also reorders imports
in that file).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

These test the newly-changed behaviour, and fix an inconsistency in
the hs-check target (versus hs-coverage).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This just ties in all the pieces so far and enables bash completion
for it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, the mon-collector binary uses the HTools/CLI module, which
is OK but mean it links in lots of htools code. By copying that module
to DataCollectors/CLI and removing the unneeded code, we reduce the
number of modules it depends on fro 20 to 12, meaning both a shorter
compiler time (24s to 9s) and a smaller binary (~9.6MB to ~7MB).

Also fixes a typo in the original HTools/CLI module, thanks Michele!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

As noted by Guido, there are problems when using the hs-prof and
hs-prof-quick targets in the default configuration (compiling all
programs). The errors manifest in the form of wrong symbols during
compilation.

I knew that the hs-prof targets, which compiled multiple objects with
the '.o' suffix, could be problematic; but the objects that are
actually needed in the Template Haskell phase are very standard and
don't differ between the binaries (Constants, BasicTypes, JSON,
THH). What I didn't realise (although it's obvious) is that also the
hs_prof_quick targets (the final binaries) are also compiled with a
single suffix ('.prof_o'), which means that the object files are
actually compiled for the last binary.

This means that targets later in the HS_ALL_PROGS list would work
correct, but early targets, especially htools/htools, would fail.

So the obvious, and the single simple solution is to make these two
rules only work on a single binary at a time. This should be fine,
since one is looking at a specific problem usually, and it has the
advantage that the hs-prof step is much faster (since it wasn't
buildable in parallel anyway).

Thanks to Guido for finding and making the initial diagnose on this!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Some confd types were not tested for serialisation, so let's move
these (type-related) tests to their own module and extend them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This single constant could be as well moved to constants.py. For now I
choose to just export the module as well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is similar to README or “doc/security.rst”. Having a comment that
needs updating for a new version is more likely going to trigger a check
of the lists of design drafts.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Use the “error” variable in multiple places and fail only once all tests
have been completed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The coverage information for mon-collector was not merged correctly.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This commit adds shelltests for the mon-collector binary and for the DRBD
data collector.

Also, it fixes a small bug in the DRBD parser found thanks to the tests.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It is implemented as a single binary receiving, as its first parameter,
the name of the actual data collector to be run.
This way, its structure can be used for all the future data collectors.

Also, factored out of bdev.py into constants.py the location of the DRBD
status file.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Implementation of the network objects and address pool. Functionality
as in the reverted commit b9a616e1, but
now using only the vector library and not the bit-vector library.
Tested with vector library version 0.9 and 0.10., which are
also installed on the buildbot machines now.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It also uses values from “lib/errors.py”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This reverts commit b9a616e1, which
depends on the "bitvec" library. We need to investigate a bit further that
dependency, as it in turns depends on "vector" 0.9.1 or *below*, since
0.10.* removed support for Data.Vector.Unboxed.Safe which it uses.

Signed-off-by: Helga Velroyen <helgav@google.com>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Implementation of the network and address pool class in
Haskell. Not complete yet. Includes unit tests that cover
all functionality that is so far implemented.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Sometimes a node is not removed properly from a cluster (especially
during development). This new tool stops all daemons and removes (after
making copies) the most critical files.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The design for this is in “doc/design-node-add.rst”. The tool receives a
JSON data structure on stdin and configures the node's daemon after
verifying the received values.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

epydoc allows overriding configuration values via environment variables.
While this might be useful in certain use cases, but as no prefix
whatsoever is used, conflicts are easily created. Some people have the
environment variable “NAME” set, effectively overriding the project name
set in epydoc.conf. A bug in epydoc causes an error if non-ASCII
characters, such as German umlauts, are used in NAME.

$ NAME=Täscht make py-apidoc
[…]
UNEXPECTED ERROR:
'ascii' codec can't decode byte 0xc3 in position 73: ordinal not in range(128)

$ parse=false make py-apidoc
[…]
epydoc: error: Invalid option combination: --parse-only and --introspect-only.

This patch changes the call in Makefile to reset the environment given
to epydoc save for PATH and PYTHONPATH.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This module implements some algorithms on Data.Graph data structures.
At the moment its main functionality is an LF-color implementation
(greedy coloring in descending order of degree). There are also a few
extra functions to calculate the degree order, and convert the node to
color mapping to color to nodes.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Just some cleanup:

- don't run hlint over hpc-htools, since it's just a symlink to
  htools.hs (so we'd get duplicate warnings)
- build rpc-test in make really-all, via HS_ALL_PROGS variable

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Since hlint doesn't check style issues but rather code only issues,
let's add a simple Makefile check for too-long lines, and fix the
single bad case we have currently.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Indentation within rules is fixed to always be 2 spaces (as opposed to
mixed tabs/2 spaces/4 spaces). Additionally, the check-dirs rule,
which was more complex than necessary, has been simplified: moved
auto4mte.cache to DIRCHECK_EXCLUDE, and simplify the find call and the
exit condition (no need for if test …; then exit 0; else exit 1; fi).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of having two different lists of directories, now there is only one.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add the design document for opportunistic locking.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The serialization itself is done by Text.JSON, so the tests deal with checking
that Text.JSON objects are created correctly from the DRBD parser data
structures.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It turns out that in newer GHC versions, the suffix shouldn't be
prefixed with a dot; otherwise name mangling is broken.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

The hpc tool, on squeeze, does not handle utf8 encoded files correctly.
The Attoparsec test is just needed to verify whether the parser handles
uft8 correctly, so the solution is to exclude it from coverage checks.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Until now ssconf would limit the amount read from files to 128 KiB and
silently ignored files larger than that. With this patch a check is
added by using fstat(2) on the file descriptor while it's being read.

There were no tests file ssconf at all, so some are added.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

It is no longer “--enable-remote-commands”, but rather
“--enable-restricted-commands”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Iustin Pop <iustin@google.com>

This adds tests that existing test files can be parsed by the Haskell
parser as well, plus one new test file.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

A new directory for haskell modules about block devices has been created
The parser is divided in two modules:
* one exports the data types describing the DRBD status
* one exports the parser itself

Signed-off-by: Michele Tartara <mtartara@google.com>
[iustin@google.com: indentation/alignment fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

Due to TemplateHaskell stage restrictions, we can't define parameters
in the same module as we're using them for TH, so we have to define
all module parameters in a separate module.

This patch therefore splits OpCodes.hs in two, adding that module and
moves most code there (types, parameters, etc.). The remaining parts
in OpCodes.hs, the actual opcode definitions, now use more parameters
instead of direct field definitions (more will come later)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

There are already three cases where we copied type definitions between
the htools-specific types into the main ganeti code. Let's stop doing
this ☺️

 and create a common types module that holds these.

Note that there already exists BasicTypes.hs, but that refers to very
low-level types, and can't use TH derivation itself.

A side effect of this unification is that there is a small conflict
between AdminStatus/AdminOffline and InstanceStatus/AdminOffline. As
such, I renamed AdminOffline and AdminDown to StatusOffline/StatusDown
in the InstanceStatus type.

The patch also moves the tests related to these types to a new test
module.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

- Remote unnecessary quote characters
- Change some uses of “test "$foo"” to “test -n "$foo"”
- Merge “if/else/if” into “if/elif”
- Wrap lines longer than 80 characters and add a test to “make check”

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>