This fixes a bug that corrups the config when one adds a
network to a node group. I wil soon extend QA by some commands
that would have detected this.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

At the moment, because 'mergeData' is pure, it may set instance auto-repair
policies that are of the form `ArSuspended $ Until timestamp_in_the_past`.
If later on the auto-repair tool notices this, it has lost access to what
would be the next-in-line policy to use (and would have to re-parse all
tags again).

To avoid this, we pass the current time to 'mergeData' from ExtLoader.hs,
and we propagate it to Loader.getArPolicy. ExtLoader.loadExternalData is in
the IO monad, so it has ready access to getClockTime.

A few other places were calling 'mergeData' directly. For Hscan.hs and
IAlloc.hs, we add appropriate calls to getClockTime. For files under test/,
we use a current time of 0.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Some auto-repair tags are composed of a prefix and then an optional
argument; the new "chompPrefix" function in Utils.hs allows to strip a
prefix, allowing the last character (the separator) to be absent if there
is no argument.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This function allows to easily convert a ClockTime object to a string
representation of its timestamp (seconds-only).

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In this commit, the AutoRepairType and AutoRepairResult types are defined,
with the possible values specified in doc/design-autorepair.rst.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

One is the --help output for kvm 1.1.2 and the second one is the same as
0.9.1 but with a fake option added between -drive and its boot=on|off
option: this tests that if boot=on|off appears in another option after
-drive it is not accepted by our regexp.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

As discussed on the list some people backport features to different kvm
versions, making the version based detection system unreliable. We
change wherever we can to detection on the help output.

This also fixes Issue 250

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be used in QA to format network interface parameters.

This is a cherry-pick of master commit eac9b7b8

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Also amended a comment of a tested method.
The tests are for methods introduced in commit 63c73073 (LVM disk creation
uses dedicated PVs).

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This commits enables the DRBD data collector to use the Confd client to
gather information about the pairing between DRBD minors and instances.

For testing purposes, the DRBD data collector now requires either zero
or two parameters: one is the DRBD file, one is the pairings file.
When no parameter is passed, the collector is in "production mode" and takes
the data from /proc/drbd and from the Confd client.

The shell tests of mon-collector are updated accordingly.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This commit modifies the DRBD parser and its data structures to include
information about the instance a DRBD minor belongs to.

Test files have been updated as well, to support the new field in the data
structures.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This was extracted from a patch by Dimitris Aragiorgis with the subject
“Add locking to _NetworkQuery”. The rest of the patch did no longer
apply and will require more work.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This removes one more tab conflict; this is the last module in our
code where we have both x.hs and x/.

Furthermore, we collapse all actual code into the new Main.hs module,
leaving the htools.hs basically empty (will allow better testing in
the future).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch unifies the functions that generated the tree
reorganisation discussion in the first place :)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The current base name of 'test' means that this binary will generate
profiling, coverage, etc. files with the name 'test.*', result in
tab-completion conflicts with the test directory. Let's slightly
change its base name for typing easiness.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

htest/data becomes test/data/htools (basically reverting commit
8feabc89), and htest/* becomes test/hs/*.

Most changes beside the rename are trivial s/…/…, with the exception
of autotools/run-in-tempdir, which needed some more changes now that
test/ is not just Python files:

- test/py is still being copied
- test/hs moves from individual symlinks to entire dir symlink
- test/data is symlinked in its entirety

Checked with make distcheck, pep8 and pylint, so at least VPATH builds
are OK.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is the first step of the test files reorganisation: moving test/*
(except test/data) to new directory test/py/.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add a new client called 'gnt-storage'.
The client interacts with the ExtStorage interface, similarly to
the way gnt-os interacts with the OS interface.

For now, only two commands are supported: 'info' and 'diagnose'.

'diagnose' calculates the node status of each provider on each node,
similarly to gnt-os diagnose. Furthermore, for every provider, it
calculates it's nodegroup validity for each nodegroup. This is done
inside the LU and not the client (marked as 'TODO' for the  global
validity of gnt-os diagnose).

In the future, gnt-storage can be used to manage storage pools,
or even be extended to diagnose other storage types supported by
Ganeti, such as lvm, drbd (INT_MIRROR) or rbd (EXT_MIRROR).

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: fixed Haskell compatibility and style fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

For people writing query filters (documented in ganeti(7)) knowing a field's
type can be useful.

$ gnt-instance list-fields name be/memory
Name      Type         Title        Description
name      Text         Instance     Instance name
be/memory Storage size ConfigMaxMem The "maxmem" backend parameter

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be used in QA to format network interface parameters.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

If opportunistic locking is requested in the opcode, but no iallocator
is used (not specified or no default), an error will be raised.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This is not a trivial s/frozenset/compat.UniqueFrozenset/, but rather
only replaces “frozenset” where appropriate. Most of the places are
“static” information that doesn't change after the module has been
loaded.

Some docstrings and code formatting (e.g. empty lines) issues are
addressed as well. Some lines got too long and were wrapped.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Unit tests updated and expanded with an inheritance check.

The flag has no effect yet.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit 141d1489 was a bit too enthusiastic. The three parameters added to
the list of parameters to be checked default to a value not evaluating
to false, leading to a failure on cluster initialization.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

“_SPICE_ADDITIONAL_PARAMS” is supposed to be the full list of
SPICE-related KVM hypervisor parameters with the exception of
“HV_KVM_SPICE_BIND”. The new test checks if all parameters starting with
“HV_KVM_SPICE_*” are included. Three previously missing parameters are
added to the list.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

For some opcodes, the output is not "stable", and depends on the exact
input values; this makes it harder to check consistency against
Haskell code.

To compensate for this, we add a way to override the formatting of the
OP_DSC_FIELD; by default, this is always "%s", but if the
OP_DSC_FORMATTER is defined (must be a callable), it is used to format
the actual value.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The watcher pause file should be set/unset on all nodes at once, not
only the master node. For that a new RPC is needed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

“TestWriteFile” was not renamed when adding the file based on
“ganeti.utils.io_unittest-runasroot.py”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This addresses issue 218. When the number of inotify watches is
exhausted, for example by being set too low from the beginning or by
other programs, waiting for a job to change would just report a lost job
(e.g. “Error checking job status: Job with id 7817 lost”).

This patch changes the job watcher to no longer catch
“errors.InotifyError” and, this is by far the larger part of this patch,
adds unittests for this situation.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

- Add more checks, some of them are deliberately redundant
- Descriptive error messages
- Add comment describing order to “tools.ensure_dirs”
- Avoid copying a list in an assertion in “tools.ensure_dirs”

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Change the "--disks" option validation, to just check the format
of the dict and do not check whether the keys are included in the
IDISK_PARAMS constant at OpCode level. This allows the passing of
arbitrary parameters at the CLI, which will then be logically
checked at LU level (inside CheckArguments).

Currently, the check wrt to IDISK_PARAMS already happens in all
related LUs (InstanceSetParams, InstanceCreate, InstanceRecreateDisks),
with the use of ForceDictType(), so there is no need for additional
checking code.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: fixed a unittest that needed adjustement]
Reviewed-by: Iustin Pop <iustin@google.com>

When used instead of a plain call to “frozenset”, this would have
avoided the issue fixed in commit e2dd6ece. The new function is located
in the “compat” module as it will be used at module load time in most
places and should therefore reside in a place with very few
dependencies.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Until now the error codes were not available from the RAPI client
module. A newly added unit test ensures all error codes are contained in
“ECODE_ALL”, as well as ensuring consistency between the RAPI client and
the authoritative “errors” module.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add verification code to mcpu to check an LU's locks. Two whitelists are
provided to exclude LUs from the two tests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Older versions use “-----BEGIN RSA PRIVATE KEY-----” instead of
“-----BEGIN PRIVATE KEY-----”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The design for this is in “doc/design-node-add.rst”. The tool receives a
JSON data structure on stdin and configures the node's daemon after
verifying the received values.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 4679547e implemented the ability to change job's priority after it
was submitted. The code contained a bug whereby it would modify the
input data for an opcode, something the job queue shouldn't do (logical
units do for historical reasons).

This patch removes the line modifying the opcode input and adjusts the
tests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In the future instance creations might have a lock on all nodes as was
the case until the implementation of opportunistic locking. Nodes for
which the lock is not held will be shown to the iallocator plugin as if
they were marked offline.

This patch adds a new parameter named “node_whitelist” to
“IAReqInstanceAlloc”. If set to a list, only nodes contained within are
shown as online.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The network management code needs to verify a MAC address prefix.
Instead of (ab)using NormalizeAndValidateMac, clean code should be used.
Unit tests for NormalizeAndValidateMac are updated and new ones for
NormalizeAndValidateThreeOctetMacPrefix are added.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>