This was extracted from a patch by Dimitris Aragiorgis with the subject
“Add locking to _NetworkQuery”. The rest of the patch did no longer
apply and will require more work.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This removes one more tab conflict; this is the last module in our
code where we have both x.hs and x/.

Furthermore, we collapse all actual code into the new Main.hs module,
leaving the htools.hs basically empty (will allow better testing in
the future).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch unifies the functions that generated the tree
reorganisation discussion in the first place :)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The current base name of 'test' means that this binary will generate
profiling, coverage, etc. files with the name 'test.*', result in
tab-completion conflicts with the test directory. Let's slightly
change its base name for typing easiness.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

htest/data becomes test/data/htools (basically reverting commit
8feabc89), and htest/* becomes test/hs/*.

Most changes beside the rename are trivial s/…/…, with the exception
of autotools/run-in-tempdir, which needed some more changes now that
test/ is not just Python files:

- test/py is still being copied
- test/hs moves from individual symlinks to entire dir symlink
- test/data is symlinked in its entirety

Checked with make distcheck, pep8 and pylint, so at least VPATH builds
are OK.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is the first step of the test files reorganisation: moving test/*
(except test/data) to new directory test/py/.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add a new client called 'gnt-storage'.
The client interacts with the ExtStorage interface, similarly to
the way gnt-os interacts with the OS interface.

For now, only two commands are supported: 'info' and 'diagnose'.

'diagnose' calculates the node status of each provider on each node,
similarly to gnt-os diagnose. Furthermore, for every provider, it
calculates it's nodegroup validity for each nodegroup. This is done
inside the LU and not the client (marked as 'TODO' for the  global
validity of gnt-os diagnose).

In the future, gnt-storage can be used to manage storage pools,
or even be extended to diagnose other storage types supported by
Ganeti, such as lvm, drbd (INT_MIRROR) or rbd (EXT_MIRROR).

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: fixed Haskell compatibility and style fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

For people writing query filters (documented in ganeti(7)) knowing a field's
type can be useful.

$ gnt-instance list-fields name be/memory
Name      Type         Title        Description
name      Text         Instance     Instance name
be/memory Storage size ConfigMaxMem The "maxmem" backend parameter

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be used in QA to format network interface parameters.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

If opportunistic locking is requested in the opcode, but no iallocator
is used (not specified or no default), an error will be raised.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This is not a trivial s/frozenset/compat.UniqueFrozenset/, but rather
only replaces “frozenset” where appropriate. Most of the places are
“static” information that doesn't change after the module has been
loaded.

Some docstrings and code formatting (e.g. empty lines) issues are
addressed as well. Some lines got too long and were wrapped.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Unit tests updated and expanded with an inheritance check.

The flag has no effect yet.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit 141d1489 was a bit too enthusiastic. The three parameters added to
the list of parameters to be checked default to a value not evaluating
to false, leading to a failure on cluster initialization.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

“_SPICE_ADDITIONAL_PARAMS” is supposed to be the full list of
SPICE-related KVM hypervisor parameters with the exception of
“HV_KVM_SPICE_BIND”. The new test checks if all parameters starting with
“HV_KVM_SPICE_*” are included. Three previously missing parameters are
added to the list.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

For some opcodes, the output is not "stable", and depends on the exact
input values; this makes it harder to check consistency against
Haskell code.

To compensate for this, we add a way to override the formatting of the
OP_DSC_FIELD; by default, this is always "%s", but if the
OP_DSC_FORMATTER is defined (must be a callable), it is used to format
the actual value.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The watcher pause file should be set/unset on all nodes at once, not
only the master node. For that a new RPC is needed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

“TestWriteFile” was not renamed when adding the file based on
“ganeti.utils.io_unittest-runasroot.py”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This addresses issue 218. When the number of inotify watches is
exhausted, for example by being set too low from the beginning or by
other programs, waiting for a job to change would just report a lost job
(e.g. “Error checking job status: Job with id 7817 lost”).

This patch changes the job watcher to no longer catch
“errors.InotifyError” and, this is by far the larger part of this patch,
adds unittests for this situation.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

- Add more checks, some of them are deliberately redundant
- Descriptive error messages
- Add comment describing order to “tools.ensure_dirs”
- Avoid copying a list in an assertion in “tools.ensure_dirs”

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Change the "--disks" option validation, to just check the format
of the dict and do not check whether the keys are included in the
IDISK_PARAMS constant at OpCode level. This allows the passing of
arbitrary parameters at the CLI, which will then be logically
checked at LU level (inside CheckArguments).

Currently, the check wrt to IDISK_PARAMS already happens in all
related LUs (InstanceSetParams, InstanceCreate, InstanceRecreateDisks),
with the use of ForceDictType(), so there is no need for additional
checking code.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: fixed a unittest that needed adjustement]
Reviewed-by: Iustin Pop <iustin@google.com>

When used instead of a plain call to “frozenset”, this would have
avoided the issue fixed in commit e2dd6ece. The new function is located
in the “compat” module as it will be used at module load time in most
places and should therefore reside in a place with very few
dependencies.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Until now the error codes were not available from the RAPI client
module. A newly added unit test ensures all error codes are contained in
“ECODE_ALL”, as well as ensuring consistency between the RAPI client and
the authoritative “errors” module.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add verification code to mcpu to check an LU's locks. Two whitelists are
provided to exclude LUs from the two tests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Older versions use “-----BEGIN RSA PRIVATE KEY-----” instead of
“-----BEGIN PRIVATE KEY-----”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The design for this is in “doc/design-node-add.rst”. The tool receives a
JSON data structure on stdin and configures the node's daemon after
verifying the received values.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 4679547e implemented the ability to change job's priority after it
was submitted. The code contained a bug whereby it would modify the
input data for an opcode, something the job queue shouldn't do (logical
units do for historical reasons).

This patch removes the line modifying the opcode input and adjusts the
tests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In the future instance creations might have a lock on all nodes as was
the case until the implementation of opportunistic locking. Nodes for
which the lock is not held will be shown to the iallocator plugin as if
they were marked offline.

This patch adds a new parameter named “node_whitelist” to
“IAReqInstanceAlloc”. If set to a list, only nodes contained within are
shown as online.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The network management code needs to verify a MAC address prefix.
Instead of (ab)using NormalizeAndValidateMac, clean code should be used.
Unit tests for NormalizeAndValidateMac are updated and new ones for
NormalizeAndValidateThreeOctetMacPrefix are added.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This code is going to be used by a new utility for setting up the node
daemon. Unit tests are updated/added.

Additionally, the certificate and key stored in “server.pem” are
verified, too.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds a new parameter to “LockSet.acquire” named
“opportunistic”. When enabled the lockset will try to acquire as many
locks as possible, but it won't wait for them (with the exception of the
lockset-internal lock in case the whole set is acquired). This is
implemented by using a timeout of 0 seconds when acquiring individual
locks. Commit 03c5291c made such acquisitions significantly cheaper.

The most complicated code included in this patch is probably the helper
function used to determine which mode to use and which timeout functions
are needed.

Full unit tests are included for the new and changed code
(“locking.py”'s overall coverage is at 97%).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Configuring a node daemon on a newly added node will need all ssconf
values.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A new utility for configuring the node daemon will support a dry-run
mode. This patch adds the necessary functionality to
“ssconf.SimpleStore” and provides comprehensive tests for
“SimpleStore.WriteFiles”. To enable the latter, a testing-only parameter
specifying the lockfile was added to the class.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The new utility for configuring the node daemon will have to check
whether it received valid ssconf names.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is not comprehensive, since in Python one can't determine what is
and what is not mutable; but I've added a few base cases (list, dict,
set).

The patch also improves (makes more uniform) the error messages in the
parameter definitions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

TValueNone checks if a value is "none" and TMaybeValueNone is a wrapper
of TOr(TValueNone, x). This is used by OpNetworkSetParam in order to
reset a network value (e.g. mac_prefix, gateway, etc.)

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Also remove a superfluous empty line in test file.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds descriptors to the “_CheckCIDR*” functions in opcodes
and improves the descriptions generated by “ht.TInstanceOf”, thereby
indirectly fixing bad type descriptions in the RAPI documentation.

Before this patch:
- (String and (<function _CheckCIDRAddrNotation at 0x2f…>))
- (Instance of (<class 'ganeti.opcodes.OpInstanceCreate'>))

After this patch:
- (String and (IPv4 address))
- (Instance of (ganeti.opcodes.OpInstanceCreate))

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

A new tool for configuring the node daemon will also have to verify the
cluster name, so it's better to have this function in a central place.
In the process of moving it to ssconf it is also changed to use
“SimpleStore” instead of reading the ssconf directly. Tests are updated.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Until now ssconf would limit the amount read from files to 128 KiB and
silently ignored files larger than that. With this patch a check is
added by using fstat(2) on the file descriptor while it's being read.

There were no tests file ssconf at all, so some are added.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A new tool to configure the node daemon will also have to load and
verify JSON data.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>