Some confd types were not tested for serialisation, so let's move
these (type-related) tests to their own module and extend them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Commit e821050d (“Switch the Luxi interface from Strings to
ByteStrings”) was designed to optimise the receive interface, but has
an unfortunate side-effect: when sending non-trivial messages, it
means that both the entire String and the ByteString versions must be
in memory at the same time, leading to much increased memory usage.

By changing the "hPut" from strict to lazy ByteStrings, it means that
both the String and the ByteString values can be evaluated lazily,
with significant effects: for a test query answer, instead of having
a peak from ~600MB to 1.4G during the entire Luxi send operation,
memory consumption actually decreased during the send operation, as
the ByteString chunks are released individually and even the backing
storage of the items that create the JSON string serialisation is
released lazily as well. So instead of slow growth 10→550MB then quick
peak to 1.4GB during Luxi send, we now have an even slower growth to
~580MB and then decrease of memory as the Luxi send progresses.

The only downside is of a small increase in CPU time of a few percents
for the above case; for our use cases, I think this is much desirable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This allows us to ensure that query results are strict as we build
them, instead of being lazy and only evaluated when the reply is sent
over the Luxi interface.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The JSValue/JSObject types don't come with a NFData instance, so let's
add one ourselves, so that we can force evaluation of JSValues (either
when building or when reading them).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

It seems that Python code generates (sometimes) absolute job
dependencies which are strings, instead of integers, so we should be
able to parse these as well. We simply change from explicit int-based
parsing (makeJobId) to the generic one (parseJobId).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This changes deps and comment fields to always be shown, to match the
Python behaviour for (at least) job listing/ops field.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This is not a required field, but rather an optional one; we add a new
parameter and use it instead.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Python code sometimes sends this, so let's support it even though it's
non-standard.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Having makeJobIdS as a separate function will allow us to use it
outside of json encoding.

The patch also exports one more function from the Types module.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

The 'temporary' library add support for create temporary
directories/files beyond what is offered in the base library (which is
somewhat limited, and hard to use in tests).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This allows tests to execute IO actions which have logging as a side
effect, without polluting the stderr too much.

A better solution would be that we have fine-grained control over
loggers, so that tests can run with their own logging, etc. etc..

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This single constant could be as well moved to constants.py. For now I
choose to just export the module as well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of always using "arbitrary", make it require a generator (that
can then use more complex rules for building the list elements), and
also convert it to use a temporary set instead of list membership.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Also adds a small helper for building the paths.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This implements in the Haskell codebase the opcode summary. As opposed
to Python, we always use custom code for formatting, since we don't
want to use dynamic attribute lookup.

To test this properly, we need to change MetaOpCode to record-syntax,
so that we get accessor functions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

For some opcodes, the output is not "stable", and depends on the exact
input values; this makes it harder to check consistency against
Haskell code.

To compensate for this, we add a way to override the formatting of the
OP_DSC_FIELD; by default, this is always "%s", but if the
OP_DSC_FORMATTER is defined (must be a callable), it is used to format
the actual value.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Also fixes indentation for OP_STATUS.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The extra space results in inaccessible names; currently GHC doesn't
flag this as an error, but I've filled an upstream bug for that
(http://hackage.haskell.org/trac/ghc/ticket/7484

).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Two constants which we use as Integer are defined as Int in
Constants.hs (coming from constants.py), so we do the conversion every
time we use it. Let's move them to top-level names, so that the
conversion is only done once.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Introduced in commit d4752785.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

this (a) allows us to test renaming in burnin even if we don't have an
extra instance name and (b) is a valid operation, as it allows to "fix"
an instance who went name-astray

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is a "better than nothing" support, just for kvm and just joining
the machine to the opevswitch bridge with the right command.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Return type is changed from Property to Bool, and the ==? True at the
end is dropped.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Also update the docstring of a function.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

As discussed on the list, these are needed to do a make hs-prof.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, we log the entire response (at debug level) in the Luxi
replies. This is not a good idea; the logging library operates on
strings, and as such it will use huge amounts of memory: without debug
logging, a certain gnt-job list invocation uses 295MB (+RTS -s) and
2m35s time, when in debug mode, it's 1525MB and 48m!

So we make two changes:

- first, we switch from "show (pp_value a)" to "encode a", which
  generates a non-formatted string rather than a indented one
- second we log only the first 2000 characters; this should be enough
  to understand the first part of the response

We could go for higher, or for logging in batches (that would be
faster, as well).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This should be done because in the case of --no-name-check there
is no 'hostname.ip' attribute, causing an execution error.
'check_ip' is always set (in CheckArguments) even if --no-name-check
is passed in the command line.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is similar to README or “doc/security.rst”. Having a comment that
needs updating for a new version is more likely going to trigger a check
of the lists of design drafts.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Use the “error” variable in multiple places and fail only once all tests
have been completed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

These are the ones I implemented and for which I'm certain they're
complete.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

- Mention user capabilities in security.rst
- Replace “query” with “read” in RAPI documentation

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It is a binary file and it should never be added to the git repository.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The master daemon should not directly read files written by the node
daemon. This patch adds a new RPC to read the watcher pause file and
changes the master code to use it.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of just setting the watcher pause file, which isn't replicated,
RPC is used to set it on all nodes (where possible). This means that
after an outage of the master node and a following master-failover, the
watcher will still be paused.

The RPC is attempted on all nodes. Errors for nodes marked offline are
ignored. All other errors are returned in the form of an exception
(there is no other option given the LUXI API).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The watcher pause file should be set/unset on all nodes at once, not
only the master node. For that a new RPC is needed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

There is no obvious need for this space--the line would wrap at the next
space, which comes soon after.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

“TestWriteFile” was not renamed when adding the file based on
“ganeti.utils.io_unittest-runasroot.py”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This addresses issue 218. When the number of inotify watches is
exhausted, for example by being set too low from the beginning or by
other programs, waiting for a job to change would just report a lost job
(e.g. “Error checking job status: Job with id 7817 lost”).

This patch changes the job watcher to no longer catch
“errors.InotifyError” and, this is by far the larger part of this patch,
adds unittests for this situation.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>