Commit 3bdbe4b3 (“Jobs.hs: move OpStatus and JobStatus ADTs to
Types.hs”) removed the TemplateHaskell language pragma from
htest/Test/Ganeti/Jobs.hs due to a hlint warning, but that is bad: it
means the testSuite call is no longer interpreted as a splice, so it results in:

  Parse error: naked expression at top level

With newer GHCs. To fix this, we restore the pragma, and add an hlint
ignore.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

configure.ac advertised that the HLINT environment variable could be set to
specify the path to the 'hlint' tool. However, HLINT was being initialized
to "no" inside configure.ac, which broke this usage.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Ganeti.Jobs now holds functions that can be used to submit and monitor the
status of jobs. In particular, execJobsWait and waitForJobs are factored
out of Hbal.hs.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Previously, functions in Hbal.hs related to execution of jobsets were
returning only IO Bool, and printing any errors they found directly to
stderr on their own.

I'm going to be moving some of these functions to a library module in
future commits, and it makes sense that they won't print to stderr, but
rather return an error condition. To make diffs more readable, I change the
return value in Hbal.hs itself, so that the next commit deals only with the
move. It's now `main` that prints any Bad result to stderr.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In Hbal.hs, it's easy to get lost in the flow of exec* functions
because their names are similar and don't convey their different
purposes (e.g. runJobSet, execJobSet, execWrapper).

This patch renames 'runJobSet' to 'execWithCancel', and 'execWrapper'
to 'execCancelWrapper', since these two functions deal, in particular,
with early termination when the user presses ^C.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

'hangleSigInt' and 'hangleSigTerm' are renamed to 'handleSigInt' and
'handleSigTerm', respectively.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This leaves Ganeti/Jobs.hs and Test/Ganeti/Jobs.hs empty, but they're the
target of a future move of some functions, so we leave them around, and
don't delete them, to avoid unnecessary delete/create diffs.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

epydoc allows overriding configuration values via environment variables.
While this might be useful in certain use cases, but as no prefix
whatsoever is used, conflicts are easily created. Some people have the
environment variable “NAME” set, effectively overriding the project name
set in epydoc.conf. A bug in epydoc causes an error if non-ASCII
characters, such as German umlauts, are used in NAME.

$ NAME=Täscht make py-apidoc
[…]
UNEXPECTED ERROR:
'ascii' codec can't decode byte 0xc3 in position 73: ordinal not in range(128)

$ parse=false make py-apidoc
[…]
epydoc: error: Invalid option combination: --parse-only and --introspect-only.

This patch changes the call in Makefile to reset the environment given
to epydoc save for PATH and PYTHONPATH.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 4679547e implemented the ability to change job's priority after it
was submitted. The code contained a bug whereby it would modify the
input data for an opcode, something the job queue shouldn't do (logical
units do for historical reasons).

This patch removes the line modifying the opcode input and adjusts the
tests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Use new NormalizeAndValidateThreeOctetMacPrefix() util function in
LUNetworkAdd/LUNetworkSetParams to validate network's MAC prefix.
Additionally, move the check in CheckArguments() in the case of
LUNetworkAdd.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

All node locks are acquired.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This opcode acquires all node resource locks, which conflicts with
instance allocations.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

See comment in code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

See comment in code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

If the lock was acquired in the first place (only when an iallocator is
used), it is released as soon as possible.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Changing instances' groups shouldn't conflict with instance allocations.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

If locking is used (usually by ganeti-watcher), node allocations must be
temporarily blocked.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Until now they are only mentioned in “doc/admin.rst” and the man page
for “ganeti-listrunner”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In the future instance creations might have a lock on all nodes as was
the case until the implementation of opportunistic locking. Nodes for
which the lock is not held will be shown to the iallocator plugin as if
they were marked offline.

This patch adds a new parameter named “node_whitelist” to
“IAReqInstanceAlloc”. If set to a list, only nodes contained within are
shown as online.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

In some cases it is useful to ignore the output of and avoid mentioning
successful commands. One would be when looking for a certain string in
a file:

  $ gnt-cluster command egrep -q '^testing$' /etc/...

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

We have to check that for each edge its vertices have different colors.

This is very easy to do with a vertex-to-color map, but not so easy with
a color-to-vertex one. Since all our coloring algorithms created a
vertex-to-color map behind the scenes and then converted it, we flip
them back to returning it directly, and do the conversion explicitly
where we need it (which for now is everywhere except when testing this
property).

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Our Dsatur implementation was incorrect: while the paper defined the
degree of saturation of a vertex as the number of different colors it is
adjacent to, we were using the number of colors, without considering
uniqueness. This effectively implemented a different algorithm, which is
very similar to the previous one, and while it performs slightly worse
on average it still beats Dsatur on some cases.

So we refactor the implementation to effectively support both algorithms
without code duplication, and then we export both the old algorithms as
"Dcolor" and the new one as "Dsatur". Since these are all fast
algorithms in hroller we will still be able to pick the best result.

Note that the new Dsatur implementation uses an IntSet to calculate the
uniqueness. Results with nub + length on a list were significantly
slower.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This function helps treating node node problems as graph problems.
It can transform a list of nodes plus a list of instances into a graph
which uses the nodes as vertices, and instances as edges connecting them
(as long as they have both a primary and a secondary node)

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This brings our coverage of Graph.hs to 100%

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Check whether coloring on a given graph makes sense. This is the case
only if there are no loops and the graph is undirected.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Implement the Dsatur algorithm for Graph coloring. This also abstracts
the neighColors function into two subfunctions that this algorithm can
reuse.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This module implements some algorithms on Data.Graph data structures.
At the moment its main functionality is an LF-color implementation
(greedy coloring in descending order of degree). There are also a few
extra functions to calculate the degree order, and convert the node to
color mapping to color to nodes.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Comments with a colon after the hash sign (“#:”) show up in the
epydoc output.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will show smaller/better error messages: full node dumps are no
longer included.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The node resource locks were not set correctly on instance import.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch changes the luxi submit job calls to use wrapped opcodes,
and therefore it changes Hbal to submit actual meta opcodes. For
nicety, hbal also submits a comment now, showing who generated the
job.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This fixes:

  Got job IDs JobId {fromJobId = 1052613}

And restores to:

  Got job IDs 1052624

Other improvements could be done here, of course.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

If TIsLength is applied to a non-container item, it will fail (type
error) due to invalid application of len(). Since this can happen on
user-supplied data, we add an explicit TList/TTuple check (the TTuple
test is a new one).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds the "meta" opcode type and the common op
params. Compatibility tests with Python are changed to pass Meta
opcodes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This was deprecated via commit 88609f00, “Switch Luxi TH code from
simple to custom fields”.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the opcode and luxi "showJSON" functions generate directly
a JSValue; in contrast, the object (single-constructor) types generate
a 'toDict' function, and then `showJSON = makeObj . toDict`. This is
useful, as the 'dict' form can be manipulated if needed.

This patch changes the opcode and luxi types to behave the same; we
generate a dict, and then (since this differs between opcodes and
luxi) generate showJSON either as `makeObj . toDict` (for opcodes), or
(for luxi) `showJSON . map sn . toDict`.

The change is needed for MetaOpCode implementation.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is reused in more than just a few places, so adding it makes the
signatures much nicer.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This will go into a separate type; this patch adds the needed
underlying types and parameters.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This mirrors the positive one, and will be needed for relative job
IDs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the job ID is a simple type alias. This is suboptimal, as
it means we can't use a custom JSON (or Arbitrary) instance for it.

The patch changes it into a newtype, and then a) simplifies some
deserialisation code and b) changes some more fields to this new type
(rather than plain 'Int').

We also move the JobId to types, since it will be needed in opcodes as
well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>