Due to TemplateHaskell stage restrictions, we can't define parameters
in the same module as we're using them for TH, so we have to define
all module parameters in a separate module.

This patch therefore splits OpCodes.hs in two, adding that module and
moves most code there (types, parameters, etc.). The remaining parts
in OpCodes.hs, the actual opcode definitions, now use more parameters
instead of direct field definitions (more will come later)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

There are already three cases where we copied type definitions between
the htools-specific types into the main ganeti code. Let's stop doing
this ☺️

 and create a common types module that holds these.

Note that there already exists BasicTypes.hs, but that refers to very
low-level types, and can't use TH derivation itself.

A side effect of this unification is that there is a small conflict
between AdminStatus/AdminOffline and InstanceStatus/AdminOffline. As
such, I renamed AdminOffline and AdminDown to StatusOffline/StatusDown
in the InstanceStatus type.

The patch also moves the tests related to these types to a new test
module.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

- Remote unnecessary quote characters
- Change some uses of “test "$foo"” to “test -n "$foo"”
- Merge “if/else/if” into “if/elif”
- Wrap lines longer than 80 characters and add a test to “make check”

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The opcode doesn't pay attention to the build-time flag to enable or
disable restricted commands. In a cluster different nodes could have
different settings.

Node locks are acquired in shared mode by default, but the use of an
exclusive lock can be forced using the “use_locking” parameter.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Expose the functionality to run restricted commands remotely via RPC.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Scheduling a pending acquisition is relatively expensive and lot of code
is involved. Unless there is already one, a new pipe needs to be opened.
Data structures need to be updated as well, only to be undone shortly
after. This patch adds a small condition to return straight away if the
timeout is shorter than 1ms and the lock couldn't be acquired right
away. Blocking acquisitions are not affected, as are acquisitions with
longer timeouts.

While I tried hard, I could not find a way of writing a test
specifically for this case which wouldn't have involved making
significant other modifications. Other tests already exercise the new
code, though.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Some versions of hlint crash on the Attoparsec test file because they
are not able to correctly handle unicode characters if the proper encoding
(--utf8) is not specified.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

More design details will be added regarding setting up the node daemon,
so this is more about adding a node than just setting up SSH.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

As per design document (doc/design-remote-commands.rst), a number of
rather strict tests is applied to any incoming request, a delay is
inserted upon errors and returned error messages are very generic
(unless it's the actual command that failed). There are unit tests for
all of the newly added code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

By default remote commands are disabled and need to be explicitely
enabled at build time.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

- Ensure query-related resources have the same access permissions
  (specifically “/2/query/*” and “/2/*/console”)
- Check access permission consistency (write implies read)

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of duplicating what is already described in the full
documentation, a reference is added instead. Man pages can't use
constants or assertions, therefore it is easier for them to get out of
sync with the code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This was requested in issue 301. Before this patch, requests to
“/2/query/*” and “/2/instances/*/console” would require authentication
with a user with write access. Since that is not strictly necessary, a
new user option named “read” is added.

Console information can also be retrieved as a normal query, therefore
the change applies there too.

This was the first user option to be added after “write”, therefore
quite a few changes were necessary. Documentation, including NEWS, is
updated as well.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

They mimic their python counterparts.

Added functions:
* getSourceDir
* testDataFilename
* readTestData
* readPythonTestData

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This allows the option to be re-used in other places.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This allows offlining nodes that don't respond if they are part of a
linux-HA cluster.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This allows controlling the cluster master role if the nodes are part of
a linux-HA cluster.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, THH.hs "injects" into the built code names of library
functions like Text.JSON.makeObj, Ganeti.JSON.fromObj, etc. built
directly from strings, via (e.g.)

  varE (mkName "makeObj")

This means that the "makeObj" name must exist in the target module,
i.o.w. must be imported there. This leads to the strange case of
having to have imports that do not appear at all in the used
(template) code, but are needed to satisfy this "hidden" dependency;
look at Ganeti/Jobs.hs before this patch, for example.

This is also not very obvious, because we usually import Text.JSON
anyway; I only stumbled upon it while doing some cleanup work.

So to clean this up, the current patch changes the THH.hs to use not
string-derived, but identifier-derived names («'identifier» versus
«mkName "identifier"»); this is better, as the names must be
resolvable when compiling THH itself (once), and not when compiling
the multiple derived modules. As you can see, this allows removal of
extraneous imports from various modules.

Background information: an `mkName "foo"` results in a name of flavour
NameS (“An unqualified name; dynamically bound”) or alternatively to a
qualified name, but still dynamically bound. Whereas what we want is a
statically bound name: `'foo` results in a NameG flavour, “Global name
bound outside of the TH AST: An original name”.

One more explanation: the change is similar to going from 'x = eval
"map"' to 'x = map'; the name is no longer dynamically evaluated, but
statically when the module is compiled. In our case, previously names
were bound at target module compile time, now they are bound at THH.hs
compile time.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is not a quickcheck property, since it doesn't have any
variable/arbitrary inputs. So let's make it a test case, and shorten a
bit the name.

Also, sorry for not catching this in the review.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Without this change, invalid lines or values would be silently ignored.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This test does not work properly if localstatedir is not “/etc”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This wasn't clearly documented until now.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Also add tests to ensure it's never allowed as a file storage path. A
constant for the lock file is also added.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The UID is there, the GID wasn't.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In some cases it's nice to verify a function has been called exactly N
times. This is going to be used in tests for remote commands.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is similar to commit f5ce7613. A stable sort order makes changes
between versions easier to find.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Unlike existing tests, this actually tests RAPI at the interface with
the HTTP server. This way authentification can also be tested. A test
for “/2/query/…” is included as it's a bit special.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This can be used to change the priority of a pending or running job (the
latter only if there are unprocessed opcodes).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

A new LUXI request is added, in both Python and Haskell.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is due to a feature request. Sometimes one wants to change the
priority of a job after it has been submitted, e.g. after submitting an
important job only to later notice many other pending jobs which will be
processed first. Priority changes only take effect at the next lock
acquisition or when the job is re-scheduled.

The design is very similar to how jobs are cancelled.

Unit tests for “_QueuedJob.ChangePriority” are included.

Also rename “TestQueuedJob.test” to “TestQueuedJob.testError”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

The job ID is re-used as the task ID, as job IDs are unique.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

When a task is deferred it should receive the same task ID upon being
returned to the pool.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Using the task ID a pending task's priority can be changed. This will be
used to change the priority of jobs in the workerpool.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

To prepare for the addition of a new function allowing changing a
pending task's priority, the internal data structure is slightly
changed. The (optional) task ID is stored as part of the task entry. A
new dictionary provides a mapping from the task ID to its task entry. If
the task ID is None, the entry is not added to the map.

Task entries used to be a tuple, but since modifying the priority
requires changing an entry, they are changed to lists in this patch.
Tuple items can not be modified.

The underlying idea is from [1].

[1]:
http://docs.python.org/library/heapq.html#priority-queue-implementation-notes



Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Some GHC flags are very useful, but only appear in more recent GHC
versions. To support the use of such flags while still supporting
older compilers, let's add conditional checks and enabling based on
the results.

Currently only `-fwarn-incomplete-uni-patterns` is enabled, which
detects refutable patterns in lambda expressions and pattern bindings
(e.g. "let (Just x) = y"); such constructs are bad as they can lead to
runtime exceptions.

Additionally, fix an existing such bad construct in a test case; we
workaround it by using error, since that should never fail.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Attoparsec is known to have had issues with parsing non-ASCII strings.
This test makes sure that parsing of Unicode characters works fine.

Signed-off-by: Michele Tartara <mtartara@google.com>
[iustin: small doc string fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

This will be needed for the data collectors of the monitoring agent.

* Detection of the library
* Creation of the appropriate variables
* Update to the installation documentation

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>