Since now rStripSpace lives in Utils.hs, we can use it directly (as
UUIDs shouldn't contain spaces anyway).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

The client queries all the master candidates in parallel, until the minimum
number of replies, defined in the constant file, is received.
A timeout prevents the waiting from being of indefinite length.

The reply to be returned to the function that made the query is decided
according to the Confd design document.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This commits adds to the Haskell codebase a function to fetch the list of
IPs of the master candidates from the SSConf database.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This function will be used by the Haskell Confd client to parse the replies
it receives from the server.

Unit test is included as well.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

With the upcoming implementation of the Haskell Confd client, requests will
no longer be the only kind of message to be parsed, therefore the parsing
function for requests is being renamed to be more specific, and to better fit
with the upcoming parseReply.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The parseRequest function of the Confd utils can be used to parse both request
(in the server) and reply (in the client, soon to be implemented) signed
messages.
This patch changes the signature of the function to allow this, and its name
accordingly.

A unit test is updated as well.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Its first use will be to generate the salt for ConfD requests of the Haskell
client, as in the Python client.

Unit test added as well.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The Confd server implementation has a getCurrentTime function.
This commit moves it to the Utils.hs file of Confd, so that it will be
available to be used by the future Confd client as well.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Sorry, didn't see this before pushing the previous commit (277a2ec9).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This just adds new constructors for the disk template and related
types, enabling the Ext storage.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The Python confd client sends newlines in messages (due to use of
DumpSignedJson/DumpJson), so that results in ugly debug output.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Also renames it in the process and adds a few tests for it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The networks field of the node groups class is a dictionary
of nic params and not a list of networks. This should fix
the broken QA.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

By abstracting the command/personality formatting, we can easily
switch htools to the same output as the rest of the commands.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This is highly incomplete, as per the manpage BUGS section.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is a new personality that for the moment doesn't do anything.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This adds a network field to the NIC params and a list of networks
to the node group type. It adjusts the nodegroup's unit tests accordingly.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Networks will be properties of NicParams and NodeGroups. Therefore
they need to be defined before those.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch is enabling split queries for jobs for gnt-node and rapi
access (only for job listing, not job waiting).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This shouldn't live in the main Query module, since it's too node
specific.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This adds support for job queries, including (basic) unit-tests.

I've tested this for memory and cpu usage as follows:

- 3600 jobs (live queue):
  - via masterd, default: ~1.1s (masterd: ~60MB ram)
  - via confd,   default: ~1.1s (hconfd:  ~25MB ram)
  - via masterd, id only: ~1.0s (masterd: ~57MB ram)
  - via confd,   id only: ~0.2s (hconfd:  ~15MB ram)

- all jobs (128K in total, around 570MB size on disk):
  - via masterd, default: 1m22s (masterd cpu: 48s), masterd: 1.4G ram
  - via confd,   default: 1m16s (hconfd  cpu: 51s), hconfd:  570MB ram peak
    (peak only right before starting luxi send, hconfd decreases in RSS as
    results are streamed over luxi, back to ~18MB after the send)
  - via masterd, id only:  ~49s (masterd cpu: 45s), masterd: 1.3G ram
  - via confd,   id only:  ~39s (hconfd cpu: 35s),  hconfd:  110MB ram peak
    (right before luxi send, decreasing as results are sent, back to ~14MB
    after the send)

Given this, and that in production it's not likely to have hundreds of
thousand of job files, I believe the implementation is safe from this
point of view.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This is required for QueryJobs implementation; we change
makeSimpleFilter to support both string and integers as names.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This patch adds implementation for a read-only job queue module,
together with "full" test (as full as can be in a lazy language…).

One note about the behaviour of the job queue is the handling of
opcodes that fail validation: the 'input' opcode actually is a
meta-type, which can hold either a real opcode or a plain JSValue, so
that we can still load jobs with invalid opcodes for querying. The
only downside of this is that, as opposed to Python code, we can't
show the correct summary for such an opcode - we try to parse the
OP_ID but not the extended OP_DSC_FIELD-equivalent.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

I am very very unhappy with this patch. We have to do this, in order
to be compatible with the Python code and behaviour, but there's lots
of duplication; both the Python and the Haskell code will need to be
cleaned up and simplified (we don't need QFF_* at all).

For now though, this implements QffTimestamp and associated code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Unit tests updated and expanded with an inheritance check.

The flag has no effect yet.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

As OpParams definitions are used at Template Haskell type, they don't
get any coverage (although the values defined by them are actually
used in unittests). However, we can at least test some of the
functions defined in the module.

This patch tests the failure modes of a few of the custom types in
this module.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

"Use comparing" was not needed, and "Use on" can be solved by actually
using on in the single case we have (the patch also reorders imports
in that file).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, this doesn't abort if multiple arguments are passed, and
the error message for errors during file read has a duplicate "Error".

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This will allow build-bash-completion to understand what is the list
of commands that each binary supports, and then provide recursively
completion support for each command.

Note: by "fake", I mean we use hand-built output and hardcoded
"--help-completion" matching, as opposed to automatically doing it via
regular options. It could be possible in the future to use a
two-staged command line parser, but for now not sure if it's worth it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

No need to do a boolean-based lookup, we can use the regular
association list lookup.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This allows usage information to display nicer help (like in Python).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, the mon-collector binary is the only multi-command binary
we have in Haskell. Not surprisingly therefore, its command line
handling is not as robust as we need, not being able to support
standard "--help" commands.

This patch introduces generic multi-command handling, similar to the
Python functionality (some parts missing, will be added in future
patches), and switches mon-collector over to it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

The signature of the personality definitions is so ugly that, at
least, we should hide it a bit behind a type alias.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, the mon-collector binary uses the HTools/CLI module, which
is OK but mean it links in lots of htools code. By copying that module
to DataCollectors/CLI and removing the unneeded code, we reduce the
number of modules it depends on fro 20 to 12, meaning both a shorter
compiler time (24s to 9s) and a smaller binary (~9.6MB to ~7MB).

Also fixes a typo in the original HTools/CLI module, thanks Michele!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

If an instance is offline we definitely shouldn't start it up.
But shutting it down, should it be up by mistake is not "that" bad.
Still, we only allow it with --force, as it still performs an action on
an instance we shouldn't touch. This should make everybody happy.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit e821050d (“Switch the Luxi interface from Strings to
ByteStrings”) was designed to optimise the receive interface, but has
an unfortunate side-effect: when sending non-trivial messages, it
means that both the entire String and the ByteString versions must be
in memory at the same time, leading to much increased memory usage.

By changing the "hPut" from strict to lazy ByteStrings, it means that
both the String and the ByteString values can be evaluated lazily,
with significant effects: for a test query answer, instead of having
a peak from ~600MB to 1.4G during the entire Luxi send operation,
memory consumption actually decreased during the send operation, as
the ByteString chunks are released individually and even the backing
storage of the items that create the JSON string serialisation is
released lazily as well. So instead of slow growth 10→550MB then quick
peak to 1.4GB during Luxi send, we now have an even slower growth to
~580MB and then decrease of memory as the Luxi send progresses.

The only downside is of a small increase in CPU time of a few percents
for the above case; for our use cases, I think this is much desirable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This allows us to ensure that query results are strict as we build
them, instead of being lazy and only evaluated when the reply is sent
over the Luxi interface.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The JSValue/JSObject types don't come with a NFData instance, so let's
add one ourselves, so that we can force evaluation of JSValues (either
when building or when reading them).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

It seems that Python code generates (sometimes) absolute job
dependencies which are strings, instead of integers, so we should be
able to parse these as well. We simply change from explicit int-based
parsing (makeJobId) to the generic one (parseJobId).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This changes deps and comment fields to always be shown, to match the
Python behaviour for (at least) job listing/ops field.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>