GitLab

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

If cluster is initialized with --no-lvm-storage then volume_group_name
does not exist in config.data. Thus we must define it as optional in
confd.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
[iustin@google.com: fixed Haskell RPC definition]
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Since 'head' is unsafe to use in most cases, this patch removes its
use from most of the code, adds a lint warning for it (and for tail as
well), and adds override annotations in the few cases where it's
actually OK to use it (mainly when using head over the result of
functions which guarantee to return a non-empty list by documentation,
not type).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Per offline discussions, this is the first patch of the
renames. Tested with "make distcheck", seems to work fine.

The only change outside of the renaming is a bit of simplification in
the .gitignore rules; otherwise, simply s/htools/src/.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This adds support for job queries, including (basic) unit-tests.

I've tested this for memory and cpu usage as follows:

- 3600 jobs (live queue):
  - via masterd, default: ~1.1s (masterd: ~60MB ram)
  - via confd,   default: ~1.1s (hconfd:  ~25MB ram)
  - via masterd, id only: ~1.0s (masterd: ~57MB ram)
  - via confd,   id only: ~0.2s (hconfd:  ~15MB ram)

- all jobs (128K in total, around 570MB size on disk):
  - via masterd, default: 1m22s (masterd cpu: 48s), masterd: 1.4G ram
  - via confd,   default: 1m16s (hconfd  cpu: 51s), hconfd:  570MB ram peak
    (peak only right before starting luxi send, hconfd decreases in RSS as
    results are streamed over luxi, back to ~18MB after the send)
  - via masterd, id only:  ~49s (masterd cpu: 45s), masterd: 1.3G ram
  - via confd,   id only:  ~39s (hconfd cpu: 35s),  hconfd:  110MB ram peak
    (right before luxi send, decreasing as results are sent, back to ~14MB
    after the send)

Given this, and that in production it's not likely to have hundreds of
thousand of job files, I believe the implementation is safe from this
point of view.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This is required for QueryJobs implementation; we change
makeSimpleFilter to support both string and integers as names.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Currently, we log the entire response (at debug level) in the Luxi
replies. This is not a good idea; the logging library operates on
strings, and as such it will use huge amounts of memory: without debug
logging, a certain gnt-job list invocation uses 295MB (+RTS -s) and
2m35s time, when in debug mode, it's 1525MB and 48m!

So we make two changes:

- first, we switch from "show (pp_value a)" to "encode a", which
  generates a non-formatted string rather than a indented one
- second we log only the first 2000 characters; this should be enough
  to understand the first part of the response

We could go for higher, or for logging in batches (that would be
faster, as well).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The QR_VIA_OP/QR_VIA_LUXI types in Python are using yet another
validation mode: QR_VIA_OP is the base type, and QR_VIA_LUXI extends
it (when doing luxi queries). But on the wire they have the same
representation.

To accommodate this properly, we split the ItemType in two: a
QueryTypeOp and a QueryTypeLuxi, joining them back together in
ItemType itself. This requires custom serialisation/deserialisation,
but allows us to express correctly that at opcode level, we only
accept a QueryTypeOp, but at Luxi query level, we accept either of
them.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

We need to change a few things, most importantly CLI options defaults,
but otherwise we already used the path to files from functions which
were already in the I/O monad, so we don't have to change much of the
code flow.

Additionally, Path.hs now has an explicit export list, to stop
accidental leakage of symbols.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The luxi binding is now annotated for better readability.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This will help with the general daemon split of prepare/run, and flag
errors earlier in the startup.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, we keep information about the "target" of a tag operation
in a data type similar to (TagKind, Maybe String). This is unsafe, as
nothing (at the type level) prevents us from accidentally having
(TagCluster, Just "instance1.example.com"), or (TagInstance, Nothing).

To fix this problem, we rename the current TagObject type to TagType
(an internal utility type), and create TagObject as a better/safer
data type (see the definition), which doesn't allow such possibilities
in the future.

The downside is that, since at encoding level (both opcode and luxi)
this is done in an ugly way (type elements spread at the same level as
level as other value), we have to add custom encoders/decoders. The
encoder is shared between the OpCode and Luxi usage, the decoder is
different however as Luxi uses custom decoding.

This also fixes the recent breakage in confd w.r.t. QueryTags.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In addition to ReqQueryTags in Luxi.hs, the TagObject ADT is also
required for the "kind" attribute of OpTagsSet and OpTagsDel, which
are coming to OpCodes.hs next. Hence, we move TagObject there, and
adjust imports accordingly.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch converts all the call paths from 'Result' (which contains
just string errors) to 'ErrorResult', which holds
GanetiException-encoded errors. We can now return proper
OpPrereq/OpExec errors to the clients of the luxi/query socket.

The patch touches many files as we had to convert the entire call
chains in a single round. But it should be pretty straightforward
otherwise:

- change 'Result' into 'ErrorResult'
- add error annotations: change "Bad msg" into "Bad (XXXEror msg)"
- add a helper function for confd, where we don't send to client
  formatted exceptions, to convert back from ErrorResult into Result
- change tests similarly, where needed
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds support for classic-style queries (before query2) to
the query socket server. The patch is rather trivial, since as in
Python we just piggy-back on the query2 implementation.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

The tests we currently have assume, that all the data required for
running the query is available - once we add live data, this will no
longer be the case.

This patch adds boolean parameter to query function, which tells it
whether to ignore live parameters gathering.
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

I assumed this was a copy-paste+forgetting to change the header comment.
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Future changes will change Path.hs to use an environment variable.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit 2cdaf225, “Re-enable standard hlint warnings”, got it almost
right. The only problem is that (confusingly) the default set of hints
is not in HLint.Default, but in HLint.HLint (it includes Default and
some built-ins).

After changing the lint file to correctly include the defaults, we had
another 128 suggestions:

  - Error: Eta reduce (2)
  - Error: Redundant bracket (4)
  - Error: Redundant do (17)
  - Error: Redundant lambda (7)
  - Error: Redundant return (1)
  - Warning: Avoid lambda (2)
  - Warning: Redundant $ (42)
  - Warning: Redundant bracket (35)
  - Warning: Use : (1)
  - Warning: Use String (4)
  - Warning: Use camelCase (10)
  - Warning: Use section (3)

which are fixed by the current patch. Note that the 10 "Use camelCase"
were all due to hlint not “knowing” the idiom of ‘case_’ (it does for
‘prop_’), for which I filled
http://code.google.com/p/ndmitchell/issues/detail?id=558

.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Per the new query module hierarchy, rename Qlang to Query/Language and
Queryd to Query/Server. This way, all query-related functionality is
now "contained" in the Query/ directory.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Since we have all the definitions already, we can easily enable
this. Manual testing shows no difference between the Python and the
Haskell versions of node list-fields.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch corrects the definitions in Qlang.hs to match what Python
expects on the wire; this means replacing some manual data type
definitions with 'buildObject' so that we get serialisation (and field
names) for free, adding (manually) JSON instances for types which are
not represented as objects in JSON, etc. Due to more TH usage, I had
to shift some definitions around, since TH breaks the "define in any
order" property (☹

).

After that, we simply add a call into the stub Query/Query.hs module
which, for all queries, responds with "query not supported". The
reason for the deep directory structure is because I want to separate
the functionality into multiple submodules, for smaller module size
and readability.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Currently, the query daemon would simply close the connection to the
client without issuing a response, if parsing the arguments failed;
the error was just logged.

Since this is very ugly from the client's point of view, we change it
so that the error is both logged and sent back to the
client. Furthermore, the execution block reporting is also cleaned up
a bit, to make the code more uniform.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch adds a simple function to ensure a socket doesn't exist
before binding to it. Additionally, we cleanup the exports of
Queryd.hs.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Since we have now access to all cluster parameters, we can "fill" the
parameters, and can finally enable it in CLI/RAPI.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This patch adds the tags field to the objects which were missing it
(all except Cluster), implements handling the LuxiCall QueryTags, and
then enables the use of the query socket in cli.ListTags, used by all
commands, and in the RAPI client.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is just a new module that exports a runQueryD function, that can
be imported to run a separate thread handling the luxi requests.

Currently it needs access just to the configuration, in the future it
will need access to an RPC runner too.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>