GitLab

Some options have defaults that depend on the environment, and we
could handle these in two ways:

- use a place-holder value (e.g. data X a = Default | Custom a) that
  is later read from the environment
- move the options list to IO monad, where it can read the
  environment, etc.

The second option allows also displaying the actual defaults in the
`--help' output, even though it's not as nice, so I went with it.

This patch only changes the option types, without actually changing
any options yet.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The "starting" message is changed to match the Python one, and in case
the preparation fails, we also log the error (beside printing it on
stderr or writing it to the error reporting pipe), as at this time
logging is usually set up.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The luxi binding is now annotated for better readability.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, the writing of the PID file uses a "standard" error
reporting: catch exception, transform it into a 'Result' type, leave
handling of that to the caller. However, for daemon startup, we
actually want exceptions to be propagated up until the handler which
will write the details to the pipe.

This patch removes the writePidFile wrapper (and stops exporting it),
and changes the code to simply annotate better the I/O error.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

It turns out that annotateIOError already exists in the standard
library, with a different purpose (this made me waste 10 minutes
trying to understand why the types were wrong…). Let's move this out
of BasicTypes, since it's used only in Daemon.hs, and also rename it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This changes the daemon reporting error to the same mechanism as in
Python: use a pipe which is written to from the forked children.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This does the address parsing earlier, before we fork, for
better/faster error reporting.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This will help with the general daemon split of prepare/run, and flag
errors earlier in the startup.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, we keep information about the "target" of a tag operation
in a data type similar to (TagKind, Maybe String). This is unsafe, as
nothing (at the type level) prevents us from accidentally having
(TagCluster, Just "instance1.example.com"), or (TagInstance, Nothing).

To fix this problem, we rename the current TagObject type to TagType
(an internal utility type), and create TagObject as a better/safer
data type (see the definition), which doesn't allow such possibilities
in the future.

The downside is that, since at encoding level (both opcode and luxi)
this is done in an ugly way (type elements spread at the same level as
level as other value), we have to add custom encoders/decoders. The
encoder is shared between the OpCode and Luxi usage, the decoder is
different however as Luxi uses custom decoding.

This also fixes the recent breakage in confd w.r.t. QueryTags.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is needed so that we have more flexibility in generating Luxi
serialisation code (deserialisation is still custom). Also, only
exceptions are now using the 'simple' field types, so we might be able
later to convert and remove that TH code as well.

Since we will use custom serialisation fields in the future, we change
the order of serialisation for custom-save fields; Luxi uses
positional as opposed to name-based ordering, so we need to keep this
stable.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This can be improved, by taking all hardcoded names as parameters, to
serve as a more-general "build save clause for a multi-constructor
data type". I'm not renaming the function as well, since I don't know
exactly how much we can abstract later.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This should be the last module rename, promise!

We rename this to conform to the other hierarchies (e.g. Query), and
to not have both Confd.hs and Confd/*.hs.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Also, adjust comment to $(genOpCode) block to avoid repetition of
"only".
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In addition to ReqQueryTags in Luxi.hs, the TagObject ADT is also
required for the "kind" attribute of OpTagsSet and OpTagsDel, which
are coming to OpCodes.hs next. Hence, we move TagObject there, and
adjust imports accordingly.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This mirrors the code in the Python code base, and is required for
clean error reporting during startup.

This patch implements the basic infrastructure; the confd daemon is
not yet modified to take advantage of this, just the types are
adjusted.

Also, the pipe-based error reporting will go in a future patch, once
we can actually use that for reporting.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is only in master, so needed to be fixed separately.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This small patch fixes compatibility with a few newer Haskell libraries:

- base 4.6, included with ghc 7.6, removed the deprecated 'catch'
  function from Prelude, so our "import Prelude hiding (catch)" is now
  an error; we workaround by using fully-qualified
  Control.Exception.catch name

- containers 0.5 changed the signature of 'deleteFindMax'; we
  workaround by using separate 'findMax' and 'deleteMax'

- QuickCheck 2.5 removed the 'maxDiscards' test parameter, replacing
  it with a much better 'maxDiscardsRatio'; however, until we can
  depend on that, we workaround by just removing it (we don't control
  anymore the maxDiscards, instead leaving it default; for our default
  test size, this is no change, as the default value is already 500,
  which is our default as well) and not printing it anymore

Tested on Squeeze (+extra libs), Wheezy and experimental, which covers
all supported GHC versions.

Also, merging this in master will be a pain, but unless we want to
stop supporting 2.6…
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This tests that the same Luxi calls are defined in Python and
Haskell. It doesn't test yet that their serialisation is correct
though.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This a bit too complex patch converts the result of Luxi calls
(submitJob, query*, etc.) from Result to ErrorResult. It then
immediately revers this in the HTools/Backend/Luxi module, where we
don't need necessarily the full error type (just a nice error
message), and does the same in Hbal's job execution functions.

While at first sight this doesn't seem to do much, what we get is
actual error messages from Ganeti, plus improvements to the result
parsing: instead of "can't parse char", we now get properly (note,
wrapped manually):

  Executing jobset for instances instance1, …
  Job submission error: Failure: the job queue is marked for drain and
    doesn't accept new requests

Or:

  Job submission error: Unhandled exception: LuxiError "parsing job
    id: cannot parse string 'a956101'"
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Five modules under the HTools/ directories are backend
implementations, so let's move them to a separate directory, to more
clearly show the hierarchy. I wanted to do this for a while, but
merging between branches is always an issue, so let's do it know since
we have an opportunity.

This patch contains the actual renames, the required changed module
names, imports, etc., but no other changes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Thanks Dato for catching this.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Testing with a newer hlint found a few minor issues; but all are real,
valid recommendations:

- don't use "if cond then f x else f y", but "f (if cond then x else y)"
- "if a then b else True" is equivalent to the simpler "not a || b"
- and as usual, one more ignore to our "testing basic properties"
  module
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This follows a conversation we had for how to deal with
optional-but-required fields in JSON serialisations: fields which are
optional (can be either a given type or 'null'), but where the 'null'
value is required. There are just a few of these in the Python code,
but we should support them nevertheless.

The patch changes the 'isOptional' attribute from boolean to a custom
ADT, three-typed. This allows us to keep the same path on load (which
deals with both cases), but use a custom save path where we explicitly
save the 'null' value.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Change {exp,act}Code to {exp,act}Ver, which gives a better idea that
the integer fields represent version numbers.

Also:

  - errors.py: update OpPrereqError's docstring to note that an error
    code is always expected as the second argument (it was previously
    optional).
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The isOptional function is no longer used after a1505857

 (“Convert
opcode TH code to the use of Field type”).
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch converts all the call paths from 'Result' (which contains
just string errors) to 'ErrorResult', which holds
GanetiException-encoded errors. We can now return proper
OpPrereq/OpExec errors to the clients of the luxi/query socket.

The patch touches many files as we had to convert the entire call
chains in a single round. But it should be pretty straightforward
otherwise:

- change 'Result' into 'ErrorResult'
- add error annotations: change "Bad msg" into "Bad (XXXEror msg)"
- add a helper function for confd, where we don't send to client
  formatted exceptions, to convert back from ErrorResult into Result
- change tests similarly, where needed
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In Python, formatError also returns the exit code, but I find that
splitting them leads to clearer code.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

As described in the module doc string, while writing this it dawned
upon me that we're mixing all errors together into a single hierarchy
(well, type on the Haskell side), which is not good. Some errors are
used purely within noded, some in the CLI frontends, etc. so these
should not be the same type; frontend functions should only be able to
raise frontend errors, not backend ones.

As to this patch itself, I've used again Template Haskell to generate
both the data type and the serialisation functions, as the initial
version, hand-written, seemed too prone to errors due to string
matching.

A small unittest for checking serialisation consistency is also added.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

These are almost generic, so let's change the signatures a bit a make
them fully so.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

This 'simple' way of defining objects will be used also for errors, so
let's make it less Luxi-specific.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch includes:

* The 'failN1' flag is now only set if there is strictly less
  memory available than required for failover.
* Unit tests for that.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Small simplifications of other unit tests using the (==?)
operator when possible, and typo fixes.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

eitherToResult now converts from `Either a b` to `GenericResult`, not
necessarily from `Either String` only. Also, fix a typo.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Since we now have the GeneralResult as a multi-purpose monad, we can
remove the custom OpResult monad, and just use 'GeneralResult
FailMode' as our type. This allows removal of a few bits of
specialised infrastructure, relying instead on the generic one.

The restriction on using OpResult as a general monad remains as
before.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Furthermore, a few messages have their capitalisation changed (fixed).
Signed-off-by: Helga Velroyen <helgav@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, our error monad—Result—has a plain string error type. This
is not good, as we don't have structured errors, we can't pass back
proper error information to Python code, etc.

To solve this, we generalise this type as 'GenericResult a', and make
Result an alias to 'GenericResult String' for compatibility with the
old code. New error hierarchies will be introduced as different
types. Furthermore, we generalise our helper functions too, so that
they can work on any 'GeneralInstance a' type, not only Result.

There are two small drawbacks to this generalisation. First, a Monad
instance requires (at least for the way we use it) a 'fail :: String
-> m a' instance, so we need to be able to build an 'a' value from a
string; therefore, we can implement the Monad instance only for a
newly-introduced typeclass, 'FromString', which requires the needed
conversion function. Second, due to the fact that 'String' is a type
alias (for [Char]) instead of an actual type, we need to enable the
FlexibleInstances language pragma; as far as I know, this has no
significant drawbacks.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This commit adds a Group.allTags field to store the tags of node groups,
and teaches each loader backend in HTools to populate it (additionally, the
IAllocator class in lib/cmdlib.py now includes tags for groups too). Test
data is updated to include an empty set of tags for node groups in all
affected test cases.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The mergeData function in Loader.hs included a step to filter an instance's
tags to include only the exclusion tags (as specified via the commandline,
or cluster-level tags). Later on, code in Node.hs assumed Instance.tags to
contain only tags to be used for exclusion.

Because in the future we will need to access the full list of an instance's
tags (and not only exclusion tags), this commits deprecates the 'tags'
field, and introduces Instance.exclTags and Instance.allTags.

Instance.allTags is now populated from the different backends (Text, Luxi,
Rapi, etc.), and Instance.exclTags is only populated from Loader.mergeData,
as was done previously. This means that loading tags from e.g. Text or Simu
and assuming that they'll be used as exclusion tags without going through
Loader.hs will no longer work; but this was already the case with other
fields, and 'mergeData' or 'loadExternalData' continue to be the only entry
points to get a consistent view of the cluster. (Additionally, there were
no tests that made this assumption that I could find.)
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This only supports test delay for now, is not built by default (only
on demand), and is also not installed anywhere.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Also add some more haddock structure to the module. The RPC call
itself is rather trivial.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>