GitLab

Some options have defaults that depend on the environment, and we
could handle these in two ways:

- use a place-holder value (e.g. data X a = Default | Custom a) that
  is later read from the environment
- move the options list to IO monad, where it can read the
  environment, etc.

The second option allows also displaying the actual defaults in the
`--help' output, even though it's not as nice, so I went with it.

This patch only changes the option types, without actually changing
any options yet.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This small patch fixes compatibility with a few newer Haskell libraries:

- base 4.6, included with ghc 7.6, removed the deprecated 'catch'
  function from Prelude, so our "import Prelude hiding (catch)" is now
  an error; we workaround by using fully-qualified
  Control.Exception.catch name

- containers 0.5 changed the signature of 'deleteFindMax'; we
  workaround by using separate 'findMax' and 'deleteMax'

- QuickCheck 2.5 removed the 'maxDiscards' test parameter, replacing
  it with a much better 'maxDiscardsRatio'; however, until we can
  depend on that, we workaround by just removing it (we don't control
  anymore the maxDiscards, instead leaving it default; for our default
  test size, this is no change, as the default value is already 500,
  which is our default as well) and not printing it anymore

Tested on Squeeze (+extra libs), Wheezy and experimental, which covers
all supported GHC versions.

Also, merging this in master will be a pain, but unless we want to
stop supporting 2.6…
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This a bit too complex patch converts the result of Luxi calls
(submitJob, query*, etc.) from Result to ErrorResult. It then
immediately revers this in the HTools/Backend/Luxi module, where we
don't need necessarily the full error type (just a nice error
message), and does the same in Hbal's job execution functions.

While at first sight this doesn't seem to do much, what we get is
actual error messages from Ganeti, plus improvements to the result
parsing: instead of "can't parse char", we now get properly (note,
wrapped manually):

  Executing jobset for instances instance1, …
  Job submission error: Failure: the job queue is marked for drain and
    doesn't accept new requests

Or:

  Job submission error: Unhandled exception: LuxiError "parsing job
    id: cannot parse string 'a956101'"
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Five modules under the HTools/ directories are backend
implementations, so let's move them to a separate directory, to more
clearly show the hierarchy. I wanted to do this for a while, but
merging between branches is always an issue, so let's do it know since
we have an opportunity.

This patch contains the actual renames, the required changed module
names, imports, etc., but no other changes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Testing with a newer hlint found a few minor issues; but all are real,
valid recommendations:

- don't use "if cond then f x else f y", but "f (if cond then x else y)"
- "if a then b else True" is equivalent to the simpler "not a || b"
- and as usual, one more ignore to our "testing basic properties"
  module
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch includes:

* The 'failN1' flag is now only set if there is strictly less
  memory available than required for failover.
* Unit tests for that.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Small simplifications of other unit tests using the (==?)
operator when possible, and typo fixes.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Since we now have the GeneralResult as a multi-purpose monad, we can
remove the custom OpResult monad, and just use 'GeneralResult
FailMode' as our type. This allows removal of a few bits of
specialised infrastructure, relying instead on the generic one.

The restriction on using OpResult as a general monad remains as
before.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Furthermore, a few messages have their capitalisation changed (fixed).
Signed-off-by: Helga Velroyen <helgav@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This commit adds a Group.allTags field to store the tags of node groups,
and teaches each loader backend in HTools to populate it (additionally, the
IAllocator class in lib/cmdlib.py now includes tags for groups too). Test
data is updated to include an empty set of tags for node groups in all
affected test cases.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The mergeData function in Loader.hs included a step to filter an instance's
tags to include only the exclusion tags (as specified via the commandline,
or cluster-level tags). Later on, code in Node.hs assumed Instance.tags to
contain only tags to be used for exclusion.

Because in the future we will need to access the full list of an instance's
tags (and not only exclusion tags), this commits deprecates the 'tags'
field, and introduces Instance.exclTags and Instance.allTags.

Instance.allTags is now populated from the different backends (Text, Luxi,
Rapi, etc.), and Instance.exclTags is only populated from Loader.mergeData,
as was done previously. This means that loading tags from e.g. Text or Simu
and assuming that they'll be used as exclusion tags without going through
Loader.hs will no longer work; but this was already the case with other
fields, and 'mergeData' or 'loadExternalData' continue to be the only entry
points to get a consistent view of the cluster. (Additionally, there were
no tests that made this assumption that I could find.)
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Before we reorganised the source tree, the 'Result' type was exported
from HTools/Types.hs. This changed during the reorg, but at that time
we didn't change the exports; instead, we kept re-exporting it from
the old module for compatibility.

In light of future changes to the Result type, let's stop this
re-export and cleanup the imports of the other modules.

One test is slightly rewritten with explicit types declaration (part
of the values already needed one, let's make it explicit).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The comments in parseData had become out of date with the implementations
of load{Group,Node,Inst}. This commit updates the field list in comments to
match the implementations.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch removes the last HTools module imports from non-htools code
(the HTools.Types module), but it requires an associated cleanup:
using luxi-specific constants for luxi timeouts (the only effect is
that one timeout decreases from 15 to 10, the default value in the
python code), and moving of the (now) RAPI specific constants to
RAPI.hs (which allows simplifying their type/usage).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This allows the command completion to treat them as separate option
types; this is not useful in build-bash-completion right now, but
might become so in the future.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This defines the arguments supported and then modifies the
--help-completion output to include them too.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is, I believe, the last non-htools specific file that still lived
in the htools directory; it's already widely used in non-htools code,
so let's move it before we add more functionality to this module.

All changes are related to the name change, imports fixup, etc.; there
are no other changes in this patch.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds support for a --help-completion option, which will
display the defined options and their completion information, in a
format designed to be parsed easily from Python, for integration into
build-bash-completion.

The alternative would have been to duplicate the build-bash-completion
functionality on the Haskell side, but that would have been lots of
duplication for not enough gain (since that is only run at build
time).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is a quite boring patch, just adding annotation information to
all existing options. Some of the annotations are not very good; but
we don't have support for more precise completion in
build-bash-completion, so this is good enough.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Currently, we test and require that each individual program (hbal,
etc.) defines/supports the generic options (currently --help and
--version). Even with the test, this is not optimal, since it requires
changes in many places whenever we modify the list of generic options,
hence we move these out of the individual programs and into the
generic CLI handling code.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

The global cluster score is less interesting than individual group
scores, for multi-group allocation purposes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This will make it easier to add new parameters.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Future changes will change Path.hs to use an environment variable.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The function `allocateOnSingle' has a bug in the calculation of the
cluster score used for deciding which of the many target nodes to use
in placing the instance: it uses the original node list for the score
calculation.

Due to this, since the original node list is the same for all target
nodes, it means that basically `allocateOnSingle' returns the same
score, no matter the target node, and hence the choosing of the node
is arbitrary, instead of being done on the basis of the algorithm.

This has gone uncaught until reported because the unittests only test
1 allocation at a time on an empty cluster, and do not check the
consistency of the score. I'll send separate patches on the master
branch for adding more checks to prevent this in the future.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Commit 2cdaf225, “Re-enable standard hlint warnings”, got it almost
right. The only problem is that (confusingly) the default set of hints
is not in HLint.Default, but in HLint.HLint (it includes Default and
some built-ins).

After changing the lint file to correctly include the defaults, we had
another 128 suggestions:

  - Error: Eta reduce (2)
  - Error: Redundant bracket (4)
  - Error: Redundant do (17)
  - Error: Redundant lambda (7)
  - Error: Redundant return (1)
  - Warning: Avoid lambda (2)
  - Warning: Redundant $ (42)
  - Warning: Redundant bracket (35)
  - Warning: Use : (1)
  - Warning: Use String (4)
  - Warning: Use camelCase (10)
  - Warning: Use section (3)

which are fixed by the current patch. Note that the 10 "Use camelCase"
were all due to hlint not “knowing” the idiom of ‘case_’ (it does for
‘prop_’), for which I filled
http://code.google.com/p/ndmitchell/issues/detail?id=558

.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

While investigating how we could test the Daemon.hs module, I realised
that we have a very, erm, sub-optimal situation:

- HTools/CLI.hs has a nice IO/pure separation testing in cmdline
  parsing, which allows some basic functionality to be tested, but
  uses direct 'read' in many options, which fails at runtime when
  evaluating the argument, and not when parsing the options
- Daemon.hs lacks that, but has a much nicer 'reqWithConversion'
  helper that can be used for nicer option parsing, and uses that +
  tryRead instead of plain 'read'

Since this situation is very bad, let's clean it up. We introduce yet
another module, Common.hs, that holds functionality common to all
command line programs (daemons or not). We move the parsing to this
module, and introduce a type class to handle option types which
support --help/--version. This allows removal of duplicated code from
CLI.hs and Daemon.hs.

The other part of the patch is cleanup/rework of the tests for this
code: we introduce some helpers (checkOpt, passFailOpt,
checkEarlyExit) that can be used from the much-slimmer now tests for
CLI and Daemon. In the common module, we just test the yes/no helper
we have. Many new tests for boolean options and numeric options are
added.

A side change is the removal of the obsolete `--replay-count',
`--test-size' options (unused since commit 95f6c931

, “Switch Haskell
test harness to test-framework”).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is another module that is generic, and not htools-specific.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

In order to have correct code coverage results, we must somehow import
all production modules into the test runner. Until now, this was done
manually (when we didn't forget) in QC.hs.

To improve the situation, we remove QC.hs and replace it with an
auto-generated file which imports all modules. This reduces the
maintenance burden and ensures we'll always have correct coverage.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This makes QC.hs obsolete, but we will remove it in a later patch,
once we automatically build a file with all the imports (to ensure
correct coverage results).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

These two files are not htools-specific, so let's move them out of the
HTools subdirectory/module hierarchy and directly under Ganeti.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Except for Ganeti.HTools.JSON, which needs rename, we split all the
other test suites into separate files.

We have to add another common test helper, due to import dependencies
(sigh), but otherwise this split is pretty straightforward.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

… from QC.hs into their own files, again mirroring the production code
source tree.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This required lots of other code moves, so I created it as a
standalone patch.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This splits the confd/utils tests, and adds the TestCommon module for
shared test code.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is the first file split out from QC.hs - an easy one, since it
has just one test.

The patch changes the way we build hpc excludes, since now we'll have
many modules that need to be excluded, and hpc doesn't seem to be able
to do wildcards (it can do packages, but all our code is
not-package-based).

Further splitting will be done in bigger batches.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch starts the move of the test haskell code from
`htools/Ganeti/HTools/' to its more proper place of
`htest/Test/Ganeti'.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This means that the verification of the correctness of the regex is
done once, at the deserialisation/creation time, as in the Python
code. To do this, we have to change the FilterRegex type from an alias
to String to a more complex data type, and we have to create manual
read/show/eq instance (phew!).

Unittests are added which test these instances, since it's the first time I do
this manually.

An additional improvement is that we now check that regex-pcre has
been compiled correctly, per the documentation (otherwise we get
runtime errors).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Per the new query module hierarchy, rename Qlang to Query/Language and
Queryd to Query/Server. This way, all query-related functionality is
now "contained" in the Query/ directory.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

In preparation for introducing filtering functionality, we convert the
'Filter' type from a '*' kind to a '* -> *' kind.

This allows us to define some general properties for the filter, and
for example introduce later an easy filter compilation, etc.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is the final part:

* Parsing the new request type
* Feed it to allocList
* Format the result
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>