GitLab

The heuristics for tiered allocation has been improved in that it
chooses to shrink such a resource next where by shrinking only this
resource a valid allocation can be made, if such a resource exists.
In order to decide whether such a resource exists, all potential
sizes of this resource are verified. We can improve performance by
cutting, as soon a potential allocation of an instance is no longer
blocked on this resource.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In tired allocation, hspace shrinks that resource of the instance
next, that causes failure on most nodes. While, this is not a bad
strategy in general, it can lead hspace into a dead end if for a large
number of nodes a particular resource blocks any further allocation of
policy compliant instances. So we improve the heuristics in that it
chooses to shrink such a resource next where by shrinking only this
resource a valid allocation can be made, if such a resource
exists. This improves the results in some cases, while still keeping
the computational complexity of the algorithm low.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

hail-alloc-invalid-network defines a cluster with two nodegroups and an
allocation request which does not fit on any of the groups. Group 1 has
invalid disk-templates while Group 2 is not connected to the right
networks.

hail-alloc-restricted-network defines a cluster wih two nodegroups.
Nodegroup Group 1 is only used as last_resort group, but is chosen by
hail as only this group is connected to the right networks.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Before trying to allocate nodes in node groups, node groups are now
filtered based on the networks they are connected to an the networks
which are required by the new instance.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

With tiered allocation, hspace uses all the max specs in turn as the
initial instance spec.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Since 'head' is unsafe to use in most cases, this patch removes its
use from most of the code, adds a lint warning for it (and for tail as
well), and adds override annotations in the few cases where it's
actually OK to use it (mainly when using head over the result of
functions which guarantee to return a non-empty list by documentation,
not type).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Per offline discussions, this is the first patch of the
renames. Tested with "make distcheck", seems to work fine.

The only change outside of the renaming is a bit of simplification in
the .gitignore rules; otherwise, simply s/htools/src/.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This just adds new constructors for the disk template and related
types, enabling the Ext storage.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Note: I'll add tests for this on the master branch, but not here.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The original htools opcodes were minimalistic and not 1:1 equivalent
with the Python ones. Let's add all missing fields and, since we
changed the order, switch to more readable record syntax for building
the opcodes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

It turns out that optimising 'read' derived instances (via -O) for
complex data types (like OpCode, or the various objects) can be slow
to very slow. Disabling such instances results in (time make
$all_our_haskell_binaries) large compile-time savings and also smaller
(unstripped) binaries (by a significant amount):

ghc 6.12:        time  htools sz  hconfd sz
  with read:    4m50s 12,244,694 14,927,928
  no read:      3m30s 10,234,305 12,536,745
ghc 7.6:
  with read:   14m45s 13,694,761 15,741,755
  no read:      3m40s 11,631,373 13,245,134

So let's remove these instances, since we never use read in production
for our custom types, and even when debugging in GHCI, we can simply
use the 'show' representation to create the types, without needing to
actually parse from strings.

Note: for the very slow ghc 7.6 compilation time, I filled a ticket
(ghc #7450), since it is surprising(ly slow).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This is the last inline-written field; I've done it separately from
the last patch due to changes being needed in HTools (field changed
type).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This does a partial conversion of existing opcodes to the new
non-empty string type, which is needed so that we don't add even more
opcodes without this safety net (and to have consistent "target_node",
etc. in opcodes).

By partial I mean that only the opcode are changed, and not the entire
htools types/workflow, so we use error to abort if, when submitting
the opcodes, we detect an empty node name. FIXME added.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Testing with a newer hlint found a few minor issues; but all are real,
valid recommendations:

- don't use "if cond then f x else f y", but "f (if cond then x else y)"
- "if a then b else True" is equivalent to the simpler "not a || b"
- and as usual, one more ignore to our "testing basic properties"
  module
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Since we now have the GeneralResult as a multi-purpose monad, we can
remove the custom OpResult monad, and just use 'GeneralResult
FailMode' as our type. This allows removal of a few bits of
specialised infrastructure, relying instead on the generic one.

The restriction on using OpResult as a general monad remains as
before.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Before we reorganised the source tree, the 'Result' type was exported
from HTools/Types.hs. This changed during the reorg, but at that time
we didn't change the exports; instead, we kept re-exporting it from
the old module for compatibility.

In light of future changes to the Result type, let's stop this
re-export and cleanup the imports of the other modules.

One test is slightly rewritten with explicit types declaration (part
of the values already needed one, let's make it explicit).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is, I believe, the last non-htools specific file that still lived
in the htools directory; it's already widely used in non-htools code,
so let's move it before we add more functionality to this module.

All changes are related to the name change, imports fixup, etc.; there
are no other changes in this patch.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The function `allocateOnSingle' has a bug in the calculation of the
cluster score used for deciding which of the many target nodes to use
in placing the instance: it uses the original node list for the score
calculation.

Due to this, since the original node list is the same for all target
nodes, it means that basically `allocateOnSingle' returns the same
score, no matter the target node, and hence the choosing of the node
is arbitrary, instead of being done on the basis of the algorithm.

This has gone uncaught until reported because the unittests only test
1 allocation at a time on an empty cluster, and do not check the
consistency of the score. I'll send separate patches on the master
branch for adding more checks to prevent this in the future.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

These two files are not htools-specific, so let's move them out of the
HTools subdirectory/module hierarchy and directly under Ganeti.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This function iterates over the tryMGAlloc function, updates the node
list and instance list, and refeeds it recursively until no new
instances are left.

This allows us then to allocate multiple instances and see if they would
fit together.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit 5a1e31b4

 (Add infrastructure for, and two extra hlint rules)
was intended to add two *extra* hlint rules, but I didn't realise at
that time that "--hint" when first used overrides the built-in
lints. As such, since then we were basically running with just those
two rules, which resulted in many uncaught warnings/errors.

This patch fixes that (by importing the standard lint rules in our
custom hints file), and then goes to fix all the warnings that a
current hlint gives me. Compared to our current style, we have just a
few additions:

- zipWithM instead of map foo . zip …
- 'exitSuccess' instead of 'exitWith ExitSuccess'
- more uses of '.'

Additionally, we have to silence a case where hlint doesn't realise
why we are using '\e -> const (return False (e :: IOError)' instead of
just '\e -> return False' or even 'const (return False').

One warning that is generated by hlint ("Use void") can't be fixed
until we deprecate GHC 6.x, as only GHC 7 has the 'void' function in
Control.Monad.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

We most always use `templateMirrorType . diskTemplate`, so let's add a
helper function for this.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch toggles the final bit and enables nodeEvacInstance to work
on shared storage instances. Diskless instances are handled the same
(as in, None is the same everywhere :).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of manually case-ing on the Either contents, let's just use
either with const functions (not really readable, but…).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The function is now generic, so let's rename it and cleanup/improve
the docstrings.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is generic enough (no longer depending on the replace type) that
it will be used for other replace modes too. So all we need to do is
to abstract it to a separate function (that can be called from
multiple branches) and to make the imove op_fn computation more
generic, instead of hard-coding it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This brings together all previous pieces and allows balancing to work
(with the known caveats related to disk space calculation) for shared
storage disk templates.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of hard-coding the instance move operation, let's accept is as
a parameter from the caller. This is the single thing that makes this
DRBD-specific, so removing it will help for future
patches. Furthermore, this abstract the imove op type into a single
place instead of two (the inner function and the nodeEvacInstance).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently the function possibleMoves always presumes DtDrbd, which is
wrong for shared storage. To make it work with all disk templates, we
additionally pass the mirror type to it (we assume that the move
behaviour only depends on the mirror type, and not the individual
disk template; currently this is true, and we should aim to keep it
that way).

For none and external mirrors, we don't give any solutions; for
internal mirrors, we keep the previous algorithm.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently we don't set a boundary on the length of the score field,
which breaks all our nicely-aligned output efforts.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently Cluster.printSolutionLine function always wants to print the
old and new secondary, and as such it fails (ugly) for moves of
non-DRBD instances. Change it so that it looks explicitly for
noSecondary nodes, and format the output different in that case.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of computing the bad/good node lists separately (in effect
iterating twice over the all_nodes list and also creating that list
twice from the container), let's use 'partition' and do a small
improvement.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This helper abstract away all the unlines and unwords call you've to do
after you got a formatTable back.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This makes balancing use the new metric.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Since we use the functions always module-qualified, let's drop the
`instance` prefix and use a shorter one.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

It prints the information which hbal prints in a nice formatted manner and of
course without doing any actions. An example output looks like this:

$ hinfo -m xen.example.com
Loaded 4 nodes, 2 instances
Cluster has 1 node group(s)
Cluster coefficients:
   Field                Value Weight
   free_mem_cv     0.00195342  x1.00
   free_disk_cv    0.00000000  x1.00
   n1_cnt          0.00000000  x1.00
   reserved_mem_cv 0.00195342  x1.00
   offline_all_cnt 0.00000000  x4.00
   offline_pri_cnt 0.00000000 x16.00
   vcpu_ratio_cv   0.12500000  x1.00
   cpu_load_cv     0.50000000  x1.00
   mem_load_cv     0.50000000  x1.00
   disk_load_cv    0.00000000  x1.00
   net_load_cv     0.50000000  x1.00
   pri_tags_score  0.00000000  x2.00
Cluster score: 1.62890685
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Notes:
  - it'd be nice if there was a way to automatically generate the
    DiskTemplate list instead of manually specifying it. After all we
    have C.diskTemplates
  - Of course this actually does nothing, as for other
    "half-implemented" templates :(
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>