This new opcode will replace LUNodeEvacStrategy, which used to return a
list of instances and new secondary nodes. With the new opcode the
iallocator (if available) is tasked to generate the necessary operations
in the form of opcodes. This moves some logic from the client to the
master daemon.

At the same time support is added to evacuate primary instances, which
are also evacuated by default.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Am I the only one to make that mistake 10 times a week?

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Haskell has two main integer types:

- Int, which is a native-type, and is guaranteed to have at least
  [-2²⁹, 2²⁹-1] range; on 64-bit platforms, it has much higher range
- Integer, which is a software type (implemented using libgmp), and
  thus unbounded

For performance reasons, the node/instance properties use Int for
their attributes (and Double for some, but that's another story). This
is all fine and doesn't cause problems. However, the CStats type which
holds the overall cluster resources starts to fail when we analyse
clusters with more than around 400 nodes and big memory/disk sizes on
32 bit platforms.

The simple fix would be to restrict cluster sizes, but that's no
nice. I've benchmarked and changing to Integer doesn't show a visible
slowdown on 64-bit platforms (as far as I can read on the internets,
GHC knows to optimise Integer and only use software types when the
values are large enough), and it also fixes the 32-bit problem. So
this patch changes the CStats types to Integer, except for the
instance count (which I don't expect to overflow 2²⁹ anytime soon).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

There were some implicit assertions in the code that all node groups
have nodes, which is not necessarily true.

Additionally, the patch does a wrapping change.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Beside the 'dscription' typo, also make the punctuation more
consistent.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Just saw this while testing the migration to QuickCheck v2: while
configure.ac detects that QuickCheck-2.x is not available, the test in
Makefile.am was against WANT_HTOOLS (overall htools compilation), not
on a more-specific WANT_HTOOLSTESTS.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch fixes a bug in the test specification where we allowed nodes
with zero free memory (hence no instance can be added, at all) and adds
a simple labeling of the way this test can fail.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This unittest had a corner case where it could fail if the same
primary/secondary node names were generated.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This just adds glue to allow replaying of tests using a given RNG state
and test size (both are needed for exact replayability).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Using new functionality in QuickCheck 2 (the suchThat function), we
generate now better test cases, such that (heh) we have no longer
incomplete tests.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently the way we generate nodes in some cases is by creating a
totally random node, then restricting the test based on whether the node
'size' (as defined by multiples of base unit) satifies some high/low
rules. This results in hard-to-satisfy conditions, so we change this
model to be able to specify node sizes directly in the generation
process, thus no longer needed post-creation filters.

This fixes prop_ClusterAllocBalance which before had at most 1-2
satisfiable tests.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The new scaffolding which replaced the batch driver of QuickCheck 1 now
shows how many passes we have for incomplete tests. Some tests show very
low pass counts, so we rework them to have more actually valid test
cases.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Since current distros don't package anymore QuickCheck 1.x, let's move
to 2.x.

This requires also a few changes to the code:

- Test.QuickCheck.Batch doesn't exist anymore, so we need to write some
  scaffolding code to replace it
- the way test sizes are generated has changed, and we need to restrict
  (in some tests) the cluster size, as our code is not yet ready for
  hundreds of thousands of nodes in a cluster and we run out of stack
  (which could be a bug somewhere by itself, needs investigation)
- at least with GHC 7, floating point errors make a perfect cluster
  score even bigger, so we need to bump up the max. rounding error
  allowed

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Removal of duplicate parantheses, removal of extra 'do', conversion from
nested if to guards, use hierarchical imports. All per hlint.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

filterMGResults was built using a sequence of map and filter calls;
while this was logically correct, it used some incomplete pattern
matching which with the new GHC 7 triggers a warning.

The patch rewrites it using a single foldl that does both the filtering
and the mapping, in a more type-safe way.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

… instead of the hardcoded test against AllocUnallocable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

… that checks if a group is allocable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently htools cannot be compiled under sid because the parallel
haskell library is version 3. Using it issues a few warning, but
compiles and passes unit tests. Ship it?

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.4:
  LUInstanceCreate: use opcodes.RequireFileStorage
  Don't add ",boot=on" to disks on kvm >= 0.14
  KVM: fix per-instance stored UID value

Conflicts:
	lib/cmdlib.py
          - use RequireSharedFileStorage there

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This function is a copy of bootstrap._InitFileStorage with the following
differences:
  - check constants.ENABLE_SHARED_FILE_STORAGE and not
    constants.ENABLE_FILE_STORAGE
  - use different local variable names
  - one different error string

Thus:
  - move the constant check outside of the function call
  - change error string so it's clear where the error is
  - call the same function twice

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Under newer kvm this prevents the vm from starting.
Ah, change!

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

When using the pool security model, _ExecuteKVMRuntime was storing the
instance's UID using str(uid), which would result in storing the
LockedUid.__repr__() result:

 $ cat /var/run/ganeti/kvm-hypervisor/uid/xxxxxxxxxxxxx
 <ganeti.uidpool.LockedUid object at 0x1f30610>

This patch restores the intended behaviour, by using LockedUid.AsStr().

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

* devel-2.4:
  Add one forgotten element to the file disk path

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This was left out during the fix/refactoring

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

* devel-2.4:
  LUInstanceCreate: fix file storage dir calculation
  Check that filestorage is enabled when requested
  Remove self.op.file_storage_dir isabs check

Conflicts:
	lib/cmdlib.py
          - use constants.DTS_FILEBASED
          - handle DT_SHARED_FILE correctly

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

- Move the calculation at the beginning of CheckPrereq, since it doesn't
  modify any state, but still keeps locks
- Only perform the calculation if the actual disk template is filebased
- Error out if there is no defined file storage dir
- Only join the optional --file-storage-dir extra-path if one is passed

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As the manpage says, and the code does, self.op.file_storage_dir is an
additional relative path under the cluster file storage dir. As such it
should not be absolute.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This also adds docstrings for the function arguments and renames exinst
to exinsts, which is how it is called in other functions, since it's a
list.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This design covers high level network block definition and pool
management.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch removes all occurrences of the “multi-relocate” iallocator
mode. Commit 25ee7fd8 updated the design document and introduced
separate modes, “change-group” and “node-evacuate”. The constants aren't
removed yet as they're still used by htools.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Stephen Shirley <diamond@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>