GitLab

In this example, there is one big instance (.75 of a node) and
otherwise small instances (.25 of a node). One node is only
used to 25% whereas the rest to 75%. Nevertheless, the mostly
empty node cannot be used for allocating a new instance, as
that would prevent the big instance from being evacuated should
its node fail. Note that the mostly empty node would be the most
balanced allocation.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

In this way we avoid a circular dependency: tryAlloc depends on
the notion of global N+1 redundancy and global N+1 redundancy for
plain instances depends on knowing how to allocate an instance on
a single node.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

While we cannot avoid data loss on node crashs if we
have plain instances, we can ensure that the cluster
has enough capacity to reinstall the instances on a
new node. Add a design describing how we enusre this.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add a simple test case for the allocate-secondary request.
In this example, hail has to avoid the node of the other
group and should prefer the free node as secondary over
the one that already hosts an instance.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

On disk conversion to DRBD, a new secondary node has to be found.
Add a function doing this search in a way similar to what instance
allocation does: all possible placements are considered and the
one resulting in the best cluster metrics is taken.

To have a uniform interface, use the same signature as all allocation
functions, even though the group list is not needed (we have to take
the group of the primary node anyway).
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Add a design describing a new Ganeti daemon to coordinate
Ganeti's task in automated repairs. While Ganeti itself will
not handle hardware swaps, it can still automatically prepare
everything, e.g., by evacuating a node.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

It can be good style to have expressions of the form

  or [ ....
     , ....
     ]

as this is quite readable and can easily be extended,
should more conditions come in place. So, as we
ignore "Use &&", let's also ingore "Use ||".
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The metadata daemon was previously running as root due to its need to
open port 80 to provide information to instances. To allow the daemon
to run in a more secure way, this patch adds a separate metadata user,
and grants the metad executable the CAP_NET_BIND_SERVICE capability.
As a result, the metadata daemon can use the port 80 without having to
acquire the full set of root capabilities and drop it later.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In this example, every node is over the spindle limit (a
soft disk error). Moreover, the cluster is unbalanced and
the only possible move is a failover.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

After cherry-picking the changes needed for GHC7.8, the former isn't
used any more.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

It turns out that GHC 7.8's -dynamic-too is forbidden not only for
profiling builds object files, but also for those using HPC coverage.

This commit accordingly renames HFLAGS_NOPROF to HFLAGS_DYNAMIC
(since it now is not conditional on profiling any more),
and makes sure that it is not used for profiling or HPC.

This way we achieve that, for profiling+coverage builds
  - .dyn_o files are available for use in TH in the following 3 cases
  - .o files are for the normal binaries
  - .hpc_o files are for coverage-enabled test binaries
  - .prof_o files are for profiling binaries

We make .hpc_o files depend on the .o files because the creation of
the .o files will also create the .so files needed for TH.
This was already in place for .prof_o files.

This requires that HFLAGS now also contains `-itest/hs` since
the rule for .o files is now also responsible for building .o/.so files
in test/hs.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Cherry-picked-from: 1ad14f38Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This fixes
  ./configure --enable-haskell-tests && make hs-tests
builds on GHC 7.8.

It changes `-osuf .ext` to `-osuf ext`; the old form was allowed in
GHC <= 7.6, but GHC 7.8 seems to have problems with it:
  https://ghc.haskell.org/trac/ghc/ticket/9760

The form without dot works in all GHC versions, and we don't have
dots in other -osuf arguments in the same file either, so this
was probably an accident.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Cherry-picked-from: 9664aff9Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

GHC 7.8 switched to dynamic linking being used for ghci, which requires
that .so file are being built for modules that are loaded for TH.

For this reason, GHC >= 7.8 has a -dynamic-too flag, which we now use.

However, -dynamic-too must not be enabled for profiling builds,
which is why this commit introduces a HFLAGS_NOPROF variable
(currently only containing -dynamic-too) that must be passed to
all GHC invocations that are not creating profiling output.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Cherry-picked-from: 083776b9Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This works around https://ghc.haskell.org/trac/ghc/ticket/9749:

GHC 7.8 (undocumentedly) changed the way in which ghc -M generated
object file dependencies, lacking the underscore that older versions
added automatically to the file names.
It also requires a -dep-suffix for the plain object file (.o).

This commit detects GHC 7.6 and older (7.7 is development only, has no
release and is treated equal to 7.8), and adjusts the -M invocation
appropriately for newer GHC versions.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Cherry-picked-from: b78a2c30Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This document was added in commit 847bc5cc; however, it was forgotten
to add it to the list of files to be distributed. Do this now.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Testing the backend functions which update SSH keys is a
pain and maintaining the tests even more. Therefore, this
patch introduces a manager for all SSH key files of a
clusters (ganeti_pub_keys and authorized_keys). It
emulates all operations on these files for all nodes in
the cluster.

This has the following advantages:
- One can query the state of the entire cluster in a
  consisten way, for example "Do all nodes have this
  master candidates' key?" instead of tediously evaluating
  a history of mock calls.
- The file manager emulates both local changes in the
  master nodes' key files and changes on other nodes'
  key files using the ssh_update tool. This way, the
  state of the cluster ssh files is managed consistently
  no matter by what mechanism they were changed.
- The file manager offers a couple of convenience
  functions to set up the test data and to query their
  state after test operation.

Note that this might look like a lot of code, but it
vastly simplifies the current unit tests and it will
make future tests (for example for invalid calls) much
more easier. As it is a test utility, it is properly
documented to make it maintainable.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

In order to be able to verify exclusion tag violations in cluster
verify, we need to decide which tags are exclusion tags. So add
two python utility functions: one for extracting exclusion prefixes
from from the cluster tags and one testing if a tag starts with
one of a set of given prefixes.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

In this way, we can export some of those constants via
Ganeti/Constants.hs to the python world while still keeping
all the special htools prefixes togehter in an HTools module.
(Note that Ganeti/Constants.hs cannot depend on
Ganeti/HTools/Tags.hs as this would cause a cylic dependency.)
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

In this example, there are 5 nodes. The nodes 1 to 3 each
host one instance as primary and one as secondary node, whereas
the nodes 4 and 5 are completely unused. Therefore, the best
allocation is on nodes 4 and 5. We check that restricted
allocation overrides the natural choice.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

And mention partial and complete designs in there.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In this example, there is one big instance (.75 of a node) and
otherwise small instances (.25 of a node). One node is only
used to 25% whereas the rest to 75%. Nevertheless, the mostly
empty node cannot be used for allocating a new instance, as
that would prevent the big instance from being evacuated should
its node fail. Note that the mostly empty node would be the most
balanced allocation.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Move all data structures related to allocation solutions to a separate
module. In this way, we can extend these in a clean way, e.g., to support
later filtering.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Bump various versions to 2.16 and remove downgrade code
from cfgupgrade. Also adapt cfgupgrade tests.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In this example, there are 4 nodes. Each has the same amount
of free memory, enough for one small instance. 3 of those nodes
host 3 small instance. One node, however, has one big instance
the size of 3 small ones. Hence that node cannot be fully evacuated.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Redundancy for DRBD nodes is already checked at every node locally.
However, in the presence of shared/external storage, we also have to
consider planning capacity for those instances---globally (within
a node group) as they can be moved to any node. Add a function that
checks whether this is true for a given node in a given cluster.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

In this way, make it available for other high-level functions
like global N+1 redundancy. For this purpose, also export
nodeEvacInstance.

As part of this splitting, move some elementary functions to
a Ganeti.HTools.Cluster.Utils module.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This is a code move. In this way, operations that need the cluster metrics
(like simulating node evacuation, e.g., in order to determine global N+1
redundancy) can use it, without having to depend on Ganeti.HTools.Cluster.
In particular, they can provide functionality that eventually will be used
by Ganeti.HTools.Cluster functions without having to go into that already
overcrowded module. Making that module smaller, and hence more manageable,
is a nice side effect.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...and export applyMoveEx. In this way, other high level operations,
like global N+1 redunancy can use it while providing functionality
that will eventually be used by the main cluster operations without
having to stuff everything into the already overcrowded
src/Ganeti/HTools/Cluster.hs.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add a design document describing how Ganeti will take care of N+1
redundancy in the presence of shared storage. As opposed to DRBD,
instances using shared storage can be started on any node. Therefore,
at no particular node we have to reserve memory---however, in total
enough memory for any node to fail has to be reserved nevertheless.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...so that it ends up in the release
tar ball as well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

...as they routinely occur in the output of 'git describe'.
To stay with the simple sed-based approach, we accept to
lose the symbol that will also cause trouble in our Haskell file,
the double quote. It is quite rare in the output of git-describe
and we just substitute it away.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

.. to the Makefile as well as to the Haskell modules where it will be
needed.

While at it, fix spacing so that command lines start with tabs, as they
should.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. as we'll be adding the version information of another package later.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This design describes an addition to the cluster
metrics that improves allocation efficiency. So
far, only the balancedness of the amount of reserved
memory was taken into account, but not its total
amount. The reserved memory, however, is not preserved
by instance moves; it can be lowered by spreading the
secondaries more widely over the cluster.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Commit dc1fe8df introduced 'kvm-ifup-os' to configure instance
communication TAP. However, the version with configure-time variables
substituted in was accidentally also distributed. Change this.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

For better LURenewCrypto unit tests, we need to be able to
mock the pathutils module. This makes it necessary to add
some support to the testutils for patching this module.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The replacement command only added dependencies for *.hi files, not
*.*_hi files that are used for testing.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

The original version didn't work when the version was just 2 numbers,
and also wasn't robust enough when there were multiple 'lens' versions
installed.

This patch fixes that, and always takes the last installed version, as
reported by ghc-pkg.

Starting with 2.14 there is a native cabal-based mechanism for
generating these definitions, so the whole 'hs-lens-version' target
should be dropped from the Makefile in that version.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The client is formed by combining the generated stub with the abstract
client implementation.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. using our existing Python -> Haskell generator.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>