GitLab

This implements the operational part of the design doc
  "Filtering of jobs for the Ganeti job queue"
(design-optables.rst).

It includes
- respecting filter rules when jobs are scheduled
- cancelling running jobs rejected by filters
- re-running the scheduler when filter rules are changed
- handling of the filter actions ACCEPT, CONTINUE, PAUSE, REJECT
  and RATE_LIMIT
- implementation of the "jobid", "opcode" and "reason" predicates
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This patch add systemd unit files used by ganeti-*.service to
EXTRA_DIST.
Signed-off-by: Jun Futagawa <jfut@integ.jp>
Reviewed-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In this example, all instances are running on the same node,
so balancing will always start migrating instances off this node.
Depending on the migrations are possible, others not.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Start a new module for htools-related tag interpretation and
move the definition of exclusion tags there. This module will
contain all the tags and tag-prefixes that have a special
meaning to htools, so that they can be found at a single place.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Add a target to manually run the python tests, and no other
tests. In the same way as the hs-tests target simplifies testing
Haskell changes while developing Haskell code, this target
simplifies developing python code.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Niklas Hambuechen <niklash@google.com>

Some test modules made use of the fact that you can

  from module import x

working even if x is not listed in module.__all__.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The cli.py module exceeded our max-module-lines limit and needs to be
split.

This pulls out all command line option definitions into a separate module.

It uses cli_opts instead of cli.opts since importing a module below
your own path causes problems if you want to re-export everything
from the lower module (which we want in this case).
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This implements the management (add, remove, replace) of job filter rules
as specified by `doc/design-optables.rst`; it does not yet implement
the filtering logic.

This commit also includes the implementation of filters in the RAPI client.py
and its tests because the client tests check that if all available RAPI
resources are being used by the client, and the only way to make them
be used is to write tests for them.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In order to update the 'ganeti_pub_keys' and the
'authorized_keys' files of various nodes via SSH, we
introduce the tool 'ssh_update'. It works similar to the
tool 'prepare_node_join', which is also a tool invoked
via SSH on a remote note.

This patch includes some refactoring to reuse code from
the 'prepare_node_join' tool and provides unit tests as
well. Note that the actual invocation of the 'ssh_update'
tool will be done in later patches of this series.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

If in the original configure, haskell-tests are disabled,
test/hs/htest.hs will not be a needed dependency, hence it
will not automatically end up in the distribution tar ball.
Add the file to EXTRA_DIST in this case.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The full test suite also includes the haskell tests.
Hence enable them when running distcheck.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Signed-off-by: Guido Serra <guido@gaikai.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Depending on the value of ENABLE_MOND it is part of the
haskell library base or not; however, in any case it should
end up in the distribution tar ball. Also, do not have whitespace
in front of Make-variable assignments.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This turns the existing `SlotMap` from the structure

  Map (label :: String, limit :: Int) (occupied :: Int)

into

  Map (label :: a) (occupied :: Int, limit :: Int)

and extracts it into its own module; this makes it possible to use
`SlotMap`s for cases outside of reason rate limiting.

There is also a semantic clean-up (not affecting how reason rate limiting
works): `joinSlotMap` (now `occupySlots`) no longer takes a `SlotMap`
as second argument, removing the need to decide how the limits of the
second `SlotMap` affect the return value.

Before `c = joinSlotMap a b`  allowed to have keys in `b` that don't appear
in `a`, and those would appear in `c` with the limits from `b` (so `a` does
not constrain them).
In the new `occupySlots`, such keys have a limit of 0 instead.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

test/hs/htest needs only be compiled if the tests are supposed to be
run.
Signed-off-by: Aaron Karper <akarper@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The autogenerated TestImports.hs depends on all HS_LIB_SRCS files and
thus the build process breaks if the dependencies of Mond are not
fulfilled, even if it is disabled.
Signed-off-by: Aaron Karper <akarper@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Unfortunately we have nothing that can notice this automatically yet.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

This design doc describes how to extend the existing network
management and make it more flexible and able to deal with more
generic use cases. It proposes support for:

 - Networks with multiple subnets
 - Subnets with multiple IP pools
 - NICs with multiple IPs from various subnets of a single network
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>

It contains only one MonadPlus-specific function and we would like
to add more Monad-specific functions.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

...instead of hard coding "xenvg".
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...instead of hard-coding the value "xen-br0".
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add a variant of the relocation hail request where the nodes
where the disk can be relocated to have a lot less memory. So any
of the relocation moves would render a node not N+1 happy.
However, there are enough resources to do it, if we accept
soft errors.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Generally a data collector can be deactivated in the config and each
data collector defines own conditions to show/hide it.

A hidden data collector is not shown in /1/list/collectors,
/1/report/all. The relevant config is requested from the RConfD.

   * Added new constant dataCollectorNames: The consistency is checked
     at compile time thanks to template haskell.
   * cfgupgrade: add datacollectors section in cluster. The section
     currently has a single entry: 'active'
   * Added Arbitrary instances for the relevant types
   * Move Ganeti.Monitoring.{Server,Types}.DataCollector
   * Move {Monitoring.Server,DataCollectors.Types}.DataCollector

Implements Issue 870: inst-status-xen should not be shown for KVM
cluster.
Signed-off-by: Aaron Karper <akarper@google.com>
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This relies on the ReqConfigQuery and takes several RFC6901 paths and
will return a table of the keys and values. The data accessible is the
same as in config.data (typically found at /var/lib/ganeti/config.data).

Implements issue 564
Signed-off-by: Aaron Karper <akarper@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This patch adds a few unit tests for cgroup management functions of
hv_lxc.
Signed-off-by: Yuto KAWAMURA(kawamuray) <kawamuray.dadada@gmail.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

The main property of rate-limited jobs is tested (jobs must not be
scheduled over the limit), as well as reason string parsing and
properties of helper functions.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This implements the "Ad-Hoc Rate Limiting" part of the design
specified in doc/design-optables.rst.

The current implementation does not try to cache how many slots are
free per bucket since the number of running jobs is typically small
(< 100).
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

We want to reuse functions for it for a future 'Test.Ganeti.JQScheduler'.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

So that we can access them from modules that will be imported into JQScheduler
(such as job queue filters).
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This design document describes a planned extension
allowing Ganeti to plan ahead for forthcoming instances.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add a design document detailing how Ganeti, and, in particular, the
balancing algorithm can be made more aware of location and other
common causes of failure for nodes.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

In particular, skip its compilation, if the required packages aren't
present.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In particular, kvmd, luxid, metad and wconfd aren't dependent on
enabling confd.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This requires to compile each object file twice, once without profiling,
for TH splicing, and then once with profiling enabled.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Remove implemented ones from the draft list and add the ones fully or
partially implemented in 2.12.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Make gnt-cluster upgrade honor the --with-backup-dir configure option.
In this way, at configure time, it can be decided in which directory
the configuration backup tar balls are put.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

During Ganeti version upgrades, Ganeti creates a backup of
the full configuration directory. Add a new configure option
to decide in which directory the backup tar ball is to be put.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

...that was accidentally removed in e8127e66

.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>