GitLab

This patch adds a QA utility function that acquires a set of locks, and
attempts to run a given function with the locks in place. Should the
given function block, this function does not detect this - later
patches will address the issue.

An example of its use is provided by having the move-instance test
modified to use it.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

(cherry picked from commit 343c9ed2)

Conflicts:
	qa/qa_rapi.py (didn't backport changes to this file)

Fix the build by including the qa/patch directory in Makefile.am.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The formatting functions in qa_utils.py cannot be used by modules
imported there, such as qa_config. This patch factors the function
calls into a separate file. Also reorders imports in touched files.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Include only the programs that were configured to be built, and let GHC
figure out the required Haskell sources by itself.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Previously they were included no matter of what the configuration was,
failing to compile if some of the dependencies were missing.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

For luxid to be feature-complete with respect to masterd, it
also needs to answer requests about locks. This includes knowing
the fields available for locks.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Cherry-picked from commit 86b04819Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The offending target 'test/hs/offline-test.sh' requires the dependencies
even when the test isn't actually run. This is because currently our
Makefile doesn't distinguish between prerequisites for building test
scripts and running them.

This is a quick fix that hides the dependencies if the Haskell tests
aren't built, but it doesn't address the core problem. This will be
fixed in a separate patch series.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

Instead of showing the long GHC command line, show which file is being
compiled and what its dependencies are. This gives more meaningful
information to developers.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This allows parallel compilation of all targets, including the Haskell
object files. While re-starting GHC for every Haskell source almost
doubles the total CPU time, allowing it run in parallel at the end cuts
down the wall-clock time to half.

Haskell targets are split into two: main ones and testing ones.
The testing targets (starting with "test/") are build with coverage
automatically. Therefore the testing object and interface files have
different extensions (this also allows their parallel compilation).

GHC is now used to produce a separate Makefile.ghc with all
dependencies. This is then included and used for compiling each object
file separately.

Note that the inclusion + generation of Makefile.ghc is turned off when
calling clean/ganeti/distclean. For some reason, Makefile infinitely
recurses when calling "make ganeti", if it includes Makefile.ghc. But
since using ifneq/endif in Makefile.am conflicts with Automake, it was
necessary to put the snippet into an Automake variable and substitute it
into the resulting Makefile.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

When the build directory differs from the source directory, the
automatic variable '$<' points to the file found by prepending the path.
This causes double '../' to be inserted when combined with
'$(abs_top_srcdir)'. This patch fixes this.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. and update the code that uses it.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Similarly to PY_UNIT, add HS_UNIT to run Haskell coverage only if we
have the required modules.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Since Ganeti now uses luxid for core operations, it needs to be always
present.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The Node deamon was executed with the wrong gid (gnt-daemons) instead of the one
assigned to it by configure.ac.

Fixes Issue 707.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The Ganeti code style has been stored on the project wiki at:

  https://code.google.com/p/ganeti/wiki/StyleGuide
  https://code.google.com/p/ganeti/wiki/HaskellStyleGuide

This commit combines the two pages into an .rst file with minimal
formatting and language changes. Note that the style guide introduced
in this commit does not fit the code base in a number of ways,
including:

* Some Haskell files have lines longer than 78 characters
* Some Haskell files have trailing whitespace
* Some Python docstring initial sentences lack punctuation at the end

The decision to either change the offending lines to fit the guidelines,
to change the guidelines to fit the codebase or to simply ignore the
discrepancies is left for other commits to solve.
Signed-off-by: Santi Raffa <rsanti@google.com>
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Export documents about 'cluster keys replacement' and 'design file
based storage' from the wiki to the repository.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Add OS reinstall design doc to the list of design docs in the
Makefile, otherwise it does not get compiled when modified, and add it
also to the index page of the documentation, where all the other
design docs are anchored.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Virtual clusters are an efficient way to test how Ganeti behaves
on a large cluster without requiring a large number of machines.
Now that more tasks like job replication are done by luxid, provide
that functionality in Haskell as well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Fix overridable configure directories, namely, --bindir, --sbindir,
and --mandir.

Variables 'bindir', 'sbindir', and 'mandir' are exported from
'configure.ac' to 'Makefile.am'. At first, it would seem to be
possible to do the following:

  MANDIR := $(mandir)
  ...
  mandir = $(versionedsharedir)$(datarootdir)/$(MANDIR)

However, this does not work in the output of 'configure' (i.e., the
'Makefile'), these variables will be reordered:

  mandir = $(versionedsharedir)$(datarootdir)/$(MANDIR)
  ...
  MANDIR := $(mandir)

As a result, 'MANDIR' will capture not the exported value from
'configure.ac' but the 'mandir' defined in the 'Makefile'.

The solution is explicitly export these variables from 'configure.ac'
with different names, namely, 'BINDIR', 'SBINDIR', and 'MANDIR':

  AC_SUBST([BINDIR], $bindir)
  AC_SUBST([SBINDIR], $sbindir)
  AC_SUBST([MANDIR], $mandir)

The rest is just fixing the paths and the symlinks in the 'Makefile'.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Add 'design-2.11.rst' which kvmd and instance shutdown to the
top-level documentation and Makefile.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Add unit tests for KVM daemon.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Add manpage for the KVM daemon.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Add KVM daemon entry point, command-line options, backgrounding, etc
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Add KVM daemon logic, which contains monitors for Qmp sockets and
directory/file watching.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* add user and group to 'configure.ac', 'Makefile.am' and
  'AutoConf.hs.in'
* extend 'Daemon' datatype with 'GanetiKvmd' and implement all related
  functions, such as, 'daemonName', etc.
* export KVM daemon name as constant
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

As 2.10 is the first version from which you can do automatic upgrades,
there is nothing to do when going to any other version in the 2.10
branch.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Design document for KVM daemon which is needed by the instance
shutdown detection for KVM.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

The RAPI workload script supplies work for the RAPI compatibility
tests. The initial version does very little, but can be expanded
as needed.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The Ganeti code style has been stored on the project wiki at:

  https://code.google.com/p/ganeti/wiki/StyleGuide
  https://code.google.com/p/ganeti/wiki/HaskellStyleGuide

This commit combines the two pages into an .rst file with minimal
formatting and language changes. Note that the style guide introduced
in this commit does not fit the code base in a number of ways,
including:

* Some Haskell files have lines longer than 78 characters
* Some Haskell files have trailing whitespace
* Some Python docstring initial sentences lack punctuation at the end

The decision to either change the offending lines to fit the guidelines,
to change the guidelines to fit the codebase or to simply ignore the
discrepancies is left for other commits to solve.
Signed-off-by: Santi Raffa <rsanti@google.com>
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch adds a couple of utility functions to manipulate
the map of master candidate SSL certificate digests.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

In this example, there are two drbd instances, rendering a total of
four nodes ineligible for being offlined. Additionally, the master
may not be offlined either, leaving a single candidate.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The generic part will be reused in WConfd.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Also create a new module for RPC errors.
This allows it to be reused for other clients as well.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Move rpc.py to rpc/node.py and modify imports in existing code.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Add Gluster to Ganeti by essentially cloning the shared file behaviour
everywhere in the code base.
Signed-off-by: Santi Raffa <rsanti@google.com>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Currently they are generated only as Strings.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

Now the file contains the type class declaration as well.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

This will allow WConfD to use the general functions without importing
Luxi.hs.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This fixes issue 634.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

In order to allow informed decissions on when to start a job,
it is necessary for luxid to keep track of the (active part
of the) job queue. Add a scheduler, similar to the config reader,
that does this, but also schedules jobs to be executed. At the
moment this is just the simple mechanism of starting jobs until
a fixed number of running jobs is reached.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>