GitLab

* Generalize 'kvm-ifup-os' to be also usable by Xen
* Include a target to build the 'xen-ifup-os' NIC script from the KVM
  one, given that they are very similar.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

It is not necessarily bad coding stile to use "and" to present
a list of conditions. In fact, a statement like

    and [ foo == foo'
        , bar == bar'
        , baz == baz'
        ]

can be more readable than the equivalent

    (foo == foo')
      && (bar == bar')
      && (baz == baz')
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add a verify first sanity check for lock waiting: a user that has a pending
request cannot modify his locks. This patch also brings in the necessary
infra structure for having arbitrary waiting structures.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

To avoid jobs polling on locks, and also to ensure that the
most important of the waiting jobs obtains a lock, add a
data structure to keep track of who is waiting for which locks.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

... as an intermediate step before moving the responsbility of
computing the final OS parameter configuration from the configuration
server to the web server.  This will allow the metadata daemon to
store a more general configuration and thus reuse that configuration
for replying to more general queries.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Move all tap-related functionality to the hv_kvm.netdev submodule. We
rename _OpenTap to OpenTap, since it will now be used as a public
function.

Also, change the hv_kvm tests to import the new code.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

We move the HMP and QMP classes to the hv_kvm.monitor module.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Weighing in at 3k LOC, it is probably time to split hv_kvm into a number
of modules before adding new code.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

The client is generated automatically by Ganeti.THH.HsRPC.

It also includes helper functions for creating a client object.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

It uses TH to create client functions with the corresponding signature
from a given list of server functions.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

... which starts the configuration and web servers.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

The configuration server listens on a Unix socket for connections from
the node daemon.  The node daemon sends the instance parameters to the
configuration server so they can be served through the metadata daemon
web server to the instances which have the communication mechanism
enabled.

The configuration server reads the instance parameters and, currently,
it extracts the instance's name and the instance's IP address on the
instance communication NIC.  The instance's name is used for logging
and the IP address is used to index the instance parameters, given
that instances do not authenticate with the metadata daemon, and the
only thing we know about them is their IP address.

The configuration server also extracts the OS parameters, including
public, private and secret, and creates an object containing those
parameters and their visibility.

The configuration is kept internally in an 'MVar' which will be shared
with the metadata daemon web server.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

The metadata daemon will be responsible for receiving the instance
parameters from the node daemon and keeping them in memory.  The type
of the instance parameters in defined in a separate module because it
will be shared between the web server and the configuration server.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Currently, the metadata daemon is mainly a web server.  However, a new
configuration server will be introduced along with other metadata
related modules.  Therefore, this patch moves the current metadata
daemon web server to its own directory.  This module is also renamed
from 'Server' to 'WebServer'.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Fix the build by including the qa/patch directory in Makefile.am.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

The formatting functions in qa_utils.py cannot be used by modules
imported there, such as qa_config. This patch factors the function
calls into a separate file. Also reorders imports in touched files.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

As our so-called unit tests for the logical units are more
like integration tests, calling the actual LU through an almost
live mcpu, we need a quite involved "mock" for the wconfd module,
called indirectly through mcpu. The difficulties arise from the
fact that both, the mcpu and the LUs verify lock ownership at
various places.

To keep things simple, we only keep track of the mocked state
of affairs where the result is actually checked. With more
consistency checks in the LUs, this module has to be implemented
further.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...as the will be used by mcpu for lock handling and that will
require tests to be adapted accordingly.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This migrates the functionality from Python
ConfigWriter._UnlockedGetSsconfValues to Haskell.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Since the Haskell 'zlib' library by default uses exceptions for
signalling errors, we need to use its internals to implement a proper,
pure decompressing function.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The module also adds type classes for lenses for common fields (serial,
mTime, uuid and tags).

Splitting the lenses into a separate module avoids problems with unused
functions.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The module currently re-exports Control.Lens and adds a function that
creates lenses from data types by appending 'L' to its field names.

All Ganeti modules should import this module instead of Control.Lens
directly.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

These helpers aren't needed by other code in Ganeti.THH and since the
module is already large, it'll be better to separate them into their
own.

No functionality is changed by this patch, just functions moved.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

To request resources from WConfD, requesters have to provide
the name of a file they own an exclusive lock on. In this way,
their death can be detected. Add utility functions to obtain
such a file name.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

With jobs running in processes different from the lock management soon,
we need to detect if a job holding locks dies, in order to release the
resources.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch adds a QA utility function that acquires a set of locks, and
attempts to run a given function with the locks in place. Should the
given function block, this function does not detect this - later
patches will address the issue.

An example of its use is provided by having the move-instance test
modified to use it.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add a module describing how abstract requests, like lock requests are
passed over the wire to WConfD.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Include only the programs that were configured to be built, and let GHC
figure out the required Haskell sources by itself.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Previously they were included no matter of what the configuration was,
failing to compile if some of the dependencies were missing.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

As we made Ganeti.Locking.Locks an instance of JSON, we better verify
that we can readJSON what we showJSON.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

For luxid to be feature-complete with respect to masterd, it
also needs to answer requests about locks. This includes knowing
the fields available for locks.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Cherry-picked from commit 86b04819Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. and include ConfigData in WConfd's ConfigState.

Currently it includes functions for asynchronous saving the
configuration and reading and (over)writing it in the daemon state.

Load the configuration from the file at daemon startup and configure an
asynchronous task to save it after writes.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Each such worker has its own background thread and performs a given
task when triggered. Triggers arriving when the worker is processing are
accumulated and processed together in a subsequent worker action.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Utils is getting too big, so better split this new set of functions into
a separate sub-module. This also allows us to use ResultG there.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Calling logging functions in a monad transformer stack requires
instances for all the transformers.
This patch adds a module with functions named just as in Ganeti.Logging,
but lifted into MonadBase. This allows using them in any transformer
stack which has MonadBase instance and whose base monad is MonadLog.
This condition is satisfied for all IO-based stacks.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The offending target 'test/hs/offline-test.sh' requires the dependencies
even when the test isn't actually run. This is because currently our
Makefile doesn't distinguish between prerequisites for building test
scripts and running them.

This is a quick fix that hides the dependencies if the Haskell tests
aren't built, but it doesn't address the core problem. This will be
fixed in a separate patch series.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

At some places a few items of lists use 8 spaces, while we use tabs
everywhere (as makefiles require anyway). This patch unifies the
indentation.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Instead of showing the long GHC command line, show which file is being
compiled and what its dependencies are. This gives more meaningful
information to developers.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This allows parallel compilation of all targets, including the Haskell
object files. While re-starting GHC for every Haskell source almost
doubles the total CPU time, allowing it run in parallel at the end cuts
down the wall-clock time to half.

Haskell targets are split into two: main ones and testing ones.
The testing targets (starting with "test/") are build with coverage
automatically. Therefore the testing object and interface files have
different extensions (this also allows their parallel compilation).

GHC is now used to produce a separate Makefile.ghc with all
dependencies. This is then included and used for compiling each object
file separately.

Note that the inclusion + generation of Makefile.ghc is turned off when
calling clean/ganeti/distclean. For some reason, Makefile infinitely
recurses when calling "make ganeti", if it includes Makefile.ghc. But
since using ifneq/endif in Makefile.am conflicts with Automake, it was
necessary to put the snippet into an Automake variable and substitute it
into the resulting Makefile.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

When the build directory differs from the source directory, the
automatic variable '$<' points to the file found by prepending the path.
This causes double '../' to be inserted when combined with
'$(abs_top_srcdir)'. This patch fixes this.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>