GitLab

* Use RPC transport to pass OS params to the metadata daemon, using
  the Luxi protocol
* Fix comment in 'ganeti.rpc.transport'
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

... which starts the configuration and web servers.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Extend metadata web server to serve the OS parameters from the shared
MVar containing the instances' configurations, indexed by their IP
address.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

The configuration server listens on a Unix socket for connections from
the node daemon.  The node daemon sends the instance parameters to the
configuration server so they can be served through the metadata daemon
web server to the instances which have the communication mechanism
enabled.

The configuration server reads the instance parameters and, currently,
it extracts the instance's name and the instance's IP address on the
instance communication NIC.  The instance's name is used for logging
and the IP address is used to index the instance parameters, given
that instances do not authenticate with the metadata daemon, and the
only thing we know about them is their IP address.

The configuration server also extracts the OS parameters, including
public, private and secret, and creates an object containing those
parameters and their visibility.

The configuration is kept internally in an 'MVar' which will be shared
with the metadata daemon web server.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

The metadata daemon will be responsible for receiving the instance
parameters from the node daemon and keeping them in memory.  The type
of the instance parameters in defined in a separate module because it
will be shared between the web server and the configuration server.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Currently, the metadata daemon is mainly a web server.  However, a new
configuration server will be introduced along with other metadata
related modules.  Therefore, this patch moves the current metadata
daemon web server to its own directory.  This module is also renamed
from 'Server' to 'WebServer'.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

The section describes how Luxi daemon should execute the jobs so that it
can ensure it can detect their death at every point.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. to 2 spaces instead of 4.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Apparently sometimes unicode strings appear out of nowhere (maybe from
the RPC calls) and then they are used as a list of characters (hail
Python). This patch adds a function that ensures we always have a
proper list.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Ganeti prouds itself of its really good retro-compatibility and API stability.
Some of our users haven't upgraded their hardware in the last 2000 years (one
century more, one century less) and their Xen-PVM (Parchment and VelluM) does
not support Arabic numerals yet.

The already existing support for printing data in Roman numerals had fallen into
disrepair lately. This patch responds to the request of our most ancient users,
quickly delivered to us via the "cursus publicus", asking for this fundamental
functionality to be restored to its old splendour.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Since LUClusterVerifyGroup issues 3 RPC calls to get information from
nodes, and during every of these calls, the configuration can be
modified by somebody else, it's necessary to acquire a shared lock on
the configuration.

All the RPC calls are grouped together in a block that holds the lock,
and the tests are only performed afterwards, after the lock is released.

This should be later moved into WConfd, which can prevent configuration
changes internally.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This makes debugging more informative and makes more clear how
configuration distribution gets aggregated.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The ConfigWriter now allows nested locking calls, and acquires
(releases) the config lock from WConfd only the first (last) time.

This is needed for blocks that acquire a lock on the configuration and
use synchronized configuration calls inside. While this should be
avoided in general, it is required for some special circumstances.

Also add a warning that ConfigWriter isn't thread-safe any more.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The get/put RAPI test tests for symmetry of what the RAPI output and
what it takes in, with exceptions for legacy differences and options
that should not be exposed. The private OS params belong to this
category, and this patch adds them as an exception.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

As, during startup, masterd uses livelock files for special
tasks, the lexicographically last livelock file for masterd
is the masterd_check_agreement_ file. However, when debugging,
we want the primary livelock file. Hence extend the prefix to
include the leading digit 1 of the time, which will work till
Wed May 18 05:33:20 CEST 2033.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Now that the product has been replaced by the sum we can, again,
identify jobs only by their job id.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Non-jobs requesting resources need to identify themselves by a string
describing uniquely what they do. For the time being (while masterd
still exists), these non-jobs are client requests to masterd. Fortunately,
they already have a unique id, the thread name, which is composed of
the description of the task (like "ClientReq") followed by a running
number.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Most requests asking for resources are jobs. However, in exceptional
cases, other requests (like currently requests to masterd requiring the
configuration) need to ask for resources. They identify themselves by
a unique string.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

* stable-2.11
  (no changes)

* stable-2.10
  Fix specification of TIDiskParams
  Add unittests for instance modify parameter renaming
  Add renaming of instance custom params
  Add RAPI symmetry tests for groups
  Extend RAPI symmetry tests with RAPI-only aliases
  Add test for group custom parameter renaming
  Add renaming of group custom ndparams, ipolicy, diskparams
  Add the RAPI symmetry test for nodes
  Add aliases for nodes
  Allow choice of HTTP method for modification
  Add cluster RAPI symmetry test
  Fix failing cluster query test
  Add aliases for cluster parameters
  Add support for value aliases to RAPI
  Provide tests for GET/PUT symmetry
  Sort imports
  Also consider filter fields for deciding if using live data
  Document the python-fdsend dependency

* stable-2.9
  Fix failing gnt-node list-drbd command

* stable-2.8
  Add reason parameter to RAPI client functions

Conflicts:
	qa/qa_rapi.py: import ALL the modules
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* stable-2.10
  Fix specification of TIDiskParams
  Add unittests for instance modify parameter renaming
  Add renaming of instance custom params
  Add RAPI symmetry tests for groups
  Extend RAPI symmetry tests with RAPI-only aliases
  Add test for group custom parameter renaming
  Add renaming of group custom ndparams, ipolicy, diskparams
  Add the RAPI symmetry test for nodes
  Add aliases for nodes
  Allow choice of HTTP method for modification
  Add cluster RAPI symmetry test
  Fix failing cluster query test
  Add aliases for cluster parameters
  Add support for value aliases to RAPI
  Provide tests for GET/PUT symmetry
  Sort imports
  Also consider filter fields for deciding if using live data
  Document the python-fdsend dependency

* stable-2.9
  Fix failing gnt-node list-drbd command

* stable-2.8
  Add reason parameter to RAPI client functions
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Conflicts:
	lib/rapi/client.py: Add both params
	lib/rapi/rlib2.py: Remove reference to missing LU, add aliases
	src/Ganeti/Query/Query.hs: Take more imports

* stable-2.9
  Fix failing gnt-node list-drbd command

* stable-2.8
  Add reason parameter to RAPI client functions
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Conflicts:
	lib/rapi/client.py: Add both parameters

* stable-2.8
  Add reason parameter to RAPI client functions
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The UUID/name switch avoided this particular bit of code, and as a
result the list-drbd command failed as it tried to compare UUIDs and
names. This patch fixes the retrieval, converts the newly returned
UUIDs to names, and modifies the QA to the results and not only the
invocation are checked.
Signed-off-by: Gerard Oskamp <gjo@google.com>
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

These tests don't work with WConfd, because the configuration is
verified automatically before each operation and it's not possible to
modify the configuration state directly.

It'll be necessary to re-create these tests for WConfd once
configuration verification is implemented there.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls that
change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls
that change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls that
change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls that
change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls that
change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls that
change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. in the configuration with comments.

Also re-read an instance object where needed.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This simplifies code that works with instance objects and needs to have
their fresh values after performing such an operation.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Instead of modifying the primary node on an object and calling 'Update',
a new method was added to ConfigWriter.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. to work with the updated version.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This test relies on bringing a configuration into an inconsistent state,
and doesn't work with WConfd any more.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Add calls to Update after modifying the node's configuration and move
some of the modifications so that they don't conflict with other calls
to ConfigWriter.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In particular, move the call to Update so that it doesn't conflict with
calls to ConfigWriter methods that change the configuration state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Since this LU intersperses calls ConfigWriter methods with modifying
objects directly, it's necessary to call 'Update' and/or re-read them at
appropriate places.

Also a new ConfigWriter method is created for modifying the instance
communication network.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. so that it's in the scope of a running WConfd daemon. This is now
required as read-only calls invoke WConfd as well.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>