GitLab

Now that the product has been replaced by the sum we can, again,
identify jobs only by their job id.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Non-jobs requesting resources need to identify themselves by a string
describing uniquely what they do. For the time being (while masterd
still exists), these non-jobs are client requests to masterd. Fortunately,
they already have a unique id, the thread name, which is composed of
the description of the task (like "ClientReq") followed by a running
number.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Most requests asking for resources are jobs. However, in exceptional
cases, other requests (like currently requests to masterd requiring the
configuration) need to ask for resources. They identify themselves by
a unique string.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

* stable-2.11
  (no changes)

* stable-2.10
  Fix specification of TIDiskParams
  Add unittests for instance modify parameter renaming
  Add renaming of instance custom params
  Add RAPI symmetry tests for groups
  Extend RAPI symmetry tests with RAPI-only aliases
  Add test for group custom parameter renaming
  Add renaming of group custom ndparams, ipolicy, diskparams
  Add the RAPI symmetry test for nodes
  Add aliases for nodes
  Allow choice of HTTP method for modification
  Add cluster RAPI symmetry test
  Fix failing cluster query test
  Add aliases for cluster parameters
  Add support for value aliases to RAPI
  Provide tests for GET/PUT symmetry
  Sort imports
  Also consider filter fields for deciding if using live data
  Document the python-fdsend dependency

* stable-2.9
  Fix failing gnt-node list-drbd command

* stable-2.8
  Add reason parameter to RAPI client functions

Conflicts:
	qa/qa_rapi.py: import ALL the modules
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* stable-2.10
  Fix specification of TIDiskParams
  Add unittests for instance modify parameter renaming
  Add renaming of instance custom params
  Add RAPI symmetry tests for groups
  Extend RAPI symmetry tests with RAPI-only aliases
  Add test for group custom parameter renaming
  Add renaming of group custom ndparams, ipolicy, diskparams
  Add the RAPI symmetry test for nodes
  Add aliases for nodes
  Allow choice of HTTP method for modification
  Add cluster RAPI symmetry test
  Fix failing cluster query test
  Add aliases for cluster parameters
  Add support for value aliases to RAPI
  Provide tests for GET/PUT symmetry
  Sort imports
  Also consider filter fields for deciding if using live data
  Document the python-fdsend dependency

* stable-2.9
  Fix failing gnt-node list-drbd command

* stable-2.8
  Add reason parameter to RAPI client functions
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Conflicts:
	lib/rapi/client.py: Add both params
	lib/rapi/rlib2.py: Remove reference to missing LU, add aliases
	src/Ganeti/Query/Query.hs: Take more imports

* stable-2.9
  Fix failing gnt-node list-drbd command

* stable-2.8
  Add reason parameter to RAPI client functions
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Conflicts:
	lib/rapi/client.py: Add both parameters

* stable-2.8
  Add reason parameter to RAPI client functions
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The UUID/name switch avoided this particular bit of code, and as a
result the list-drbd command failed as it tried to compare UUIDs and
names. This patch fixes the retrieval, converts the newly returned
UUIDs to names, and modifies the QA to the results and not only the
invocation are checked.
Signed-off-by: Gerard Oskamp <gjo@google.com>
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

These tests don't work with WConfd, because the configuration is
verified automatically before each operation and it's not possible to
modify the configuration state directly.

It'll be necessary to re-create these tests for WConfd once
configuration verification is implemented there.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls that
change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls
that change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls that
change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls that
change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls that
change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Re-read the instance in-memory objects from WConfd after calls that
change its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. in the configuration with comments.

Also re-read an instance object where needed.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This simplifies code that works with instance objects and needs to have
their fresh values after performing such an operation.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Instead of modifying the primary node on an object and calling 'Update',
a new method was added to ConfigWriter.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. to work with the updated version.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This test relies on bringing a configuration into an inconsistent state,
and doesn't work with WConfd any more.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Add calls to Update after modifying the node's configuration and move
some of the modifications so that they don't conflict with other calls
to ConfigWriter.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In particular, move the call to Update so that it doesn't conflict with
calls to ConfigWriter methods that change the configuration state.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Since this LU intersperses calls ConfigWriter methods with modifying
objects directly, it's necessary to call 'Update' and/or re-read them at
appropriate places.

Also a new ConfigWriter method is created for modifying the instance
communication network.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. so that it's in the scope of a running WConfd daemon. This is now
required as read-only calls invoke WConfd as well.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This involves:

- Updating how a mocked configuration object is set up.
- Adding calls to refresh in-memory objects from the configuration
  (because they're not shared any more).
- Adding calls to Update to save modified in-memory objects so that the
  changes are visible by other code.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Previously there was one shared configuration object for all jobs,
threads and other tasks. This patch creates separate ConfigWrite
instances for distinct jobs/threads.

All exported methods of ConfigWriter are now wrapped in calls that
obtain the ConfigLock from WConfD, read the current configuration, and
optionally write it back to WConfD.

_OpenConfig is now called at each such request (instead of just once at
the creation time of ConfigWriter). A new method _CloseConfig is added
that performs the necessary cleanup (saving the configuration, releasing
the lock).

_UpgradeConfig needs to be called every time a configuration is received
from WConfd, to fix parts that aren't persisted by the Python code. This
requires that it doesn't use any methods that acquire locks, and it must
not save the configuration at the end (unless it's called just after
creating a ConfigWriter instance in "offline" mode).

The semantics of Update changes slightly. Before it just checked its
argument existed in the configuration. Now it also checks that the
its serial number is the same as in the master configuration state, to
avoid overwriting changes in other threads. This will require fixing all
calls to Update, in particular to avoid interspersing calls to Update
and other ConfigWriter methods. In the future, we should aim to
eliminate Update completely.

All LUs now carry their own instance of ConfigWriter, with their
corresponding job ID. Other cide that uses ConfigWriter identifies with
job ID 'None' and thread ID.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Allow reading the configuration only if the client has acquired
ConfigLock in shared or exclusive mode, and allow writing it only if the
client has acquired ConfigLock in exclusive mode.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This allows the clients to protect configuration operations, as required
by the current implementation of Python 'ConfigWriter'.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This convenience function examines 'listLocks' for a particular owner
and checks, if it holds a given lock at a given ownership level (or
higher). Therefore checking for a shared locks returns 'True' if the
owner holds a shared or exclusive lock.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. using the DEBUG channel.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This way, each call to AddNode or RemoveNode uses a passed configuration
object that belongs to the appropriate job.

This is required to subsequently get rid of the single shared
ConfigWriter object.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

It was only required for RPC calls, which are now handled by WConfD.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

There was just a locking version, and to keep things consistent in
_OpenConfig, we need also a non-locking one.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In most cases, this is what has been already implemented manually.
In some cases, the configuration write is issued even if the
configuration hasn't been modified, but since WConfd checks for such
cases, neither file write nor distribution to nodes is performed.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. so that it can be enhanced later.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The lock is used for protecting concurrent operations on the
configuration on the Python side.

The lock must be last wrt to the lock ordering so that it can be always
obtained regardless of other locks held by a client. During operations
that hold this lock, no other locks are ever acquired.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This will be needed later to add WConfd locking for the synchronization.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This is a prerequisite for having thread-local configuration states.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Commit 580b1fdd

 incorrectly assumes that disk parameters are
just the standard ones, whereas the man page explicitly states
that additional parameters can be passed as well, if they make
sense for the chosen storage type. Fix this.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Cherry-picked from b26a275a

. Fixes issue 779.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

This was an unnoticed regression when SSConf has been refactored into
WConfd.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>