GitLab

We add a second category of daemons in daemon-util, those that are
started on demand and should not be started by the start_all action, but
should be stopped during service shutdown. This category currently
includes metad and kvmd.

To this end we introduce two more array variables:

 - ON_DEMAND_DAEMONS: currently holds kvmd and metad, which should only
   be stopped during stop-all.

 - ALL_DAEMONS: is the union of DAEMONS and ON_DEMAND_DAEMONS, i.e. the
   list of all known daemons.

We also change the daemon-util test to expect kvmd and metad in the stop
daemon list.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

Since _SetDrbdHelper calls methods on self.cfg, it's necessary to update
the configuration before it is called.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

* stable-2.11
  Make GlusterFS tests dual-stack-aware
  Moving release date of 2.11.0 to 25th of April
  Revision bump for 2.11.0
  Prepare NEWS file for release of 2.11.0
  Fix Xen instance state

* stable-2.10
  Fix failed DRBD disk creation cleanup
  Fix lint errors introduced during cherry-pick
  Hooking up verification for shared file storage
  Fix --shared-file-storage-dir option of gnt-cluster modify
  Clarify default setting of 'metavg'
  Fix invocation of GetCommandOutput in QA
  Clean up RunWithLocks
  Add an exception-trapping thread class
  Wait for delay to provide interruption information
  Add an expected block option to RunWithLocks
  Track if a QA test was blocked by locks
  Add a RunWithLocks QA utility function

* stable-2.9
  If Automake version > 1.11, force serial tests

Conflicts:
	NEWS: take all additions
	configure.ac: ignore suffix bump
	lib/client/gnt_cluster.py: union of additions
	lib/cmdlib/cluster.py: union of additions
	lib/hypervisor/hv_xen.py: union of additions
	qa/qa_job_utils.py: take version of master
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

* stable-2.10
  Fix failed DRBD disk creation cleanup
  Fix lint errors introduced during cherry-pick
  Hooking up verification for shared file storage
  Fix --shared-file-storage-dir option of gnt-cluster modify
  Clarify default setting of 'metavg'
  Fix invocation of GetCommandOutput in QA
  Clean up RunWithLocks
  Add an exception-trapping thread class
  Wait for delay to provide interruption information
  Add an expected block option to RunWithLocks
  Track if a QA test was blocked by locks
  Add a RunWithLocks QA utility function

* stable-2.9
  If Automake version > 1.11, force serial tests
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

* stable-2.9
  If Automake version > 1.11, force serial tests
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Resolving "localhost" may either return "127.0.0.1" or "::1",
so we have to check for both cases in the relevant tests.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This fixes broken compilation on Debian Jessie (#802).
See also http://stackoverflow.com/questions/15820844/



Thanks to Apollon Oikonomopoulos for finding the solution
and Klaus Aehlig for helping to do it conditionally.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

When forking off jobs, make them inherit the debug level
of the parent process (i.e., of luxid). In this way, we
can debug jobs in test clusters without cluttering production
logs. We pass the debug level through the environment instead
of the startup dialog with the parent, as we would like to
be able to debug the startup dialog.

While there, fix the call of SetupLogging: the debug parameter
is supposed to be an int.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

While jobs are still handled by what used to be masterd,
the log message should be assigned to the respective
job.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

When creating a DRBD disk, Ganeti reserves minor numbers on a per-node
basis. In case of a failed disk creation, these reservations should be
released. During the name/uuid refactoring, the invocation of the
function that releases the minors was not updated, resulting in no
minors being released in case of a DRBD disk creation failure.
Ganeti's verification utilities catch this error and rightly complain.
This patch fixes the issue by providing the UUID and not the name.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Due to unforseen circumstances, we are moving the
release of 2.11.0 to tomorrow and thus the NEWS
file needs to be updated.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch adds support for the native aio on KVM
hypervisor.

Basically, It adds a new HV-KVM optional parameter
"disk_aio" that can be set with the followings values:
threads (the default for KVM) or native. If not set, it
uses the KVM default that is "threads". The patch also
updates the man page for gnt-instance.
Signed-off-by: Marco Casavecchia (aka MCM) <marcolinuz@gmail.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. so that they are displayed properly in logs.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Otherwise a job that is being started is falsely reported as dead.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. and add a reason trail message. Otherwise failed jobs hang, never
finishing.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In this example, the cluster has two nodes and four instances, two with
primary on each of the nodes. The most scarce resource on this cluster
are (virtual) CPUs and the second node has 3 times the CPU speed of the
first one. So distributing the instances 1 and 3 gives a more balanced
cluster.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...as described in doc/design-cpu-speed.rst
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add a derived parameter for nodes, providing the ratio of
virtual CPUs per CPU-speed weighted physical CPU.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Make the htools luxi backend also query for cpu_speed
and take the result into account.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Extend the text format by an optional column for each node
containing the relative CPU speed, if provided.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add a function on nodes modifying the CPU speed
parameter.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add an additional parameter to the representation of a node
for the relative CPU speed, initially set to 1.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

In other words, remove "cpu_speed" from all "nodeparams"
where it is present, be it cluster, group, or node. Note
that upgrading is no problem, as the default value will
be used implicitly.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This parameter will describe the speed of the CPU relative
to the speed of a "normal" node in this node group.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...in order not to have to declare floating point
values as VTypeInt and rely on the sloppiness of
the JSON specification to not distinguish between
integers and floating point numbers.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This document really only talks about CPU speed.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

In this case, the call trying to acquire a shared lock always succeeds,
because the daemon already has an exclusive lock, which falsely reports
that the job has died.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In particular, distinguish the cases when a job could not have been
cancelled and when a job has already finished.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. because modifying the queue inside the handler can have unexpected
consequences.

Since Python 2 doesn't have a nice way how to modify a variable from
an inner function, we have to use a list as a wrapper.
(Python 3 has the "nonlocal" keyword for it.)
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Calm a few lint errors introduced during cherry-picking code in
qa_job_utils.py. The fixes were intentionally made in a way which should
produce merge conflicts later on, so it's not forgotten to undo them.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

When failing a job, add an entry to the reason trail, indicating
what made the job fail (e.g., failed to fork or detected job death).
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...to simplify manipulation of them.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...to be able to operate on the MetaOpCode that
is behind an InputOpCode (if we're in the right
component of the sum).
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...so that manipulations deep within such an object
get more simple.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Move all the definition of objects to a spearate file. In
this way, the lense module for JQueue can use these objects,
while JQueue can use the lenses. For use outside, we reexport
the objects.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>