This adds support for a callback returning name resolver options. This
is required for powercycling offline nodes. While it would be possible
to implement this using a separate RPC client class, doing so would
require some code duplication with the standard “RpcRunner” class.

The callback can generate the options based on the call arguments.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It is always used and can be set to “None” if intended.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

On my work for instance policies we decided not to overload --force
more, instead we go with a new flag --ignore-ipolicy. This is just a
small patch to reflect this decision.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

autotools/testrunner was missing in distcheck.  Now some changes from
70041061 have been reverted, and testrunner
is not built anymore.  coverage now uses fakeroot when needed.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Running “make -jN distcheck” for any N > 1 would reliably break when
building the RPC client wrappers. This was due to wrong dependencies in
Makefile.am, fixed by this patch.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Signed-off-by: Michael Hanselmann <hansmi@google.com>

* devel-2.5:
  jqueue: Factorize checking job processor's result
  jqueue unittest: Rename simple fake-job class
  jqueue: Fix epylint errors introduced in 37d76f1e
  jqueue: Fix deadlock between job queue and dependency manager
  locking: Add “__repr__” to SharedLock and PipeCondition
  daemon.GenericMain: Don't generate backtrace on conflicting daemons
  utils.io.WritePidFile: Improve error reporting
  utils.ListVisibleFiles: Hide “lost+found” directories
  Fix race condition in test for *FileID functions

Conflicts:
	lib/utils/io.py: Trivial

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* stable-2.5:
  jqueue: Fix epylint errors introduced in 37d76f1e
  jqueue: Fix deadlock between job queue and dependency manager

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

I had an idea for improving locking performance. To see if it worked I
wrote this tool. Unfortunately the idea didn't quite work (broke
unittests left and right), but the tool is still handy for evaluating
future changes to the “SharedLock” class.

It is not installed or run at build/test time. In its current form it is
intended for manual use. Example output:

---
Total number of acquisitions: 32642
Per-thread acquisitions:
  Thread 0: 6536 (20.0%)
  Thread 1: 6488 (19.9%)
  Thread 2: 6536 (20.0%)
  Thread 3: 6529 (20.0%)
  Thread 4: 6553 (20.1%)
Benchmark CPU time: 5.010s
Average time per lock acquisition: 0.15348ms
Process:
  User time: 4.160s
  System time: 1.030s
  Total time: 5.190s
---

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This allows for more unittesting.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Also add a parameter for priority, to be used in an upcoming
patch.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

When Ganeti was compiled with the option of running as a different
user/group, the group ownership of /etc/hosts was set to gnt-daemons.  Now
permissions for /etc/hosts are preserved (or set correctly).

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

These tests need fakeroot.  If it's not present, they are not run (they
don't fail).

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Tests ending in "-runasroot" or "-runasroot.py" are run through fakeroot,
so activity that needs root privileges can be tested.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Even for a small 4-node node-group, this reduces the payload of this
RPC call by half for each node; for a big node group, previous testing
(using a different patch) showed a general reduction in traffic over
the entire cluster verify from 9MB to ~2MB.

We only need to change the RPC call, not its return value, since that
was already in an "optimised" form.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, all RPC payloads sent by the client to the remote node
daemons must be identical, due to how the data is passed
internally. This is deficient in both use (from the programmer's point
of view) and from the network traffic (cluster verify/disk data
gathering has a total payload which is O(n²) in size of the nodes
being queried, instead of O(n)).

This patch changes the RPC internals so that we always pass
dictionaries indexed by target node name. For the default use case,
when the payload is identical, we only serialise the payload once, so
the extra overhead is just a dict with the node names and values all
pointing to the same object. For different payloads, we will encode
the body multiple times, but hopefully the bodies will be smaller.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This will be used in a later patch for parameter pre-processing. For
now we just add the parameter as 'None' and add a consistency check
for the parameter definitions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Change a few lines to match rest of code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Change code style to match rest of Ganeti code and add
two tests checking for single-line messages.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

No need to look up the key twice.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Serializing to JSON using “simplejson” is significantly slower when
indentation and/or sorting of dictionary keys is used. In simplejson 1.x
the difference isn't that big, but with simplejson 2.x the difference
can be up to a factor of 7.5. The reason is that the latter no longer
uses C functions when sorting or indentation is used.

With this patch we revert everything to simplejson's defaults, which
should provide us with the best performance available.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

When an opcode is about to be processed its dependencies are
evaluated using “_JobDependencyManager.CheckAndRegister”. Due
to its nature that function requires a lock on the manager's
internal structures. All of this happens while the job queue
lock is held in shared mode (required for the job processor).

When a job has been processed any pending dependencies are re-added
to the job workerpool. Before this patch that would require
the manager's lock and then, for adding the jobs, the job queue
lock. Since this is in reverse order it will lead to deadlocks.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

When gnt-job submit is used with the --each option, it sends as
parameters for QueryJobs a list of lists, each containing a job ID.
While this works now, it is not the intended format for QueryJobs.

This patch fixes this behavior, by making it send a list of job IDs.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

These help when debugging.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Andrea Spadaccini <spadaccio@google.com>

Cluster destroy would fail if there were errors while deactivating the
master IP address. This patch demotes such errors to warnings, allowing
the cluster to be destroyed even if the master IP address turndown
fails.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Document the --each option of gnt-debug submit-job.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Users of the RAPI client may want to use feature strings too, therefore
they should be explicitely public.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Instead of repeating a “if cond: query.append(…)” pattern, utility
functions are added and used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

By default parameters existing for all opcodes (e.g. “debug_level”) are
not listed in the RAPI documentation. With this change the “depends”
parameter will be listed unless explicitely excluded.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The documentation string is used in the RAPI description and
having more details is good.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>