We call _OpenRealBD during the process and this needs disk parameters to
work. This was reported by Constantinos.

The fix is very ugly though.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch moves the “call_version” to a new RPC client definition and
then adds a new runner using the DNS resolver for getting the host
address.

The standard “BootstrapRunner”, where the call was before, tries to
resolve node names using ssconf first, which doesn't work properly when
re-adding a node with a new primary IP address.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In at least two cases "%s" is replaced with str(), too.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, noded requires PUT, even though the semantics of the RPC
calls do not match a PUT. We change the code accept both PUT and POST,
with the intention to remove the PUT support in a later version.

On the client side, the RPC code is changed to send POST requests.

Additionally, we add a message to the HttpBadRequest exception to make
clear the failure mode (not seeing any error message was what made me
send this patch…). This was the only description-less use of this
exception, by the way.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Commit 601dfcbb made some changes to the default encoders. This made
“ConfigRunner.call_upload_file” fail due to a missing encoder. This
patch applies the necessary changes to ConfigRunner and updates the
unittest for “call_upload_file”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Use fstat(2) on the file handle instead of doing a separate stat(2).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

After some preparing patches this unittest can finally be added.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Instead of receiving the whole masterd context, it only gets the
configuration and the lock monitor callback. This simplifies
unittesting.

Additionaly, two new arguments for testing are added.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch adds a number of unittests for the RPC client base class.
Some small changes were necessary in “rpc.py” to allow for better
testing.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Just an additional check, useful in unittests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

“assert” is a statement, not a function.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is needed to powercycle a node marked offline.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This adds support for a callback returning name resolver options. This
is required for powercycling offline nodes. While it would be possible
to implement this using a separate RPC client class, doing so would
require some code duplication with the standard “RpcRunner” class.

The callback can generate the options based on the call arguments.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

It is always used and can be set to “None” if intended.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, all RPC payloads sent by the client to the remote node
daemons must be identical, due to how the data is passed
internally. This is deficient in both use (from the programmer's point
of view) and from the network traffic (cluster verify/disk data
gathering has a total payload which is O(n²) in size of the nodes
being queried, instead of O(n)).

This patch changes the RPC internals so that we always pass
dictionaries indexed by target node name. For the default use case,
when the payload is identical, we only serialise the payload once, so
the extra overhead is just a dict with the node names and values all
pointing to the same object. For different payloads, we will encode
the body multiple times, but hopefully the bodies will be smaller.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This will be used in a later patch for parameter pre-processing. For
now we just add the parameter as 'None' and add a consistency check
for the parameter definitions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Serializing to JSON using “simplejson” is significantly slower when
indentation and/or sorting of dictionary keys is used. In simplejson 1.x
the difference isn't that big, but with simplejson 2.x the difference
can be up to a factor of 7.5. The reason is that the latter no longer
uses C functions when sorting or indentation is used.

With this patch we revert everything to simplejson's defaults, which
should provide us with the best performance available.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

… instead of calling it with the whole results dictionary. This
fixes an issue when replacing disks (and all other cases where
result processors are used).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This avoids having to override the function in the RPC runner.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This way the generated code no longer contains arbitrary code.
Post-processing functions are used by reference.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Until now “autotools/build-rpc” would read the definition of all RPCs
and write them to a new file, “lib/_generated_rpc.py” with some
modifications. With this patch the generated code loads the definitions
and, at module load time, gets references to the actual definitions.
Further patches will even remove the calls to the post-processing
functions from the generated code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Until now it would just call itself, eventually failing.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

With this patch all RPC calls at runtime of masterd will show up in the
lock monitor. There is a chicken-and-egg issue with initializing the
configuration with a context since the lock manager, containing the
monitor, requires the configuration. This is worked around by setting
the config's context only once the lock monitor is available.

Example:
rpc/node19.example.com/write_ssconf_files  Jq9/Job32/N_SET_PARAMjobqueue_update  Jq2/Job27/C_VERIFY_CONFIG

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of in-place code now each argument has an (optional) kind
assigned. The RPC client will then take care of encoding each argument
according to its definition. A new base (_RpcClientBase) is added to
take care of encoding and dispatching calls.

Future patches will also use these definitions in the node daemon.

A nice side-effect of this change is the ability to use stand-alone
functions, as opposed to class member methods, for most of the encoding.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This simplifies the RPC argument encoding code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Add the instance_get_migration_status RPC definition and the needed
helper.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

These are no longer needed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

These two calls, “upload_file” and “write_ssconf_files” are treated
separated as they're used by the configuration, where we can't use the
normal resolver.

There's still some duplicated code in rpc.py, but that will be taken
care of in future patches.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Instead of filling the parameters in the RPC layer, that is now done
before the wrapper is called, thereby simplifying the wrapper.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Interface changes were necessary as these took more parameters than were
actually passed over the wire. Those parameters were just passed to the
instance serialization function.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

With this patch, only 6 RPC are left as old-style code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

With these changes job queue RPC will finally show up on the lock
monitor. See below for an example. A job queue-specific class is used to
restrict the use of a static list for name resolution to the job queue.
Further improvements can be made to not re-create the whole RPC client
for every call (e.g. by using a more dynamic resolver), but for now this
works.

rpc/node8.example.com/jobqueue_update Jq8/Job9/TEST_DELAY

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>