This is an ugly patch, sorry. It adds the following features to
rpc-test, to help with (stress) testing the Haskell RPC client:

- customisable repeat count for the RPCs
- customisable parallelisation factor
- options to show timing stats and other information
- capability to execute any RPCs already defined, as opposed to having
  the RPC hardcoded; this requires a data file (defaulting to
  "rpc.json") that has the serialised RPC call

The polymorphism/way the RPC library is implemented requires us to
have some duplicate code (operating on different types); I wasn't able
to found a way to make this more abstract.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a simple query as it has only two fields, however it's the
first query that doesn't have a clear 'base' object and 1:1
correspondence between such objects and the results (groups, nodes and
networks do so).

We keep nodes as the 'base' object, since that's what we want to
filter on for RPC selection, and we have a very simple type as the
runtime type, since we only have one other field. The
'collectLiveData' function is the one that does the expansion from
[node] to [(node, path)], with the help of 'rpcExtractor'.

Also, this patch introduces a slightly different naming structure for
the exported names out of the module, to reduce naming conflicts.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

When the RPC to a node has failed (for whatever reason), we currently
return _FS_UNAVAIL, which means "this node doesn't support
exports". However, that is misleading: all nodes support exports, but
we failed to list them at the moment, so _FS_NODATA (temporary
failure) is more appropriate here.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is straightforward, as the call has no parameters and a very simple return type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, an empty objects will generate warnings as the arguments of
various functions are unused. By adding conditional code for this, we
can support generation of empty objects, e.g. like needed in Rpc code.

Additionally, the patch also converts RpcCallVersion to THH, now that
it can build it. We change the serialisation for this (from JSNull to
JSObject []), but this shouldn't matter as this is not used in
production.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is an autogenerated file, so it has to go into gitignore, thanks
Michael!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, the choice between masterd and conf/queryd is made
statically, based on whether a query is converted or not and whether
split queries are enabled. This makes it hard to test and develop
these new queries, so this patch adds an override for the socket.

The choice for going with two hardcoded strings and arbitrary socket
fallback is done so that it's easy to use this in virtual clusters as
well.

Additionally, the patch improves the error message on failed connects
when we use arbitrary sockets; before, it always "blamed" confd for
non-master sockets.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

- Document fields in “rapi/users” file in the order they appear
  (username, password, options)
- Mention new “--require-authentication” option
- Clarify use of “write” on “GET” method

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Other parts of the documentation directly refer to the constants through
“pyeval”. There's no need for this map as it's just additional work to
maintain.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Most RAPI resources do not require authentication for the “GET” method.
In some setups it can be desirable to always require authentication.
This patch adds a command line parameter to always require it.

Some unrelated minor typos in the “ganeti-rapi” man page are also fixed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The previous patch was incomplete when erroneously pushed.

This patch, originally intended as part of the previous one, fixes it.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Update the monitoring agent design document adding the format of the HTTP
queries that will have to be used to query the monitoring agent.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This will remove the tests for the getNetworkUuid method.
The method will be obsolete after the changes regarding
identification of networks by UUID and not name are merged
into master. Will send another patch for removing the
actual method.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This replaces the very-basic parMap of IO actions (fully serialised,
as parMap won't work here), to the multi interface.

This makes a simple "time gnt-node list" on a 6-node cluster go from
3.2s to ~0.9s, and allows even better parallelisation - before,
curlGetString was blocking, whereas the new interface does allow some
interleaving.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

As we want to be able to run queries against multiple nodes in
parallel, and furthermore in parallel with other work, we need to
implement the Curl Multi interface (see libcurl-multi(3)).

This patch adds a Ganeti-specific such implementation, to be used
until upstream Curl bindings provide it. The implemented interface
(there are two 'multi' interfaces) is the older curl_multi_perform(3).

It has one downside (which is also, somewhat, and advantage): we use
polling instead of more properly select() or poll. This is due to how
waiting for FDs is implemented in Haskell: currently, it's not
possible to wait for multiple FDs at once nicely, so we'd have to fork
many threads for each FD to be watched, or alternatively one could use
FFI select, but that would block the entire runtime.

With the current poll method, the implementation achieves consistent
~100 RPC/s per second (with 10 multi interfaces running in parallel,
each with 10 easy handles), and over ~1 hour of runtime the memory
usage is stable, so memory allocation/deallocation (manual when
dealing with FFI) _seems_ to be well handled.

Future optimisations could be to move to curl_multi_socket_action(3),
which might allow better integration with the Haskell runtime.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

These two functions permit operating in bulk on only the Left or Right
values in the original list, then reassembling the list back in the
original order.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This patch removes the node from the RPC error constructurs
CurlLayerError and OfflineNodeError. The rationale is that we anyway
return tuples (node, result), and removing this duplication allows
simplified signatures/calls in the execution of RPC calls.

Note: this was sent before as well, resending on top of current codebase.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Some curl option are request-specific, but not node specific: e.g. rpc
timeout, etc. The patch changes the HttpClientRequest type so that we
can pre-seed such options, instead of rebuilding the list in each
individual request execution.

Note: this was sent before and LGTMed, but on a different codebase, so
resending an updated version.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, we support curl being optional via some sporting exercises:
ifdefs in the code, data types that represent 'Curl is disabled'
state, etc. However, with the future work on RPC, we would have to
even make the dependencies list conditional on it, etc. This is too
much work, when the curl library is included even in stable
distributions.

This patch changes curl from option to required, the same as the other
base libraries.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Use a sort key function instead of using a comparing function (“cmp=…”).
The latter is not supported in Python 3 and using a sort key function is
easier.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Having a readable representation makes debugging easier.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This patch adds unit tests for success and failure of the “GetNodeInfo”
method for the Xen hypervisor abstractions.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Also make “used” a property.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

* devel-2.7:
  NEWS: Fix the fix in commit 82b6f9ac
  NEWS: Fix release date for 2.7.0 beta1
  Minor fixes regarding nic.network change
  Fix issue 378
  NEWS: Mention multi-alloc and fix typo
  Add cfgupgrade for changing nic.network to uuid
  Remove useless code in backend for network hooks
  Show network name and not uuid in instance info
  Add IQ_NETWORKS in query tests
  Implement network locking in Instance queries
  Changes in query to support nic.network as uuid
  Modify query LUs to supoprt nic.network as uuid
  Add GetInstanceNetworks() config method
  cmdlib changes to support nic.network as uuid
  Make network config methods take uuid as argument
  Document possible hypervisor bug in serial_console
  Update man pages wrt ExtStorage

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

February 2, 2013 was a Wednesday, not a Thursday.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Commit b6ad806f used a wrong date.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Sometimes it can be difficult to determine the access permissions needed
for a certain RAPI resource without looking at code. This table, added
at the end of “rapi.rst”, shows all resources and the permissions needed
for their methods.

Another nice side-effect of this change is that there's an automatic
cross-checking between implemented resources and methods and the
documentation.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Until now the RAPI resource table returned by
“rapi.connector.GetHandlers” always uses fixed strings or compiled
regular expressions as dictionary keys. Now that a table of all RAPI
resources (and their access rights) should be written for the
documentation, a plain string version of the URIs must be available. The
compiled pattern might contain escape sequences and is not suitable.

This patch changes “GetHandlers” to accept a hook function which is used
to translate non-static resource URIs such as “/2/jobs/[job_id]/wait”.
The default implemention retains the original behaviour by translating
to compiled regular expressions.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

- Specify that there will be options for selecting nodes by at least
  nodegroups and tags, rather than just individually.
- Specify a better handling for non-redundant instances (eg. plain or
  file) which today are simply ignored
- Specify that the rolling maintenance behavior is triggered by
  instances being up, but also overridable
- Remove execution of rolling maintenances altogether, as it is deemed
  unsafe in the current version, and move it to future work, discuss the
  requirements that were pointed out for it to be safe.

Cosmetic:
- Fix numbered list, which were rendered incorrectly in the HTML version

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Make LookupNetwork() return None in case target is None. This fixes
Issue 380. Rapi passes network=None and the lookup should not fail.

Make network client aware of new nic.network.gnt-network info shows
the IPs of each instance inside the network.  It parses nic.networks
field of QueryInstances() output, which now is a list of uuids an
not names.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>

"_OPCODE_ATTRS" will also be used in the Sphinx extension.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The RAPI documentation now depends on “connector.py” (handler
registration) and “rlib2.py” (access permissions).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A newly added table of RAPI access permissions will link to all
resources and their supported methods. For this reason link targets need
to be added.

The GET method on “/2/nodes/[node_name]” was not separately mentioned.
This is also fixed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In case a NIC is not inside a network then netinfo None. Thus
netinfo["name"] fails.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>

Merge similar code into a helper function.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Use a module-level constant for the tab width.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Assert that the documented fields are equal to those in the actual
object.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add support to the Reboot command for specifying the reason for the last
status change.

Some features are implemented as functions, even if used only once, because
they will be used by the future patches introducing reason support for all
the others commands able to alter the state of instances.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch introduces some infrastructural modifications that will be used by
the following commits to implement the support for specifying the reason for
the last status change of an instance.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>