Until now all tests depended on simple AND/OR conditions. With the
introduction of virtual cluster support, some tests can only run if
virtual clusters are not in use (e.g. not yet supported or not
possible). Instead of introducing more logic for enabling/disabling
tests, callable values are supported instead. “ht.WithDesc” should be
used on them to continue to provide a readable description.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

If these new options are set, a virtual cluster is created and used.
Virtual clusters consist of 1..N virtual nodes on a single physical (or
virtual) node. The design of virtual clusters is described in
“doc/design-virtual-clusters.rst”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>
Reviewed-by: Michele Tartara <mtartara@google.com>

Many resources, but not all, list their supported methods. With a new
reST directive, this list is now automatically generated and also lists
the required permissions.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Accept the -G option, and if it's passed require that it matches a
nodegroup, then only output nodes belonging to that group.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

To make the graphs work even when instances live on offline nodes (eg.
because we're offlining them just to exclude them, or because they have
instance still on them) we just filter them out at the end, when we're
going to print out the result.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

We have no way to test Python 2.4 compatibility anymore, and that
version is long deprecated. From version 2.8+ require Python 2.6 or
above (but still not 3.0).

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In some situations after restarting the SSH daemon by
“prepare-node-join” it's not immediately reachable. Like with the master
and node daemons, waiting for it to become reachable instead of failing
hard can help in such situations.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Log every attempt to reach a port on another node. Don't silently return
False if a target not recognized as an IP address is given (e.g. a
hostname).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

In issue 353 Sascha Lucas reported that disks are not wiped when added
through “gnt-instance modify”. This patch adds this functionality and
updates the docstring for “_WipeDisks”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This directory is required by “cfgupgrade” and some other tools.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Moving the SSH/SCP constants around was necessary for this.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

- Commit 4a90bd4f contained a rather large number of style violations:
  - Indentation/sequence formatting
  - Wrapping of long lines
  - Docstrings whose first line was wrapped
  - A stray backslash in a docstring
- opcodes: Don't check for list or tuple. None of the other opcodes does
  it explicitely. As long as the length and the items match the value is
  accepted.
- server/noded: “if variable” doesn't test for None, but False

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Until now many resources, but not all and not consistently list their
supported methods (e.g. “Supports the following commands: ``GET`` …”).
Not only is it easy for this list to get out of date, but it would also
be nice if it listed the required access permissions.

This patch adds a new directive to insert a table listing all supported
methods and their access permissions for a resource.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The list of resources and handlers will also be used for generating
a per-resource table.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The factorized parts will be used to show a small table with methods and
required permissions for each resource.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The list of required access permissions uses the word “everyone” to mean
“no special permissions are needed”. The word “none” fits better.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Dato Simó <dato@google.com>

As it was 80 chars, and you know that bothers me :)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Rapi doesn't seem to export backup lists, so we enable the split query
only in gnt-backup.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is an ugly patch, sorry. It adds the following features to
rpc-test, to help with (stress) testing the Haskell RPC client:

- customisable repeat count for the RPCs
- customisable parallelisation factor
- options to show timing stats and other information
- capability to execute any RPCs already defined, as opposed to having
  the RPC hardcoded; this requires a data file (defaulting to
  "rpc.json") that has the serialised RPC call

The polymorphism/way the RPC library is implemented requires us to
have some duplicate code (operating on different types); I wasn't able
to found a way to make this more abstract.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a simple query as it has only two fields, however it's the
first query that doesn't have a clear 'base' object and 1:1
correspondence between such objects and the results (groups, nodes and
networks do so).

We keep nodes as the 'base' object, since that's what we want to
filter on for RPC selection, and we have a very simple type as the
runtime type, since we only have one other field. The
'collectLiveData' function is the one that does the expansion from
[node] to [(node, path)], with the help of 'rpcExtractor'.

Also, this patch introduces a slightly different naming structure for
the exported names out of the module, to reduce naming conflicts.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

When the RPC to a node has failed (for whatever reason), we currently
return _FS_UNAVAIL, which means "this node doesn't support
exports". However, that is misleading: all nodes support exports, but
we failed to list them at the moment, so _FS_NODATA (temporary
failure) is more appropriate here.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is straightforward, as the call has no parameters and a very simple return type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, an empty objects will generate warnings as the arguments of
various functions are unused. By adding conditional code for this, we
can support generation of empty objects, e.g. like needed in Rpc code.

Additionally, the patch also converts RpcCallVersion to THH, now that
it can build it. We change the serialisation for this (from JSNull to
JSObject []), but this shouldn't matter as this is not used in
production.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is an autogenerated file, so it has to go into gitignore, thanks
Michael!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, the choice between masterd and conf/queryd is made
statically, based on whether a query is converted or not and whether
split queries are enabled. This makes it hard to test and develop
these new queries, so this patch adds an override for the socket.

The choice for going with two hardcoded strings and arbitrary socket
fallback is done so that it's easy to use this in virtual clusters as
well.

Additionally, the patch improves the error message on failed connects
when we use arbitrary sockets; before, it always "blamed" confd for
non-master sockets.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

- Document fields in “rapi/users” file in the order they appear
  (username, password, options)
- Mention new “--require-authentication” option
- Clarify use of “write” on “GET” method

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Other parts of the documentation directly refer to the constants through
“pyeval”. There's no need for this map as it's just additional work to
maintain.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Most RAPI resources do not require authentication for the “GET” method.
In some setups it can be desirable to always require authentication.
This patch adds a command line parameter to always require it.

Some unrelated minor typos in the “ganeti-rapi” man page are also fixed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The previous patch was incomplete when erroneously pushed.

This patch, originally intended as part of the previous one, fixes it.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Update the monitoring agent design document adding the format of the HTTP
queries that will have to be used to query the monitoring agent.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This will remove the tests for the getNetworkUuid method.
The method will be obsolete after the changes regarding
identification of networks by UUID and not name are merged
into master. Will send another patch for removing the
actual method.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This replaces the very-basic parMap of IO actions (fully serialised,
as parMap won't work here), to the multi interface.

This makes a simple "time gnt-node list" on a 6-node cluster go from
3.2s to ~0.9s, and allows even better parallelisation - before,
curlGetString was blocking, whereas the new interface does allow some
interleaving.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

As we want to be able to run queries against multiple nodes in
parallel, and furthermore in parallel with other work, we need to
implement the Curl Multi interface (see libcurl-multi(3)).

This patch adds a Ganeti-specific such implementation, to be used
until upstream Curl bindings provide it. The implemented interface
(there are two 'multi' interfaces) is the older curl_multi_perform(3).

It has one downside (which is also, somewhat, and advantage): we use
polling instead of more properly select() or poll. This is due to how
waiting for FDs is implemented in Haskell: currently, it's not
possible to wait for multiple FDs at once nicely, so we'd have to fork
many threads for each FD to be watched, or alternatively one could use
FFI select, but that would block the entire runtime.

With the current poll method, the implementation achieves consistent
~100 RPC/s per second (with 10 multi interfaces running in parallel,
each with 10 easy handles), and over ~1 hour of runtime the memory
usage is stable, so memory allocation/deallocation (manual when
dealing with FFI) _seems_ to be well handled.

Future optimisations could be to move to curl_multi_socket_action(3),
which might allow better integration with the Haskell runtime.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

These two functions permit operating in bulk on only the Left or Right
values in the original list, then reassembling the list back in the
original order.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This patch removes the node from the RPC error constructurs
CurlLayerError and OfflineNodeError. The rationale is that we anyway
return tuples (node, result), and removing this duplication allows
simplified signatures/calls in the execution of RPC calls.

Note: this was sent before as well, resending on top of current codebase.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Some curl option are request-specific, but not node specific: e.g. rpc
timeout, etc. The patch changes the HttpClientRequest type so that we
can pre-seed such options, instead of rebuilding the list in each
individual request execution.

Note: this was sent before and LGTMed, but on a different codebase, so
resending an updated version.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, we support curl being optional via some sporting exercises:
ifdefs in the code, data types that represent 'Curl is disabled'
state, etc. However, with the future work on RPC, we would have to
even make the dependencies list conditional on it, etc. This is too
much work, when the curl library is included even in stable
distributions.

This patch changes curl from option to required, the same as the other
base libraries.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Use a sort key function instead of using a comparing function (“cmp=…”).
The latter is not supported in Python 3 and using a sort key function is
easier.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>