Most RAPI resources do not require authentication for the “GET” method.
In some setups it can be desirable to always require authentication.
This patch adds a command line parameter to always require it.

Some unrelated minor typos in the “ganeti-rapi” man page are also fixed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The previous patch was incomplete when erroneously pushed.

This patch, originally intended as part of the previous one, fixes it.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Update the monitoring agent design document adding the format of the HTTP
queries that will have to be used to query the monitoring agent.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This will remove the tests for the getNetworkUuid method.
The method will be obsolete after the changes regarding
identification of networks by UUID and not name are merged
into master. Will send another patch for removing the
actual method.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This replaces the very-basic parMap of IO actions (fully serialised,
as parMap won't work here), to the multi interface.

This makes a simple "time gnt-node list" on a 6-node cluster go from
3.2s to ~0.9s, and allows even better parallelisation - before,
curlGetString was blocking, whereas the new interface does allow some
interleaving.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

As we want to be able to run queries against multiple nodes in
parallel, and furthermore in parallel with other work, we need to
implement the Curl Multi interface (see libcurl-multi(3)).

This patch adds a Ganeti-specific such implementation, to be used
until upstream Curl bindings provide it. The implemented interface
(there are two 'multi' interfaces) is the older curl_multi_perform(3).

It has one downside (which is also, somewhat, and advantage): we use
polling instead of more properly select() or poll. This is due to how
waiting for FDs is implemented in Haskell: currently, it's not
possible to wait for multiple FDs at once nicely, so we'd have to fork
many threads for each FD to be watched, or alternatively one could use
FFI select, but that would block the entire runtime.

With the current poll method, the implementation achieves consistent
~100 RPC/s per second (with 10 multi interfaces running in parallel,
each with 10 easy handles), and over ~1 hour of runtime the memory
usage is stable, so memory allocation/deallocation (manual when
dealing with FFI) _seems_ to be well handled.

Future optimisations could be to move to curl_multi_socket_action(3),
which might allow better integration with the Haskell runtime.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

These two functions permit operating in bulk on only the Left or Right
values in the original list, then reassembling the list back in the
original order.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This patch removes the node from the RPC error constructurs
CurlLayerError and OfflineNodeError. The rationale is that we anyway
return tuples (node, result), and removing this duplication allows
simplified signatures/calls in the execution of RPC calls.

Note: this was sent before as well, resending on top of current codebase.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Some curl option are request-specific, but not node specific: e.g. rpc
timeout, etc. The patch changes the HttpClientRequest type so that we
can pre-seed such options, instead of rebuilding the list in each
individual request execution.

Note: this was sent before and LGTMed, but on a different codebase, so
resending an updated version.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Currently, we support curl being optional via some sporting exercises:
ifdefs in the code, data types that represent 'Curl is disabled'
state, etc. However, with the future work on RPC, we would have to
even make the dependencies list conditional on it, etc. This is too
much work, when the curl library is included even in stable
distributions.

This patch changes curl from option to required, the same as the other
base libraries.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Use a sort key function instead of using a comparing function (“cmp=…”).
The latter is not supported in Python 3 and using a sort key function is
easier.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Having a readable representation makes debugging easier.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This patch adds unit tests for success and failure of the “GetNodeInfo”
method for the Xen hypervisor abstractions.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Also make “used” a property.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

* devel-2.7:
  NEWS: Fix the fix in commit 82b6f9ac
  NEWS: Fix release date for 2.7.0 beta1
  Minor fixes regarding nic.network change
  Fix issue 378
  NEWS: Mention multi-alloc and fix typo
  Add cfgupgrade for changing nic.network to uuid
  Remove useless code in backend for network hooks
  Show network name and not uuid in instance info
  Add IQ_NETWORKS in query tests
  Implement network locking in Instance queries
  Changes in query to support nic.network as uuid
  Modify query LUs to supoprt nic.network as uuid
  Add GetInstanceNetworks() config method
  cmdlib changes to support nic.network as uuid
  Make network config methods take uuid as argument
  Document possible hypervisor bug in serial_console
  Update man pages wrt ExtStorage

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

February 2, 2013 was a Wednesday, not a Thursday.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Commit b6ad806f used a wrong date.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Sometimes it can be difficult to determine the access permissions needed
for a certain RAPI resource without looking at code. This table, added
at the end of “rapi.rst”, shows all resources and the permissions needed
for their methods.

Another nice side-effect of this change is that there's an automatic
cross-checking between implemented resources and methods and the
documentation.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Until now the RAPI resource table returned by
“rapi.connector.GetHandlers” always uses fixed strings or compiled
regular expressions as dictionary keys. Now that a table of all RAPI
resources (and their access rights) should be written for the
documentation, a plain string version of the URIs must be available. The
compiled pattern might contain escape sequences and is not suitable.

This patch changes “GetHandlers” to accept a hook function which is used
to translate non-static resource URIs such as “/2/jobs/[job_id]/wait”.
The default implemention retains the original behaviour by translating
to compiled regular expressions.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

- Specify that there will be options for selecting nodes by at least
  nodegroups and tags, rather than just individually.
- Specify a better handling for non-redundant instances (eg. plain or
  file) which today are simply ignored
- Specify that the rolling maintenance behavior is triggered by
  instances being up, but also overridable
- Remove execution of rolling maintenances altogether, as it is deemed
  unsafe in the current version, and move it to future work, discuss the
  requirements that were pointed out for it to be safe.

Cosmetic:
- Fix numbered list, which were rendered incorrectly in the HTML version

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Make LookupNetwork() return None in case target is None. This fixes
Issue 380. Rapi passes network=None and the lookup should not fail.

Make network client aware of new nic.network.gnt-network info shows
the IPs of each instance inside the network.  It parses nic.networks
field of QueryInstances() output, which now is a list of uuids an
not names.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>

"_OPCODE_ATTRS" will also be used in the Sphinx extension.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The RAPI documentation now depends on “connector.py” (handler
registration) and “rlib2.py” (access permissions).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A newly added table of RAPI access permissions will link to all
resources and their supported methods. For this reason link targets need
to be added.

The GET method on “/2/nodes/[node_name]” was not separately mentioned.
This is also fixed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In case a NIC is not inside a network then netinfo None. Thus
netinfo["name"] fails.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>

Merge similar code into a helper function.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Use a module-level constant for the tab width.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Assert that the documented fields are equal to those in the actual
object.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add support to the Reboot command for specifying the reason for the last
status change.

Some features are implemented as functions, even if used only once, because
they will be used by the future patches introducing reason support for all
the others commands able to alter the state of instances.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch introduces some infrastructural modifications that will be used by
the following commits to implement the support for specifying the reason for
the last status change of an instance.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

As far as I could see, the ability to allocate multiple instances via
RAPI was not mentioned at all. A superfluous word is also removed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds a couple of unit tests for Query/Network.hs.
Note that they'll need to be adapted, once issue 362 is addressed.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Allow the Confd client to be able to connect to an arbitrary server
instead of just the real one running on a cluster.

This is meant to be used for testing.

If a data collector using the Confd client is tested with shelltest, it will
need to receive through the CLI the address and port of the server, and also the
FQDN for which information is being requested. In normal usage it will likely be
the name of the node the data collector is running on, but for testing we need a
way to explicitly specify it so that it corresponds to the one used inside the
configuration files that the test confd server is reading.

So, optConfdAddr and optConfdPort and optNode are provided by this patch, even
if not used, because they are an important part of the modifications for making
the Confd client testable.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In order to fetch precise information about the uptime of the VMs
running in Xen, we need to analyze the output of the "xm uptime" command.

This commit adds the parser to do that, and its tests.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In order to fetch precise information about the status of the VMs running in
Xen, we need to analyze the output of the "xm list --long" command.

This commit adds the parser to do that, and its tests.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add to Confd server a new request type (and its implementation) to ask for
the list of instances in a node.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This commit expands the monitoring agent design document improving the
description of the general report file format.

There can be two kinds of data collectors (performance and status collectors).

Every collector can now be part of a category of collectors (such as
storage, instance, daemon, hypervisor, etc.) that defines a minimum basic
structure that has to be common to all the collectors part of the same class.

Also, such basic structure is defined for all the currently envisioned class
of collectors.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This gets all network info from config_data and parses
all instances and their NICs and makes the substitution.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>

In backend NICs arrive with netinfo filled. If nic.network is not None
nic.netinfo is not too. Thus all the info is derived from HooksDict()
and nic.network must not be checked.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>