This patch implements recreation of instance disk symlinks when the
activate-disks operation is run. Until now, it was not possible to
re-create these symlinks without stopping and starting or migrating an
instance as the RPC call where this is done was in instance startup
and migration.

In order to do this, the blockdev_assemble rpc call needs the disk
index too, which is added to the protocol. This is a change from 2.3
and makes instance startup incompatible (FYI).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Recently, we've seen more and more cases of a specific breakage
pattern in Ganeti: master candidates which are semi-alive (as in, they
respond to ping, they can complete a TCP/SSL handshake, but otherwise
the root filesystem is broken) cause lots of confusion within masterd.

My analysis shows that waiting up to 5 minutes for a reply from such a
broken master candidate is too long, and this long wait breaks other
timeouts (e.g. the Luxi timeout), making standard recovery from this
situation very hard. It's much easier to kill the master daemon, edit
manually the config file and mark the node as regular, then restart
the master daemon.

The proposal is therefore to reduce the timeout for the job queue
functions to TMO_URGENT (1 minute), which should be more balanced
between a working but overloaded node and a broken node.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

data ≫ code, eom.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Tests have shown that the changes in commit b8d26c6e don't work as
wanted. If any disk wasn't found on the node, all disks located on the
same node would show as faulty. The cause was incorrect exception
handling on the node.

This patch changes the RPC call to return a per-disk success/error
status, avoiding the problem.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Luca Bigliardi <shammash@google.com>

This will show a warning if, for example, one side of a DRBD
disk becomes unavailable. The data is collected separately
from the other verification data.

Example output:

* Verifying instance status
 - ERROR: instance inst1: disk/0 on node2 is faulty

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Aliasing the “threading” module allows us to avoid the “No information
available for ganeti.rpc._RpcThreadLocal's base threading.local” warning
by epydoc.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This allows OS installation scripts to make use of special parameters,
e.g. to retain some data on reinstallation.

The RAPI resource is not updated as it takes all parameters via the
query string and encoding arbitrary data in a query string is tricky.
The resource will need to be changed to use the POST body instead.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch solves the very slow (~8-9 seconds) gnt-instance modify
behaviour. Well, it solves in general the slow RPC behaviour, but it was
most visible in that LU.

It seems that curl's behaviour with regard to file uploads (via PUT) and
the 'Expect' header are interacting badly with our http server.

First, our http server doesn't properly handle this header. According to
RFC 2616:

  Requirements for HTTP/1.1 origin servers: Upon receiving a request
  which includes an Expect request-header field with the "100-continue"
  expectation, an origin server MUST either respond with 100 (Continue)
  status and continue to read from the input stream, or respond with a
  final status code.

Our server doesn't do this, and hence it triggers this behaviour in curl
(from the curl FAQ):

  4.16 My HTTP POST or PUT requests are slow!

  libcurl makes all POST and PUT requests (except for POST requests with a
  very tiny request body) use the "Expect: 100-continue" header. This header
  allows the server to deny the operation early so that libcurl can bail out
  already before having to send any data. This is useful in authentication
  cases and others.

  However, many servers don't implement the Expect: stuff properly and if the
  server doesn't respond (positively) within 1 second libcurl will continue
  and send off the data anyway.

  You can disable libcurl's use of the Expect: header the same way you disable
  any header, using -H / CURLOPT_HTTPHEADER, or by forcing it to use HTTP 1.0.

This behaviour was detected by watching the captured traffic (in non-SSL
mode), where between the initial HTTP headers (ending with the Expect
one), there was a ~1-2 second pause until curl was sending the body.
Properly RTFM-ing would have saved ~1 day of digging around, but hey…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch enables IPv6 name resolution by using socket.getaddrinfo
instead of socket.gethostbyname_ex.

It renames the HostInfo class to Hostname and unifies its use throughout
the code. This is achieved by using static calls where no object is
needed and removes some obsolete code.

For now, we just resolve to IPv4 addresses, but this will change once it
is needed.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In light of the upcoming IPv6 support, this patch enables the rpc.Client
to always use a node's address to connect to it. This is necessary as we
do not want to rely on name resolution to connect to the correct IP
address on a dual-stack machine.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is needed because not just the cli scripts need this decorator, but
the master daemon too (and it already duplicated the code once).

In cli.py we just leave a stub, so that we don't have to modify all the
scripts to import rpc.py.

We then change the master daemon code to reuse this decorator, instead
of duplicating it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Instead of using our custom HTTP client, using PycURL's multi
interface allows us to get rid of the HTTP client threadpool.
The majority of the code is still in the ganeti.http.client
module.

A simple per-thread HTTP client pool gives cURL a chance to
cache and retain as much information as possible (e.g. SSL certs).
Unused HTTP clients (e.g. due to removed nodes) are deleted after
25 requests going through the pool.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch moves network utility functions to a dedicated module.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The patch also modifies the instance RPC calls to fill the osparameters
correctly with the cluster defaults, and exports the OS parameters in
the instance/OS environment.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

While we only support the 'parameters' check today, the RPC call is
generic enough that will be able to support other checks in the future.
The backend function will both validate the parameters list (so as to
make sure we don't pass in extra parameters that the OS validation
doesn't care about) and the parameter values, via the OS verify script.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A typo in the Raise() method of rpc.RpcResult means that any remote
errors will lack an appropriate error code; this will confuse e.g. RAPI
users.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This call was introduced but never used. In two years.
Since it's just creating/removing a file it can also be in simpler ways,
without a special rpc call, if/when we need it again. In the meantime,
let's give it to history.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch adds a table with per-opcode timeouts. They were chosen in an
empiric, rather than scientific, way - see the comments in lib/rpc.py.

The patch also shows how custom timeouts can be used - call_test_delay
explicitly overrides the timeout with one computed from the delay
parameters.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, the read timeout is hardcoded in the
HttpClientRequestExecutor class. The patch changes the timeout so that
it's a per-request property, and makes the rpc.Client class pass one
explicitly in. Furthermore, we modify the rpc.RpcRunner class to support
per-call explicit timeouts.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The X509 key name and CA are passed from cmdlib all the way to
the backend import/export daemon. With the addition of an option
to choose the compression method, another parameter would have
to be passed all the way. By moving these options to a separate
object, adding new ones will become much easier.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Both of these functions, “snapshot_export” and “instance_os_import”,
have been replaced by the instance import/export daemon.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The X509 certificate RPC functions were also renamed, but I
forgot to rename them in rpc.py. Fortunately they aren't used
yet.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be used to stop the daemon without doing complete cleanup (yet).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

These RPC calls can be used to start, monitor and stop the instance data
import/export daemon.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Certificates and keys generated using these functions will be used for
inter-cluster instance moves. As per design, the private key should never
leave the node.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

To be consistent with RAPI_CERT_FILE, the rather generic named
“SSL_CERT_FILE” constant is renamed to “NODED_CERT_FILE”. The actual file
name is not changed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This doesn't implement the full functionality, we need to add the debug
level to the opcodes too, but at least won't require changing the RPC
calls during the 2.1 series.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch should have only:

- pylint disables
- docstring changes
- whitespace changes

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

This patch adds targeted pylint disables, where it makes sense (either
due to limitations in pylint or due to historical usage), and also a few
blanket ones in rapi where all the names are… “different”.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

By checking for this explicitly, the errors (SSLEAY_RAND_BYTES, “PRNG
not seeded”) will happen in the start-up phase of the daemon and not
only when executing remote procedure calls.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>