GitLab

Currently, we have one structual validation for opcode attributes: the
_OP_REQP, which checks that a given attribute is not 'None', and the
rest of the checks are done at runtime. This means our type system has
two types: None versus Not-None.

We have been hit many times by small, trivial bugs in this area, and
only a huge amount of unittest and/or hand-written checks would ensure
that we cover all possibilities. This patch attempts to redress the
needs for manual checks by introducing a micro-type system for the
validation of the opcode attributes. What we lose, from the start, are
the custom error messages (e.g. "Invalid reboot mode, choose one of …",
or "The disk index must be a positive integer"). What we gain is the
ability to express easily things as:

- this parameter must be None or an int
- this parameter must be a non-empty list
- this parameter must be either none or a list of dictionaries with keys
  from the list of valid hypervisors and the values dictionaries with
  keys strings and values either None or strings; furthermore, the list
  must be non-empty

These examples show that we have a composable (as opposed to just a few
static types) system, and that we can nest it a few times (just for
sanity; we could nest it up to stack depth).

We also gain lots of ))))))), which is not that nice :)

The current patch moves the existing _OP_REQP to the new framework, but
if accepted, a lot more validations should move to it. In the end, we
definitely should declare a type for all the opcode parameters
(eventually moving _OP_REQP directly to opcodes.py and validating in the
load/init case, and build __slots__ from it).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

For this, we needed to extend the NodeImage class with a few extra
variables, and we do a trick in the node verification where we pick the
first node that returned valid OS data as the reference node, and then
we compare all other nodes against it.

The checks added are:

- consistency of DiagnoseOS responses
- multiple paths for an OS
- inconsistent OS between a reference node and the current node
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Nothing special here, just copy/adjust the beparams code.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

While we only support the 'parameters' check today, the RPC call is
generic enough that will be able to support other checks in the future.
The backend function will both validate the parameters list (so as to
make sure we don't pass in extra parameters that the OS validation
doesn't care about) and the parameter values, via the OS verify script.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

"adopt" was missing from bd061c35

, thus breaking disk adoption.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

"adopt" was missing from bd061c35

, thus breaking disk adoption.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Many RPC calls have changed in Ganeti 2.2, hence bumping the RPC protocol
version.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Older OpenSSL versions include DES-CBC3-* ciphers when specifying the
HIGH group of ciphers. Removing potentially weak ciphers from the list
of allowed ciphers ensures only strong ciphers are considered for SSL
connections.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This “magic” value will be used to ensure that we don't accidentially
connect to the wrong daemon (e.g. due to a bug), comparable to DRBD's
per-disk secret. Just depending on the SSL certificate isn't enough
as it's always per instance and not per disk.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently each time we submit a job we check the job queue size, and the
drained file. With this change we keep these pieces of information in
memory and don't read them from the filesystem each time.

Significant changes include:
  - The drained value can only be properly set by calling the
    appropriate cluster command "gnt-cluster queue drain/undrain" and
    not by removing/creating the file in the job queue directory. Not
    that anybody would have done it in this undocumented way before.
  - We get rid of the soft limit for the job queue, which we haven't
    ever used anyway.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

With this change unknown disk and nic parameters will be refused, rather
than silently ignored, so that one can't pass them in by mistake and not
realize what went wrong.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Once we have a size for an export (in the context of the
import/export daemon), we can provide the user with a
percentage and ETA.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds a new boolean hypervisor parameter to the KVM hypervisor,
named 'use_chroot'.
If it's turned on for an instance, than KVM is started in "chroot mode":
Ganeti creates an empty directory for the instance and passes the path
of this dir to KVM via the -chroot flag.
KVM changes its root to this directory after starting up.

It also adds a "quarantine" feature for moving any unexpected files to
a separate directory for later analysis.

This has been backported from master,
commit 84c08e4e

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Introduce 2 new hypervisor options, migration_bandwidth and migration_downtime
and implement KVM migration bandwidth and downtime control.

migration_bandwidth controls KVM's maximal bandwidth during migration, in
MiB/s. Default value is 32 MiB/s, same as KVM's internal default. This option
is a global hypervisor option.

migration_downtime sets the amount of time (in ms) a KVM instance is allowed to
freeze while copying memory pages. This is useful when migrating busy guests,
as KVM's internal default of 30ms is too low for the page-copying algorithm to
converge. This is a per-instance option, with a default of 30ms, same as KVM's
internal default.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This patch adds a new boolean hypervisor parameter to the KVM hypervisor,
named 'use_chroot'.
If it's turned on for an instance, than KVM is started in "chroot mode":
Ganeti creates an empty directory for the instance and passes the path
of this dir to KVM via the -chroot flag.
KVM changes its root to this directory after starting up.

It also adds a "quarantine" feature for moving any unexpected files to
a separate directory for later analysis.
Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

For example, exports on the same node shouldn't be compressed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

To prepare a remote export, the X509 key and certificate need to be generated.
A handshake value is also returned for an easier check whether both clusters
share the same cluster domain secret.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently the EOM terminator is hardcoded on the server side, and is
customizable in the Transport object (with the default being the same as
the value found in the server), but not in the luxi client.

With this patch we move the value to constants, and remove the "fake"
customizability, which would just break client/server communication. If
we ever need to have a luxi transport with a different terminator it's
easy enough to add it back.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will only work on patched or newer (>= 2.6.34) kernels and with a
patched version of qemu-kvm.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This also updates a comment in the unittest for utils.py. We unittest
the new function for two things: correct reporting on real case (for
localhost), and correct reporting with a mocked-out TcpPing that returns
false.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

We also create a generic SECURE_DIR_MODE constant, rather than
hardcoding 0700 in the code.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This function is not used anymore, so there's no point in keeping it
around.

This reverts commit 3f71b464

, apart from
a one empty line conflict in ssconf.py
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Interpreting the backend's import/export daemon status is a bit tricky.
This utility code keeps track of multiple transfers at the same time.
Users can supply callback functions to react to events.

Timeouts are currently hardcoded. Intra-cluster instance moves will likely
require other timeouts.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

These RPC calls can be used to start, monitor and stop the instance data
import/export daemon.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This backend daemon for instance import and export will be used to
transfer instance data to other machines. It is implemented in a generic
way to support different ways of data input and output. The third-party
program “socat”, which is already used by the KVM hypervisor abstraction,
is used to connect to remote machines using SSL/TLS. After starting the
child processes in a separate process group, the import/export daemon
monitors their output and updates a status file regularily. This status
file can then be read by ganeti-noded (not in this patch).

Three I/O methods are supported: Raw disk, file and script. Each of these
can be used for import and export.

Similar to daemon-util, an incomplete set of tests written in Bash is
included.

Two future enhancements are planned:
- Run parts of the command chain as a dedicated user (privilege
  separation).
- Currently users of this daemon have to poll the status file while data
  is transferred. This is inefficient and creates unnecessary delays. By
  adding “dd” into the chain and sending it SIGUSR1 regularily, we can get
  some statistics, optimize the polling frequenc and even provide the user
  with an ETA (which isn't available with all current methods to
  import/export instance data).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently the backend parameters are not exported automatically, but
only a few directly in the '[instance]' section. Hypervisor type and
hypervisor parameters are not exported at all.

This patch creates two separate sections for the be and hv parameters,
and stores the parameters (including ones that come from the cluster
defaults, but not the hypervisor globals for example) in the export.
The import code is not changed yet.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This can be used by nodes to know which hypervisors they are supposed to
support.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Abstract the growable disk types in a ganeti constants, and only run
disk grow, from burnin, on them.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>