With this commit we introduce the External Storage Interface
to Ganeti, abbreviated: ExtStorage Interface.

The ExtStorage Interface provides Ganeti with the ability to interact
with externally connected shared storage pools, visible by all
VM-capable nodes. This means that Ganeti is able to handle VM disks
that reside inside a NAS/SAN or any distributed block storage provider.

The ExtStorage Interface provides a clear API, heavily inspired by the
gnt-os-interface API, that can be used by storage vendors or sysadmins
to write simple ExtStorage Providers (correlated to gnt-os-interface's
OS Definitions). Those Providers will glue externally attached shared
storage with Ganeti, without the need of preprovisioned block devices
on Ganeti VM-capable nodes as confined be the current `blockdev' disk
template.

To do so, we implement a new disk template called `ext' (of type
DTS_EXT_MIRROR) that passes control to externally provided scripts
(the ExtStorage Provider) for the template's basic functions:

 create / attach / detach / remove / grow

The scripts reside under ES_SEARCH_PATH (correlated to OS_SEARCH_PATH)
and only one ExtStorage Provider is supported called `ext'.

The disk's logical id is the tuple ('ext', UUID.ext.diskX), where UUID
is generated as in disk template `plain' and X is the disk's index.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>

This would have caught our previous issue, since QA otherwise doesn't
do add, remove or changes of NICs.

Signed-off-by: Iustin Pop <iustin@google.com>
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Since Ganeti 2.6 does warn about disks with wrong iv_names, it would
be better to fix them when upgrading to 2.6.

Not sure if this is too brittle and whether we want to add more
safeguards on the contents of the config/instance/disks before
touching them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

For epydoc raise statement, it already expects an exception which will
be hyperlinked; if passing an L{…} construct, it will instead break
the formatting (the @raise will not be recognised, but merged into the
previous paragraph).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

As reported by Ben Beuchler, kvm-ifup uses bash-specific constructs and
doesn't work with Ubuntu's default shell. We use bash all over the place
anyway, so changing the shebang line is the easiest solution.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Also does some abstracting of the versions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Closes: #201 - KVM_IFUP does not set bridge-MTU on tap devices
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
(cherry picked from commit a1ec8695)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Extend the kernel version check to also accept Linux 3.x as valid.

Signed-off-by: Alexander Schreiber <als@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit 1bf72492)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

So while testing my previous patch I run setup-ssh manually without
any arguments, and it did nothing (no complains, no messages,
anything). That was very surprising, so let's make it behave better if
no nodes are passed.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This works around a problem in Paramiko, but which looks like it's a
Ganeti problem:

[cluster] root@node4:~# gnt-node add --readd node3
/usr/lib/python2.6/dist-packages/Crypto/Util/randpool.py:40:
RandomPool_DeprecationWarning: This application uses RandomPool, which
is BROKEN in older releases.  See http://www.pycrypto.org/randpool-broken


  RandomPool_DeprecationWarning)

To silence this, we do an ugly thing (since Python 2.4 doesn't support
context managers): manually reset the warnings filter, which resets
any -W command line flags (hopefully noone calls this tool manually so
often).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Extend the kernel version check to also accept Linux 3.x as valid.

Signed-off-by: Alexander Schreiber <als@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In at least two cases "%s" is replaced with str(), too.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

While debugging an issue recently I had the pleasure of looking at a
long, single-line string of JSON data. This simple utility makes it
possible to format such data into a readable format.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Introduce the rbd disk template, which handles provisioning and
management of instance disks as block devices mapped to rbd volumes
on a RADOS cluster.

The rbd template is of type DTS_EXT_MIRROR, meaning rbd-based
instances can failover or migrate to any node, without
primary-secondary limitations.

Use of this template requires the rbd kernel driver and rbd tool
on all VM-capable nodes, as well as an operational RADOS cluster,
accessible by them.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add new --maxmem-size and --minmem-size options.
The old --mem-size option allows setting both at the same time.
If --minmem-size is lower than --maxmem-size execute a runtime memory
change on all instances, bringing them to their minimum memory size.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Oops, forgot to check this before initial commit, sorry!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This can be used to test live servers; currently there's not direct
way to interact with a confd server, except for burnin's builtin tests
(which were the source of this file).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add a shell script that will be the default replacement for the body of
backend.ActivateMasterIp and backend.DeactivateMasterIp.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This reverts commit b6267745.
This commit will be readded on master.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Closes: #201 - KVM_IFUP does not set bridge-MTU on tap devices
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

node readd can take a long time, it's good to have info messages to see
progress.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Andrea Spadaccini <spadaccio@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Andrea Spadaccini <spadaccio@google.com>

Use the gnt-cluster deactivate-master-ip command in cluster-merge to
disable the master IP.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
(cherry picked from commit e87e5afb)

Use the gnt-cluster deactivate-master-ip command in cluster-merge to
disable the master IP.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

If no arguments were specified the “exec_args” variable was “None”,
leading to the command being run as “… ./… None”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Added support for TLS-protected SPICE connections:

client/gnt_cluster.py, cli.py:
* added three new parameters to renew-crypto (--new-spice-certificate,
  --spice-certificate, --spice-ca-certificate) and their validation.

utils/x509.py:
* changed GenerateSelfSignedSslCert so that now also returns the
  generated key and certificate;
* added missing return value in the docstring of
  GenerateSelfSignedX509Cert.

lib/bootstrap.py:
* changed the signatures of the relevant functions and implemented
  certificates generation/writing.

tools/cfupgrade:
* changed GenerateClusterCrypto invocation to reflect the new signature;
* added SPICE certificate names.

lib/errors.py:
* added the X509CertError class.

lib/hypervisor/hv_kvm.py:
* silenced pylint warning R0915

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This wasn't possible until now.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In version 0.21, pylint unified all the disable-* (and enable-*)
directives to disable (resp. enable). This leads to a lot of
DeprecationWarning being emitted even if one uses the recommended
version of pylint (0.21.1, as stated in devnotes.rst).

This commit changes all the disable-msg directives to disable.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

- str.split("/").pop() should be os.path.basename
- str.split("\n") should be str.splitlines()

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Added a step in cluster-merge that removes the cluster IP from the
master node of the mergee clusters.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Identified using the “pep8” utility.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Had to break it as well, today! ;)

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Handle exceptions gracefully when trying to read the command's output.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Andrea Spadaccini <spadaccio@google.com>

Fixed a wrong method name in the last patch.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>