LVM version 2.02.93 (or at least, sometimes after .88) has extend the
lv_attr field with two more flag; we only care about the first digit,
so let's change the "!= 6" check to "< 6".

Thanks to Robin H Johnson <robbat2@gentoo.org> for finding this issue.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

In at least two cases "%s" is replaced with str(), too.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Introduce the rbd disk template, which handles provisioning and
management of instance disks as block devices mapped to rbd volumes
on a RADOS cluster.

The rbd template is of type DTS_EXT_MIRROR, meaning rbd-based
instances can failover or migrate to any node, without
primary-secondary limitations.

Use of this template requires the rbd kernel driver and rbd tool
on all VM-capable nodes, as well as an operational RADOS cluster,
accessible by them.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add the c-plan-ahead, c-fill-target, c-delay-target, c-max-rate,
c-min-rate parameters; report errors while setting the
synchronization speed.

Change the error signaling model of SetSyncParams. Instead of returning
True or False, return a list of errors (to make possible error reporting
during the recursive call).

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

constants.py, cmdlib.py:
- add the dynamic-resync parameter, both at DT and LD levels;

lib/bdev.py:
- change SetSyncSpeed to SetSyncParams, and _SetMinorSyncSpeed to
  _SetMinorSyncParams;
- use the dynamic-resync parameter.

Later patches will implement the rest of the parameters.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Those parameters can be used to pass options directly to drbdsetup disk
and drbdsetup net.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add the LDP_ prefix to the LD parameters-related constants, in order to
avoid pollution in the global constants namespace.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add the disk-barriers and meta-barriers parameters described in the
design doc.

constants.py:
* add the needed LD and DT-level parameters, use the defaults provided
  at ./configure time;
* add constants representing which barriers should be disabled and the
  set of valid options.

lib/bdev.py:
* factor the barriers handling code to a class method, for testing
  purposes;
* implement the more granular version checking logic;
* use the LD level parameters;
* add stricter check on DRBD version (8.0, 8.2 or 8.3), as we do not
  support 8.4 yet.

lib/cmdlib.py:
* translate DT level parameters to LD level ones.

configure.ac, Makefile.am:
* set both disk and meta barriers parameters depending on the value of
  --enable-drbd-barriers.

test/ganeti.bdev_unittest.py:
* unit tests for the code that sets DRBD barrier parameters depending on
  the version.

doc/design-resource-model.rst:
* reword the description of meta-barriers;
* change all disk parameters names to use dashes instead of underscores.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

configure.ac:
* change the documentation of --with-lvm-stripecount parameter to
  reflect the change

doc/design-resource-model.rst:
* change drbd/stripes to drbd/data-stripes and drbd/metastripes to
  drbd/meta-stripes

rest of files:
* add the plain/stripes, drbd/data-stripes and drbd/meta-stripes disk
  parameters

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* add the params attribute to BlockDev, and add the corresponding
  parameter to all the BlockDev classes;
* change the Create, Assemble and FindDevice factory functions interface
  to accept as parameters an objects.Disk instance and a list of
  children block devices; update their callers;
* make the factory functions provide default values for params if
  needed;
* factor out a check in the block device factory functions.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Instead of relying on clients of the class for setting the device speed
(and, in general, the DRBD parameters), move this responsibility inside
the Assemble method.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In version 0.21, pylint unified all the disable-* (and enable-*)
directives to disable (resp. enable). This leads to a lot of
DeprecationWarning being emitted even if one uses the recommended
version of pylint (0.21.1, as stated in devnotes.rst).

This commit changes all the disable-msg directives to disable.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Identified using the “pep8” utility.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In the Linux kernel commit 4b0715f096 introduced a display bug into
/proc/drbd which broke our regex.

The bug was first introduced into Linux 2.6.39-rc1. This bug is still
unfixed as of today.

This patch adapt the regular expression to workaround this bug for the
time being.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The docstring of the DRBD8 class says:

  … The meta device is checked for valid size and is zeroed on create.

which is not done today, hence we have
http://code.google.com/p/ganeti/issues/detail?id=182

:

  node1# mkreiserfs -f /dev/xenvg/t8
  …
  ReiserFS is successfully created on /dev/xenvg/t8.
  node1# drbdmeta --force /dev/drbd256 v08 /dev/xenvg/t8 0 create-md
  md_offset 0
  al_offset 4096
  bm_offset 36864

  Found reiser filesystem

  This would corrupt existing data.
  If you want me to do this, you need to zero out the first part
  of the device (destroy the content).
  You should be very sure that you mean it.
  Operation refused.

I've tested and even just 1MB is enough to wipe the meta, but let's be
safer and pass a 'clean' meta to drbd.

Note: I didn't copy _WipeDevice from backend.py since it seemed more
complex than needed here.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

s/'/"/ in (hopefully) the right places.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is always called with False from backend for now.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch introduces basic shared block storage support.

It introduces a new storage backend, bdev.PersistentBlockDevice, to
use as a backend for shared block storage. The new bdev requires a new
BLOCKDEV_DRIVER_MANUAL constant with the value "manual" and uses it as
the first part of the block device unique_id.

A new disk template, DT_BLOCK is introduced as well and added to
DTS_EXT_MIRROR and DTS_MAY_ADOPT. Also added DTS_MUST_ADOPT constant
and use it to check for the presence of the adopt keyword during LU
invocation. We enforce the /dev/disk limitation upon adoption, but we
allow block devices to reside anywhere under /dev.

This is very basic support and includes no storage manipulation (provisioning,
resizing, renaming) which will have to be implemented through a "driver"
framework.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
[iustin@google.com: slight changes to bdev.py]
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch introduces core file storage support, consisting of the following:

A configure-time switch for enabling/disabling shared file storage
support and controlling the shared file storage location:
--with-shared-file-storage-dir=.  Shared file storage configuration is then
available as _autoconf.ENABLE_SHARED_FILE_STORAGE and
_autoconf.SHARED_FILE_STORAGE_DIR and there is a cluster-wide ssconf
key named "shared_file_storage_dir" for changing the file location.

A new disk template named "sharedfile" (DT_SHARED_FILE), using
ganeti.bdev.FileStorage.

Auxiliary functions in lib/config.py to handle shared file storage.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
[iustin@google.com: small style fixes]
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, the Snapshot() function of LogicalVolume returns only the
logical volume path, with the assumption that we only have one VG. But
with the recent changes, it makes more sense to return the full data (vg
and lv) from it, so as to not require computing it in the master.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch removes duplicate code found in backend which also needs to
get VG infos. To make it simpler we moved to bdev.LogicalVolume.GetVGInfo.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Even for snapshot we looked at the biggest free pv space even though
the vg might have fit the snapshot we aborted if one of the pvs was too
small. This patch fixes this by looking at the vg size instead of the pv
size.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch unifies the netutils functions dealing with IP addresses to
three classes:
- IPAddress: Common IP address functionality
- IPv4Address: IPv4 specific functionality
- IPv6address: IPv6-specific functionality

Furthermore it adds methods to check whether an address is a loopback
address, replacing the .startswith("127") for IPv4 and adding IPv6
support.

It also provides the basis for future IPv6 address handling. Methods to
convert IP strings to their corresponding interger values will allow to
canonicalize IPv6 addresses.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch fixes a bug reported in [0]. Newer drbd versions can have
another digit beyond the regular major, minor and point release digits.
We modify the regex used to match that with an optional parts which is
not saved.

Furthermore it adds unittests that test for these different cases. Now
the data read from /proc is passed into the _GetVersion method, which
makes testing easier.

[0] http://code.google.com/p/ganeti/issues/detail?id=110



Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch moves network utility functions to a dedicated module.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Support IPv6 configuration for 'drbdsetup show' parser and add unittests
concerning that case. Renames some data files to use consistent names
clarifying their usage.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This makes it compatible with the python builtin, and we can even use
the builtin when running under the right version of python. The all and
any functions are renamed to _all and _any, so that they can be tested,
and (non)existing unittests are updated (translation: there are no unit
tests, so none are updated).

This patch also moves the code deciding which partial to use next to
the definition of the _partial function, rather than after TryToRoman.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

'format' is a new built-in function, and 'bytes' is a new builtin type.
We rename this to make pylint happy (and remove potential bugs).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

utils.py, where they were before, is already huge.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Rather than checking that the file doesn't exist, and then creating it,
we create it with O_CREAT | O_EXCL, making sure the checking/creation is
atomic.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

As described in issue 93, just saying ':' is not a valid char can be
confusing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently we don't enforce name validation for the LVM backend, on the
idea that LVM itself will reject invalid names and we catch those
errors.

However, recent LVM documents the accepted VG/LV name space, so it's
easy to add this in the LVM backend code.

In addition, the patch changes some hardcoded /dev/ constructions with
utils.PathJoin().

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rationale: the file-based storage backend can add/remove files under a
certain directory. However, the master node is also controlling the
setting of the file-based root directory, so basically it means we can't
prevent arbitrary modifications by the master of the node's filesystem.

In order to mitigate this for setups where the file-based storage is not
used, we introduce a new setting at ./configure time, that controls the
enable/disable of file-based storage. Since this is not modifiable by
the master (over RPC), it is now possible in this case to prevent
unintended modifications of the node's filesystem from the master.

The new setting is used in bdev.py to not expose the file-based storage
at all, and in cmdlib.py to prevent attempts at creation of such
instances.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

These are both cleanups and, in the case of _MassageProcData, switching
from a weaker RE to a stronger one (we now need cs: in the line,
previosuly any line starting with \d+: was accepted).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Ideally we want to/will have per-device DRBD controls of disk/metadata
flushes. In the meantime, we want at least a disable of the barrier
functionality for cases where one has battery-backed caches.

Background: DRBD has four mechanism of handling ordered disk-writes.
From the drbdsetup man-page, these are: barrier, flush, drain and none.
DRBD prior to 8.2 only has drain and none. This patch makes all 8.x
versions of DRBD disable all methods, and revert to none, in case one
fully trusts batteries (either UPS for the whole system or battery for
NVRAM).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

int()/float() can raise either ValueError (in case of int("a")), or
TypeError (in case of int(None)). We had many bugs over time due to
this, and a recent one was just diagnosed, so we go over the codebase
and replace all 'except ValueError' with 'except (TypeError,
ValueError)' that protect such conversions (there were no 'except
TypeError' cases that needed a ValueError added).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>