Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, the Snapshot() function of LogicalVolume returns only the
logical volume path, with the assumption that we only have one VG. But
with the recent changes, it makes more sense to return the full data (vg
and lv) from it, so as to not require computing it in the master.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch removes duplicate code found in backend which also needs to
get VG infos. To make it simpler we moved to bdev.LogicalVolume.GetVGInfo.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Even for snapshot we looked at the biggest free pv space even though
the vg might have fit the snapshot we aborted if one of the pvs was too
small. This patch fixes this by looking at the vg size instead of the pv
size.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch unifies the netutils functions dealing with IP addresses to
three classes:
- IPAddress: Common IP address functionality
- IPv4Address: IPv4 specific functionality
- IPv6address: IPv6-specific functionality

Furthermore it adds methods to check whether an address is a loopback
address, replacing the .startswith("127") for IPv4 and adding IPv6
support.

It also provides the basis for future IPv6 address handling. Methods to
convert IP strings to their corresponding interger values will allow to
canonicalize IPv6 addresses.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch fixes a bug reported in [0]. Newer drbd versions can have
another digit beyond the regular major, minor and point release digits.
We modify the regex used to match that with an optional parts which is
not saved.

Furthermore it adds unittests that test for these different cases. Now
the data read from /proc is passed into the _GetVersion method, which
makes testing easier.

[0] http://code.google.com/p/ganeti/issues/detail?id=110



Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch moves network utility functions to a dedicated module.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Support IPv6 configuration for 'drbdsetup show' parser and add unittests
concerning that case. Renames some data files to use consistent names
clarifying their usage.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This makes it compatible with the python builtin, and we can even use
the builtin when running under the right version of python. The all and
any functions are renamed to _all and _any, so that they can be tested,
and (non)existing unittests are updated (translation: there are no unit
tests, so none are updated).

This patch also moves the code deciding which partial to use next to
the definition of the _partial function, rather than after TryToRoman.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

'format' is a new built-in function, and 'bytes' is a new builtin type.
We rename this to make pylint happy (and remove potential bugs).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

utils.py, where they were before, is already huge.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Rather than checking that the file doesn't exist, and then creating it,
we create it with O_CREAT | O_EXCL, making sure the checking/creation is
atomic.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

As described in issue 93, just saying ':' is not a valid char can be
confusing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently we don't enforce name validation for the LVM backend, on the
idea that LVM itself will reject invalid names and we catch those
errors.

However, recent LVM documents the accepted VG/LV name space, so it's
easy to add this in the LVM backend code.

In addition, the patch changes some hardcoded /dev/ constructions with
utils.PathJoin().

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rationale: the file-based storage backend can add/remove files under a
certain directory. However, the master node is also controlling the
setting of the file-based root directory, so basically it means we can't
prevent arbitrary modifications by the master of the node's filesystem.

In order to mitigate this for setups where the file-based storage is not
used, we introduce a new setting at ./configure time, that controls the
enable/disable of file-based storage. Since this is not modifiable by
the master (over RPC), it is now possible in this case to prevent
unintended modifications of the node's filesystem from the master.

The new setting is used in bdev.py to not expose the file-based storage
at all, and in cmdlib.py to prevent attempts at creation of such
instances.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

These are both cleanups and, in the case of _MassageProcData, switching
from a weaker RE to a stronger one (we now need cs: in the line,
previosuly any line starting with \d+: was accepted).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Ideally we want to/will have per-device DRBD controls of disk/metadata
flushes. In the meantime, we want at least a disable of the barrier
functionality for cases where one has battery-backed caches.

Background: DRBD has four mechanism of handling ordered disk-writes.
From the drbdsetup man-page, these are: barrier, flush, drain and none.
DRBD prior to 8.2 only has drain and none. This patch makes all 8.x
versions of DRBD disable all methods, and revert to none, in case one
fully trusts batteries (either UPS for the whole system or battery for
NVRAM).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

int()/float() can raise either ValueError (in case of int("a")), or
TypeError (in case of int(None)). We had many bugs over time due to
this, and a recent one was just diagnosed, so we go over the codebase
and replace all 'except ValueError' with 'except (TypeError,
ValueError)' that protect such conversions (there were no 'except
TypeError' cases that needed a ValueError added).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A piece of old code in bdev.py uses a for loop over a single variable
because we can 'break' out of the loop or exit on the 'else' path. This
is not a nice usage of the for loop, it should be converted to a
standard if...elif...else structure.

In the meantime we silence a warning from pylint (it is actually
invalid, IMHO) and add a TODO.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

This patch adds explicit errors (instead of notimplemented) in
FileStorage (and the associated TODOs).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Note there are some cases left which need extra cleanup.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

This patch adds targeted pylint disables, where it makes sense (either
due to limitations in pylint or due to historical usage), and also a few
blanket ones in rapi where all the names are… “different”.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

The DRBD driver can ignore dead disks but not dead meta devices (for
which it refuses to configure the minor). To handle this case, we check
that the meta device is readable and if not we ignore it (the same as
when backend._RecursiveAssembleBD didn't find it).

A needed change is the move of some checks and initialisers before this
test (which is before the super().__init__ call), but that should be
fine.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A newer version of pylint, more warnings…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

We will need to enumerate selectively the PVs of (possible) many VGs and
not only the allocatable ones. For this we make the VG selection and the
allocatable filtering optional. The two callers are modified for this
new calling syntax.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

While ':' is not actually a supporte character in PV names (it has a
special meaning for commands like lvcreate), we should throw specific
errors for this case instead of generic “Can't create LV”.

This patch does two things:

- modifies the separator used when listing PVs to be '|' such that we
  can actually parse ':' as part of PV names
- check if any of the discovered PVs have ':' in their name when
  creating LVs, and if so throw a specific error

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Also replaces a hardcoded limit of 15 seconds with 1/4
of NET_RECONFIG_TIMEOUT.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This survived QA, burnin and unittests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Luca Bigliardi <shammash@google.com>

In case there are gaps in /proc/drbd related to the minors sequence,
this can lead to empty lines (see the attached data file). In this case,
we currently fail to parse the file correctly.

The patch fixes this by skipping empty lines in
bdev.BaseDRBD._MassageProcData and adds a data file and unittest for
this condition.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This allows us to report “uncertain” states (LDS_UNKNOWN) for cases
where the code can't easily detect or report what's wrong with a
block device.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch changes the return type for backend.BlockdevFind to an object
(objects.BlockDevStatus). Before a tuple was used. Adding more values to
this tuple causes a lot of work.  Converting the result to an object with
properties will make this a bit simpler.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds a GetActualSize for block devices that returns the
actual disk size. It is done using blockdev (and stat for file storage).

While this could be done via reading /sys/block/N/size, that is not as
simple as running blockdev, as the correspondence between an LV and its
sys entry is not straightforward.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch changes the DRBD8 class (the only one to use the size in
Assemble) to ignore the size in Assemble when a zero size is passed.
This will allow activation of disks even when the size recorded in the
configuration is wrong.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

LVM has issues when growing stripped volumes, so it's best to specify
the growth in exact multiples of the full stripe size (as precise as
possible). For this we need to do a couple of changes:
  - in LVM Attach(), we query additionally the VG extent size and the LV
    stripe count; since this makes lvs return a (possibly) multi-line
    output, we now split it into lines and only take the last one
  - in LVM Grow(), we round up the increase in multiples of the full
    stripe size

The patch also sets the correct target size in DRBD growth.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

With the change to stripped LVs, the actual size of a meta device (which
is small) can be more than we expected (for non-stripped LVs). This
patch increases from 160MB to 1GB the accepted size, and updates the
comment with the rationale behind this change.

Note that we do want even meta devices stripped, since it can increase
metadata update.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

This patch enables stripped LVs, falling back to non-stripped if the
stripped creation fails. If the configure-time lvm-stripecount is 1,
this patch becomes a noop (with an insignificant python-level overhead,
but no extra lvm calls).

The effect of this patch is that new instances will get stripped LVs
from the start, whereas old instances will have their LVs stripped as
soon as replace-disks is run for them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds constants for the connection status, peer roles and disk
status, and it changes the rules for when the disk is considered as
“resyncing” - previously it was only for syncsource/synctarget, but
there are many other transient statuses which could be misinterpreted as
‘degraded’ (because they where not considered as resyncing, but the disk
is not consistent in these statuses).

Furthermore, cmdlib.py:WaitForSync determines if a device is syncing or
not based on sync_percent being not none. Not all DRBD resync statuses
offer a percent done, so if we are syncing but don't have a sync
percent, we'll report a zero sync percent (and no time estimate).

The patch also removes a few unused variables (is_sync_target,
peer_sync_target, is_resync) whose value doesn't make sense anymore with
the new sync rules.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>