This patch introduces basic shared block storage support.

It introduces a new storage backend, bdev.PersistentBlockDevice, to
use as a backend for shared block storage. The new bdev requires a new
BLOCKDEV_DRIVER_MANUAL constant with the value "manual" and uses it as
the first part of the block device unique_id.

A new disk template, DT_BLOCK is introduced as well and added to
DTS_EXT_MIRROR and DTS_MAY_ADOPT. Also added DTS_MUST_ADOPT constant
and use it to check for the presence of the adopt keyword during LU
invocation. We enforce the /dev/disk limitation upon adoption, but we
allow block devices to reside anywhere under /dev.

This is very basic support and includes no storage manipulation (provisioning,
resizing, renaming) which will have to be implemented through a "driver"
framework.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
[iustin@google.com: slight changes to bdev.py]
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch introduces core file storage support, consisting of the following:

A configure-time switch for enabling/disabling shared file storage
support and controlling the shared file storage location:
--with-shared-file-storage-dir=.  Shared file storage configuration is then
available as _autoconf.ENABLE_SHARED_FILE_STORAGE and
_autoconf.SHARED_FILE_STORAGE_DIR and there is a cluster-wide ssconf
key named "shared_file_storage_dir" for changing the file location.

A new disk template named "sharedfile" (DT_SHARED_FILE), using
ganeti.bdev.FileStorage.

Auxiliary functions in lib/config.py to handle shared file storage.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
[iustin@google.com: small style fixes]
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Stephen Shirley <diamond@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

With this patch, the exporting node will retry to connect a few times.
The receiving node will make use of the master's increased timeout (see
previous patch).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Patch 03941a26 made disk growth VG-aware, but the code has a bug: it
only works for LD_LV disks, and not LD_DRBD. To fix it, we add a
generic function to compute the per-VG growth requirements, and switch
to using that instead of manual build.

Furthermore, since the function _CheckNodesFreeDiskPerVG always takes
its input from this new function or _ComputeDiskSizePerVG, we change
the latter to always return a dict (possibly empty), instead of either
a dict or None. This makes the types more consistent.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

For an instance with a single disk the message should be “… disks 0 to
0”, not “… 0 to 1”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

With this patch OpConnectConsole will no longer just return a command
with arguments, but rather a detailed description about how to connect
to an instance's console. Unittests for some parts are included. Another
patch will follow to adjust the hypervisor abstractions for the new
model.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is just a state of record field and does not necessary
reflect the reality.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This can be set at group creation time and via OpSetGroupParams. The default
is "preferred", and existing node groups from previous Ganeti version will
get the attribute set to this value.

Signed-off-by: Adeodato Simo <dato@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

… and use them in cmdlib.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Also update description of QueryFieldDefinition.name.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

The defaults are filled in from the node group defaults, not cluster
defaults (presumably copy & paste error).

Signed-off-by: Adeodato Simo <dato@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In particular:

  - add AddNodeGroup and RemoveNodeGroup in config.py

  - refactor _UpgradeConfig to make use of _UnlockedAddNodeGroup

  - add support for objects.NodeGroup in ConfigWriter.Update; add missing
    "serial_no" attribute to NodeGroup objects

  - update NodeGroup.UpgradeConfig to set mtime and serial_no if missing
    (the refactored code in _UpgradeConfig was not settting mtime nor ctime)

Signed-off-by: Adeodato Simo <dato@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This has been deprecated and unused since 2.1.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Changes to backend.GetVolumeList():
- now accepts a list of VGs instead of one VG
- returns LV names in the form "vg_name/lv_name"

Corresponding changes are done in: VerifyDisks, VerifyNode,
LUCreateInstance (for both disk creation and adoption cases)

Now the syntax
"gnt-instance add ... --disk N:adopt=LV_NAME,vg=VG_NAME"
as was described earlier in the man page works.

Signed-off-by: Dmitry Chernyak <dmi.chernyak@gmail.com>
[iustin@google.com: QA changes for reserved LVs, style fixes and a few
 extra error checks, reviewed by hansmi/rn]
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This adds the flag and some initial handling. The rest of the changes,
for cmdlib, come in a separate patch.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In the context of a node, its group has (at least today) only one
meaning, that is the node's node group. As such, we rename
node.nodegroup to just node.group.

Note: if we want to keep node in there, it should be at least
node_group, for consistency with the other node attributes.

Similarly, we rename the OpAddNode nodegroup attribute to group.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is the first step for the support of wiping block devices prior
to creation of the instance.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Per the mailing list discussion, rename _oss to _os, both in cluster parameters
and in the rest of the code.

This is just an s/_oss/_os, with the exception of a small bit of cleanup
around the helper_os function in cmdlib.py.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The new variables are:

- a list of hidden OSes, that should not be displayed to the users in
  interactive selection (e.g. reinstall); however, if they are selected, they
  can be used
- a list of OSes that should be hidden and blocked from install-time selection

The filtering will apply at pure OS name level, not OS+variant level.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the computation of the 'pure' name or the variant is
hardcoded and spread around the functions that need it. This is not
nice, and in the future we'd spread it even more with more usage of
variants/pure os names.

This patch abstracts these functions into the OS class, and then
replaces the hardcoded uses with the new functions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

If a node doesn't have a node group we'll upgrade the config making it
the cluster default. Also the node add and removal operations are
changed to set/clear the node group correctly. Finally we populate the
"members" list of nodegroups on config load with  the value from the
nodes.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Also:
  - reformat the __slots__ declaration of ConfigData
  - call UpgradeConfig on each node groups to handle future upgrades
  - add nodegroups special case in {To,From}Dict
  - add nodegroups to _AllUUIDObjects

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

When _ContainerFromDicts is called on some element which doesn't exist
in the config, because it is yet to be upgraded, it will receive its
value as None. We take care of this case by using an empty element of
the required target type.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The "members" slot of this object is not serialized, and is discarded
when deserializing, initializing it explicitly to [].

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

We expose the ip_version (4, 6) to the external interface and internally
we convert it to ip_family (AF_INET=2, AF_INET6=10). This makes the code
more concise as all functions deal with family rather than version.

This patch does not yet expose this parameter via gnt-cluster, but only uses
the constant IP4_VERSION. This will be enabled in a future patch.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This parameter, which is a list of regular expression patterns, will
make cluster verify ignore any such LVs. It will not prevent creation or
removal of such volumes by the backend code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Add a cluster parameter to hold the iallocator that will be used by default
when required and no alternative (manually-specified iallocator or
manually-specified node(s)) is given.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The comments seem to imply we can remove deprecated slots after 2.1 is
released. This is not quite true, if we still want to support upgrading
straight from 2.0. So let's specify better that they can be removed only
after upgrading from 2.0 can be deprecated.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The patch also modifies the instance RPC calls to fill the osparameters
correctly with the cluster defaults, and exports the OS parameters in
the instance/OS environment.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>