Now instance policies can contain more than one min/max specs.  This is the
main element of the "Constrained instance sizes" section in the
"Partitioned Ganeti" design doc.

This is a big patch, but changing the type of a configuration item requires
to change all the code that handles it.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This is needed to be able to validate the std spec against multiple min/max
spec pairs (appearing in next patches).

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Specs used to specify limits in instance policies are always complete, even
at group level (only the whole limit set can be overridden). This is in
preparation for introducing multiple limits.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This makes the code more modular, in preparation for the following patches.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This patch adds "uuid" and "name" slots to Disk and NIC ConfigObjects.

Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>

Since managing of different storage units is now done using
disk templates and not storage types, we remove the obsolete
enabled storage types.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michele Tartara <mtarara@google.com>

This patch adds constants for enabling disk templates to the constants,
and the cluster configuration in haskell and python. It removes the
obsolete preference list for enabled storage types.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michele Tartara <mtarara@google.com>

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Minimum and maximum instance specs are put together into a single element
of the instance policy. This is in preparation for introducing multiple
min/max specs.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This implements an online update of the configuration for when a configuration
is loaded that does not yet contain the 'enabled_storage_types' attribute.
Note that this will require more changes as dis/enabling of (shared) file
storage at configure time will be removed.
This patch includes adding a mapping from disk templates to storage types. While
I was on it, I also reordered the disk template and storage type parameters
alphabetically, to make dealing with those easier.
Also, this mentions the configuration change in the NEWS file.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds the cluster's new field 'enabled_storage_types'
to the configuration objects in python and haskell.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Tests for:
  objects.InstancePolicy.CheckParameterSyntax()
  objects.InstancePolicy.CheckDiskTemplates()
  objects.InstancePolicy.CheckISpecSyntax()

Instance policies with an empty disk-template list now are reported.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Unknown elements were silently removed on startup. This means that a
software upgrade could result in lost configuration information if
cfgupgrade wasn't run promptly.

Added unit test for Cluster.UpgradeConfig() to cover this case.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In this method self.tags might be None and cannot be used
in join(). Use GetTags instead.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Back when this was introduced, I mentioned that it breaks heavily tab
completion (ob<TAB> doesn't work anymore), but at that moment I didn't
have a suggestion what to name it. I think outils is good and short
enough, and doesn't conflict with anything else, so here it goes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This can be used in hypervisor code as well. For consistency
export *NETWORK_NAME and not *NETWORK throughout the code.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This info is not used by Ganeti and therefore is removed.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This slot was not used by Ganeti so the same info can be
provided via tags. In order not to break configuration data
we add a FromDict() method in Network config object that
removes the deprecated network_type (if found) and then invoke
the parent's method (which does the validation).

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

There's never been support for a configuration where nodes in the same node
group have different values of the exclusive_storage flag. This patch
disables the possibility to change the flag for individual nodes.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

1) Move the hooks environment dict generator inside the object. This
also adds missing values such as network family and uuid.
2) Use the same generator both for the os environment and for the
instance hooks.
3) Update manpage and hooks documentation.

Also update a forgotten docstring about a tuple's content.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Although this does nothing for now, running it is safe, and consistent
with how other objects behave.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

“objects.ConfigObject” contains two useful functions for working with
containers of serialized objects, “_ContainerToDicts” and
“_ContainerFromDicts”. This patch separates those functions and moves
them into “objectutils” as they'll be useful for converting parts of the
QA configuration to objects. Unittests are added to cover the parts not
already tested through other code paths.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

- Handle de-serialization correctly when pool is not defined
- Serialize to empty list when the attribute is None (this should never
  happen in reality as the attribute is always set when de-serializing)
- Add tests

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be used for checks related to exclusive storage.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This makes the code more readable and easier to upgrade.
bdev.LogicalVolume.GetPVInfo and the code that depends on it have been
refactored to use the new class.

utils.CheckVolumeGroupSize() has been moved to lib/utils/lvm.py, where more
functions will be added.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Look up “NIC_MODE” only once, capitalize “NIC” in error messages.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add support for passing parameters to the ext template (ext-params).
Take advantage of disk-params, that don't seem to make much sense in
this template (ExtStorage Providers are not predefined and we don't
know their needs) and use them to pass the ext-params dynamically to
the template.

ext-params are correlated with gnt-os-interface's os-params.
All ext-params are exported to the ExtStorage Provider through it's
environment, with variables prefixed with 'EXTP_' (similarly to the
OS interface's 'OSP_' params).

ext-params are passed via the --disk option. If the disk template
is of type `ext', then any additional options passed to --disk and
are not in IDISK_PARAMS are considered ext-params e.g.:

 gnt-instance add -t ext --disk=0:size=2G,param1=value1,param2=value2

Finally, we introduce a new IDISK_PARAM called IDISK_PROVIDER, that is
mandatory for template `ext' and is used to select the desired
ExtStorage Provider. This parameter is not a valid --disk option for
any other template type.

The IDISK_PROVIDER parameter becomes the first element of the
disk's unique_id tuple e.g.:

 unique_id = ('sample_provider1', 'UUID.ext.diskX')

Example selecting different ExtStorage Providers for each disk and
passing different ext-params to them:

 -t ext --disk=0:size=2G,provider=sample_provider1,param1=value1
        --disk=1:size=3G,provider=sample_provider2,param2=value2

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: small simplification in bdev code, pylint fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

With this commit we introduce the External Storage Interface
to Ganeti, abbreviated: ExtStorage Interface.

The ExtStorage Interface provides Ganeti with the ability to interact
with externally connected shared storage pools, visible by all
VM-capable nodes. This means that Ganeti is able to handle VM disks
that reside inside a NAS/SAN or any distributed block storage provider.

The ExtStorage Interface provides a clear API, heavily inspired by the
gnt-os-interface API, that can be used by storage vendors or sysadmins
to write simple ExtStorage Providers (correlated to gnt-os-interface's
OS Definitions). Those Providers will glue externally attached shared
storage with Ganeti, without the need of preprovisioned block devices
on Ganeti VM-capable nodes as confined be the current `blockdev' disk
template.

To do so, we implement a new disk template called `ext' (of type
DTS_EXT_MIRROR) that passes control to externally provided scripts
(the ExtStorage Provider) for the template's basic functions:

 create / attach / detach / remove / grow

The scripts reside under ES_SEARCH_PATH (correlated to OS_SEARCH_PATH)
and only one ExtStorage Provider is supported called `ext'.

The disk's logical id is the tuple ('ext', UUID.ext.diskX), where UUID
is generated as in disk template `plain' and X is the disk's index.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: small simplification in bdev code, pylint fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

Some fixes, added more information in a few points, removed a stale (5+
year old) TODO comment.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Iustin Pop <iustin@google.com>

Support:
 - gnt-network add --tags.
 - gnt-network list-tags/add-tags/remove-tags/.
 - gnt-network list -o +tags

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Iustin Pop <iustin@google.com>

If a nic has a network field then encapsulate a network object in
netinfo slot for every rpc. This is needed to pass network info to
scripts managing nics (kvm-vif-bridge).

Introduce _BuildNetworkEnv().

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Iustin Pop <iustin@google.com>

Config objects:
 * Introduce L{Network} with
  - IPv4 network field (mandatory)
  - IPv4 gateway, IPv6 (network/gateway), mac prefix, type (optional)
 * Modify existing config objects to support networks:
  - Add new slot 'network' to L{NIC} config object
  - Add new slot 'networks' to L{NodeGroup} config object

Opcodes:
 * Introduce new opcodes for networks
  - add/remove/modify/query/connect/disconnect.
 * In InstanceCreate/InstanceSetParams add conflicts_check option

Constants:
 * INIC_PARAM 'INIC_NETWORK'
 * NIC_IP_POOL for automaticaly obtain an IP from a pool
 * NETWORK_TYPE_PUBLIC/PRIVATE for network types

Checking of network_type handled by the opcode parameter validation.
Introduce _CheckCIDR*Notation() functions for network parameters
validation.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Iustin Pop <iustin@google.com>

objects.Disk.ComputeLDParams: Simplify structure, don't use and
overwrite local variables.

rpc.AnnotateDiskParams: Replace explicit loop.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As we will introduce another set of containers using the __slots__ trick
we abstract away as much as possible to separate bases classes. The
child classes then adapt them for their needs. This leads to less code
duplication.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:

- bump version in the docs
- silence some new checks that are wrong due to our indent=2 instead of 4
- fix lots of errors in the code where the indentation was wrong by 1
  or 2 spaces
- fix a few cases of == True, False, None and replace with 'is'
- re-indent some cases where the code is OK, but pep8 complains

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Since this is an inheritable parameter, having it as a MABYE_STRING
creates only problems (per our derivation rules). We change it to
STRING, with the default "", meaning no program. Note that most of the
code already accepts this as valid for "no program", and some comments
even say that this is the expected value.

We have some other parameters like this, I'll have to investigate
whether they need to be changed too. But right now I need this for the
hconfd changes (it's a prerequisite for them, I forgot to send it in
that patch series).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

It seems that 'explicitely' is wrong, and that the right form is
'explicitly'. This is just fixing the typo plus adjusting affected
paragraphs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

On node groups we don't have the std field. However, the InstancePolicy
object always verifies that the std value is within a given range. As we
fill it up with defaults if not set (as it happens to be on node groups)
and the min value is higher than the default std value (taken from
constants.py) we fail.

We overcome this situation by simply let the function know if we want to
verify the std value at all. If we don't want to verify std, we just set
it to a compliant value (min_v) and continue.

We also slightly adapt the error message provided, as we don't have std
values on groups.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit cce46164 fixed upgrading from other 2.6 versions, but
accidentally broke upgrading from 2.5 (disk.params was left as None,
which breaks FillDict). The simplest way to handle params is to always
set them to an empty dict (disregarding what they currently contain).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>