GitLab

Initial support for the `ext' template in gnt-instance modify.
Take care of all needed checks in CheckArguments and CheckPrereqs
wrt ExtStorage Providers and ext-params.

ExtStorage disk addition and specification of arbitrary ext-params
during addition is supported.

Currently, one cannot modify an existing disk's ExtStorage Provider,
neither can change an existing disk's ext-params. Only an existing
disk's `mode' can be modified.

Removal is transparent of the disk template and is also supported.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: fix bug in disk changes, pylint fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

Add support for passing parameters to the ext template (ext-params).
Take advantage of disk-params, that don't seem to make much sense in
this template (ExtStorage Providers are not predefined and we don't
know their needs) and use them to pass the ext-params dynamically to
the template.

ext-params are correlated with gnt-os-interface's os-params.
All ext-params are exported to the ExtStorage Provider through it's
environment, with variables prefixed with 'EXTP_' (similarly to the
OS interface's 'OSP_' params).

ext-params are passed via the --disk option. If the disk template
is of type `ext', then any additional options passed to --disk and
are not in IDISK_PARAMS are considered ext-params e.g.:

 gnt-instance add -t ext --disk=0:size=2G,param1=value1,param2=value2

Finally, we introduce a new IDISK_PARAM called IDISK_PROVIDER, that is
mandatory for template `ext' and is used to select the desired
ExtStorage Provider. This parameter is not a valid --disk option for
any other template type.

The IDISK_PROVIDER parameter becomes the first element of the
disk's unique_id tuple e.g.:

 unique_id = ('sample_provider1', 'UUID.ext.diskX')

Example selecting different ExtStorage Providers for each disk and
passing different ext-params to them:

 -t ext --disk=0:size=2G,provider=sample_provider1,param1=value1
        --disk=1:size=3G,provider=sample_provider2,param2=value2
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: small simplification in bdev code, pylint fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

With this commit we introduce the External Storage Interface
to Ganeti, abbreviated: ExtStorage Interface.

The ExtStorage Interface provides Ganeti with the ability to interact
with externally connected shared storage pools, visible by all
VM-capable nodes. This means that Ganeti is able to handle VM disks
that reside inside a NAS/SAN or any distributed block storage provider.

The ExtStorage Interface provides a clear API, heavily inspired by the
gnt-os-interface API, that can be used by storage vendors or sysadmins
to write simple ExtStorage Providers (correlated to gnt-os-interface's
OS Definitions). Those Providers will glue externally attached shared
storage with Ganeti, without the need of preprovisioned block devices
on Ganeti VM-capable nodes as confined be the current `blockdev' disk
template.

To do so, we implement a new disk template called `ext' (of type
DTS_EXT_MIRROR) that passes control to externally provided scripts
(the ExtStorage Provider) for the template's basic functions:

 create / attach / detach / remove / grow

The scripts reside under ES_SEARCH_PATH (correlated to OS_SEARCH_PATH)
and only one ExtStorage Provider is supported called `ext'.

The disk's logical id is the tuple ('ext', UUID.ext.diskX), where UUID
is generated as in disk template `plain' and X is the disk's index.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: small simplification in bdev code, pylint fixes]
Reviewed-by: Iustin Pop <iustin@google.com>

Error message formatting, else/if merging to elif, docstrings.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

If opportunistic locking is requested in the opcode, but no iallocator
is used (not specified or no default), an error will be raised.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Currently gnt-instance shutdown --force of an offline instance
resets the offline status. Which of course is bad and completely
unintended. Fix. (caught by the new qa tests)
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, most network LUs simply use EINVAL instead of correct error
codes; this patch tries to adjust the obvious cases.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The code can loop directly over “self.wanted”, saving a lookup. An
explicit loop is also replaced with a generator.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

LUNetworkConnect and LUNetworkDisconnect had very similar code to detect
conflicts between instance's network interfaces and networks. This code
factorizes the common part and does some cleanup:

- Remove single-letter variable for lambda (“l”)
- Don't repeat instance name in warning output
- Fixed error messages to match style guide
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is not a trivial s/frozenset/compat.UniqueFrozenset/, but rather
only replaces “frozenset” where appropriate. Most of the places are
“static” information that doesn't change after the module has been
loaded.

Some docstrings and code formatting (e.g. empty lines) issues are
addressed as well. Some lines got too long and were wrapped.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch removes code which is no longer used due to refactoring:

- http.InitSsl, last usage removed in commit 33231500 (“Convert RPC
  client to PycURL”)
- rapi.baserlib.MakeParamsDict, last usage remove in commit 4e5a68f8
  (“RAPI: rlib1 removal”)
- rapi.baserlib.ExtractField, last usage removed in commit 028c6b76
  (“RAPI:Fix root list and unittest for it.”)
- qa.qa_utils.AssertNotEqual, last real usage removed in commit
  2f4b4f78 (“Simplify QA commands”) (but it was still imported for a
  while, until we extended pylint to run over the QA code as well)
- qlang._IsHostname, added in commit f8638e28 (“Detect globbing
  patterns as query arguments”) but never used
- cmdlib._BuildNetworkHookEnvByObject, last usage removed in commit
  f22433c0

 (“Locking related fixes for networks”)
- NLD constants, which are obsolete—the nbma project is no longer
  maintained and it's not working with current ganeti
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

If an instance is offline we definitely shouldn't start it up.
But shutting it down, should it be up by mistake is not "that" bad.
Still, we only allow it with --force, as it still performs an action on
an instance we shouldn't touch. This should make everybody happy.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

If an instance is running (eg. ERROR_up) and at the same time offline,
there's no way to either shut it down or reonline it. This allows
onlining it. Offlining is still disabled for running instances.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Some fixes, added more information in a few points, removed a stale (5+
year old) TODO comment.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

LEVEL_NODE_ALLOC should be aquired too if LEVEL_NODE is ALL_SET.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

When non-DRBD disks are used for an instance,
“lu.needed_locks[locking.LEVEL_NODE]” is set to “locking.ALL_SET” (which
is None). The assertion will then fail as None evaluates to False.

Reported by Constantinos Venetsanopoulos.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Doing so avoids triggering an assertion in mcpu which cross-checks the
node lock and node allocation lock acquisition mode.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Commit f0f8d060

 (“Show old primary/secondary node on disk
replacement”) added two wrong uses of feedback_fn, which results in
log entries such like these:

      "log": [
        [
          7, [1351258326, 466214], "message", "Replacing disk(s) 0 for instance 'instance1.example.com'"
        ],
        [
          8, [1351258326, 749391], "Current primary node: %s", "node1"
        ],
        [
          9, [1351258326, 774829], "Current seconary node: %s", "node4"
        ],
        [
          10, [1351258329, 620410], "message", "STEP 1/6 Check device existence"
        ],

(the first and fourth are OK, the 2nd and 3rd are broken).

The API of jqueue._OpExecCallbacks.Feedback should probably be changed
to prevent such an occurrence in the future.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Introduced in commit d4752785

.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a "better than nothing" support, just for kvm and just joining
the machine to the opevswitch bridge with the right command.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This should be done because in the case of --no-name-check there
is no 'hostname.ip' attribute, causing an execution error.
'check_ip' is always set (in CheckArguments) even if --no-name-check
is passed in the command line.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The master daemon should not directly read files written by the node
daemon. This patch adds a new RPC to read the watcher pause file and
changes the master code to use it.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

When an instance creation uses opportunistic locks, the iallocator might
not be able to find an allocation solution if not enough node locks (or
a suboptimal subset thereof) were acquired. As per the design document
<doc/design-opportunistic-locking.rst> a new error code denoting
temporary failures is added.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Also actually start using opportunistic locks (if requested).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Forward the node whitelist to the iallocator plugin.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Avoid conflicts between instance allocations.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

If no node names are given, all node locks are acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Use GetNetwork() only when having already aquired the lock,
i.e. in CheckPrereq().

In LUNetworkConnect/Disconnect do not include Network info in Hooks
environment, so that network locking can be avoided if conflicts
are not checked.

Invoke _CheckNodeGroupInstances() only in case of conflicts check,
because otherwise owned instances are None, which results to an error.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Use new NormalizeAndValidateThreeOctetMacPrefix() util function in
LUNetworkAdd/LUNetworkSetParams to validate network's MAC prefix.
Additionally, move the check in CheckArguments() in the case of
LUNetworkAdd.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

All node locks are acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This opcode acquires all node resource locks, which conflicts with
instance allocations.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

See comment in code.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

See comment in code.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>