Commit d759a02b authored by Klaus Aehlig's avatar Klaus Aehlig

Merge branch 'stable-2.11' into master

* stable-2.11
  (no changes)

* stable-2.10
  Fix 'JobIdListOnly' type from 'List' to 'Map'
  Remove NEWS entry for 2.9.4
  Workaround for monitor bug related to greeting msg
  hotplug: Verify if a command succeeded or not
  hotplug: Call each qemu commmand with an own socat
  upgrade: start daemons after ensure-dirs
  upgrade design: ensure-dirs before starting daemons
  Fix network management section in admin.rst
  Adapt release date for 2.10.0-rc2
  Revision bump for 2.10.0-rc2
  Update NEWS file in preparation of 2.10.0rc2
  Add Network Management section in admin.rst

* stable-2.9
  Revision bump for 2.9.4
  Set release date for 2.9.4
  Note UUID identification change in NEWS file
  Allow classic queries to use either names or UUIDs
  Document the change of noded's group in NEWS
  Make the LUInstanceCreate return node names, not UUIDs
  Document new handling of degraded instances in NEWS
  Gracefully handle degraded instances in verification
  Be aware of the degraded case when cleaning up an instance
  Document changes to file-based disks in NEW
  Preserve disk basename on instance rename
  Update NEWS file
  Modify test to reflect RAPI operation changes
  Add QA tests for RAPI multi-instance allocatio
  Fix multi-allocation RAPI method
  Assign unique filenames to filebased disks

* stable-2.8
  Fix execution group of NodeD
Signed-off-by: default avatarKlaus Aehlig <>
Reviewed-by: default avatarHelga Velroyen <>
parents a28216b0 adcccd43
......@@ -1963,6 +1963,7 @@ $(REPLACE_VARS_SED): $(SHELL_ENV_INIT) Makefile stamp-directories
echo 's#@''GNTRAPIGROUP@#$(RAPI_GROUP)#g'; \
echo 's#@''GNTADMINGROUP@#$(ADMIN_GROUP)#g'; \
echo 's#@''GNTCONFDGROUP@#$(CONFD_GROUP)#g'; \
echo 's#@''GNTNODEDGROUP@#$(NODED_GROUP)#g'; \
echo 's#@''GNTLUXIDGROUP@#$(LUXID_GROUP)#g'; \
echo 's#@''GNTMONDGROUP@#$(MOND_GROUP)#g'; \
......@@ -90,10 +90,10 @@ For Haskell:
at least version
Version 2.10.0 rc1
Version 2.10.0 rc2
*(Released Tue, 17 Dec 2013)*
*(Released Fri, 31 Jan 2014)*
Incompatible/important changes
......@@ -159,6 +159,34 @@ Python
- The version requirements for ``python-mock`` have increased to at least
version 1.0.1. It is still used for testing only.
Since 2.10.0 rc1
- Documentation improvements
- Run drbdsetup syncer only on network attach
- Include target node in hooks nodes for migration
- Fix configure dirs
- Support post-upgrade hooks during cluster upgrades
Inherited from the 2.9 branch:
- Ensure that all the hypervisors exist in the config file (Issue 640)
- Correctly recognise the role as master node (Issue 687)
- configure: allow detection of Sphinx 1.2+ (Issue 502)
- gnt-instance now honors the KVM path correctly (Issue 691)
Inherited from the 2.8 branch:
- Change the list separator for the usb_devices parameter from comma to space.
Commas could not work because they are already the hypervisor option
separator (Issue 649)
- Add support for blktap2 file-driver (Issue 638)
- Add network tag definitions to the haskell codebase (Issue 641)
- Fix RAPI network tag handling
- Add the network tags to the tags searched by gnt-cluster search-tags
- Fix caching bug preventing jobs from being cancelled
- Start-master/stop-master was always failing if ConfD was disabled. (Issue 685)
Since 2.10.0 beta1
......@@ -197,6 +225,15 @@ Inherited from the 2.8 branch:
- Refactor reading live data in htools (future proofing cherry-pick)
Version 2.10.0 rc1
*(Released Tue, 17 Dec 2013)*
This was the first RC release of the 2.10 series. All important changes
are listed in the latest 2.10 entry.
Version 2.10.0 beta1
......@@ -217,6 +254,20 @@ before rc1.
- Issue 623: IPv6 Masterd <-> Luxid communication error
Version 2.9.4
*(Released Mon, 10 Feb 2014)*
- Fix the RAPI instances-multi-alloc call
- assign unique filenames to file-based disks
- gracefully handle degraded non-diskless instances with 0 disks (issue 697)
- noded now runs with its specified group, which is the default group,
defaulting to root (issue 707)
- make using UUIDs to identify nodes in gnt-node consistently possible
(issue 703)
Version 2.9.3
......@@ -91,7 +91,7 @@ _daemon_usergroup() {
......@@ -833,6 +833,167 @@ needed to do two replace-disks operation (two copies of the instance
disks), because we needed to get rid of both the original nodes (A and
Network Management
Ganeti used to describe NICs of an Instance with an IP, a MAC, a connectivity
link and mode. This had three major shortcomings:
* there was no easy way to assign a unique IP to an instance
* network info (subnet, gateway, domain, etc.) was not available on target
node (kvm-ifup, hooks, etc)
* one should explicitly pass L2 info (mode, and link) to every NIC
Plus there was no easy way to get the current networking overview (which
instances are on the same L2 or L3 network, which IPs are reserved, etc).
All the above required an external management tool that has an overall view
and provides the corresponding info to Ganeti.
gnt-network aims to support a big part of this functionality inside Ganeti and
abstract the network as a separate entity. Currently, a Ganeti network
provides the following:
* A single IPv4 pool, subnet and gateway
* Connectivity info per nodegroup (mode, link)
* MAC prefix for each NIC inside the network
* IPv6 prefix/Gateway related to this network
* Tags
IP pool management ensures IP uniqueness inside this network. The user can
pass `ip=pool,network=test` and will:
1. Get the first available IP in the pool
2. Inherit the connectivity mode and link of the network's netparams
3. NIC will obtain the MAC prefix of the network
4. All network related info will be available as environment variables in
kvm-ifup scripts and hooks, so that they can dynamically manage all
networking-related setup on the host.
Hands on with gnt-network
To create a network do::
# gnt-network add --network= --gateway= test
Please see all other available options (--add-reserved-ips, --mac-prefix,
--network6, --gateway6, --tags).
Currently, IPv6 info is not used by Ganeti itself. It only gets exported
to NIC configuration scripts and hooks via environment variables.
To make this network available on a nodegroup you should specify the
connectivity mode and link during connection::
# gnt-network connect test bridged br100 default nodegroup1
To add a NIC inside this network::
# gnt-instance modify --net -1:add,ip=pool,network=test inst1
This will let a NIC obtain a unique IP inside this network, and inherit the
nodegroup's netparams (bridged, br100). IP here is optional. If missing the
NIC will just get the L2 info.
To move an existing NIC from a network to another and remove its IP::
# gnt-instance modify --net -1:ip=none,network=test1 inst1
This will release the old IP from the old IP pool and the NIC will inherit the
new nicparams.
On the above actions there is a extra option `--no-conflicts-ckeck`. This
does not check for conflicting setups. Specifically:
1. When a network is added, IPs of nodes and master are not being checked.
2. When connecting a network on a nodegroup, IPs of instances inside this
nodegroup are not checked whether they reside inside the subnet or not.
3. When specifying explicitly a IP without passing a network, Ganeti will not
check if this IP is included inside any available network on the nodegroup.
External components
All the aforementioned steps assure NIC configuration from the Ganeti
perspective. Of course this has nothing to do, how the instance eventually will
get the desired connectivity (IPv4, IPv6, default routes, DNS info, etc) and
where will the IP resolve. This functionality is managed by the external
Let's assume that the VM will need to obtain a dynamic IP via DHCP, get a SLAAC
address, and use DHCPv6 for other configuration information (in case RFC-6106
is not supported by the client, e.g. Windows). This means that the following
external services are needed:
1. A DHCP server
2. An IPv6 router sending Router Advertisements
3. A DHCPv6 server exporting DNS info
4. A dynamic DNS server
These components must be configured dynamically and on a per NIC basis.
The way to do this is by using custom kvm-ifup scripts and hooks.
The snf-network package [1,3] includes custom scripts that will provide the
aforementioned functionality. `kvm-vif-bridge` and `vif-custom` is an
alternative to `kvm-ifup` and `vif-ganeti` that take into account all network
info being exported. Their actions depend on network tags. Specifically:
`dns`: will update an external DDNS server (nsupdate on a bind server)
`ip-less-routed`: will setup routes, rules and proxy ARP
This setup assumes a pre-existing routing table along with some local
configuration and provides connectivity to instances via an external
gateway/router without requiring nodes to have an IP inside this network.
`private-filtered`: will setup ebtables rules to ensure L2 isolation on a
common bridge. Only packets with the same MAC prefix will be forwarded to the
corresponding virtual interface.
`nfdhcpd`: will update an external DHCP server
snf-network works with nfdhcpd [2,3]: a custom user space DHCP
server based on NFQUEUE. Currently, nfdhcpd replies on BOOTP/DHCP requests
originating from a tap or a bridge. Additionally in case of a routed setup it
provides a ra-stateless configuration by responding to router and neighbour
solicitations along with DHCPv6 requests for DNS options. Its db is
dynamically updated using text files inside a local dir with inotify
(snf-network just adds a per NIC binding file with all relevant info if the
corresponding network tag is found). Still we need to mangle all these
packets and send them to the corresponding NFQUEUE.
Known shortcomings
Currently the following things are some know weak points of the gnt-network
design and implementation:
* Cannot define a network without an IP pool
* The pool defines the size of the network
* Reserved IPs must be defined explicitly (inconvenient for a big range)
* Cannot define an IPv6 only network
Future work
Any upcoming patches should target:
* Separate L2, L3, IPv6, IP pool info
* Support a set of IP pools per network
* Make IP/network in NIC object take a list of entries
* Introduce external scripts for node configuration
(dynamically create/destroy bridges/routes upon network connect/disconnect)
[3] deb http:/ wheezy/
Node operations
......@@ -218,10 +218,10 @@ following actions.
- If the action is an upgrade to a higher minor version, the configuration
is upgraded now, using ``cfgupgrade``.
- All daemons are started on all nodes.
- ``ensure-dirs --full-run`` is run on all nodes.
- All daemons are started on all nodes.
- ``gnt-cluster redist-conf`` is run on the master node.
- All daemons are restarted on all nodes.
......@@ -1957,12 +1957,6 @@ def _UpgradeAfterConfigurationChange(oldversion):
returnvalue = 0
ToStdout("Starting daemons everywhere.")
badnodes = _VerifyCommand([pathutils.DAEMON_UTIL, "start-all"])
if badnodes:
ToStderr("Warning: failed to start daemons on %s." % (", ".join(badnodes),))
returnvalue = 1
ToStdout("Ensuring directories everywhere.")
badnodes = _VerifyCommand([pathutils.ENSURE_DIRS])
if badnodes:
......@@ -1970,6 +1964,12 @@ def _UpgradeAfterConfigurationChange(oldversion):
(", ".join(badnodes)))
returnvalue = 1
ToStdout("Starting daemons everywhere.")
badnodes = _VerifyCommand([pathutils.DAEMON_UTIL, "start-all"])
if badnodes:
ToStderr("Warning: failed to start daemons on %s." % (", ".join(badnodes),))
returnvalue = 1
ToStdout("Redistributing the configuration.")
if not _RunCommandAndReport(["gnt-cluster", "redist-conf", "--yes-do-it"]):
returnvalue = 1
......@@ -2869,6 +2869,7 @@ class LUClusterVerifyGroup(LogicalUnit, _VerifyErrors):
node_disks = {}
node_disks_dev_inst_only = {}
diskless_instances = set()
nodisk_instances = set()
diskless = constants.DT_DISKLESS
for nuuid in node_uuids:
......@@ -2881,6 +2882,8 @@ class LUClusterVerifyGroup(LogicalUnit, _VerifyErrors):
for disk in instanceinfo[inst_uuid].disks]
if not disks:
nodisk_instances.update(uuid for uuid in node_inst_uuids
if instanceinfo[uuid].disk_template != diskless)
# No need to collect data
......@@ -2937,6 +2940,10 @@ class LUClusterVerifyGroup(LogicalUnit, _VerifyErrors):
for inst_uuid in diskless_instances:
assert inst_uuid not in instdisk
instdisk[inst_uuid] = {}
# ...and disk-full instances that happen to have no disks
for inst_uuid in nodisk_instances:
assert inst_uuid not in instdisk
instdisk[inst_uuid] = {}
assert compat.all(len(statuses) == len(instanceinfo[inst].disks) and
len(nuuids) <= len(instanceinfo[inst].all_nodes) and
......@@ -1552,7 +1552,7 @@ class LUInstanceCreate(LogicalUnit):
False, self.op.reason)
result.Raise("Could not start instance")
return list(iobj.all_nodes)
return self.cfg.GetNodeNames(list(iobj.all_nodes))
class LUInstanceRename(LogicalUnit):
......@@ -52,6 +52,8 @@ _DISK_TEMPLATE_NAME_PREFIX = {
constants.DT_PLAIN: "",
constants.DT_RBD: ".rbd",
constants.DT_EXT: ".ext",
constants.DT_FILE: ".file",
constants.DT_SHARED_FILE: ".sharedfile",
......@@ -468,8 +470,8 @@ def GenerateDiskTemplate(
elif template_name in constants.DTS_FILEBASED: # Gluster handled above
logical_id_fn = \
lambda _, disk_index, disk: (file_driver,
"%s/disk%d" % (file_storage_dir,
"%s/%s" % (file_storage_dir,
elif template_name == constants.DT_BLOCK:
logical_id_fn = \
lambda idx, disk_index, disk: (constants.BLOCKDEV_DRIVER_MANUAL,
......@@ -293,7 +293,15 @@ def RemoveDisks(lu, instance, target_node_uuid=None, ignore_failures=False):
CheckDiskTemplateEnabled(lu.cfg.GetClusterInfo(), instance.disk_template)
if instance.disk_template in [constants.DT_FILE, constants.DT_SHARED_FILE]:
file_storage_dir = os.path.dirname(instance.disks[0].logical_id[1])
if len(instance.disks) > 0:
file_storage_dir = os.path.dirname(instance.disks[0].logical_id[1])
if instance.disk_template == constants.DT_SHARED_FILE:
file_storage_dir = utils.PathJoin(lu.cfg.GetSharedFileStorageDir(),
file_storage_dir = utils.PathJoin(lu.cfg.GetFileStorageDir(),
if target_node_uuid:
tgt = target_node_uuid
......@@ -1580,13 +1580,13 @@ class ConfigWriter(object):
inst = self._config_data.instances[inst_uuid] = new_name
for (idx, disk) in enumerate(inst.disks):
for (_, disk) in enumerate(inst.disks):
if disk.dev_type in [constants.DT_FILE, constants.DT_SHARED_FILE]:
# rename the file paths in logical and physical id
file_storage_dir = os.path.dirname(os.path.dirname(disk.logical_id[1]))
disk.logical_id = (disk.logical_id[0],
"disk%s" % idx))
# Force update of ssconf files
self._config_data.cluster.serial_no += 1
......@@ -513,6 +513,10 @@ class QmpConnection(MonitorSocket):
raise errors.HypervisorError("kvm: QMP communication error (wrong"
" server greeting")
# This is needed because QMP can return more than one greetings
# see
self._buf = ""
# Let's put the monitor in command mode using the qmp_capabilities
# command, or else no command will be executable.
# (As per the QEMU Protocol Specification 0.1 - section 4)
......@@ -784,6 +788,11 @@ class KVMHypervisor(hv_base.BaseHypervisor):
_INFO_PCI_RE = re.compile(r'Bus.*device[ ]*(\d+).*')
_INFO_PCI_CMD = "info pci"
lambda pci, devid: re.compile(r'Bus.*device[ ]*%d,(.*\n){5,6}.*id "%s"' %
(pci, devid), re.M))
re.compile(r'^QEMU (\d+)\.(\d+)(\.(\d+))?.*monitor.*', re.M)
_INFO_VERSION_CMD = "info version"
......@@ -2217,11 +2226,34 @@ class KVMHypervisor(hv_base.BaseHypervisor):
if (int(v_major), int(v_min)) < (1, 0):
raise errors.HotplugError("Hotplug not supported for qemu versions < 1.0")
def _CallHotplugCommand(self, name, cmd):
output = self._CallMonitorCommand(name, cmd)
# TODO: parse output and check if succeeded
for line in output.stdout.splitlines():"%s", line)
def _CallHotplugCommands(self, name, cmds):
for c in cmds:
self._CallMonitorCommand(name, c)
def _VerifyHotplugCommand(self, instance_name, device, dev_type,
"""Checks if a previous hotplug command has succeeded.
It issues info pci monitor command and checks depending on should_exist
value if an entry with PCI slot and device ID is found or not.
@raise errors.HypervisorError: if result is not the expected one
output = self._CallMonitorCommand(instance_name, self._INFO_PCI_CMD)
kvm_devid = _GenerateDeviceKVMId(dev_type, device)
match = \
self._FIND_PCI_DEVICE_RE(device.pci, kvm_devid).search(output.stdout)
if match and not should_exist:
msg = "Device %s should have been removed but is still there" % kvm_devid
raise errors.HypervisorError(msg)
if not match and should_exist:
msg = "Device %s should have been added but is missing" % kvm_devid
raise errors.HypervisorError(msg)"Device %s has been correctly hot-plugged", kvm_devid)
def HotAddDevice(self, instance, dev_type, device, extra, seq):
""" Helper method to hot-add a new device
......@@ -2236,21 +2268,22 @@ class KVMHypervisor(hv_base.BaseHypervisor):
kvm_devid = _GenerateDeviceKVMId(dev_type, device)
runtime = self._LoadKVMRuntime(instance)
if dev_type == constants.HOTPLUG_TARGET_DISK:
command = "drive_add dummy file=%s,if=none,id=%s,format=raw\n" % \
(extra, kvm_devid)
command += ("device_add virtio-blk-pci,bus=pci.0,addr=%s,drive=%s,id=%s" %
(hex(device.pci), kvm_devid, kvm_devid))
cmds = ["drive_add dummy file=%s,if=none,id=%s,format=raw" %
(extra, kvm_devid)]
cmds += ["device_add virtio-blk-pci,bus=pci.0,addr=%s,drive=%s,id=%s" %
(hex(device.pci), kvm_devid, kvm_devid)]
elif dev_type == constants.HOTPLUG_TARGET_NIC:
(tap, fd) = _OpenTap()
self._ConfigureNIC(instance, seq, device, tap)
self._PassTapFd(instance, fd, device)
command = "netdev_add tap,id=%s,fd=%s\n" % (kvm_devid, kvm_devid)
cmds = ["netdev_add tap,id=%s,fd=%s" % (kvm_devid, kvm_devid)]
args = "virtio-net-pci,bus=pci.0,addr=%s,mac=%s,netdev=%s,id=%s" % \
(hex(device.pci), device.mac, kvm_devid, kvm_devid)
command += "device_add %s" % args
cmds += ["device_add %s" % args]
utils.WriteFile(self._InstanceNICFile(, seq), data=tap)
self._CallHotplugCommand(, command)
self._CallHotplugCommands(, cmds)
self._VerifyHotplugCommand(, device, dev_type, True)
# update relevant entries in runtime file
index = _DEVICE_RUNTIME_INDEX[dev_type]
entry = _RUNTIME_ENTRY[dev_type](device, extra)
......@@ -2269,13 +2302,14 @@ class KVMHypervisor(hv_base.BaseHypervisor):
kvm_device = _RUNTIME_DEVICE[dev_type](entry)
kvm_devid = _GenerateDeviceKVMId(dev_type, kvm_device)
if dev_type == constants.HOTPLUG_TARGET_DISK:
command = "device_del %s\n" % kvm_devid
command += "drive_del %s" % kvm_devid
cmds = ["device_del %s" % kvm_devid]
cmds += ["drive_del %s" % kvm_devid]
elif dev_type == constants.HOTPLUG_TARGET_NIC:
command = "device_del %s\n" % kvm_devid
command += "netdev_del %s" % kvm_devid
cmds = ["device_del %s" % kvm_devid]
cmds += ["netdev_del %s" % kvm_devid]
utils.RemoveFile(self._InstanceNICFile(, seq))
self._CallHotplugCommand(, command)
self._CallHotplugCommands(, cmds)
self._VerifyHotplugCommand(, kvm_device, dev_type, False)
index = _DEVICE_RUNTIME_INDEX[dev_type]
self._SaveKVMRuntime(instance, runtime)
......@@ -2292,8 +2326,6 @@ class KVMHypervisor(hv_base.BaseHypervisor):
if dev_type == constants.HOTPLUG_TARGET_NIC:
# putting it back in the same pci slot
device.pci = self.HotDelDevice(instance, dev_type, device, _, seq)
# TODO: remove sleep when socat gets removed
self.HotAddDevice(instance, dev_type, device, _, seq)
def _PassTapFd(self, instance, fd, nic):
......@@ -975,12 +975,19 @@ class R_2_instances_multi_alloc(baserlib.OpcodeResource):
raise http.HttpBadRequest("Request is missing required 'instances' field"
" in body")
op_id = {
"OP_ID": self.POST_OPCODE.OP_ID, # pylint: disable=E1101
# Unlike most other RAPI calls, this one is composed of individual opcodes,
# and we have to do the filling ourselves
"os": "os_type",
"name": "instance_name",
body = objects.FillDict(self.request_body, {
"instances": [objects.FillDict(inst, op_id)
for inst in self.request_body["instances"]],
"instances": [
baserlib.FillOpcode(opcodes.OpInstanceCreate, inst, {},
for inst in self.request_body["instances"]
return (body, {
......@@ -871,6 +871,7 @@ def RunQa():
if (qa_config.TestEnabled("instance-add-plain-disk")
and qa_config.IsTemplateSupported(constants.DT_PLAIN)):
# Normal instance allocation via RAPI
for use_client in [True, False]:
rapi_instance = RunTest(qa_rapi.TestRapiInstanceAdd, pnode,
......@@ -882,6 +883,16 @@ def RunQa():
del rapi_instance
# Multi-instance allocation
rapi_instance_one, rapi_instance_two = \
RunTest(qa_rapi.TestRapiInstanceMultiAlloc, pnode)
RunTest(qa_rapi.TestRapiInstanceRemove, rapi_instance_one, True)
RunTest(qa_rapi.TestRapiInstanceRemove, rapi_instance_two, True)
......@@ -27,6 +27,7 @@ import tempfile
import random
import re
import itertools
import functools
from ganeti import utils
from ganeti import constants
......@@ -656,6 +657,76 @@ def TestRapiInstanceAdd(node, use_client):
def _GenInstanceAllocationDict(node, instance):
"""Creates an instance allocation dict to be used with the RAPI"""
disks = [{"size": utils.ParseUnit(d.get("size")),
"name": str(d.get("name"))}
for d in qa_config.GetDiskOptions()]
nic0_mac = instance.GetNicMacAddr(0, constants.VALUE_GENERATE)
nics = [{
constants.INIC_MAC: nic0_mac,
beparams = {
constants.BE_MAXMEM: utils.ParseUnit(qa_config.get(constants.BE_MAXMEM)),
constants.BE_MINMEM: utils.ParseUnit(qa_config.get(constants.BE_MINMEM)),
return _rapi_client.InstanceAllocation(constants.INSTANCE_CREATE,,
disks, nics,
def TestRapiInstanceMultiAlloc(node):
"""Test adding two new instances via the RAPI instance-multi-alloc method"""
if not qa_config.IsTemplateSupported(constants.DT_PLAIN):
JOBS_KEY = "jobs"
instance_one = qa_config.AcquireInstance()
instance_two = qa_config.AcquireInstance()
instance_list = [instance_one, instance_two]