GitLab

Move all tap-related functionality to the hv_kvm.netdev submodule. We
rename _OpenTap to OpenTap, since it will now be used as a public
function.

Also, change the hv_kvm tests to import the new code.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

We move the HMP and QMP classes to the hv_kvm.monitor module.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Weighing in at 3k LOC, it is probably time to split hv_kvm into a number
of modules before adding new code.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

The vlan nicparam is used in openvswitch mode. This patch
exports it as VLAN environment variable so that NIC
configuration scripts can later use it.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Latest QEMU versions support various migration capabilities. Each
can be enabled/disabled with 'migrate_set_capability' monitor
command.

Version 1.7.0 defines x-rdma-pin-all, auto-converge, zero-blocks,
and xbzrle migration capabilities.

Since migration capabilities are more than one, and because hvparams
accept only strings/integers and not dicts we decide to provide
the capabilities that should be on via a ":" separated string.

In other words one can use the following to enable xbzrle and
auto-converge:

-H kvm:migration_caps=xbzrle:auto-converge

IMPORTANT: xbzrle may result to BSOD for instances running
Windows 2008r8 on drbd.

Update man page of gnt-instance to include the new hvparam.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

With regard to PCI slot occupied by a KVM instance we have
observed the following:

1) Slot 0 will always be Host bridge.
2) Slot 1 will always be ISA bridge.
3) Slot 2 will always be VGA controller (even with -display none).
4) If soundhw=hda|ac97|es1370 an extra PCI slot is occupied.
   This slot *MUST* be the No. 3.

5) Option '-balloon virtio' gets an extra PCI slot.
   Still it can take id, bus, and addr args and be placed anywhere

6) If spice is used instead of vnc we have:
   * No extra PCI slot gets occupied without vdagent
   * Otherwise we have the following extra optionsa [1]
     a) -device virtio-serial-pci
        (this can take id, bus, and addr args too)
     b) -device virtserialport,chardev=spicechannel0,...
     c) -chardev spicevmc,id=spicechannel0

This patch does the following:

1) Change _DEFAULT_PCI_RESERVATIONS to "1110...."
2) Move soundhw option early in the command line and if hda etc.
   reserve slot 3.
3) Add id, bus, and addr in balloon option and reserve next slot.
4) Add id, bus, and addr in -device virtio-serial-pci option and if
   passed reserve next slot.

[1] http://www.linux-kvm.org/page/SPICE



Fixes issue 757.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

Introduce new method `_GetFreeSlot()` responsible only for bitarray
operations. It fixes search in case of bitarray is either '0000..'
or '1111..'.

Use it instead of `_UpdatePCISlots()` and in `_GetFreePCISlot()`.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

Add the timeout parameter to the StopInstance function of the hypervisor base
class and to all its implementations.

Also, change the tests as required by this change.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Modify the function that sends commands to the KVM monitor so that it is
possible to specify an optional timeout after which the command is killed.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

* Add helper function that generates names of the form 'gnt.com.%d'
  which are the names for the TAP interfaces meant to be used by
  instance communication.  These names are unique within the node.

* Use the previous helper function to actually generate the name for
  the NICs of KVM instance being created.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This is necessary in order to create TAP interfaces for the
communication mechanism, which are named 'gnt.com.%d', where '%d' is a
unique number within a given node.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

QMP may return multiple greeting messages upon connection.
This is reported on qemu-devel. The fix is one-liner but
until it get's released this is a quick and dirty workaround
that flushes the client's buffer after getting the first
greeting message.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Just after issuing _CallHoplugCommands() we invoke
_VerifyHotplugCommand() which parses `info pci` result
and searches for given PCI slot and device id.

If we previously had removed a device but it is still there
abort. Do the same if we had add a device but it is not found.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Previously we issued one socat command with two "\n" separated
actions (e.g. netdev_add ...\ndevice_add...)

After having observed a strange monitor behavior [1] splitting
those commands and introducing a sleep time in between, may reduce
any race possibilities. Additionally it will be a step forward
parsing monitor output of each command and decide whether to
modify runtime files or raise a HotplugError.

This patch simply changes _CallHotplugCommand() to take a
list of commands and issue them separately. Introduces one second
sleep after each command and removes sleep from HotModDevice().

[1] https://groups.google.com/d/msg/ganeti-devel/hCQnmuqKtU0/UWGnw251cskJ

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

... because it is a protected definition.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* The previous patch introduced cdrom image URLs for KVM cdrom drives.
  However, it worked only for the first drive and not the second.
  This patch generalizes the KVM cdrom command line option to allow
  cdrom image URLs to be passed to both drives.

* Simplify some functions and fix some doc strings
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Perform a simple urllib2 check on ISO images specified as URL before
instance start, so as to work around qemu bug #597575 [1].

[1] https://bugs.launchpad.net/qemu/+bug/597575

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

New versions of KVM support booting from HTTP-hosted ISO images, via
libcurl. This patch adds a proper check to allow defining either a sane,
absolute path or an HTTP URL as an iso image path.

Remove "format=raw" from the cdrom device options when iso_image starts
with "(https?|ftps?)://", to allow latest KVM versions (qemu-kvm) to boot
from HTTP-hosted iso images. Added FILE_OR_URL check to allow booting
from HTTP (kvm).
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

* modify the KVM hypervisor to look for the shutdown files created by
  the KVM daemon, which determine whether an instance was shutdown by
  the user
* modify the KVM hypervisor to spawn the KVM daemon if it is not
  running
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Add user shutdown parameter for KVM.  Based on this parameter, decide
what information to report for a KVM instance, for example,
distinguish between 'ADMIN_down' and 'USER_down'.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Commit de253f14 of QEMU repo "BREAKS QMP's compatibility for
the error response" as it removes "data" key from qmp error
response messages.  To this end we only log "class" and "desc"
values of the message.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Stratos Psomadakis <psomas@grnet.gr>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

Commit e6e4ff4c

 fixed Issue 649 by switching the
separator for usb_devices from comma to space. That solved the problem with
the command line, but RAPI was able to work with commas too, so, for backwards
compatibility we need to keep supporting that as well.

Also, in order to avoid changing the format of the config file, the default
internal representation is brought back to being comma-based, and it is changed
at the interface level (CLI or RAPI) before being passed on.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The usb_devices parameter was using comma as a list separator, but this cannot
work because comma is already used as the hypervisor parameter separator.

Change it to use whitespace as a separator, in accordance to what already done
for the extra parameters.

The NEWS file is updated accordingly.

Fixes Issue 649.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This method is invoked during _AnalizeSerializedRuntime() and is
meant to modify runtime files in the way cfgupgrade does for
config.data.  This could remove deprecated fields, change the
format of the file, add/remove sections, etc.

There is a possibility data included inside runtime
files to cause migration failures after a Ganeti upgrade.
Use this method to avoid such cases.

Commit cc6fd3da

 exports NICs' UUIDs to configuration scripts.
It assumes UUIDs as an non optional slot in NIC object.
This is true for latest Ganeti versions. Still there might
be instances created back in Ganeti 2.7 missing that slot.

Ganeti 2.10 supports hot-plugging of devices. A prerequisite
for this feature was to change the format of runtime files and
add another entry containing block devices. This was done in
_AnalizeSerializedRuntime(). Move this logic in the new method.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Fixes issue 616.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This commit fixes two TODOs from 2008 about using the hardcoded
"default" path for KVM where a custom one could've been set through
`gnt-cluster modify`.

As a result, `gnt-cluster verify` will no longer fail if a custom
path was set in such a manner.
Signed-off-by: Santi Raffa <rsanti@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

In case of kvm None values are not allowed in env dict
so we have to add name only if not None.

In case of Xen since we are writing on a file that
is going to be sourced we should not add INTERFACE_NAME=None.

Fix IP=None case.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Use GetTags() instance method in order to export instance tags
to NIC configuration scripts and files of kvm and xen hypervisors.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This will be useful for an external entity using RAPI that
wants to modify devices of instances.

The common use case for that is:
"I want to add a NIC/disk to an instance. If it is running
then try to hotplug the device. If not, then just add it to config."

Another use case is that if we are unaware of the instance's
hypervisor: "I want do add a NIC/disk to an instance and if
its hypervisor is KVM try to hotplug it."

For all those cases just print a warning in CheckPrereq() and
disable any further hotplug related actions (and let
modifications take place after reboot).

Add NoHoplugSupport and HotplugIfPossible unittests an modify
existing by explicitly creating successful RPC results for
call_hotplug_supported.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This is accomplished by passing the corresponding node group to
hv_*.py. Tests for hv_*.py that call GetInstanceConsole updated.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Several methods in KVM hypervisor behave like 'classmethods', however,
they did not have the '@classmethod' annotation.  This patch adds
'@classmethod' annotations to several of these methods with the goal
of making '_StopInstance' available to the QA.  This step is necessary
for the QA test for instance shutdown detection on KVM.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Add 'HvInstanceState' which represents the state of an instance
(either 'running' or 'shutdown') and fix the remaining hypervisor
backends to return a value of this type.  Before this patch, each
backend was returning their own value, some were returning strings,
others integers.  However, the value that represents the state of an
instance was never actually used in Ganeti, until instance shutdown
detection became necessary.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Introduce new RPC hotplug_supported that invokes the corresponding
hypervisor's method which checks if hotplug is generally
supported. Call this RPC early in CheckPrereq() and abort if
hotplug is not supported. Currently only KVM hypervisor with
qemu versions > 1.0. Please note that this check is unaware of
target device and hotplug action. This latter check is still
done during hotplug_device() RPC.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This is needed so that qemu process really releases the
corresponding file descriptor and thus we can dmsetup/lvremove
the device without errors.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

and a couple more styple fixes.
block_devices are misleading so rename them to kvm_disk (just like kvm_nics)
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Hotplug is not supported by default and thus we raise
HotplugError() for hotplug related methods.
If a hypervisor wants to support it, we must override VerifyHotplugSupport()
and implement the Hot* method.

Backend invokes VerifyHotplugSupport() and raises RPCFail if an
exception is raised.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

* Add docstring.
* fix missed usecase of boot_disk=False if kernel_path is given.
* Use devlist arg in GenerateKVMBlockDeviceOptions()
  This is previously calculated in _ExecuteKVMRuntime()
  so there is no need to rerun the same command.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

4 new methods: HotAddDevice, HotDelDevice, HotModDevice, HotplugSupported
All these are the hypervisor interface with the backend.
In general they read current runtime info, do the actual hotplug
and update the runtime file.

NIC hot-add:
 - open a tap and get its file descriptor.
 - pass fd with SCM rights (using python-fdsend) via monitor socket
 - create netdev and device with id=kvm_devid and proper pci info

Disk hot-add:
 - create drive with id=kvm_devid
 - create device with id=kvm_devid and corresponding pci info

Use MonitorSocket in _PassTapFd in order to connect to monitor socket
and pass the tap's file descriptor using fdsend.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Separate unix socket related code from QmpMonitor class and
make the latter extend the newly introduced one: MonitorSocket
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Hotplug is currently *not* supported for:

 - KVM < 1.0
 - existing devices in the cluster
 - python-fdsend module is not installed (NIC hotplug)
 - chroot (Disk hotplug)
 - security mode other than None (Disk hotplug)

For the above reasons raise HotplugError exception and
let masterd handle it.

TODO: modify CallHotplugCommand() so that it parses
monitor output and reports whether the command
succeeded or not.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>