This allows a lot of simplification in the TestIcmpPing, as fping can
take multiple arguments so we don't need anymore to create many
commands joined with &&.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This will change the result of this LU to a query like result. A list of
tuples with information about the state of the data.

It also includes the modification to the commands calling this opcode.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.4:
  Another fix for LUClusterVerifyDisks
  QA: also run gnt-cluster verify-disks
  Fix disk adoption breakage
  Fix typo in query2 design document
  Improve documentation for QRFS_UNAVAIL
  lvmstrap: add PV-on-partition support
  lvmstrap: abstract a little the sysfs paths
  lvmstrap: ignore small-sized partitions
  lvmstrap: add explicit test for swap backends
  lvmstrap: add an explicit test for mounted devices
  lvmstrap: add more excluded FS types
  lvmstrap: fix very old contact information
  Instance query: replace duplicates with aliases
  query: Add alias support in _PrepareFieldList
  Fix disk count check in LUSetInstanceParams
  Document iallocator change (alloc_policy)

Conflicts:
        lib/constants.py (due to QRFS→RS on master and doc on 2.4)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch renames QRFS_* to RS_* fields so they can be used in other
places (i.e. LUs) without confusion, as this was initially meant for
query operations.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The LVM queries should only be done for vm_capable nodes. In order to
do this, we also add a new ConfigWriter method to abstract that query.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The bug recently reported by Apollon Oikonomopoulos was missed because
we don't test this command at all.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Disk adoption is currently broken by 84d7e26b, which added multiple LVM
volume group support. This patch fixes the calls to rpc.call_vg_list,
which are multi-node calls but were handled as single-node calls in
84d7e26b.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

IMHO this should have been named QRFS_NA or QRFS_UNSUPPORTED, but
UNAVAIL is good enough.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is a not-so-nice change, adding support for partitions to be used
as PVs.

The not-nice part is that partitions live in a separate place in
sysfs, whereas in dev they live at the same level as disks. We
workaround this via a new SysfsName function that computes the correct
sysfs base path for a given disk.

The other rule is that if a disk is not in use, we ignore its
partitions completely, as the disk will be re-partitioned anyway. Only
if the disk is busy, we consider each of its partitions for the
free/busy list.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch changes lvmstrap to ignore “small” partitions. Currently
extended partitions are reported as unused as with a size of 1024
(bytes), and this confuses lvmstrap. Since a very small partition
won't help anyway (below hundred of PE size is not helpful), let's
restrict it to 1GB.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Similar to mounted filesystems, recent kernel/userland report swap
backends:

    root@node4:~# fuser -avm /dev/sda6
                         USER        PID ACCESS COMMAND
    /dev/sda6:           root     kernel swap  /dev/sda6

But old ones not:

    node1# fuser -avm /dev/sda6
                         USER        PID ACCESS COMMAND
    /dev/sda6:

So we add an explicit test for swap backends using /proc/swaps.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Recent kernels/userland report a mounted filesystem as follows:

    root@node4:~# fuser -avm /dev/sda5
                         USER        PID ACCESS COMMAND
    /dev/sda5:           root     kernel mount /srv/ganeti

But old ones do not:

    node1# fuser -avm /dev/sda5
                         USER        PID ACCESS COMMAND
    /dev/sda5:

So in order to cover both, we add an explicit test for this. We
already compute the device numbers of all mounted filesystems, so it
is a trivial addition (GetMountInfo() is cheap, so we don't need to
cache it).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Also moves the list of excluded types to the top level and make it a
frozenset.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

A memory from the past: this was left over from before the 1.2 release
or so…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is just a plain stupid and simple out-of-band helper without
anything fancy. It uses plain ssh to power off / power cycle the
machine, does not support power on. It support power status using fping
to check if the host replies.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

LUSetInstanceParams checked instance.nics (and not instance.disks)
against constants.MAX_DISKS.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This was missing from commit 7385c51d.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will detect human errors when setting a release date in NEWS.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch exposes the VTYPE kind of BE/HV parameters, instead of
returning QFT_OTHER. The current situation makes a query like:

    gnt-instance list -o name,be/memory,oper_ram

very strange looking.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

If a HV parameter is required that does not apply for an instance,
currently the code returns None. This is bad, as it means we cannot
switch to the actual HV parameter types and validate correctly this
field.

This patch changes it so that in this case we return QRFS_UNAVAIL;
ideally we would use a NOT_APPROPRIATE or similar field, but UNAVAIL is
good enough (the call cannot fail in another way).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, the single assert just checks that the entire row is
consistent (true/false), and dumps the row and field definitions as an
accompanying message. This makes it very hard to understand what failed.

This patch changes this validation to show descriptive messages, which
makes it much faster in diagnosing invalid result.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The conversion to querylib introduced unique field titles, which however
did an inconsistent rename of the instance vcpus/memory fields.

This patch makes both BE_MEMORY and _VCPUS be named with prefix “Config”
(which before was “Configured_”), and drops the “Runtime” (note no
underscore) prefix on the oper_ ones.

Rationale: for numeric fields, the width of the title defines the width
of the columns, as the numbers are usually small. Hence these columns
(more likely used than the BE ones) would be very wide, leading to a
not-so-nice display. I happened upon this when my terminals didn't
manage to fit a gnt-instance list anymore.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

While looking at the query library, I realized that while we have five
field statuses, making this a 5-dimensional space, four of them are
shrunk to a single possible value (None). Hence it should be possible to
convert this into a single value space plus extra 4 special constants.

This patch implements this, making (IMHO) the return value of normal
functions much simpler: you simply return the desired value, instead of
(QRFS_NORMAL, value); for the special results, you simply return
_FS_UNAVAIL, instead of (QRFS_UNAVAIL, None). This I believe does
simplify the code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, the QRFT/QFT constants are used with the module (e.g.
constants.QRFS_NORMAL) in the query library. Given that there are 176
uses of these constants in a module that has roughly ~1200 lines in
total (including whitespace), it makes sense to import these directly to
improve readability.

This patch does this and (IMHO) it does indeed make the code more to the
point; the entire query library is about queries, so using these
constants directly doesn't cause any misunderstandings.

Note: the epydoc links are not changed, since we want the generated docs
to point to the right place.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This shouldn't normally happen, but if it does no need to fail.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch introduces network configuration for KVM in Ganeti.

There are three problems with having KVM perform network configuration via ifup
scripts:
  a) Ganeti never gets to know the tap interface that is associated with an
     instance's NIC
  b) Migration of routed instances will cause network problems because the
     incoming KVM side configures the network as soon as it is spawned and not
     as soon as the migration finishes. This means that all routing
     configuration will be present in both, primary and secondary, nodes at the
     same time, possibly causing network disruption during the migration.
  c) We never get to know if the network configuration succeeded or not.

This patch moves network configuration from KVM to Ganeti, using KVM's ability
to receive already open tap devices as file descriptors.

_WriteNetScript is removed from hv_kvm.py, together with its unit tests.

Minor modifications are made to _ExecKVMRuntime to handle tap device
initialization. NIC <-> tap associations are stored under a new directory,
_ROOT_DIR/nic in a file-per-nic fashion.

The end-user semantics remain the same: The user can override the network
configuration by providing _KVM_NET_SCRIPT. If this is not present or
executable, the default constants.KVM_IFUP script is run.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In order to retain backwards compatibility, we ship the shell script previously
generated by _WriteNetScript as a stand-alone script under PKGLIBDIR.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add two new functions, _OpenTap and _ProbeTapVnetHdr, to
hypervisors.hv_kvm.

_ProbeTapVnetHdr checks if the host kernel supports the virtio net header
(IFF_VNET_HDR) feature. This feature allows KVM to ask the the host kernel to
perform checksumming and segmentation offload of the data passing through a
guest's virtio NIC, using a special header[1] prepended to data sent through
the tap interface. A qemu-kvm version later than 0.12 is required for this
feature to work.

_OpenTap opens a new tap device suitable for use with qemu-kvm's "-net
tap,fd=n" option. It also enables vnet_hdr support if requested and safe to do
so.

[1] struct virtio_net_hdr in include/linux/virtio_net.h

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This way we can add there non-python scripts

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The removal requires adding a few pylint ignores, as it cannot know that
OP_ID will be defined at runtime.

Also switch the asserts to check for non-declaration.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As the class names should be now consistent with the OP_IDs, we add a
check for wrongly-defined OP_IDs.

However, the future removal of the hand-coded OP_IDs will render this
obsolete, so this check is introduced just to make sure that the
previous renaming patches did the right job, and it will then be
removed.

The consistency checks require renaming the test opcodes, which were
using arbitrary names, depending on test author. They are now all
standardized on OpTest (local scope).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>