admin.rst

Ganeti administrator's guide

Documents Ganeti version |version|

Contents

• Introduction
  • Ganeti terminology
    • Cluster
    • Node
    • Instance
    • Disk template
    • IAllocator
    • “Primary” and “secondary” concepts
    • Tags
    • Jobs and OpCodes
  • Prerequisites
• Instance management
  • Adding an instance
  • Regular instance operations
    • Removal
    • Startup/shutdown
    • Querying instances
  • Changing an instance's runtime memory
  • Export/Import
  • Import of foreign instances
  • Instance kernel selection
    • Xen-PVM
    • KVM
• Instance HA features
  • Changing the primary node
    • Failing over an instance
    • Live migrating an instance
    • Moving an instance (offline)
  • Disk operations
    • Preparing for disk operations
    • Restoring redundancy for DRBD-based instances
    • Re-creating disks for non-redundant instances
    • Conversion of an instance's disk type
  • Debugging instances
    • Accessing an instance's disks
    • Accessing an instance's console
  • Other instance operations
  • Instance OS definitions debugging
    • Instance relocation
• Node operations
  • Add/readd
  • Changing the node role
    • Failing over the master node
    • Changing between the other roles
  • Evacuating nodes
    • Primary instance conversion
    • Secondary instance evacuation
  • Removal
  • Replication network changes
  • Storage handling
    • Logical volumes
    • Generalized storage handling
    • Use of the storage commands
• Cluster operations
  • Standard operations
  • Global node commands
  • Cluster verification
  • Configuration redistribution
  • Cluster renaming
  • Queue operations
  • Watcher control
  • Node auto-maintenance
  • Removing a cluster entirely
• Tags handling
  • Limitations
  • Operations
  • Global tag search
• Job operations
• Special Ganeti deployments
  • Running Ganeti under Ganeti
  • Multi-site model
• Ganeti tools
  • lvmstrap
  • cfgupgrade
  • cfgshell
  • burnin
  • sanitize-config
  • move-instance
• Other Ganeti projects
  • NBMA tools
  • ganeti-htools

Introduction

Ganeti is a virtualization cluster management software. You are expected to be a system administrator familiar with your Linux distribution and the Xen or KVM virtualization environments before using it.

The various components of Ganeti all have man pages and interactive help. This manual though will help you getting familiar with the system by explaining the most common operations, grouped by related use.

After a terminology glossary and a section on the prerequisites needed to use this manual, the rest of this document is divided in sections for the different targets that a command affects: instance, nodes, etc.

Ganeti terminology

This section provides a small introduction to Ganeti terminology, which might be useful when reading the rest of the document.

Cluster

A set of machines (nodes) that cooperate to offer a coherent, highly available virtualization service under a single administration domain.

Node

A physical machine which is member of a cluster. Nodes are the basic cluster infrastructure, and they don't need to be fault tolerant in order to achieve high availability for instances.

Node can be added and removed (if they host no instances) at will from the cluster. In a HA cluster and only with HA instances, the loss of any single node will not cause disk data loss for any instance; of course, a node crash will cause the crash of its primary instances.

A node belonging to a cluster can be in one of the following roles at a given time:

• master node, which is the node from which the cluster is controlled
• master candidate node, only nodes in this role have the full cluster configuration and knowledge, and only master candidates can become the master node
• regular node, which is the state in which most nodes will be on bigger clusters (>20 nodes)
• drained node, nodes in this state are functioning normally but the cannot receive new instances; the intention is that nodes in this role have some issue and they are being evacuated for hardware repairs
• offline node, in which there is a record in the cluster configuration about the node, but the daemons on the master node will not talk to this node; any instances declared as having an offline node as either primary or secondary will be flagged as an error in the cluster verify operation

Depending on the role, each node will run a set of daemons:

• the :command:`ganeti-noded` daemon, which controls the manipulation of this node's hardware resources; it runs on all nodes which are in a cluster
• the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all nodes, but is only functional on master candidate nodes; this daemon can be disabled at configuration time if you don't need its functionality
• the :command:`ganeti-rapi` daemon which runs on the master node and offers an HTTP-based API for the cluster
• the :command:`ganeti-masterd` daemon which runs on the master node and allows control of the cluster

Beside the node role, there are other node flags that influence its behaviour:

• the master_capable flag denotes whether the node can ever become a master candidate; setting this to 'no' means that auto-promotion will never make this node a master candidate; this flag can be useful for a remote node that only runs local instances, and having it become a master is impractical due to networking or other constraints
• the vm_capable flag denotes whether the node can host instances or not; for example, one might use a non-vm_capable node just as a master candidate, for configuration backups; setting this flag to no disallows placement of instances of this node, deactivates hypervisor and related checks on it (e.g. bridge checks, LVM check, etc.), and removes it from cluster capacity computations

Instance

A virtual machine which runs on a cluster. It can be a fault tolerant, highly available entity.

An instance has various parameters, which are classified in three categories: hypervisor related-parameters (called hvparams), general parameters (called beparams) and per network-card parameters (called nicparams). All these parameters can be modified either at instance level or via defaults at cluster level.

Disk template

The are multiple options for the storage provided to an instance; while the instance sees the same virtual drive in all cases, the node-level configuration varies between them.

There are five disk templates you can choose from:

diskless

The instance has no disks. Only used for special purpose operating systems or for testing.

file

The instance will use plain files as backend for its disks. No redundancy is provided, and this is somewhat more difficult to configure for high performance.

plain

The instance will use LVM devices as backend for its disks. No redundancy is provided.

drbd

Note

This is only valid for multi-node clusters using DRBD 8.0+

A mirror is set between the local node and a remote one, which must be specified with the second value of the --node option. Use this option to obtain a highly available instance that can be failed over to a remote node should the primary one fail.

Note

Ganeti does not support DRBD stacked devices: DRBD stacked setup is not fully symmetric and as such it is not working with live migration.

rbd

The instance will use Volumes inside a RADOS cluster as backend for its disks. It will access them using the RADOS block device (RBD).

IAllocator

A framework for using external (user-provided) scripts to compute the placement of instances on the cluster nodes. This eliminates the need to manually specify nodes in instance add, instance moves, node evacuate, etc.

In order for Ganeti to be able to use these scripts, they must be place in the iallocator directory (usually lib/ganeti/iallocators under the installation prefix, e.g. /usr/local).

“Primary” and “secondary” concepts

An instance has a primary and depending on the disk configuration, might also have a secondary node. The instance always runs on the primary node and only uses its secondary node for disk replication.

Similarly, the term of primary and secondary instances when talking about a node refers to the set of instances having the given node as primary, respectively secondary.

Tags

Tags are short strings that can be attached to either to cluster itself, or to nodes or instances. They are useful as a very simplistic information store for helping with cluster administration, for example by attaching owner information to each instance after it's created:

$ gnt-instance add … %instance1%
$ gnt-instance add-tags %instance1% %owner:user2%

And then by listing each instance and its tags, this information could be used for contacting the users of each instance.

Jobs and OpCodes

While not directly visible by an end-user, it's useful to know that a basic cluster operation (e.g. starting an instance) is represented internally by Ganeti as an OpCode (abbreviation from operation code). These OpCodes are executed as part of a Job. The OpCodes in a single Job are processed serially by Ganeti, but different Jobs will be processed (depending on resource availability) in parallel. They will not be executed in the submission order, but depending on resource availability, locks and (starting with Ganeti 2.3) priority. An earlier job may have to wait for a lock while a newer job doesn't need any locks and can be executed right away. Operations requiring a certain order need to be submitted as a single job, or the client must submit one job at a time and wait for it to finish before continuing.

For example, shutting down the entire cluster can be done by running the command gnt-instance shutdown --all, which will submit for each instance a separate job containing the “shutdown instance” OpCode.

Prerequisites

You need to have your Ganeti cluster installed and configured before you try any of the commands in this document. Please follow the :doc:`install` for instructions on how to do that.

Instance management

Adding an instance

The add operation might seem complex due to the many parameters it accepts, but once you have understood the (few) required parameters and the customisation capabilities you will see it is an easy operation.

The add operation requires at minimum five parameters:

• the OS for the instance
• the disk template
• the disk count and size
• the node specification or alternatively the iallocator to use
• and finally the instance name

The OS for the instance must be visible in the output of the command gnt-os list and specifies which guest OS to install on the instance.

The disk template specifies what kind of storage to use as backend for the (virtual) disks presented to the instance; note that for instances with multiple virtual disks, they all must be of the same type.

The node(s) on which the instance will run can be given either manually, via the -n option, or computed automatically by Ganeti, if you have installed any iallocator script.

With the above parameters in mind, the command is:

$ gnt-instance add \
  -n %TARGET_NODE%:%SECONDARY_NODE% \
  -o %OS_TYPE% \
  -t %DISK_TEMPLATE% -s %DISK_SIZE% \
  %INSTANCE_NAME%

The instance name must be resolvable (e.g. exist in DNS) and usually points to an address in the same subnet as the cluster itself.

The above command has the minimum required options; other options you can give include, among others:

• The maximum/minimum memory size (-B maxmem, -B minmem) (-B memory can be used to specify only one size)
• The number of virtual CPUs (-B vcpus)
• Arguments for the NICs of the instance; by default, a single-NIC instance is created. The IP and/or bridge of the NIC can be changed via --nic 0:ip=IP,bridge=BRIDGE

See the manpage for gnt-instance for the detailed option list.

For example if you want to create an highly available instance, with a single disk of 50GB and the default memory size, having primary node node1 and secondary node node3, use the following command:

$ gnt-instance add -n node1:node3 -o debootstrap -t drbd -s 50G \
  instance1

There is a also a command for batch instance creation from a specification file, see the batch-create operation in the gnt-instance manual page.

Regular instance operations

Removal

Removing an instance is even easier than creating one. This operation is irreversible and destroys all the contents of your instance. Use with care:

$ gnt-instance remove %INSTANCE_NAME%

Startup/shutdown

Instances are automatically started at instance creation time. To manually start one which is currently stopped you can run:

$ gnt-instance startup %INSTANCE_NAME%

Ganeti will start an instance with up to its maximum instance memory. If not enough memory is available Ganeti will use all the available memory down to the instance minimum memory. If not even that amount of memory is free Ganeti will refuse to start the instance.

Note, that this will not work when an instance is in a permanently stopped state offline. In this case, you will first have to put it back to online mode by running:

$ gnt-instance modify --online %INSTANCE_NAME%

The command to stop the running instance is:

$ gnt-instance shutdown %INSTANCE_NAME%

If you want to shut the instance down more permanently, so that it does not require dynamically allocated resources (memory and vcpus), after shutting down an instance, execute the following:

$ gnt-instance modify --offline %INSTANCE_NAME%

Warning

Do not use the Xen or KVM commands directly to stop instances. If you run for example xm shutdown or xm destroy on an instance Ganeti will automatically restart it (via the :command:`ganeti-watcher(8)` command which is launched via cron).

Querying instances

There are two ways to get information about instances: listing instances, which does a tabular output containing a given set of fields about each instance, and querying detailed information about a set of instances.

The command to see all the instances configured and their status is:

$ gnt-instance list

The command can return a custom set of information when using the -o option (as always, check the manpage for a detailed specification). Each instance will be represented on a line, thus making it easy to parse this output via the usual shell utilities (grep, sed, etc.).

To get more detailed information about an instance, you can run:

$ gnt-instance info %INSTANCE%

which will give a multi-line block of information about the instance, it's hardware resources (especially its disks and their redundancy status), etc. This is harder to parse and is more expensive than the list operation, but returns much more detailed information.

Changing an instance's runtime memory

Ganeti will always make sure an instance has a value between its maximum and its minimum memory available as runtime memory. As of version 2.6 Ganeti will only choose a size different than the maximum size when starting up, failing over, or migrating an instance on a node with less than the maximum memory available. It won't resize other instances in order to free up space for an instance.

If you find that you need more memory on a node any instance can be manually resized without downtime, with the command:

$ gnt-instance modify -m %SIZE% %INSTANCE_NAME%

The same command can also be used to increase the memory available on an instance, provided that enough free memory is available on its node, and the specified size is not larger than the maximum memory size the instance had when it was first booted (an instance will be unable to see new memory above the maximum that was specified to the hypervisor at its boot time, if it needs to grow further a reboot becomes necessary).

Export/Import

You can create a snapshot of an instance disk and its Ganeti configuration, which then you can backup, or import into another cluster. The way to export an instance is:

$ gnt-backup export -n %TARGET_NODE% %INSTANCE_NAME%

The target node can be any node in the cluster with enough space under /srv/ganeti to hold the instance image. Use the --noshutdown option to snapshot an instance without rebooting it. Note that Ganeti only keeps one snapshot for an instance - any previous snapshot of the same instance existing cluster-wide under /srv/ganeti will be removed by this operation: if you want to keep them, you need to move them out of the Ganeti exports directory.

Importing an instance is similar to creating a new one, but additionally one must specify the location of the snapshot. The command is:

$ gnt-backup import -n %TARGET_NODE% \
  --src-node=%NODE% --src-dir=%DIR% %INSTANCE_NAME%

By default, parameters will be read from the export information, but you can of course pass them in via the command line - most of the options available for the command :command:`gnt-instance add` are supported here too.

Import of foreign instances

There is a possibility to import a foreign instance whose disk data is already stored as LVM volumes without going through copying it: the disk adoption mode.

For this, ensure that the original, non-managed instance is stopped, then create a Ganeti instance in the usual way, except that instead of passing the disk information you specify the current volumes:

$ gnt-instance add -t plain -n %HOME_NODE% ... \
  --disk 0:adopt=%lv_name%[,vg=%vg_name%] %INSTANCE_NAME%

This will take over the given logical volumes, rename them to the Ganeti standard (UUID-based), and without installing the OS on them start directly the instance. If you configure the hypervisor similar to the non-managed configuration that the instance had, the transition should be seamless for the instance. For more than one disk, just pass another disk parameter (e.g. --disk 1:adopt=...).

Instance kernel selection

The kernel that instances uses to bootup can come either from the node, or from instances themselves, depending on the setup.

Xen-PVM

With Xen PVM, there are three options.

First, you can use a kernel from the node, by setting the hypervisor parameters as such:

• kernel_path to a valid file on the node (and appropriately initrd_path)
• kernel_args optionally set to a valid Linux setting (e.g. ro)
• root_path to a valid setting (e.g. /dev/xvda1)
• bootloader_path and bootloader_args to empty

Alternatively, you can delegate the kernel management to instances, and use either pvgrub or the deprecated pygrub. For this, you must install the kernels and initrds in the instance and create a valid GRUB v1 configuration file.

For pvgrub (new in version 2.4.2), you need to set:

• kernel_path to point to the pvgrub loader present on the node (e.g. /usr/lib/xen/boot/pv-grub-x86_32.gz)
• kernel_args to the path to the GRUB config file, relative to the instance (e.g. (hd0,0)/grub/menu.lst)
• root_path must be empty
• bootloader_path and bootloader_args to empty

While pygrub is deprecated, here is how you can configure it:

• bootloader_path to the pygrub binary (e.g. /usr/bin/pygrub)
• the other settings are not important

More information can be found in the Xen wiki pages for pvgrub and pygrub.

KVM

For KVM also the kernel can be loaded either way.

For loading the kernels from the node, you need to set:

• kernel_path to a valid value
• initrd_path optionally set if you use an initrd
• kernel_args optionally set to a valid value (e.g. ro)

If you want instead to have the instance boot from its disk (and execute its bootloader), simply set the kernel_path parameter to an empty string, and all the others will be ignored.

Instance HA features

Note

This section only applies to multi-node clusters

Changing the primary node

There are three ways to exchange an instance's primary and secondary nodes; the right one to choose depends on how the instance has been created and the status of its current primary node. See :ref:`rest-redundancy-label` for information on changing the secondary node. Note that it's only possible to change the primary node to the secondary and vice-versa; a direct change of the primary node with a third node, while keeping the current secondary is not possible in a single step, only via multiple operations as detailed in :ref:`instance-relocation-label`.

Failing over an instance

If an instance is built in highly available mode you can at any time fail it over to its secondary node, even if the primary has somehow failed and it's not up anymore. Doing it is really easy, on the master node you can just run:

$ gnt-instance failover %INSTANCE_NAME%

That's it. After the command completes the secondary node is now the primary, and vice-versa.

The instance will be started with an amount of memory between its maxmem and its minmem value, depending on the free memory on its target node, or the operation will fail if that's not possible. See :ref:`instance-startup-label` for details.

If the instance's disk template is of type rbd, then you can specify the target node (which can be any node) explicitly, or specify an iallocator plugin. If you omit both, the default iallocator will be used to determine the target node:

$ gnt-instance failover -n %TARGET_NODE% %INSTANCE_NAME%

Live migrating an instance

If an instance is built in highly available mode, it currently runs and both its nodes are running fine, you can migrate it over to its secondary node, without downtime. On the master node you need to run:

$ gnt-instance migrate %INSTANCE_NAME%

The current load on the instance and its memory size will influence how long the migration will take. In any case, for both KVM and Xen hypervisors, the migration will be transparent to the instance.

If the destination node has less memory than the instance's current runtime memory, but at least the instance's minimum memory available Ganeti will automatically reduce the instance runtime memory before migrating it, unless the --no-runtime-changes option is passed, in which case the target node should have at least the instance's current runtime memory free.

If the instance's disk template is of type rbd, then you can specify the target node (which can be any node) explicitly, or specify an iallocator plugin. If you omit both, the default iallocator will be used to determine the target node:

$ gnt-instance migrate -n %TARGET_NODE% %INSTANCE_NAME%

Moving an instance (offline)

If an instance has not been create as mirrored, then the only way to change its primary node is to execute the move command:

$ gnt-instance move -n %NEW_NODE% %INSTANCE%

This has a few prerequisites:

• the instance must be stopped
• its current primary node must be on-line and healthy
• the disks of the instance must not have any errors

Since this operation actually copies the data from the old node to the new node, expect it to take proportional to the size of the instance's disks and the speed of both the nodes' I/O system and their networking.

Disk operations

Disk failures are a common cause of errors in any server deployment. Ganeti offers protection from single-node failure if your instances were created in HA mode, and it also offers ways to restore redundancy after a failure.

Preparing for disk operations

It is important to note that for Ganeti to be able to do any disk operation, the Linux machines on top of which Ganeti runs must be consistent; for LVM, this means that the LVM commands must not return failures; it is common that after a complete disk failure, any LVM command aborts with an error similar to:

$ vgs
/dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 't30jmN-4Rcf-Fr5e-CURS-pawt-z0jU-m1TgeJ'.
Couldn't find all physical volumes for volume group xenvg.

Before restoring an instance's disks to healthy status, it's needed to fix the volume group used by Ganeti so that we can actually create and manage the logical volumes. This is usually done in a multi-step process:

1. first, if the disk is completely gone and LVM commands exit with “Couldn't find device with uuid…” then you need to run the command:

   $ vgreduce --removemissing %VOLUME_GROUP%

2. after the above command, the LVM commands should be executing normally (warnings are normal, but the commands will not fail completely).

3. if the failed disk is still visible in the output of the pvs command, you need to deactivate it from allocations by running:

   $ pvs -x n /dev/%DISK%

At this point, the volume group should be consistent and any bad physical volumes should not longer be available for allocation.

Note that since version 2.1 Ganeti provides some commands to automate these two operations, see :ref:`storage-units-label`.

Restoring redundancy for DRBD-based instances

A DRBD instance has two nodes, and the storage on one of them has failed. Depending on which node (primary or secondary) has failed, you have three options at hand:

• if the storage on the primary node has failed, you need to re-create the disks on it
• if the storage on the secondary node has failed, you can either re-create the disks on it or change the secondary and recreate redundancy on the new secondary node

Of course, at any point it's possible to force re-creation of disks even though everything is already fine.

For all three cases, the replace-disks operation can be used:

# re-create disks on the primary node
$ gnt-instance replace-disks -p %INSTANCE_NAME%
# re-create disks on the current secondary
$ gnt-instance replace-disks -s %INSTANCE_NAME%
# change the secondary node, via manual specification
$ gnt-instance replace-disks -n %NODE% %INSTANCE_NAME%
# change the secondary node, via an iallocator script
$ gnt-instance replace-disks -I %SCRIPT% %INSTANCE_NAME%
# since Ganeti 2.1: automatically fix the primary or secondary node
$ gnt-instance replace-disks -a %INSTANCE_NAME%

Since the process involves copying all data from the working node to the target node, it will take a while, depending on the instance's disk size, node I/O system and network speed. But it is (barring any network interruption) completely transparent for the instance.

Re-creating disks for non-redundant instances

For non-redundant instances, there isn't a copy (except backups) to re-create the disks. But it's possible to at-least re-create empty disks, after which a reinstall can be run, via the recreate-disks command: