-
Guido Trotter authored
As we know, those are just defaults, mostly chosen by replicating the only behavior we supported before allowing customization. They may need changes, to work in specific environments (which is why we introduced them in the first place). This tries to address issue 83, which was caused by the default parameters not being correct for the target cluster. Signed-off-by:
Guido Trotter <ultrotter@google.com> Reviewed-by:
Iustin Pop <iustin@google.com>
b8313b29
Ganeti installation tutorial
Documents Ganeti version |version|
Contents
Introduction
Ganeti is a cluster virtualization management system based on Xen or KVM. This document explains how to bootstrap a Ganeti node (Xen dom0, the host Linux system for KVM), create a running cluster and install virtual instances (Xen domUs, KVM guests). You need to repeat most of the steps in this document for every node you want to install, but of course we recommend creating some semi-automatic procedure if you plan to deploy Ganeti on a medium/large scale.
A basic Ganeti terminology glossary is provided in the introductory section of the :doc:`admin`. Please refer to that document if you are uncertain about the terms we are using.
Ganeti has been developed for Linux and should be distribution-agnostic. This documentation will use Debian Lenny as an example system but the examples can be translated to any other distribution. You are expected to be familiar with your distribution, its package management system, and Xen or KVM before trying to use Ganeti.
This document is divided into two main sections:
- Installation of the base system and base components
- Configuration of the environment for Ganeti
Each of these is divided into sub-sections. While a full Ganeti system will need all of the steps specified, some are not strictly required for every environment. Which ones they are, and why, is specified in the corresponding sections.
Installing the base system and base components
Hardware requirements
Any system supported by your Linux distribution is fine. 64-bit systems are better as they can support more memory.
Any disk drive recognized by Linux (IDE/SCSI/SATA/etc.) is
supported in Ganeti. Note that no shared storage (e.g. SAN) is
needed to get high-availability features (but of course, one can be used
to store the images). It is highly recommended to use more than one disk
drive to improve speed. But Ganeti also works with one disk per machine.
Installing the base system
Mandatory on all nodes.
It is advised to start with a clean, minimal install of the operating
system. The only requirement you need to be aware of at this stage is to
partition leaving enough space for a big (minimum 20GiB) LVM volume
group which will then host your instance filesystems, if you want to use
all Ganeti features. The volume group name Ganeti uses (by default) is
xenvg.
You can also use file-based storage only, without LVM, but this setup is not detailed in this document.
While you can use an existing system, please note that the Ganeti installation is intrusive in terms of changes to the system configuration, and it's best to use a newly-installed system without important data on it.
Also, for best results, it's advised that the nodes have as much as possible the same hardware and software configuration. This will make administration much easier.
Hostname issues
Note that Ganeti requires the hostnames of the systems (i.e. what the
hostname command outputs to be a fully-qualified name, not a short
name. In other words, you should use node1.example.com as a hostname
and not just node1.
Debian
Debian Lenny and Etch configures the hostname differently than you
need it for Ganeti. For example, this is what Etch puts in
/etc/hosts in certain situations:
127.0.0.1 localhost
127.0.1.1 node1.example.com node1but for Ganeti you need to have:
127.0.0.1 localhost
192.168.1.1 node1.example.com node1replacing 192.168.1.1 with your node's address. Also, the file
/etc/hostname which configures the hostname of the system
should contain node1.example.com and not just node1 (you
need to run the command /etc/init.d/hostname.sh start after
changing the file).
Why a fully qualified host name
Although most distributions use only the short name in the /etc/hostname file, we still think Ganeti nodes should use the full name. The reason for this is that calling 'hostname --fqdn' requires the resolver library to work and is a 'guess' via heuristics at what is your domain name. Since Ganeti can be used among other things to host DNS servers, we don't want to depend on them as much as possible, and we'd rather have the uname() syscall return the full node name.
We haven't ever found any breakage in using a full hostname on a Linux system, and anyway we recommend to have only a minimal installation on Ganeti nodes, and to use instances (or other dedicated machines) to run the rest of your network services. By doing this you can change the /etc/hostname file to contain an FQDN without the fear of breaking anything unrelated.
Installing The Hypervisor
Mandatory on all nodes.
While Ganeti is developed with the ability to modularly run on different virtualization environments in mind the only two currently useable on a live system are Xen and KVM. Supported Xen versions are: 3.0.3, 3.0.4 and 3.1. Supported KVM version are 72 and above.
Please follow your distribution's recommended way to install and set up Xen, or install Xen from the upstream source, if you wish, following their manual. For KVM, make sure you have a KVM-enabled kernel and the KVM tools.
After installing Xen, you need to reboot into your new system. On some distributions this might involve configuring GRUB appropriately, whereas others will configure it automatically when you install the respective kernels. For KVM no reboot should be necessary.
Xen on Debian
Under Lenny or Etch you can install the relevant xen-linux-system
package, which will pull in both the hypervisor and the relevant
kernel. Also, if you are installing a 32-bit Lenny/Etch, you should
install the libc6-xen package (run apt-get install
libc6-xen).
Xen settings
It's recommended that dom0 is restricted to a low amount of memory
(512MiB or 1GiB is reasonable) and that memory ballooning is disabled in
the file /etc/xen/xend-config.sxp by setting the value
dom0-min-mem to 0, like this:
(dom0-min-mem 0)For optimum performance when running both CPU and I/O intensive
instances, it's also recommended that the dom0 is restricted to one CPU
only, for example by booting with the kernel parameter nosmp.
It is recommended that you disable xen's automatic save of virtual
machines at system shutdown and subsequent restore of them at reboot.
To obtain this make sure the variable XENDOMAINS_SAVE in the file
/etc/default/xendomains is set to an empty value.
If you want to use live migration make sure you have, in the xen config file, something that allows the nodes to migrate instances between each other. For example:
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address '')
(xend-relocation-hosts-allow '^192\\.168\\.3\\.[0-9]+$')The second line assumess that the hypervisor parameter
migration_port is set 8002, otherwise modify it to match. The last
line assumes that all your nodes have secondary IPs in the
192.168.3.0/24 network, adjust it accordingly to your setup.
Debian
Besides the ballooning change which you need to set in
/etc/xen/xend-config.sxp, you need to set the memory and nosmp
parameters in the file /boot/grub/menu.lst. You need to modify
the variable xenhopt to add dom0_mem=1024M like this:
## Xen hypervisor options to use with the default Xen boot option
# xenhopt=dom0_mem=1024Mand the xenkopt needs to include the nosmp option like this:
## Xen Linux kernel options to use with the default Xen boot option
# xenkopt=nosmpAny existing parameters can be left in place: it's ok to have
xenkopt=console=tty0 nosmp, for example. After modifying the
files, you need to run:
/sbin/update-grubIf you want to run HVM instances too with Ganeti and want VNC access to
the console of your instances, set the following two entries in
/etc/xen/xend-config.sxp:
(vnc-listen '0.0.0.0') (vncpasswd '')You need to restart the Xen daemon for these settings to take effect:
/etc/init.d/xend restartSelecting the instance kernel
After you have installed Xen, you need to tell Ganeti exactly what
kernel to use for the instances it will create. This is done by creating
a symlink from your actual kernel to /boot/vmlinuz-2.6-xenU, and one
from your initrd to /boot/initrd-2.6-xenU [1]. Note that
if you don't use an initrd for the domU kernel, you don't need to create
the initrd symlink.
Debian
After installation of the xen-linux-system package, you need to
run (replace the exact version number with the one you have):
cd /boot
ln -s vmlinuz-2.6.26-1-xen-amd64 vmlinuz-2.6-xenU
ln -s initrd.img-2.6.26-1-xen-amd64 initrd-2.6-xenUInstalling DRBD
Recommended on all nodes: DRBD is required if you want to use the high availability (HA) features of Ganeti, but optional if you don't require them or only run Ganeti on single-node clusters. You can upgrade a non-HA cluster to an HA one later, but you might need to export and re-import all your instances to take advantage of the new features.
Supported DRBD versions: 8.0+. It's recommended to have at least version
8.0.12. Note that for version 8.2 and newer it is needed to pass the
usermode_helper=/bin/true parameter to the module, either by
configuring /etc/modules or when inserting it manually.
Now the bad news: unless your distribution already provides it installing DRBD might involve recompiling your kernel or anyway fiddling with it. Hopefully at least the Xen-ified kernel source to start from will be provided (if you intend to use Xen).
The good news is that you don't need to configure DRBD at all. Ganeti
will do it for you for every instance you set up. If you have the DRBD
utils installed and the module in your kernel you're fine. Please check
that your system is configured to load the module at every boot, and
that it passes the following option to the module:
minor_count=NUMBER. We recommend that you use 128 as the value of
the minor_count - this will allow you to use up to 64 instances in total
per node (both primary and secondary, when using only one disk per
instance). You can increase the number up to 255 if you need more
instances on a node.
Debian
On Debian, you can just install (build) the DRBD module with the following commands, making sure you are running the target (Xen or KVM) kernel:
apt-get install drbd8-source drbd8-utils
m-a update
m-a a-i drbd8
echo drbd minor_count=128 usermode_helper=/bin/true >> /etc/modules
depmod -a
modprobe drbd minor_count=128 usermode_helper=/bin/trueIt is also recommended that you comment out the default resources in
the /etc/drbd.conf file, so that the init script doesn't try to
configure any drbd devices. You can do this by prefixing all
resource lines in the file with the keyword skip, like this:
skip resource r0 {
...
}
skip resource "r1" {
...
}Other required software
See :doc:`install-quick`.
Setting up the environment for Ganeti
Configuring the network
Mandatory on all nodes.
You can run Ganeti either in "bridge mode" or in "routed mode". In bridge mode, the default, the instances network interfaces will be attached to a software bridge running in dom0. Xen by default creates such a bridge at startup, but your distribution might have a different way to do things, and you'll definitely need to manually set it up under KVM.
Beware that the default name Ganeti uses is xen-br0 (which was used
in Xen 2.0) while Xen 3.0 uses xenbr0 by default. The default bridge
your Ganeti cluster will use for new instances can be specified at
cluster initialization time.
If you want to run in "routing mode" you need to specify that at cluster init time (using the --nicparam option), and then no bridge will be needed. In this mode instance traffic will be routed by dom0, instead of bridged.
In order to use "routing mode" under Xen, you'll need to change the relevant parameters in the Xen config file. Under KVM instead, no config change is necessary, but you still need to set up your network interfaces correctly.
By default, under KVM, the "link" parameter you specify per-nic will represent, if non-empty, a different routing table name or number to use for your instances. This allows insulation between different instance groups, and different routing policies between node traffic and instance traffic.
You will need to configure your routing table basic routes and rules outside of ganeti. The vif scripts will only add /32 routes to your instances, through their interface, in the table you specified (under KVM, and in the main table under Xen).
Bridging under Debian
The recommended way to configure the Xen bridge is to edit your
/etc/network/interfaces file and substitute your normal
Ethernet stanza with the following snippet:
auto xen-br0
iface xen-br0 inet static
address YOUR_IP_ADDRESS
netmask YOUR_NETMASK
network YOUR_NETWORK
broadcast YOUR_BROADCAST_ADDRESS
gateway YOUR_GATEWAY
bridge_ports eth0
bridge_stp off
bridge_fd 0The following commands need to be executed on the local console:
ifdown eth0 ifup xen-br0
To check if the bridge is setup, use the ip and brctl show
commands:
# ip a show xen-br0
9: xen-br0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc noqueue
link/ether 00:20:fc:1e:d5:5d brd ff:ff:ff:ff:ff:ff
inet 10.1.1.200/24 brd 10.1.1.255 scope global xen-br0
inet6 fe80::220:fcff:fe1e:d55d/64 scope link
valid_lft forever preferred_lft forever
# brctl show xen-br0
bridge name bridge id STP enabled interfaces
xen-br0 8000.0020fc1ed55d no eth0