install.rst 31.6 KB
Newer Older
1 2 3
Ganeti installation tutorial
============================

Iustin Pop's avatar
Iustin Pop committed
4
Documents Ganeti version |version|
5 6 7

.. contents::

Iustin Pop's avatar
Iustin Pop committed
8
.. highlight:: shell-example
Iustin Pop's avatar
Iustin Pop committed
9

10 11 12 13
Introduction
------------

Ganeti is a cluster virtualization management system based on Xen or
Iustin Pop's avatar
Iustin Pop committed
14 15 16 17 18 19
KVM. This document explains how to bootstrap a Ganeti node (Xen *dom0*,
the host Linux system for KVM), create a running cluster and install
virtual instances (Xen *domUs*, KVM guests).  You need to repeat most of
the steps in this document for every node you want to install, but of
course we recommend creating some semi-automatic procedure if you plan
to deploy Ganeti on a medium/large scale.
20 21

A basic Ganeti terminology glossary is provided in the introductory
Iustin Pop's avatar
Iustin Pop committed
22 23
section of the :doc:`admin`. Please refer to that document if you are
uncertain about the terms we are using.
24

Iustin Pop's avatar
Iustin Pop committed
25
Ganeti has been developed for Linux and should be distribution-agnostic.
Iustin Pop's avatar
Iustin Pop committed
26
This documentation will use Debian Squeeze as an example system but the
Iustin Pop's avatar
Iustin Pop committed
27 28 29
examples can be translated to any other distribution. You are expected
to be familiar with your distribution, its package management system,
and Xen or KVM before trying to use Ganeti.
30 31 32 33 34 35 36 37

This document is divided into two main sections:

- Installation of the base system and base components

- Configuration of the environment for Ganeti

Each of these is divided into sub-sections. While a full Ganeti system
Iustin Pop's avatar
Iustin Pop committed
38 39 40
will need all of the steps specified, some are not strictly required for
every environment. Which ones they are, and why, is specified in the
corresponding sections.
41 42 43 44 45 46 47

Installing the base system and base components
----------------------------------------------

Hardware requirements
+++++++++++++++++++++

Iustin Pop's avatar
Iustin Pop committed
48 49
Any system supported by your Linux distribution is fine. 64-bit systems
are better as they can support more memory.
50

Iustin Pop's avatar
Iustin Pop committed
51 52 53
Any disk drive recognized by Linux (``IDE``/``SCSI``/``SATA``/etc.) is
supported in Ganeti. Note that no shared storage (e.g. ``SAN``) is
needed to get high-availability features (but of course, one can be used
Iustin Pop's avatar
Iustin Pop committed
54 55 56
to store the images). Whilte it is highly recommended to use more than
one disk drive in order to improve speed, Ganeti also works with one
disk per machine.
57 58 59 60 61 62 63

Installing the base system
++++++++++++++++++++++++++

**Mandatory** on all nodes.

It is advised to start with a clean, minimal install of the operating
Iustin Pop's avatar
Iustin Pop committed
64 65 66 67 68
system. The only requirement you need to be aware of at this stage is to
partition leaving enough space for a big (**minimum** 20GiB) LVM volume
group which will then host your instance filesystems, if you want to use
all Ganeti features. The volume group name Ganeti uses (by default) is
``xenvg``.
69

Iustin Pop's avatar
Iustin Pop committed
70 71
You can also use file-based storage only, without LVM, but this setup is
not detailed in this document.
72

73
If you choose to use RBD-based instances, there's no need for LVM
Iustin Pop's avatar
Iustin Pop committed
74
provisioning. However, this feature is experimental, and is not yet
75 76
recommended for production clusters.

77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
While you can use an existing system, please note that the Ganeti
installation is intrusive in terms of changes to the system
configuration, and it's best to use a newly-installed system without
important data on it.

Also, for best results, it's advised that the nodes have as much as
possible the same hardware and software configuration. This will make
administration much easier.

Hostname issues
~~~~~~~~~~~~~~~

Note that Ganeti requires the hostnames of the systems (i.e. what the
``hostname`` command outputs to be a fully-qualified name, not a short
name. In other words, you should use *node1.example.com* as a hostname
and not just *node1*.

.. admonition:: Debian

Iustin Pop's avatar
Iustin Pop committed
96 97 98
   Debian usually configures the hostname differently than you need it
   for Ganeti. For example, this is what it puts in ``/etc/hosts`` in
   certain situations::
99 100 101 102 103 104 105

     127.0.0.1       localhost
     127.0.1.1       node1.example.com node1

   but for Ganeti you need to have::

     127.0.0.1       localhost
Iustin Pop's avatar
Iustin Pop committed
106
     192.0.2.1       node1.example.com node1
107

108
   replacing ``192.0.2.1`` with your node's address. Also, the file
109 110 111 112 113
   ``/etc/hostname`` which configures the hostname of the system
   should contain ``node1.example.com`` and not just ``node1`` (you
   need to run the command ``/etc/init.d/hostname.sh start`` after
   changing the file).

114 115
.. admonition:: Why a fully qualified host name

116 117 118 119 120 121 122 123 124 125 126 127 128 129 130
   Although most distributions use only the short name in the
   /etc/hostname file, we still think Ganeti nodes should use the full
   name. The reason for this is that calling 'hostname --fqdn' requires
   the resolver library to work and is a 'guess' via heuristics at what
   is your domain name. Since Ganeti can be used among other things to
   host DNS servers, we don't want to depend on them as much as
   possible, and we'd rather have the uname() syscall return the full
   node name.

   We haven't ever found any breakage in using a full hostname on a
   Linux system, and anyway we recommend to have only a minimal
   installation on Ganeti nodes, and to use instances (or other
   dedicated machines) to run the rest of your network services. By
   doing this you can change the /etc/hostname file to contain an FQDN
   without the fear of breaking anything unrelated.
131 132


133 134
Installing The Hypervisor
+++++++++++++++++++++++++
135 136 137

**Mandatory** on all nodes.

138
While Ganeti is developed with the ability to modularly run on different
139
virtualization environments in mind the only two currently useable on a
Iustin Pop's avatar
Iustin Pop committed
140 141 142
live system are Xen and KVM. Supported Xen versions are: 3.0.3 and later
3.x versions, and 4.x (tested up to 4.1).  Supported KVM versions are 72
and above.
143

Iustin Pop's avatar
Iustin Pop committed
144 145 146 147
Please follow your distribution's recommended way to install and set up
Xen, or install Xen from the upstream source, if you wish, following
their manual. For KVM, make sure you have a KVM-enabled kernel and the
KVM tools.
148

149
After installing Xen, you need to reboot into your new system. On some
150 151 152
distributions this might involve configuring GRUB appropriately, whereas
others will configure it automatically when you install the respective
kernels. For KVM no reboot should be necessary.
153

154
.. admonition:: Xen on Debian
155

Iustin Pop's avatar
Iustin Pop committed
156
   Under Debian you can install the relevant ``xen-linux-system``
Iustin Pop's avatar
Iustin Pop committed
157
   package, which will pull in both the hypervisor and the relevant
Iustin Pop's avatar
Iustin Pop committed
158
   kernel. Also, if you are installing a 32-bit system, you should
Iustin Pop's avatar
Iustin Pop committed
159 160
   install the ``libc6-xen`` package (run ``apt-get install
   libc6-xen``).
161 162 163 164 165

Xen settings
~~~~~~~~~~~~

It's recommended that dom0 is restricted to a low amount of memory
Iustin Pop's avatar
Iustin Pop committed
166 167 168
(512MiB or 1GiB is reasonable) and that memory ballooning is disabled in
the file ``/etc/xen/xend-config.sxp`` by setting the value
``dom0-min-mem`` to 0, like this::
169 170 171 172

  (dom0-min-mem 0)

For optimum performance when running both CPU and I/O intensive
Iustin Pop's avatar
Iustin Pop committed
173
instances, it's also recommended that the dom0 is restricted to one CPU
174 175 176 177 178
only. For example you can add ``dom0_max_vcpus=1,dom0_vcpus_pin`` to your
kernels boot command line and set ``dom0-cpus`` in
``/etc/xen/xend-config.sxp`` like this::

  (dom0-cpus 1)
179 180 181 182 183 184

It is recommended that you disable xen's automatic save of virtual
machines at system shutdown and subsequent restore of them at reboot.
To obtain this make sure the variable ``XENDOMAINS_SAVE`` in the file
``/etc/default/xendomains`` is set to an empty value.

Guido Trotter's avatar
Guido Trotter committed
185 186
If you want to use live migration make sure you have, in the xen config
file, something that allows the nodes to migrate instances between each
Iustin Pop's avatar
Iustin Pop committed
187 188 189
other. For example:

.. code-block:: text
190 191 192 193

  (xend-relocation-server yes)
  (xend-relocation-port 8002)
  (xend-relocation-address '')
194
  (xend-relocation-hosts-allow '^192\\.0\\.2\\.[0-9]+$')
195

196

197
The second line assumes that the hypervisor parameter
198 199
``migration_port`` is set 8002, otherwise modify it to match. The last
line assumes that all your nodes have secondary IPs in the
200
192.0.2.0/24 network, adjust it accordingly to your setup.
201

Iustin Pop's avatar
Iustin Pop committed
202 203
If you want to run HVM instances too with Ganeti and want VNC access to
the console of your instances, set the following two entries in
Iustin Pop's avatar
Iustin Pop committed
204 205 206
``/etc/xen/xend-config.sxp``:

.. code-block:: text
207 208 209 210 211

  (vnc-listen '0.0.0.0') (vncpasswd '')

You need to restart the Xen daemon for these settings to take effect::

Iustin Pop's avatar
Iustin Pop committed
212
  $ /etc/init.d/xend restart
213 214 215 216 217

Selecting the instance kernel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

After you have installed Xen, you need to tell Ganeti exactly what
Iustin Pop's avatar
Iustin Pop committed
218
kernel to use for the instances it will create. This is done by creating
Iustin Pop's avatar
Iustin Pop committed
219 220
a symlink from your actual kernel to ``/boot/vmlinuz-3-xenU``, and one
from your initrd to ``/boot/initrd-3-xenU`` [#defkernel]_. Note that
Iustin Pop's avatar
Iustin Pop committed
221 222
if you don't use an initrd for the domU kernel, you don't need to create
the initrd symlink.
223 224 225 226 227 228

.. admonition:: Debian

   After installation of the ``xen-linux-system`` package, you need to
   run (replace the exact version number with the one you have)::

Iustin Pop's avatar
Iustin Pop committed
229 230 231 232 233 234 235 236 237 238
     $ cd /boot
     $ ln -s vmlinuz-%2.6.26-1%-xen-amd64 vmlinuz-3-xenU
     $ ln -s initrd.img-%2.6.26-1%-xen-amd64 initrd-3-xenU

   By default, the initrd doesn't contain the Xen block drivers needed
   to mount the root device, so it is recommended to update the initrd
   by following these two steps:

   - edit ``/etc/initramfs-tools/modules`` and add ``xen_blkfront``
   - run ``update-initramfs -u``
239 240 241 242

Installing DRBD
+++++++++++++++

Iustin Pop's avatar
Iustin Pop committed
243 244 245
Recommended on all nodes: DRBD_ is required if you want to use the high
availability (HA) features of Ganeti, but optional if you don't require
them or only run Ganeti on single-node clusters. You can upgrade a
Iustin Pop's avatar
Iustin Pop committed
246 247
non-HA cluster to an HA one later, but you might need to convert all
your instances to DRBD to take advantage of the new features.
248 249 250

.. _DRBD: http://www.drbd.org/

251
Supported DRBD versions: 8.0-8.4. It's recommended to have at least
Iustin Pop's avatar
Iustin Pop committed
252 253
version 8.0.12. Note that for version 8.2 and newer it is needed to pass
the ``usermode_helper=/bin/true`` parameter to the module, either by
254 255 256 257 258
configuring ``/etc/modules`` or when inserting it manually. When using
Xen and DRBD 8.3.2 or higher, it is recommended_ to use the
``disable_sendpage=1`` setting as well.

.. _recommended: https://drbd.linbit.com/users-guide/s-xen-drbd-mod-params.html
259 260

Now the bad news: unless your distribution already provides it
Iustin Pop's avatar
Iustin Pop committed
261 262 263
installing DRBD might involve recompiling your kernel or anyway fiddling
with it. Hopefully at least the Xen-ified kernel source to start from
will be provided (if you intend to use Xen).
264 265

The good news is that you don't need to configure DRBD at all. Ganeti
Iustin Pop's avatar
Iustin Pop committed
266 267 268 269 270 271 272 273 274 275
will do it for you for every instance you set up.  If you have the DRBD
utils installed and the module in your kernel you're fine. Please check
that your system is configured to load the module at every boot, and
that it passes the following option to the module:
``minor_count=NUMBER``. We recommend that you use 128 as the value of
the minor_count - this will allow you to use up to 64 instances in total
per node (both primary and secondary, when using only one disk per
instance). You can increase the number up to 255 if you need more
instances on a node.

276 277 278

.. admonition:: Debian

Iustin Pop's avatar
Iustin Pop committed
279 280 281
   On Debian, you can just install (build) the DRBD module with the
   following commands, making sure you are running the target (Xen or
   KVM) kernel::
282

Iustin Pop's avatar
Iustin Pop committed
283 284 285
     $ apt-get install drbd8-source drbd8-utils
     $ m-a update
     $ m-a a-i drbd8
286 287 288 289 290 291 292

   Or on newer versions, if the kernel already has modules:

     $ apt-get install drbd8-utils

   Then to configure it for Ganeti::

293
     $ echo "options drbd minor_count=128 usermode_helper=/bin/true" \
294
        > /etc/modprobe.d/drbd.conf
295
     $ echo "drbd" >> /etc/modules
Iustin Pop's avatar
Iustin Pop committed
296
     $ depmod -a
297
     $ modprobe drbd
298

299 300
   It is also recommended that you comment out the default resources (if any)
   in the ``/etc/drbd.conf`` file, so that the init script doesn't try to
Iustin Pop's avatar
Iustin Pop committed
301
   configure any drbd devices. You can do this by prefixing all
Iustin Pop's avatar
Iustin Pop committed
302 303 304
   *resource* lines in the file with the keyword *skip*, like this:

   .. code-block:: text
305

306 307 308 309
     skip {
       resource r0 {
         ...
       }
310 311
     }

312 313 314 315
     skip {
       resource "r1" {
         ...
       }
316 317
     }

318
Installing RBD
Iustin Pop's avatar
Iustin Pop committed
319
++++++++++++++
320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373

Recommended on all nodes: RBD_ is required if you want to create
instances with RBD disks residing inside a RADOS cluster (make use of
the rbd disk template). RBD-based instances can failover or migrate to
any other node in the ganeti cluster, enabling you to exploit of all
Ganeti's high availabilily (HA) features.

.. attention::
   Be careful though: rbd is still experimental! For now it is
   recommended only for testing purposes.  No sensitive data should be
   stored there.

.. _RBD: http://ceph.newdream.net/

You will need the ``rbd`` and ``libceph`` kernel modules, the RBD/Ceph
userspace utils (ceph-common Debian package) and an appropriate
Ceph/RADOS configuration file on every VM-capable node.

You will also need a working RADOS Cluster accessible by the above
nodes.

RADOS Cluster
~~~~~~~~~~~~~

You will need a working RADOS Cluster accesible by all VM-capable nodes
to use the RBD template. For more information on setting up a RADOS
Cluster, refer to the `official docs <http://ceph.newdream.net/>`_.

If you want to use a pool for storing RBD disk images other than the
default (``rbd``), you should first create the pool in the RADOS
Cluster, and then set the corresponding rbd disk parameter named
``pool``.

Kernel Modules
~~~~~~~~~~~~~~

Unless your distribution already provides it, you might need to compile
the ``rbd`` and ``libceph`` modules from source. You will need Linux
Kernel 3.2 or above for the kernel modules. Alternatively you will have
to build them as external modules (from Linux Kernel source 3.2 or
above), if you want to run a less recent kernel, or your kernel doesn't
include them.

Userspace Utils
~~~~~~~~~~~~~~~

The RBD template has been tested with ``ceph-common`` v0.38 and
above. We recommend using the latest version of ``ceph-common``.

.. admonition:: Debian

   On Debian, you can just install the RBD/Ceph userspace utils with
   the following command::

Iustin Pop's avatar
Iustin Pop committed
374
      $ apt-get install ceph-common
375

376 377 378 379 380 381 382 383 384 385 386
KVM userspace access
~~~~~~~~~~~~~~~~~~~~

If your cluster uses a sufficiently new version of KVM (you will need at
least QEMU 0.14 with RBD support compiled in), you can take advantage of
KVM's native support for ceph in order to have better performance and
avoid potential deadlocks_ in low memory scenarios.

.. _deadlocks: http://tracker.ceph.com/issues/3076

To initialize a cluster with support for this feature, use a command
387 388 389
like the following. Note, that you possibly need to follow the more
general installation instructions before invoking this command (see
`Initializing the cluster`_ ).
390 391

  $ gnt-cluster init \
392 393
      --enabled-disk-templates=rbd \
      --ipolicy-disk-templates=rbd \
394 395 396 397 398 399 400 401 402
      --enabled-hypervisors=kvm \
      -D rbd:access=userspace

(You may want to enable more templates than just ``rbd``.)

You can also change this setting on a live cluster by giving the same
switches to ``gnt-cluster modify``, or change those settings at the node
group level with ``gnt-group modify``.

403 404 405 406 407 408 409 410 411
Configuration file
~~~~~~~~~~~~~~~~~~

You should also provide an appropriate configuration file
(``ceph.conf``) in ``/etc/ceph``. For the rbd userspace utils, you'll
only need to specify the IP addresses of the RADOS Cluster monitors.

.. admonition:: ceph.conf

Iustin Pop's avatar
Iustin Pop committed
412 413 414
   Sample configuration file:

   .. code-block:: text
415 416 417 418 419 420 421 422 423 424 425 426 427 428

    [mon.a]
           host = example_monitor_host1
           mon addr = 1.2.3.4:6789
    [mon.b]
           host = example_monitor_host2
           mon addr = 1.2.3.5:6789
    [mon.c]
           host = example_monitor_host3
           mon addr = 1.2.3.6:6789

For more information, please see the `Ceph Docs
<http://ceph.newdream.net/docs/latest/>`_

429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451
Installing Gluster
++++++++++++++++++

For Gluster integration, Ganeti requires that ``mount.glusterfs`` is
installed on each and every node. On Debian Wheezy and newer, you can
satisfy this requirement with the ``glusterfs-client`` package; see
`this guide
<http://gluster.org/community/documentation/index.php/Gluster_3.2:_Installing_the_Gluster_Native_Client>`_
for details.

KVM userspace access
~~~~~~~~~~~~~~~~~~~~

If your cluster uses a sufficiently new version of KVM (you will need at
least QEMU 1.3 with Gluster support compiled in), you can take advantage
of KVM's native support for gluster in order to have better performance
and avoid potential deadlocks in low memory scenarios.

Please be aware that QEMU 1.3 was released in December 3, 2012, and as
such this feature is not available out of the box in any distribution
older than Ubuntu 13.04; this excludes Ubuntu 12.04 LTS and Debian
Wheezy.

452 453 454
Other required software
+++++++++++++++++++++++

455 456 457
Please install all software requirements mentioned in :doc:`install-quick`.
If you want to build Ganeti from source, don't forget to follow the steps
required for that as well.
458 459 460 461 462 463 464 465 466

Setting up the environment for Ganeti
-------------------------------------

Configuring the network
+++++++++++++++++++++++

**Mandatory** on all nodes.

467 468 469 470 471 472
You can run Ganeti either in "bridged mode", "routed mode" or
"openvswitch mode". In bridged mode, the default, the instances network
interfaces will be attached to a software bridge running in dom0. Xen by
default creates such a bridge at startup, but your distribution might
have a different way to do things, and you'll definitely need to
manually set it up under KVM.
473

Iustin Pop's avatar
Iustin Pop committed
474
Beware that the default name Ganeti uses is ``xen-br0`` (which was used
475 476 477
in Xen 2.0) while Xen 3.0 uses ``xenbr0`` by default. See the
`Initializing the cluster`_ section to learn how to choose a different
bridge, or not to use one at all and use "routed mode".
478

479
In order to use "routed mode" under Xen, you'll need to change the
480 481 482
relevant parameters in the Xen config file. Under KVM instead, no config
change is necessary, but you still need to set up your network
interfaces correctly.
483 484

By default, under KVM, the "link" parameter you specify per-nic will
485
represent, if non-empty, a different routing table name or number to use
Iustin Pop's avatar
Iustin Pop committed
486
for your instances. This allows isolation between different instance
487 488
groups, and different routing policies between node traffic and instance
traffic.
489

490 491 492 493
You will need to configure your routing table basic routes and rules
outside of ganeti. The vif scripts will only add /32 routes to your
instances, through their interface, in the table you specified (under
KVM, and in the main table under Xen).
494

495 496 497 498
Also for "openvswitch mode" under Xen a custom network script is needed.
Under KVM everything should work, but you'll need to configure your
switches outside of Ganeti (as for bridges).

499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518
.. admonition:: Bridging issues with certain kernels

    Some kernel versions (e.g. 2.6.32) have an issue where the bridge
    will automatically change its ``MAC`` address to the lower-numbered
    slave on port addition and removal. This means that, depending on
    the ``MAC`` address of the actual NIC on the node and the addresses
    of the instances, it could be that starting, stopping or migrating
    instances will lead to timeouts due to the address of the bridge
    (and thus node itself) changing.

    To prevent this, it's enough to set the bridge manually to a
    specific ``MAC`` address, which will disable this automatic address
    change. In Debian, this can be done as follows in the bridge
    configuration snippet::

      up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE

    which will "set" the bridge address to the initial one, disallowing
    changes.

519
.. admonition:: Bridging under Debian
520 521 522 523 524 525 526

   The recommended way to configure the Xen bridge is to edit your
   ``/etc/network/interfaces`` file and substitute your normal
   Ethernet stanza with the following snippet::

     auto xen-br0
     iface xen-br0 inet static
Iustin Pop's avatar
Iustin Pop committed
527 528 529 530 531
        address %YOUR_IP_ADDRESS%
        netmask %YOUR_NETMASK%
        network %YOUR_NETWORK%
        broadcast %YOUR_BROADCAST_ADDRESS%
        gateway %YOUR_GATEWAY%
532 533 534
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
535 536
        # example for setting manually the bridge address to the eth0 NIC
        up ip link set addr $(cat /sys/class/net/eth0/address) dev $IFACE
537

Iustin Pop's avatar
Iustin Pop committed
538
The following commands need to be executed on the local console::
539

Iustin Pop's avatar
Iustin Pop committed
540 541
  $ ifdown eth0
  $ ifup xen-br0
542 543 544 545

To check if the bridge is setup, use the ``ip`` and ``brctl show``
commands::

Iustin Pop's avatar
Iustin Pop committed
546
  $ ip a show xen-br0
547 548 549 550 551 552
  9: xen-br0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc noqueue
      link/ether 00:20:fc:1e:d5:5d brd ff:ff:ff:ff:ff:ff
      inet 10.1.1.200/24 brd 10.1.1.255 scope global xen-br0
      inet6 fe80::220:fcff:fe1e:d55d/64 scope link
         valid_lft forever preferred_lft forever

Iustin Pop's avatar
Iustin Pop committed
553
  $ brctl show xen-br0
554 555 556
  bridge name     bridge id               STP enabled     interfaces
  xen-br0         8000.0020fc1ed55d       no              eth0

557 558 559 560 561 562 563 564 565 566 567 568 569 570 571
In order to have a custom and more advanced networking configuration in Xen
which can vary among instances, after having successfully installed Ganeti
you have to create a symbolic link to the vif-script provided by Ganeti
inside /etc/xen/scripts (assuming you installed Ganeti under /usr/lib)::

  $ ln -s /usr/lib/ganeti/vif-ganeti /etc/xen/scripts/vif-ganeti

This has to be done on all nodes. Afterwards you can set the ``vif_script``
hypervisor parameter to point to that script by::

  $ gnt-cluster modify -H xen-pvm:vif_script=/etc/xen/scripts/vif-ganeti

Having this hypervisor parameter you are able to create your own scripts
and create instances with different networking configurations.

Iustin Pop's avatar
Iustin Pop committed
572 573
.. _configure-lvm-label:

574 575 576 577 578 579 580
Configuring LVM
+++++++++++++++

**Mandatory** on all nodes.

The volume group is required to be at least 20GiB.

Iustin Pop's avatar
Iustin Pop committed
581 582 583 584
If you haven't configured your LVM volume group at install time you need
to do it before trying to initialize the Ganeti cluster. This is done by
formatting the devices/partitions you want to use for it and then adding
them to the relevant volume group::
585

Iustin Pop's avatar
Iustin Pop committed
586 587
  $ pvcreate /dev/%sda3%
  $ vgcreate xenvg /dev/%sda3%
588 589 590

or::

Iustin Pop's avatar
Iustin Pop committed
591 592 593
  $ pvcreate /dev/%sdb1%
  $ pvcreate /dev/%sdc1%
  $ vgcreate xenvg /dev/%sdb1% /dev/%sdc1%
594 595 596 597

If you want to add a device later you can do so with the *vgextend*
command::

Iustin Pop's avatar
Iustin Pop committed
598 599
  $ pvcreate /dev/%sdd1%
  $ vgextend xenvg /dev/%sdd1%
600 601 602

Optional: it is recommended to configure LVM not to scan the DRBD
devices for physical volumes. This can be accomplished by editing
Iustin Pop's avatar
Iustin Pop committed
603
``/etc/lvm/lvm.conf`` and adding the ``/dev/drbd[0-9]+`` regular
Iustin Pop's avatar
Iustin Pop committed
604 605 606
expression to the ``filter`` variable, like this:

.. code-block:: text
607 608 609

  filter = ["r|/dev/cdrom|", "r|/dev/drbd[0-9]+|" ]

Iustin Pop's avatar
Iustin Pop committed
610 611 612 613 614
Note that with Ganeti a helper script is provided - ``lvmstrap`` which
will erase and configure as LVM any not in-use disk on your system. This
is dangerous and it's recommended to read its ``--help`` output if you
want to use it.

615 616 617 618 619 620
Installing Ganeti
+++++++++++++++++

**Mandatory** on all nodes.

It's now time to install the Ganeti software itself.  Download the
621
source from the project page at `<http://downloads.ganeti.org/releases/>`_,
Iustin Pop's avatar
Iustin Pop committed
622
and install it (replace 2.6.0 with the latest version)::
623

Iustin Pop's avatar
Iustin Pop committed
624 625 626 627 628 629
  $ tar xvzf ganeti-%2.6.0%.tar.gz
  $ cd ganeti-%2.6.0%
  $ ./configure --localstatedir=/var --sysconfdir=/etc
  $ make
  $ make install
  $ mkdir /srv/ganeti/ /srv/ganeti/os /srv/ganeti/export
630

Iustin Pop's avatar
Iustin Pop committed
631 632
You also need to copy the file ``doc/examples/ganeti.initd`` from the
source archive to ``/etc/init.d/ganeti`` and register it with your
633 634
distribution's startup scripts, for example in Debian::

635
  $ chmod +x /etc/init.d/ganeti
Iustin Pop's avatar
Iustin Pop committed
636
  $ update-rc.d ganeti defaults 20 80
637

Iustin Pop's avatar
Iustin Pop committed
638 639 640
In order to automatically restart failed instances, you need to setup a
cron job run the *ganeti-watcher* command. A sample cron file is
provided in the source at ``doc/examples/ganeti.cron`` and you can copy
641 642 643 644
that (eventually altering the path) to ``/etc/cron.d/ganeti``. Finally,
a sample logrotate snippet is provided in the source at
``doc/examples/ganeti.logrotate`` and you can copy it to
``/etc/logrotate.d/ganeti`` to have Ganeti's logs rotated automatically.
Iustin Pop's avatar
Iustin Pop committed
645 646 647 648 649 650 651 652 653 654 655

What gets installed
~~~~~~~~~~~~~~~~~~~

The above ``make install`` invocation, or installing via your
distribution mechanisms, will install on the system:

- a set of python libraries under the *ganeti* namespace (depending on
  the python version this can be located in either
  ``lib/python-$ver/site-packages`` or various other locations)
- a set of programs under ``/usr/local/sbin`` or ``/usr/sbin``
656
- if the htools component was enabled, a set of programs under
Iustin Pop's avatar
Iustin Pop committed
657
  ``/usr/local/bin`` or ``/usr/bin/``
Iustin Pop's avatar
Iustin Pop committed
658 659 660 661 662 663 664 665 666
- man pages for the above programs
- a set of tools under the ``lib/ganeti/tools`` directory
- an example iallocator script (see the admin guide for details) under
  ``lib/ganeti/iallocators``
- a cron job that is needed for cluster maintenance
- an init script for automatic startup of Ganeti daemons
- provided but not installed automatically by ``make install`` is a bash
  completion script that hopefully will ease working with the many
  cluster commands
667 668 669 670 671 672 673 674 675

Installing the Operating System support packages
++++++++++++++++++++++++++++++++++++++++++++++++

**Mandatory** on all nodes.

To be able to install instances you need to have an Operating System
installation script. An example OS that works under Debian and can
install Debian and Ubuntu instace OSes is provided on the project web
Iustin Pop's avatar
Iustin Pop committed
676
site.  Download it from the project page and follow the instructions in
677
the ``README`` file.  Here is the installation procedure (replace 0.14
Iustin Pop's avatar
Iustin Pop committed
678
with the latest version that is compatible with your ganeti version)::
679

Iustin Pop's avatar
Iustin Pop committed
680
  $ cd /usr/local/src/
681 682 683
  $ wget http://ganeti.googlecode.com/files/ganeti-instance-debootstrap-%0.14%.tar.gz
  $ tar xzf ganeti-instance-debootstrap-%0.14%.tar.gz
  $ cd ganeti-instance-debootstrap-%0.14%
684
  $ ./configure --with-os-dir=/srv/ganeti/os
Iustin Pop's avatar
Iustin Pop committed
685 686
  $ make
  $ make install
687 688 689 690 691 692 693 694 695 696 697 698

In order to use this OS definition, you need to have internet access
from your nodes and have the *debootstrap*, *dump* and *restore*
commands installed on all nodes. Also, if the OS is configured to
partition the instance's disk in
``/etc/default/ganeti-instance-debootstrap``, you will need *kpartx*
installed.

.. admonition:: Debian

   Use this command on all nodes to install the required packages::

Iustin Pop's avatar
Iustin Pop committed
699 700 701 702 703
     $ apt-get install debootstrap dump kpartx

   Or alternatively install the OS definition from the Debian package::

     $ apt-get install ganeti-instance-debootstrap
704

705 706 707
.. admonition:: KVM

   In order for debootstrap instances to be able to shutdown cleanly
Iustin Pop's avatar
Iustin Pop committed
708 709
   they must install have basic ACPI support inside the instance. Which
   packages are needed depend on the exact flavor of Debian or Ubuntu
710
   which you're installing, but the example defaults file has a
Iustin Pop's avatar
Iustin Pop committed
711 712
   commented out configuration line that works for Debian Lenny and
   Squeeze::
713 714 715

     EXTRA_PKGS="acpi-support-base,console-tools,udev"

Iustin Pop's avatar
Iustin Pop committed
716 717
   ``kbd`` can be used instead of ``console-tools``, and more packages
   can be added, of course, if needed.
718

719 720 721
Please refer to the ``README`` file of ``ganeti-instance-debootstrap`` for
further documentation.

722
Alternatively, you can create your own OS definitions. See the manpage
723
:manpage:`ganeti-os-interface(7)`.
724 725 726 727

Initializing the cluster
++++++++++++++++++++++++

Iustin Pop's avatar
Iustin Pop committed
728
**Mandatory** once per cluster, on the first node.
729

Iustin Pop's avatar
Iustin Pop committed
730
The last step is to initialize the cluster. After you have repeated the
731 732 733
above process on all of your nodes and choose one as the master. Make sure
there is a SSH key pair on the master node (optionally generating one using
``ssh-keygen``). Finally execute::
734

Iustin Pop's avatar
Iustin Pop committed
735
  $ gnt-cluster init %CLUSTERNAME%
736

Iustin Pop's avatar
Iustin Pop committed
737 738 739 740 741 742 743
The *CLUSTERNAME* is a hostname, which must be resolvable (e.g. it must
exist in DNS or in ``/etc/hosts``) by all the nodes in the cluster. You
must choose a name different from any of the nodes names for a
multi-node cluster. In general the best choice is to have a unique name
for a cluster, even if it consists of only one machine, as you will be
able to expand it later without any problems. Please note that the
hostname used for this must resolve to an IP address reserved
744 745 746
**exclusively** for this purpose, and cannot be the name of the first
(master) node.

747
If you want to use a bridge which is not ``xen-br0``, or no bridge at
748
all, change it with the ``--nic-parameters`` option. For example to
Iustin Pop's avatar
Iustin Pop committed
749
bridge on br0 you can add::
750 751 752 753 754 755

  --nic-parameters link=br0

Or to not bridge at all, and use a separate routing table::

  --nic-parameters mode=routed,link=100
756

Iustin Pop's avatar
Iustin Pop committed
757 758
If you don't have a ``xen-br0`` interface you also have to specify a
different network interface which will get the cluster IP, on the master
759
node, by using the ``--master-netdev <device>`` option.
760 761 762

You can use a different name than ``xenvg`` for the volume group (but
note that the name must be identical on all nodes). In this case you
763 764
need to specify it by passing the *--vg-name <VGNAME>* option to
``gnt-cluster init``.
765

Iustin Pop's avatar
Iustin Pop committed
766
To set up the cluster as an Xen HVM cluster, use the
767
``--enabled-hypervisors=xen-hvm`` option to enable the HVM hypervisor
Iustin Pop's avatar
Iustin Pop committed
768 769
(you can also add ``,xen-pvm`` to enable the PVM one too). You will also
need to create the VNC cluster password file
770 771 772 773 774 775 776 777 778
``/etc/ganeti/vnc-cluster-password`` which contains one line with the
default VNC password for the cluster.

To setup the cluster for KVM-only usage (KVM and Xen cannot be mixed),
pass ``--enabled-hypervisors=kvm`` to the init command.

You can also invoke the command with the ``--help`` option in order to
see all the possibilities.

779 780 781 782 783
Hypervisor/Network/Cluster parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Please note that the default hypervisor/network/cluster parameters may
not be the correct one for your environment. Carefully check them, and
Iustin Pop's avatar
Iustin Pop committed
784 785
change them either at cluster init time, or later with ``gnt-cluster
modify``.
786 787 788 789

Your instance types, networking environment, hypervisor type and version
may all affect what kind of parameters should be used on your cluster.

790 791 792 793 794 795 796 797 798 799 800 801 802
.. admonition:: KVM

  Instances are by default configured to use a host kernel, and to be
  reached via serial console, which works nice for Linux paravirtualized
  instances. If you want fully virtualized instances you may want to
  handle their kernel inside the instance, and to use VNC.

  Some versions of KVM have a bug that will make an instance hang when
  configured to use the serial console (which is the default) unless a
  connection is made to it within about 2 seconds of the instance's
  startup. For such case it's recommended to disable the
  ``serial_console`` option.

803

804 805 806 807 808
Joining the nodes to the cluster
++++++++++++++++++++++++++++++++

**Mandatory** for all the other nodes.

Iustin Pop's avatar
Iustin Pop committed
809 810 811
After you have initialized your cluster you need to join the other nodes
to it. You can do so by executing the following command on the master
node::
812

Iustin Pop's avatar
Iustin Pop committed
813
  $ gnt-node add %NODENAME%
814 815 816 817 818 819 820 821 822 823 824 825

Separate replication network
++++++++++++++++++++++++++++

**Optional**

Ganeti uses DRBD to mirror the disk of the virtual instances between
nodes. To use a dedicated network interface for this (in order to
improve performance or to enhance security) you need to configure an
additional interface for each node.  Use the *-s* option with
``gnt-cluster init`` and ``gnt-node add`` to specify the IP address of
this secondary interface to use for each node. Note that if you
Iustin Pop's avatar
Iustin Pop committed
826 827
specified this option at cluster setup time, you must afterwards use it
for every node add operation.
828 829 830 831

Testing the setup
+++++++++++++++++

Iustin Pop's avatar
Iustin Pop committed
832
Execute the ``gnt-node list`` command to see all nodes in the cluster::
833

Iustin Pop's avatar
Iustin Pop committed
834
  $ gnt-node list
835 836 837
  Node              DTotal  DFree MTotal MNode MFree Pinst Sinst
  node1.example.com 197404 197404   2047  1896   125     0     0

Iustin Pop's avatar
Iustin Pop committed
838
The above shows a couple of things:
839

Iustin Pop's avatar
Iustin Pop committed
840 841 842
- The various Ganeti daemons can talk to each other
- Ganeti can examine the storage of the node (DTotal/DFree)
- Ganeti can talk to the selected hypervisor (MTotal/MNode/MFree)
843

Iustin Pop's avatar
Iustin Pop committed
844 845
Cluster burnin
~~~~~~~~~~~~~~
846

Iustin Pop's avatar
Iustin Pop committed
847 848 849 850 851
With Ganeti a tool called :command:`burnin` is provided that can test
most of the Ganeti functionality. The tool is installed under the
``lib/ganeti/tools`` directory (either under ``/usr`` or ``/usr/local``
based on the installation method). See more details under
:ref:`burnin-label`.
852

Iustin Pop's avatar
Iustin Pop committed
853 854
Further steps
-------------
855

Iustin Pop's avatar
Iustin Pop committed
856 857 858 859
You can now proceed either to the :doc:`admin`, or read the manpages of
the various commands (:manpage:`ganeti(7)`, :manpage:`gnt-cluster(8)`,
:manpage:`gnt-node(8)`, :manpage:`gnt-instance(8)`,
:manpage:`gnt-job(8)`).
860

Iustin Pop's avatar
Iustin Pop committed
861
.. rubric:: Footnotes
862

Iustin Pop's avatar
Iustin Pop committed
863 864 865
.. [#defkernel] The kernel and initrd paths can be changed at either
   cluster level (which changes the default for all instances) or at
   instance level.
866 867

.. vim: set textwidth=72 :
Iustin Pop's avatar
Iustin Pop committed
868 869 870 871
.. Local Variables:
.. mode: rst
.. fill-column: 72
.. End: