Commit bced76fd authored by Jose A. Lopes's avatar Jose A. Lopes
Browse files

Export docs from wiki to repo

Export documents about 'cluster keys replacement' and 'design file
based storage' from the wiki to the repository.
Signed-off-by: default avatarJose A. Lopes <>
Reviewed-by: default avatarKlaus Aehlig <>
parent 1a705513
......@@ -487,6 +487,7 @@ utils_PYTHON = \
docinput = \
doc/admin.rst \
doc/cluster-keys-replacement.rst \
doc/cluster-merge.rst \
doc/ \
doc/css/style.css \
......@@ -510,6 +511,7 @@ docinput = \
doc/design-daemons.rst \
doc/design-device-uuid-name.rst \
doc/design-draft.rst \
doc/design-file-based-storage.rst \
doc/design-glusterfs-ganeti-support.rst \
doc/design-hotplug.rst \
doc/design-hroller.rst \
......@@ -1340,6 +1340,13 @@ SSH changes and log directories:
Otherwise, if you plan to re-create the cluster, you can just go ahead
and rerun ``gnt-cluster init``.
Replacing the SSH and SSL keys
Ganeti uses both SSL and SSH keys, and actively modifies the SSH keys on
the nodes. As result, in order to replace these keys, a few extra steps
need to be followed: :doc:`cluster-keys-replacement`
Monitoring the cluster
Cluster Keys Replacement
Ganeti uses both SSL and SSH keys, and actively modifies the SSH keys
on the nodes. As result, in order to replace these keys, a few extra
steps need to be followed.
For an example when this could be needed, see the thread at
`Regenerating SSL and SSH keys after the security bug in Debian's
Ganeti uses OpenSSL for encryption on the RPC layer and SSH for
executing commands. The SSL certificate is automatically generated
when the cluster is initialized and it's copied to added nodes
automatically together with the master's SSH host key.
Note that paths below may vary depending on your distribution. In
general, modifications should be done on the master node and then
distributed to all nodes of a cluster (possibly using a pendrive - but
don't forget to use "shred" to remove files securely afterwards).
Replacing SSL keys
The cluster SSL key is stored in ``/var/lib/ganeti/server.pem``.
Run the following command to generate a new key::
gnt-cluster renew-crypto --new-cluster-certificate
# Older version, which don't have this command, can instead use:
chmod 0600 /var/lib/ganeti/server.pem &&
openssl req -new -newkey rsa:1024 -days 1825 -nodes \
-x509 -keyout /var/lib/ganeti/server.pem \
-out /var/lib/ganeti/server.pem -batch &&
chmod 0400 /var/lib/ganeti/server.pem &&
/etc/init.d/ganeti restart
gnt-cluster copyfile /var/lib/ganeti/server.pem
gnt-cluster command /etc/init.d/ganeti restart
Replacing SSH keys
There are two sets of SSH keys in the cluster: the host keys (both DSA
and RSA, though Ganeti only uses the RSA one) and the root's DSA key
(Ganeti uses DSA for historically reasons, in the future RSA will be
host keys
These are the files named ``/etc/ssh/ssh_host_*``. You need to
manually recreate them; it's possibly that the startup script of
OpenSSH will generate them if they don't exist, or that the package
system regenerates them.
Also make sure to copy the master's SSH host keys to all other nodes.
cluster public key file
The new public rsa host key created in the previous step must be added
in two places:
#. known hosts file, ``/var/lib/ganeti/known_hosts``
#. cluster configuration file, ``/var/lib/ganeti/``
Edit these two files and update them with newly generated SSH host key
(in the previous step, take it from the
For the ```` file, please look for an entry named
``rsahostkeypub`` and replace the value for it with the contents of
the ``.pub`` file. For the ``known_hosts`` file, you need to replace
the old key with the new one on each line (for each host).
root's key
These are the files named ``~root/.ssh/id_dsa*``.
Run this command to rebuild them::
ssh-keygen -t dsa -f ~root/.ssh/id_dsa -q -N ""
root's ``authorized_keys``
This is the file named ``~root/.ssh/authorized_keys``.
Edit file and update it with the newly generated root key, from the
```` file generated in the previous step.
In the end, the files mentioned above should be identical for all
nodes in a cluster. Also do not forget to run ``gnt-cluster verify``.
File-based Storage
This page describes the proposed file-based storage for the 2.0 version
of Ganeti. The project consists in extending Ganeti in order to support
a filesystem image as Virtual Block Device (VBD) in Dom0 as the primary
storage for a VM.
* file-based storage for virtual machines running in a Xen-based
Ganeti cluster
* failover of file-based virtual machines between cluster-nodes
* export/import file-based virtual machines
* reuse existing image files
* allow Ganeti to initalize the cluster without checking for a volume
group (e.g. xenvg)
Non Goals:
* any kind of data mirroring between clusters for file-based instances
(this should be achieved by using shared storage)
* special support for live-migration
* encryption of VBDs
* compression of VBDs
Ganeti is a virtual server management software tool built on top of Xen
VM monitor and other Open Source software.
Since Ganeti currently supports only block devices as storage backend
for virtual machines, the wish came up to provide a file-based backend.
Using this file-based option provides the possibility to store the VBDs
on basically every filesystem and therefore allows to deploy external
data storages (e.g. SAN, NAS, etc.) in clusters.
Xen (and other hypervisors) provide(s) the possibility to use a file as
the primary storage for a VM. One file represents one VBD.
Advantages of file-backed VBD:
* support of sparse allocation
* easy from a management/backup point of view (e.g. you can just copy
the files around)
* external storage (e.g. SAN, NAS) can be used to store VMs
Disadvantages of file-backed VBD:
* possible performance loss for I/O-intensive workloads
* using sparse files requires care to ensure the sparseness is
preserved when copying, and there is no header in which metadata
relating back to the VM can be stored
Xen-related specifications
There are several ways to realize the required functionality with an
underlying Xen hypervisor.
1) loopback driver
* available in most precompiled kernels
* stable, since it is in kernel tree for a long time
* easy to set up
* buffer writes very aggressively, which can affect guest filesystem
correctness in the event of a host crash
* can even cause out-of-memory kernel crashes in Dom0 under heavy
write load
* substantial slowdowns under heavy I/O workloads
* the default number of supported loopdevices is only 8
* doesn't support QCOW files
``blktap`` driver
* higher performance than loopback driver
* more scalable
* better safety properties for VBD data
* Xen-team strongly encourages use
* already in Xen tree
* supports QCOW files
* asynchronous driver (i.e. high performance)
* not enabled in most precompiled kernels
* stable, but not as much tested as loopback driver
3) ubklback driver
The Xen Roadmap states "Work is well under way to implement a
``ublkback`` driver that supports all of the various qemu file format
Furthermore, the Roadmap includes the following:
"... A special high-performance qcow plugin is also under
development, that supports better metadata caching, asynchronous IO,
and allows request reordering with appropriate safety barriers to
enforce correctness. It remains both forward and backward compatible
with existing qcow disk images, but makes adjustments to qemu's
default allocation policy when creating new disks such as to
optimize performance."
File types
Raw disk image file
* Resizing supported
* Sparse file (filesystem dependend)
* simple and easily exportable
* Underlying filesystem needs to support sparse files (most
filesystems do, though)
QCOW disk image file
* Smaller file size, even on filesystems which don't support holes
(i.e. sparse files)
* Snapshot support, where the image only represents changes made to an
underlying disk image
* Optional zlib based compression
* Optional AES encryption
* Resizing not supported yet (it's on the way)
VMDK disk image file
This file format is directly based on the qemu vmdk driver, which is
synchronous and thus slow.
Detailed Design
* **VBD** (Virtual Block Device): Persistent storage available to a
virtual machine, providing the abstraction of an actual block
storage device. VBDs may be actual block devices, filesystem images,
or remote/network storage.
* **Dom0** (Domain 0): The first domain to be started on a Xen
machine. Domain 0 is responsible for managing the system.
* **VM** (Virtual Machine): The environment in which a hosted
operating system runs, providing the abstraction of a dedicated
machine. A VM may be identical to the underlying hardware (as in
full virtualization, or it may differ, as in paravirtualization). In
the case of Xen the domU (unprivileged domain) instance is meant.
* **QCOW**: QEMU (a processor emulator) image format.
Managing file-based instances
The option for file-based storage will be added to the 'gnt-instance'
Add Instance
gnt-instance add -t file:[path\ =[,driver=loop[,reuse[,...]]]] \
--disk 0:size=5G --disk 1:size=10G -n node -o debian-etch instance2
This will create a file-based instance with e.g. the following files:
* ``/sda`` -> 5GB
* ``/sdb`` -> 10GB
The default directory where files will be stored is
``/srv/ganeti/file-storage/``. This can be changed by setting the
``<path>`` option. This option denotes the full path to the directory
where the files are stored. The filetype will be "raw" for the first
release of Ganeti 2.0. However, the code will be extensible to more
file types, since Ganeti will store information about the file type of
each image file. Internally Ganeti will keep track of the used driver,
the file-type and the full path to the file for every VBD. Example:
"logical_id" : ``[FD_LOOP, FT_RAW, "/instance1/sda"]`` If the
``--reuse`` flag is set, Ganeti checks for existing files in the
corresponding directory (e.g. ``/xen/instance2/``). If one or more
files in this directory are present and correctly named (the naming
conventions will be defined in Ganeti version 2.0) Ganeti will set a
VM up with these. If no file can be found or the names or invalid the
operation will be aborted.
Remove instance
The instance removal will just differ from the actual one by deleting
the VBD-files instead of the corresponding block device (e.g. a logical
Starting/Stopping Instance
Here nothing has to be changed, as the xen tools don't differentiate
between file-based or blockdevice-based instances in this case.
Export/Import instance
Provided "dump/restore" is used in the "export" and "import" guest-os
scripts, there are no modifications needed when file-based instances are
exported/imported. If any other backup-tool (which requires access to
the mounted file-system) is used then the image file can be temporaily
mounted. This can be done in different ways:
Mount a raw image file via loopback driver::
mount -o loop /srv/ganeti/file-storage/instance1/sda1 /mnt/disk\
Mount a raw image file via blkfront driver (Dom0 kernel needs this
module to do the following operation)::
xm block-attach 0 tap:aio:/srv/ganeti/file-storage/instance1/sda1 /dev/xvda1 w 0\
mount /dev/xvda1 /mnt/disk
Mount a qcow image file via blkfront driver (Dom0 kernel needs this
module to do the following operation)
xm block-attach 0 tap:qcow:/srv/ganeti/file-storage/instance1/sda1 /dev/xvda1 w 0
mount /dev/xvda1 /mnt/disk
High availability features with file-based instances
Failing over an instance
Failover is done in the same way as with block device backends. The
instance gets stopped on the primary node and started on the secondary.
The roles of primary and secondary get swapped. Note: If a failover is
done, Ganeti will assume that the corresponding VBD(s) location (i.e.
directory) is the same on the source and destination node. In case one
or more corresponding file(s) are not present on the destination node,
Ganeti will abort the operation.
Replacing an instance disks
Since there is no data mirroring for file-backed VM there is no such
Evacuation of a node
Since there is no data mirroring for file-backed VMs there is no such
Live migration
Live migration is possible using file-backed VBDs. However, the
administrator has to make sure that the corresponding files are exactly
the same on the source and destination node.
Xen Setup
File creation
Creation of a raw file is simple. Example of creating a sparse file of 2
Gigabytes. The option "seek" instructs "dd" to create a sparse file::
dd if=/dev/zero of=vm1disk bs=1k seek=2048k count=1
Creation of QCOW image files can be done with the "qemu-img" utility (in
debian it comes with the "qemu" package).
Config file
The Xen config file will have the following modification if one chooses
the file-based disk-template.
1) loopback driver and raw file
disk = ['file:</path/to/file>,sda1,w']
2) blktap driver and raw file
disk = ['tap:aio:,sda1,w']
3) blktap driver and qcow file
disk = ['tap:qcow:,sda1,w']
Other hypervisors
Other hypervisors have mostly differnet ways to make storage available
to their virtual instances/machines. This is beyond the scope of this
......@@ -71,8 +71,8 @@ in this section.
A description of the locking strategy and, in particular, lock order
dependencies is presented in :doc:`locking`.
Build dependencies and other useful development-related information are provided
in the :doc:`devnotes`.
Build dependencies and other useful development-related information
are provided in the :doc:`devnotes`.
All the features implemented in Ganeti are described in a design document before
being actually implemented. Designs can be implemented in a released version, or
......@@ -108,12 +108,14 @@ Draft designs
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment