Commit 18240dbc authored by Ilias Tsitsimpis's avatar Ilias Tsitsimpis
Browse files

Merge pull request #34 from cstavr/feature-volume-snapshots

Add support for volumes & snapshots

This patch set extends Cyclades and Pithos to support Volumes and Snapshots.
Each virtual server can have multiple virtual disks (of the same disk
template), which can be dynamically created or removed. Also, in case of
Archipelago volumes, the user can create snapshots of an instance's volumes
which are stored as files on Pithos. The volumes and snapshots are handled
via the OpenStack Cinder API, which is exposed by the Cyclades 'volume' service.
parents 6589759a a26cedd6
......@@ -13,6 +13,7 @@ synnefo_repo =
# branch/sha will result from the current repository.
synnefo_branch =
build_pithos_webclient = True
# pithos-web-client git repo
pithos_webclient_repo =
# Git branch to use for pithos-web-client
......@@ -35,6 +36,8 @@ git_config_mail =
accept_ssh_from =
# Config file to save temporary options (eg IPs, passwords etc)
temporary_config = /tmp/ci_temp_conf
# Install x2go and firefox
setup_x2go = True
# File to save the x2goplugin html file
x2go_plugin_file = /tmp/x2go.html
......@@ -32,10 +32,12 @@ pithos_dir = /srv/pithos
flavor_cpu = 1,2,4,8
flavor_ram = 128,256,512,1024,2048,4096,8192
flavor_disk = 2,5,10,20,40,60,80,100
flavor_storage = file
flavor_storage = file,ext_archipelago
vm_public_bridge = br0
vm_private_bridge = prv0
common_bridge = br0
debian_base_url =
segment_size = 2048
......@@ -25,6 +25,7 @@ python-django =
drbd8-utils =
collectd =
dnsutils =
python-svipc =
......@@ -60,3 +61,17 @@ nfdhcpd = wheezy
kamaki =
python-bitarray = wheezy
python-nfqueue = 0.4+physindev-1~wheezy
librados2 =
archipelago = experimental
archipelago-dbg = experimental
archipelago-modules-dkms = experimental
archipelago-modules-source = experimental
archipelago-rados = experimental
archipelago-rados-dbg = experimental
libxseg0 = experimental
libxseg0-dbg = experimental
python-archipelago = experimental
python-xseg = experimental
archipelago-ganeti = experimental
......@@ -390,16 +390,19 @@ class SynnefoCI(object):
_run(cmd, False)
# Setup apt, download packages
self.logger.debug("Setup apt. Install x2goserver and firefox")
self.logger.debug("Setup apt")
cmd = """
echo 'APT::Install-Suggests "false";' >> /etc/apt/apt.conf
echo 'precedence ::ffff:0:0/96 100' >> /etc/gai.conf
apt-get update
apt-get install curl --yes --force-yes
apt-get install -q=2 curl --yes --force-yes
echo -e "\n\n{0}" >> /etc/apt/sources.list
# Synnefo repo's key
curl | apt-key add -
""".format(self.config.get('Global', 'apt_repo'))
_run(cmd, False)
cmd = """
# X2GO Key
apt-key adv --recv-keys --keyserver E1F958385BFE2B6E
apt-get install x2go-keyring --yes --force-yes
......@@ -418,9 +421,10 @@ class SynnefoCI(object):
echo 'Encoding=UTF-8' >> /usr/share/applications/xterm.desktop
echo 'Icon=xterm-color_48x48' >> /usr/share/applications/xterm.desktop
echo 'Categories=System;TerminalEmulator;' >> \
""".format(self.config.get('Global', 'apt_repo'))
_run(cmd, False)
if self.config.get("Global", "setup_x2go") == "True":
self.logger.debug("Install x2goserver and firefox")
_run(cmd, False)
def _find_flavor(self, flavor=None):
"""Find a suitable flavor to use
......@@ -695,7 +699,9 @@ class SynnefoCI(object):
synnefo_repo=synnefo_repo, synnefo_branch=synnefo_branch,
local_repo=local_repo, pull_request=pull_request)
# Clone pithos-web-client
if self.config.get("Global", "build_pithos_webclient") == "True":
# Clone pithos-web-client
def clone_synnefo_repo(self, synnefo_repo=None, synnefo_branch=None,
......@@ -908,7 +914,8 @@ class SynnefoCI(object):
# Build synnefo packages
# Build pithos-web-client packages
if self.config.get("Global", "build_pithos_webclient") == "True":
def build_synnefo(self):
Volume snapshots
The goal of this design document is to record and present the proposed method
for Volume snapshotting in Synnefo. We will describe the whole work-flow from
the user's API down to Archipelago and up again.
Snapshot functionality
The snapshot functionality aims to provide the user with the following:
a. A thin snapshot of an Archipelago volume: |br|
The snapshot should capture the volume's data state at the time the
snapshot was requested. |br|
**Note:** The VM user is accountable for the consistency of its volume.
Journaled filesystems or prior shutdown of the VM is advised.
#. Presenting the snapshot instantly as a regular file
on Pithos: |br|
Users can view their snapshots in Pithos as any other file that they have
uploaded and can subsequently download them.
#. A registered Synnefo snapshot, ready to be deployed: |br|
Essentially, if a snapshotted volume includes an OS installation, it can be
used as any other Synnefo OS image. This allows users to create
*restoration points* for their VMs.
Side goals
In order to make the snapshot process as slim as possible, the following goals
must also be met:
a. Low computational/storage overhead: |br|
If many users send snapshot requests, the service should be able to respond
quick. Also, a snapshotted volume should not incur any significant storage
#. Solid reconciliation process: |br|
If a snapshot request fails, the system should do the necessary cleanups or
at least make it easy to reconcile the affected databases.
Snapshot creation
An illustration of the proposed method follows below:
Each step of the procedure is explained below:
#. The Cyclades App receives a snapshot request. The request is expected to
originate from the user and be sent via an API client or the Cyclades UI. A
valid snapshot request should target an existing volume ID.
#. The Cyclades App uses its Pithos Backend to create a `snapshot record`_ in
the Pithos database. The snapshot record is explained in the following
section. It is initially set with the values that are seen in the diagram.
#. The Cyclades App creates a snapshot job and sends it to the Ganeti
#. The Ganeti Master in turn, designates the snapshot job to the appropriate
Ganeti Noded.
#. The Ganeti Noded runs the corresponding Archipelago ``ExtStorage`` script
and invokes the Vlmc tool to create the snapshot.
#. The Vlmc tool instructs Archipelago to create a snapshot by sending a
snapshot request.
Once Archipelago has (un)successfully created the snapshot, the response is
propagated to the Ganeti Master which in turn creates an event about this
snapshot job and its execution result.
7. snf-ganeti-eventd is informed about this event, using an ``inotify()``
mechanism, and forwards it to the snf-dispatcher.
#. The snf-dispatcher uses its Pithos Backend to update the ``snapshot status``
property of the snapshot record that was created in Step 2. According to the
result of the snapshot request, the snapshot status is set as ``Ready`` or
``Error``. This means that, as far as Cinder is concerned, the snapshot is
#. The ``Available`` attribute however is still ``0``. Swift (Pithos) will make
it available (``1``) and thus usable, the first time it will try to ping
back (a.k.a. the first time someone tries to access it).
Snapshot record
The snapshot record has the following attributes:
| Key | Value | Service |
| file_name | Generated (string - see below) | Swift |
| available | Generated (boolean) | Swift |
and the following properties:
| Key | Value | Service |
| snapshot_name | User-provided (string) | Cinder |
| snapshot_status | Generated (see Cinder API) | Cinder |
| EXCLUDE_ALL_TASKS | Generated (string) | Glance |
The file_name is a string that has the following form::
* ``<vol_id>`` is the Volume ID and
* ``<counter>`` is the number of times that the volume has been snapshotted and
increases monotonically.
The snapshot name should thus be unique.
"Available" attribute
The ``available`` attribute is a Swift attribute that is introduced with
snapshots and applies to all Pithos files. When a Pithos file is "available",
it means that its map file has been created and points to the correct data.
Normally, all Pithos files have their map file properly created before adding a
record in the Pithos database. Snapshots however are an exception to this rule,
since their map file is created asynchronously.
Therefore, the creation of a Pithos file has the following rules:
* If the file is a snapshot, the ``available`` attribute is set to "0".
* For all other files, the ``available`` attribute is set to "1".
**Note:** ``available`` can change from "0" to "1", but never the opposite.
The update of the ``available`` attribute happens implicitly after the creation
of the snapshot, when a request reads the file record from the
Pithos database. The following diagram shows how can a request (download
request for example) update the ``available`` attribute.
In short, what happens is:
#. The user asks to download the file from the Pithos App.
#. The Pithos App checks the file record and finds that the ``available``
attribute is "0".
#. It then pings Archipelago to check the status of the map.
#. If the map exists, it sets ``available`` to "1" and can use the map file to
serve the data.
VM creation from snapshot
The following diagram illustrates the VM creation process from a snapshot
The steps are explained in detail below:
#. A user who has the registered Images list (which includes all Snapshots
too), requests to create a VM from the Cyclades App, using one of the
registered snapshots of the list.
#. The Cyclades App sends a VM creation job to the Ganeti Master with the
appropriate data. The data differ according to the disk template:
* If the template is ext, then the "origin" field has the archipelago map
* For any other template, the archipelago map name is passed in the "img_id"
property of OS parameters.
#. The Ganeti Master designates the job to the appropriate Ganeti Noded.
#. The Ganeti Noded will create the disks, according to the requested disk
a. If the disk template is "ext", the following execution path is taken:
1. The Ganeti Noded runs the corresponding Archipelago ``ExtStorage``
script and invokes the Vlmc tool.
#. The Vlmc tool instructs Archipelago to create a volume from a
b. If the disk template is other than "ext", e.g. "drbd", Ganeti Noded
creates a new, empty disk.
#. After the volume has been created, Ganeti Noded instructs snf-image to
prepare it for deployment. The parameters that are passed to snf-image are
the OS parameters of Step 2, and the main ones for each disk template are
shown in the diagram, next to the "Ext?" decision box. According to the disk
template, snf-image has two possible execution paths:
a. If the disk template is "ext", there is no need to copy any data.
Also, the image has the "EXCLUDE_ALL_TASKS" property set to "yes", so
snf-image will run no customization scripts and will simply return.
b. If the disk template is other than "ext", e.g. "drbd", snf-image will
copy the snapshot's data from Pithos to the created DRBD disk. As above,
since the image has the "EXCLUDE_ALL_TASKS" property set to "yes",
snf-image will run no customization scripts.
.. |br| raw:: html
<br />
.. |create_snapshot| image:: /images/create-snapshot.png
.. |available_attribute| image:: /images/available-attribute.png
.. |spawn_from_snapshot| image:: /images/spawn-from-snapshot.png
Cyclades Volumes
This document describes the extension of Cyclades to handle block storage
devices, referred to as Volumes.
Current state and shortcomings
Currently one block storage device is created and destroyed together with the
virtual servers. One disk is created per server with the size and the disk
template that are described by the server's flavor. The disk is destroyed when
the server is destroyed. Also, There is no way to attach/detach disks to
existing servers and there is no API to expose information about the server's
Proposed changes
Cyclades will be able to manage volumes so that users can create volumes and
dynamically attach them to servers. Volumes can be created either empty or
filled with data from a specified source (image, volume or snapshot). Also,
users can dynamically remove volumes. Finally, a server can be created with
multiple volumes at once.
Implementation details
Known Limitations
While addition and removal of disks is supported by Ganeti, it is not supported
to handle disks that are not attached to an instance. There is no way to create
a disk without attaching it to an instance, neither a way to detach a disk from
an instance without deleting it. Future versions of Ganeti will make disks
separate entities, thus overcoming the above mentioned issues. Until then,
this issues will also force a limitation to the way Cyclades will be handling
Volumes. Specifically, Cyclades volumes will not be attached/detached from
servers; they will be only be added and removed from them.
Apart from Ganeti's inability to manage disks as separate entities, attaching
and detaching a disk is not meaningful for storage solutions that are not
shared between Ganeti nodes and clusters, because this would require copying
the data of the disks between nodes. Thus, the ability to attach/detach a disk
will only be available for disks of externally mirrored template (EXT_MIRRORED
disk templates).
Also, apart from the root volume of an instance, initializing a volume with
data is currently only available for Archipelago volumes, since there is no
mechanism in Ganeti for filling data in other type of volumes (e.g. file, lvm,
drbd). Until then, creating a volume from a source, other than the root volume
of the instance, will only be supported for Archipelago.
Finally, an instance can not have disks of different disk template.
Cyclades internal representation
Each volume will be represented by the new `Volume` database model, containing
information about the Volume, like the following:
* name: User defined name
* description: User defined description
* userid: The UUID of the owner
* size: Volume size in GB
* disk_template: The disk template of the volume
* machine: The server the volume is currently attached, if any
* source: The UUID of the source of the volume, prefixed by it's type, e.g. "snapshot:41234-43214-432"
* status: The status of the volume
Each Cyclades Volume corresponds to the disk of a Ganeti instance, uniquely
named `snf-vol-$VOLUME_ID`.
API extensions
The Cyclades API will be extended with all the needed operations to support
Volume management. The API will be compatible with OpenStack Cinder API.
Volume API
The new `volume` service will be implemented and will provide the basic
operations for volume management. This service provides the `/volumes` and
`/types` endpoints, almost as described in the OS Cinder API, with the only
difference that we will extend `POST /volumes/volumes` with a required
field to represent the server that the volume will be attached, and which is
named `server_id`.
Compute API extensions
The API that is exposed by the `volume` service is enough for the addition and
removal of volumes to existing servers. However, it does not cover creating a
server from a volume or creating a server with more than one volumes. For this
reason, the `POST /servers` API call will be extended with the
`block_device_mapping_v2` attribute, as defined in the OS Cinder API, which is
a list of dictionaries with the following keys:
* source_type: The source of the volume (`volume` | `snapshot` | `image` | `blank`)
* uuid: The uuid of the source (if not blank)
* size: The size of the volume
* delete_on_termination: Whether to preserve the volume on instance deletion.
If the type is `volume`, then the volume that is specified by the `uuid` field
will be used. Otherwise, a new volume will be created which will contain the
data of the specified source (`image` or `snapshot`). If the source type
is `blank` then the new volume will be empty.
Also, we will implement `os_volume-attachments` extension, containing the
following operations:
* List server attachments(volumes)
* Show server attachment information
* Attach volume to server (notImplemented until Ganeti supports detachable volumes)
* Detach volume from server (notImplemented until Ganeti supports detachable volumes)
The total size of volumes will be quotable, and will be directly mapped to
existing `cyclades.disk` resource. In the future, we may implement having
different quotas for each volume type.
Command-line interface
The following management commands will be implemented:
* `volume-create`: Create a new volume
* `volume-remove`: Remove a volume
* `volume-list`: List volumes
* `volume-info`: Show volume details
* `volume-inspect`: Inspect volume in DB and Ganeti
* `volume-import`: Import a volume from an existing Ganeti disk
The following commands will be extended:
* `server-create`: extended to specify server's volumes
* `server-inspect`: extended to display server's volumes
* `reconcile-servers`: extended to reconcile server's volumes
......@@ -156,6 +156,8 @@ Drafts
:maxdepth: 1
Resource-pool projects design <design/resource-pool-projects>
Volumes design <design/volumes>
Volume Snapshots design <design/volume-snapshots>
......@@ -46,6 +46,10 @@
## than 'max:nic-count' option of Ganeti's ipolicy.
## Maximum number of disks per Ganeti instance. This value must be less or
## equal than 'max:disk-count' option of Ganeti's ipolicy.
## The following setting defines a dictionary with key-value parameters to be
## passed to each Ganeti ExtStorage provider. The setting defines a mapping
## from the provider name, e.g. 'archipelago' to a dictionary with the actual
......@@ -103,20 +103,31 @@ def metadata_item_demux(request, image_id, key):
def image_to_dict(image, detail=True):
d = dict(id=image['id'], name=image['name'])
if detail:
d['updated'] = utils.isoformat(date_parse(image['updated_at']))
d['created'] = utils.isoformat(date_parse(image['created_at']))
d['status'] = 'DELETED' if image['deleted_at'] else 'ACTIVE'
d['progress'] = 100 if image['status'] == 'available' else 0
img_status = image.get("status", "").upper()
status = API_STATUS_FROM_IMAGE_STATUS.get(img_status, "UNKNOWN")
d['status'] = status
d['progress'] = 100 if status == 'ACTIVE' else 0
d['user_id'] = image['owner']
d['tenant_id'] = image['owner']
d['public'] = image["is_public"]
d['links'] = util.image_to_links(image["id"])
if image["properties"]:
d['metadata'] = image['properties']
d['metadata'] = {}
d["is_snapshot"] = image["is_snapshot"]
return d
......@@ -44,7 +44,8 @@ from snf_django.lib.api import faults, utils
from synnefo.api import util
from synnefo.db.models import (VirtualMachine, VirtualMachineMetadata)
from synnefo.logic import servers, utils as logic_utils
from synnefo.logic import servers, utils as logic_utils, server_attachments
from synnefo.volume.util import get_volume
from logging import getLogger
log = getLogger(__name__)
......@@ -61,8 +62,17 @@ urlpatterns = patterns(
(r'^/(\d+)/metadata/(.+?)(?:.json|.xml)?$', 'metadata_item_demux'),
(r'^/(\d+)/stats(?:.json|.xml)?$', 'server_stats'),
(r'^/(\d+)/diagnostics(?:.json)?$', 'get_server_diagnostics'),
(r'^/(\d+)/os-volume_attachments(?:.json)?$', 'demux_volumes'),
(r'^/(\d+)/os-volume_attachments/(\d+)(?:.json)?$', 'demux_volumes_item'),
def demux(request):
if request.method == 'GET':
......@@ -112,6 +122,26 @@ def metadata_item_demux(request, server_id, key):
def demux_volumes(request, server_id):
if request.method == 'GET':
return get_volumes(request, server_id)
elif request.method == 'POST':
return attach_volume(request, server_id)
return api.api_method_not_allowed(request,
allowed_methods=['GET', 'POST'])
def demux_volumes_item(request, server_id, volume_id):
if request.method == 'GET':
return get_volume_info(request, server_id, volume_id)
elif request.method == 'DELETE':
return detach_volume(request, server_id, volume_id)
return api.api_method_not_allowed(request,
allowed_methods=['GET', 'DELETE'])
def nic_to_attachments(nic):
"""Convert a NIC object to 'attachments attribute.
......@@ -194,6 +224,8 @@ def vm_to_dict(vm, detail=False):
d['attachments'] = attachments
d['addresses'] = attachments_to_addresses(attachments)
d['volumes'] = [ for v in vm.volumes.order_by('id')]
# include the latest vm diagnostic, if set
diagnostic = vm.get_last_diagnostic()
if diagnostic:
......@@ -389,10 +421,13 @@ def create_server(request):
except (KeyError, AssertionError):
raise faults.BadRequest("Malformed request")
volumes = None
dev_map = server.get("block_device_mapping_v2")
if dev_map is not None:
volumes = parse_block_device_mapping(dev_map)
# Verify that personalities are well-formed
# Get image information
image = util.get_image_dict(image_id, user_id)
# Get flavor (ensure it is active)
flavor = util.get_flavor(flavor_id, include_deleted=False)
if not flavor.allow_create:
......@@ -402,9 +437,9 @@ def create_server(request):
# Generate password
password = util.random_password()
vm = servers.create(user_id, name, password, flavor, image,
vm = servers.create(user_id, name, password, flavor, image_id,
metadata=metadata, personality=personality,
project=project, networks=networks)
project=project, networks=networks, volumes=volumes)
server = vm_to_dict(vm, detail=True)
server['status'] = 'BUILD'
......@@ -415,6 +450,65 @@ def create_server(request):
return response
def parse_block_device_mapping(dev_map):