Commit ffdca3ac authored by Christos Stavrakakis's avatar Christos Stavrakakis

Merge branch 'admin-guide'

parents fbfa27c7 edc6e95e
......@@ -21,6 +21,7 @@ Guide before proceeding.
:target: _images/synnefo-architecture1.png
Identity Service (Astakos)
......@@ -230,63 +231,315 @@ Image Overview
RabbitMQ is used as a generic message broker for Cyclades. It should be
installed on two seperate :ref:`QUEUE <QUEUE_NODE>` nodes in a high
availability configuration as described here:
Asynchronous communication with Ganeti backends
Synnefo uses Google Ganeti backends for VM cluster management. In order for
Cyclades to be able to handle thousands of user requests, Cyclades and Ganeti
communicate asynchronously. Briefly, requests are submitted to Ganeti through
Ganeti's RAPI/HTTP interface, and then asynchronous notifications about the
progress of Ganeti jobs are being created and pushed upwards to Cyclades. The
architecture and communication with a Ganeti backend is shown in the graph
.. image:: images/cyclades-ganeti-communication.png
:width: 50%
:target: _images/cyclades-ganeti-communication.png
The Cyclades API server is responsible for handling user requests. Read-only
requests are directly served by looking up the Cyclades DB. If the request
needs an action in the Ganeti backend, Cyclades submit jobs to the Ganeti
master using the `Ganeti RAPI interface
While Ganeti executes the job, `snf-ganeti-eventd`, `snf-ganeti-hook` and
`snf-progress-monitor` are monitoring the progress of the job and send
corresponding messages to the RabbitMQ servers. These components are part
of `snf-cyclades-gtools` and must be installed on all Ganeti nodes. Specially:
* *snf-ganeti-eventd* sends messages about operations affecting the operating
state of instances and networks. Works by monitoring the Ganeti job queue.
* *snf-ganeti_hook* sends messages about the NICs of instances. It includes a
number of `Ganeti hooks <>`_
for customisation of operations.
* *snf-progress_monitor* sends messages about the progress of the Image deployment
phase which is done by the Ganeti OS Definition `snf-image`.
Finally, `snf-dispatcher` consumes messages from the RabbitMQ queues, processes
these messages and properly updates the state of the Cyclades DB. Subsequent
requests to the Cyclades API, will retrieve the updated state from the DB.
The values set for the user and password must be mirrored in the ``RABBIT_*``
variables in your settings, as managed by :ref:`snf-common <snf-common>`.
.. todo:: Document an active-active configuration based on the latest version
of RabbitMQ.
Work in progress. Please refer to :ref:`quick administrator quide <quick-install-admin-guide>`.
Work in progress. Please refer to :ref:`quick administrator quide <quick-install-admin-guide>`.
Work in progress. Please refer to :ref:`quick administrator quide <quick-install-admin-guide>`.
Working with Cyclades
Managing Ganeti Backends
Since v0.11, Synnefo is able to manage multiple Ganeti clusters (backends)
making it capable to scale linearly to tens of thousands of VMs. Backends
can be dynamically added or removed via `snf-manage` commands.
Each newly created VM is allocated to a Ganeti backend by the Cyclades backend
allocator. The VM is "pinned" to this backend, and can not change through its
lifetime. The backend allocator decides in which backend to spawn the VM based
on the available resources of each backend, trying to balance the load between
Handling of Networks, as far as backends are concerned, is based on whether the
network is public or not. Public networks are created through the `snf-manage
network-create` command, and are only created on one backend. Private networks
are created on all backends, in order to ensure that VMs residing on different
backends can be connected to the same private network.
Listing existing backends
To list all the Ganeti backends known to Synnefo, we run:
.. code-block:: console
$ snf-manage backend-list
Adding a new Ganeti backend
Backends are dynamically added under the control of Synnefo with `snf-manage
backend-add` command. In this section it is assumed that a Ganeti cluster,
named ```` is already up and running and configured to be
able to host Synnefo VMs.
To add this Ganeti cluster, we run:
.. code-block:: console
$ snf-manage backend-add --user="synnefo_user" --pass="synnefo_pass"
where ``clustername`` is the Cluster hostname of the Ganeti cluster, and
``user`` and ``pass`` are the credentials for the `Ganeti RAPI user
<>`_. All
backend attributes can be also changed dynamically using the `snf-manage
backend-modify` command.
``snf-manage backend-add`` will also create all existing private networks to
the new backend. You can verify that the backend is added, by running
`snf-manage backend-list`.
Note that no VMs will be spawned to this backend, since by default it is in a
``drained`` state after addition and also it has no public network assigned to
So, first you need to create its public network, make sure everything works as
expected and finally make it active by un-setting the ``drained`` flag. You can
do this by running:
.. code-block:: console
$ snf-manage backend-modify --drained=False <backend_id>
Removing an existing Ganeti backend
In order to remove an existing backend from Synnefo, we run:
.. code-block:: console
# snf-manage backend-remove <backend_id>
This command will fail if there are active VMs on the backend. Also, the
backend is not cleaned before removal, so all the Synnefo private networks
will be left on the Ganeti nodes. You need to remove them manually.
Allocation of VMs in Ganeti backends
As already mentioned, the Cyclades backend allocator is responsible for
allocating new VMs to backends. This allocator does not choose the exact Ganeti
node that will host the VM but just the Ganeti backend. The exact node is
chosen by the Ganeti cluster's allocator (hail).
The decision about which backend will host a VM is based on the available
resources. The allocator computes a score for each backend, that shows its load
factor, and the one with the minimum score is chosen. The admin can exclude
backends from the allocation phase by marking them as ``drained`` by running:
.. code-block:: console
$ snf-manage backend-modify --drained=True <backend_id>
The backend resources are periodically updated, at a period defined by
the ``BACKEND_REFRESH_MIN`` setting, or by running `snf-manage backend-update-status`
command. It is advised to have a cron job running this command at a smaller
interval than ``BACKEND_REFRESH_MIN`` in order to remove the load of refreshing
the backends stats from the VM creation phase.
Finally, the admin can decide to have a user's VMs being allocated to a
specific backend, with the ``BACKEND_PER_USER`` setting. This is a mapping
between users and backends. If the user is found in ``BACKEND_PER_USER``, then
Synnefo allocates all his/hers VMs to the specific backend in the variable,
even if is marked as drained (useful for testing).
Managing Virtual Machines
As mentioned, Cyclades uses Ganeti for management of VMs. The administrator can
handle Cyclades VMs just like any other Ganeti instance, via `gnt-instance`
commands. All Ganeti instances that belong to Synnefo, are separated from
others, by a prefix in their names. This prefix is defined in
``BACKEND_PREFIX_ID`` setting in
Apart from handling instances directly in the Ganeti level, a number of `snf-manage`
commands are available:
* ``snf-manage server-list``: List servers
* ``snf-manage server-show``: Show information about a server in the Cyclades DB
* ``snf-manage server-inspect``: Inspect the state of a server both in DB and Ganeti
* ``snf-manage server-modify``: Modify the state of a server in the Cycldes DB
* ``snf-manage server-create``: Create a new server
* ``snf-manage server-import``: Import an existing Ganeti instance to Cyclades
Managing Virtual Networks
Cyclades is able to create and manage Virtual Networks. Networking is
desployment specific and must be customized based on the specific needs of the
system administrator. For better understanding of networking please refer to
the :ref:`Network <networks>` section.
Exactly as Cyclades VMs can be handled like Ganeti instances, Cyclades Networks
can also by handled as Ganeti networks, via `gnt-network commands`. All Ganeti
networks that belong to Synnefo are named with the prefix
There are also the following `snf-manage` commands for managing networks:
* ``snf-manage network-list``: List networks
* ``snf-manage network-show``: Show information about a network in the Cyclades DB
* ``snf-manage network-inspect``: Inspect the state of the network in DB and Ganeti backends
* ``snf-manage network-modify``: Modify the state of a network in the Cycldes DB
* ``snf-manage network-create``: Create a new network
* ``snf-manage network-remove``: Remove an existing network
Managing Network Resources
Proper operation of the Cyclades Network Service depends on the unique
assignment of specific resources to each type of virtual network. Specifically,
these resources are:
* IP addresses. Cyclades creates a Pool of IPs for each Network, and assigns a
unique IP address to each VM, thus connecting it to this Network. You can see
the IP pool of each network by running `snf-manage network-inspect
<network_ID>`. IP pools are automatically created and managed by Cyclades,
depending on the subnet of the Network.
* Bridges corresponding to physical VLANs, which are required for networks of
* One Bridge corresponding to one physical VLAN which is required for networks of
Cyclades allocates those resources from pools that are created by the
administrator with the `snf-manage pool-create` management command.
Pool Creation
Pools are created using the `snf-manage pool-create` command:
.. code-block:: console
# snf-manage pool-create --type=bridge --base=prv --size=20
will create a pool of bridges, containing bridges prv1, prv2,..prv21.
You can verify the creation of the pool, and check its contents by running:
.. code-block:: console
# snf-manage pool-list
# snf-manage pool-show --type=bridge 1
With the same commands you can handle a pool of MAC prefixes. For example:
.. code-block:: console
# snf-manage pool-create --type=mac-prefix --base=aa:00:0 --size=65536
will create a pool of MAC prefixes from ``aa:00:1`` to ``b9:ff:f``. The MAC
prefix pool is responsible for providing only unicast and locally administered
MAC addresses, so many of these prefixes will be externally reserved, to
exclude from allocation.
Cyclades advanced operations
Reconciliation mechanism
On certain occasions, such as a Ganeti or RabbitMQ failure, the VM state in the
system's database may differ from that in the Ganeti installation. The
reconciliation process is designed to bring the system's database in sync with
what Ganeti knows about each VM, and is able to detect the following three
On certain occasions, such as a Ganeti or RabbitMQ failure, the state of
Cyclades database may differ from the real state of VMs and networks in the
Ganeti backends. The reconciliation process is designed to synchronize
the state of the Cyclades DB with Ganeti. There are two management commands
for reconciling VMs and Networks
Reconciling Virtual Machines
Reconciliation of VMs detects the following conditions:
* Stale DB servers without corresponding Ganeti instances
* Orphan Ganeti instances, without corresponding DB entries
* Out-of-sync operstate for DB entries wrt to Ganeti instances
* Out-of-sync state for DB entries wrt to Ganeti instances
The reconciliation mechanism runs as a management command, e.g., as follows:
[PYTHONPATH needs to contain the parent of the synnefo Django project
To detect all inconsistencies you can just run:
.. code-block:: console
$ snf-manage reconcile --detect-all -v 2
$ snf-manage reconcile-servers
Adding the `--fix-all` option, will do the actual synchronization:
.. code-block:: console
$ snf-manage reconcile --fix-all
Please see ``snf-manage reconcile --help`` for all the details.
The administrator can also trigger reconciliation of operating state manually,
by issuing a Ganeti ``OP_INSTANCE_QUERY_DATA`` command on a Synnefo VM, using
gnt-instance info.
Reconciling Networks
Reconciliation of Networks detects the following conditions:
* Stale DB networks without corresponding Ganeti networks
* Orphan Ganeti networks, without corresponding DB entries
* Private networks that are not created to all Ganeti backends
* Unsynchronized IP pools
To detect all inconsistencies you can just run:
.. code-block:: console
$ snf-manage reconcile-networks
Adding the `--fix-all` option, will do the actual synchronization:
.. code-block:: console
$ snf-manage reconcile-networks --fix-all
Please see ``snf-manage reconcile-networks --help`` for all the details.
......@@ -315,20 +568,6 @@ Archipelago advanced operations
.. _mail-server:
Configure mail server
In order to be able to send email (for example activation emails),
synnefo needs access to a running mail server. Your mail server should
be defined in the ``/etc/synnefo/00-snf-common-admins.conf``
related constants. At least:
.. code-block:: console
The "kamaki" API client
......@@ -411,6 +650,90 @@ To verify that the image was registered successfully use:
.. RabbitMQ
RabbitMQ Broker
Queue nodes run the RabbitMQ sofware, which provides AMQP functionality. To
guarantee high-availability, more than one Queue nodes should be deployed, each
of them belonging to the same `RabbitMQ cluster
<>`_. Synnefo uses the RabbitMQ
active/active `High Available Queues <>`_ which
are mirrored between two nodes within a RabbitMQ cluster.
The RabbitMQ nodes that form the cluster, are declared to Synnefo through the
`AMQP_HOSTS` setting. Each time a Synnefo component needs to connect to
RabbitMQ, one of these nodes is chosen in a random way. The client that Synnefo
uses to connect to RabbitMQ, handles connection failures transparently and
tries to reconnect to a different node. As long as one of these nodes are up
and running, functionality of Synnefo should not be downgraded by the RabbitMQ
node failures.
All the queues that are being used are declared as durable, meaning that
messages are persistently stored to RabbitMQ, until they get successfully
processed by a client.
Currently, RabbitMQ is used by the following components:
* `snf-ganeti-eventd`, `snf-ganeti-hook` and `snf-progress-monitor`:
These components send messages concerning the status and progress of
jobs in the Ganeti backend.
* `snf-dispatcher`: This daemon, consumes the messages that are sent from
the above components, and updates the Cyclades DB accordingly.
Please check the RabbitMQ documentation which covers extensively the
`installation of RabbitMQ server <>`_ and
the setup of a `RabbitMQ cluster <>`_.
Also, check out the `web management plugin
<>`_ that can be useful for managing and
monitoring RabbitMQ.
For a basic installation of RabbitMQ on two nodes (node1 and node2) you can do
the following:
On both nodes, install rabbitmq-server and create a Synnefo user:
.. code-block:: console
$ apt-get install rabbitmq-server
$ rabbitmqctl add_user synnefo "example_pass"
$ rabbitmqctl set_permissions synnefo ".*" ".*" ".*"
Also guarantee that both nodes share the same cookie, by running:
.. code-block:: console
$ scp node1:/var/lib/rabbitmq/.erlang.cookie node2:/var/lib/rabbitmq/.erlang.cookie
and restart the nodes:
.. code-block:: console
$ /etc/init.d/rabbitmq-server restart
To setup the RabbitMQ cluster run:
.. code-block:: console
root@node2: rabbitmqctl stop_app
root@node2: rabbitmqctl reset
root@node2: rabbitmqctl cluster rabbit@node1 rabbit@node2
root@node2: rabbitmqctl start_app
You can verify that the cluster is set up correctly by running:
.. code-block:: console
root@node2: rabbitmqctl cluster_status
Admin tool: snf-manage
......@@ -437,30 +760,16 @@
Note that this is a feature of Python 2.7 that we have backported for use in
Python 2.6.
The logging configuration dictionary is defined in settings.d/00-logging.conf
and is broken in 4 separate dictionaries:
* LOGGING is the logging configuration used by the web app. By default all
loggers fall back to the main 'synnefo' logger. The subloggers can be
changed accordingly for finer logging control. e.g. To disable debug
messages from the API set the level of 'synnefo.api' to 'INFO'.
* DISPATCHER_LOGGING is the logging configuration of the logic/
command line tool.
* SNFADMIN_LOGGING is the logging configuration of the snf-admin tool.
Consider using matching configuration for snf-admin and the synnefo.admin
logger of the web app.
Please note the following:
* As of Synnefo v0.7, by default the Django webapp logs to syslog, the
dispatcher logs to /var/log/synnefo/dispatcher.log and the console,
snf-admin logs to the console.
* Different handlers can be set to different logging levels:
for example, everything may appear to the console, but only INFO and higher
may actually be stored in a longer-term logfile
The logging configuration dictionary is defined in
The administrator can have finer logging control by modifying the
``LOGGING_SETUP`` dictionary, and defining subloggers with different handlers
and log levels. e.g. To enable debug messages only for the API set the level
of 'synnefo.api' to ``DEBUG``
By default, the Django webapp and snf-manage logs to syslog, while
`snf-dispatcher` logs to `/var/log/synnefo/dispatcher.log`.
Scaling up to multiple nodes
......@@ -73,7 +73,7 @@ Level section):
- assigning one physical VLAN per network
- assigning one MAC prefix per network, so that every NIC attached to this
network will have this prefix. Isolation is then achieved by filtering
rules (via `ebtables`) based on a specific mask (ff:ff:ff:00:00:00, see Node
rules (via `ebtables`) based on a specific mask (ff:ff:f0:00:00:00, see Node
Level section for more details).
Having this in mind and in order to prevent assignment of duplicate VLAN/MAC
......@@ -94,28 +94,40 @@ Ganeti Level section):
Existing network flavors are the following:
- ``DEFAULT``: { bridged, br0, aa:00:00, [] }
- ``IP_LESS_ROUTED``: { routed, snf_public, aa:00:00, [ip-less-routed] }
- ``MAC_FILTERED``: { bridged, br0, pool, [mac-filtered] }
- ``PHYSICAL_VLAN``: { bridged, pool, aa:00:00, [physical-vlan] }
- ``CUSTOM``: {}
============== ======= =============================== ====================== ==================
Flavor Name Mode Link MAC prefix Tags
============== ======= =============================== ====================== ==================
MAC_FILTERED bridged ``DEFAULT_MAC_FILTERED_BRIDGE`` 'pool' 'private'filtered'
PHYSICAL_VLAN bridged 'pool' ``DEFAULT_MAC_PREFIX`` 'physical-vlan'
============== ======= =============================== ====================== ==================
are all configurable settings in ``/etc/synnefo/20-snf-cyclades-app-api.conf``. 'pool' is used
to denote that a link or MAC prefix will be allocated from the corresponging Pool.
The administrator is able to create any of the above flavors
and override their default values by explicitly passing mode, link, etc. using
the `snf-manage network-create` command.
The end-user is allowed to create only networks of flavor ``MAC_FILTERED`` and
``PHYSICAL_VLAN``. The administrator is able to create any of the above flavors or
explicitly define any of their options (mode, link, etc..) using the
`snf-manage network-create` command. In this case the flavor of the network is
marked as ``CUSTOM`` and cannot make use of existing pools. Because of that
link or mac uniqueness cannot be guaranteed.
``PHYSICAL_VLAN``. Currently, only ``MAC_FILTERED`` and ``PHYSICAL_VLAN`` can
use existing pools and cannot be overriden.
Network @ Ganeti level
Currently, Ganeti does not support IP Pool management. However, we've been
actively in touch with the official Ganeti team, who are reviewing a relatively
big patchset that implements this functionality. We hope that the functionality
will be merged to the Ganeti master branch soon and appear on Ganeti 2.7.
You can find it in stable-2.6-grnet
(among with hotplug and external storage interface support).
Currently, stable Ganeti does not support IP Pool management. However, the
functionality has been merged in the official Ganeti master branch and will
appear on Ganeti 2.7.0. So, you can either checkout the Ganeti master branch
and build your packages, or clone our local repo and checkout the
`stable-2.6-ippool-hotplug-esi` branch. This is the Ganeti stable branch with
IP pool management, Hotplugging and ExtStorage Interface features merged on top
of it. The last two features are not a hard Synnefo requirement, but will
enable you to do neat things when you get experienced with Synnefo. They are
going to be pushed for review upstream sometime soon.
Any network created in Synnefo is also created in one (for public networks) or
all (for private networks) Ganeti backends. In Ganeti a network can have the
......@@ -147,7 +159,7 @@ networks in Synnefo. According to which flavors you want to support, you should
have already setup all your physical hosts correspondingly. This means you
- one bridge for the ``DEFAULT`` flavor (br0, see Fig. 1)
- one bridge for the ``CUSTOM`` flavor (br0, see Fig. 1)
- one bridge for the ``MAC_FILTERED`` flavor (prv0, see Fig. 2)
- a number of bridges and their corresponding VLANs (bridged to them) for
the ``PHYSICAL_VLAN`` flavor (prv1..prv100, see Fig. 3)
......@@ -163,24 +175,27 @@ FLAVORS
As mentioned earlier supported flavors are:
In the following sections we mention what configuration imposes each flavor from
Synnefo, Ganeti and Physical host perspective.
In this case we will bridge all primary interfaces of the VMs on one bridge that must
be the same collition domain with the router. The router sould then forward packets
(if a public IPv4 Subnet is available) or do NAT in order to provide internet access to
the VMs.
To create a network with DEFAULT flavor run you have to pre-provision in each Ganeti
node one bridge (e.g. ``br100``) that will be on the same collition domain with the
router. To this end if we assume that ``eth0`` is the public interface run:
To this end we will use the CUSTOM flavor and pre-provision in each Ganeti
node one bridge (e.g. ``br100``). If we assume that ``eth1`` is the physical interface
connected to the router, run:
.. image:: images/network-bridged.png
:align: right
......@@ -190,14 +205,12 @@ router. To this end if we assume that ``eth0`` is the public interface run:
.. code-block:: console
# brctl addbr br100
# vconfig add eth0 100
# ip link set eth0.100 up
# brctl addif br100 eth0.100
# brctl addif br100 eth1
# ip link set br100 up
# brctl show
bridge name bridge id STP enabled interfaces
br100 8000.8a3c3ede3583 no eth0.100
br100 8000.8a3c3ede3583 no eth1
......@@ -205,11 +218,11 @@ Then in Cyclades run:
.. code-block:: console
# snf-manage network-create --subnet= --gateway= --subnet6=2001:648:2FFC:1322::/64 --gateway6=2001:648:2FFC:1322::1 --public --dhcp --flavor=DEFAULT --name=default --backend-id=1
# snf-manage network-create --subnet= --gateway= --subnet6=2001:648:2FFC:1322::/64 --gateway6=2001:648:2FFC:1322::1 --public --dhcp --flavor=CUSTOM --link=br100 ----name=default --backend-id=1
# snf-manage network-list
id name flavor owner mac_prefix dhcp state link vms public IPv4 Subnet IPv4 Gateway
1 default DEFAULT True ACTIVE br100 True
1 default CUSTOM True ACTIVE br100 True
This will add a network in Synnefo DB and create a network in Ganeti backend by
......@@ -225,11 +238,12 @@ issuing:
To enable NAT in a Internal Router if you do not have a public IP range available
but only a public routable IP (e.g
but only a public routable IP (e.g
.. code-block:: console
# iptables -t nat -A POSTROUTING -o eth0.100 --to-source -j SNAT
# ip addr add dev eth1
# iptables -t nat -A POSTROUTING -o eth1 --to-source -j SNAT
......@@ -242,22 +256,17 @@ IP_LESS_ROUTED
To create a network with IP_LESS_ROUTED flavor run you have to pre-provision in
each Ganeti node one routing table (e.g. ``snf_public``) that will do all the
routing from/to the VMs' taps. Additionally you must enable ``Proxy-ARP``
support. All traffic will be on a single VLAN (e.g. ``.201``). To this end if
we assume that ``eth0`` is the public interface run: