admin-guide.rst 106 KB
Newer Older
1 2 3 4 5 6 7 8
.. _admin-guide:

Synnefo Administrator's Guide

This is the complete Synnefo Administrator's Guide.

.. _syn+archip:

11 12 13
General Synnefo Architecture

14 15 16 17
The following figure shows a detailed view of the whole Synnefo architecture
and how it interacts with multiple Ganeti clusters. We hope that after reading
the Administrator's Guide you will be able to understand every component and
all the interactions between them.

.. image:: images/synnefo-arch2.png
   :width: 100%
   :target: _images/synnefo-arch2.png

23 24 25
Synnefo also supports RADOS as an alternative storage backend for
Files/Images/VM disks. You will find the :ref:`corresponding figure
<syn+archip+rados>` later in this guide.


28 29
Identity Service (Astakos)
30 31

Authentication methods

Astakos supports multiple authentication methods:

37 38 39 40 41 42
 * local username/password
 * LDAP / Active Directory
 * SAML 2.0 (Shibboleth) federated logins
 * Google
 * Twitter
 * LinkedIn
43 44 45 46

.. _shibboleth-auth:

Shibboleth Authentication
48 49 50 51 52 53 54 55 56 57 58 59 60 61

Astakos can delegate user authentication to a Shibboleth federation.

To setup shibboleth, install package::

  apt-get install libapache2-mod-shib2

Change appropriately the configuration files in ``/etc/shibboleth``.

Add in ``/etc/apache2/sites-available/synnefo-ssl``::

  ShibConfig /etc/shibboleth/shibboleth2.xml
  Alias      /shibboleth-sp /usr/share/shibboleth

  <Location /ui/login/shibboleth>
63 64 65 66 67 68 69 70 71 72 73 74 75
    AuthType shibboleth
    ShibRequireSession On
    ShibUseHeaders On
    require valid-user

and before the line containing::

  ProxyPass        / http://localhost:8080/ retry=0


  ProxyPass /Shibboleth.sso !

77 78 79
Then, enable the shibboleth module::

  a2enmod shib2

81 82 83 84 85 86 87 88 89 90 91
After passing through the apache module, the following tokens should be
available at the destination::

  eppn # eduPersonPrincipalName

92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112
Astakos keeps a map of shibboleth users using the value of the ``REMOTE_USER``
header, passed by the ``mod_shib2`` module. This happens in order to be able to
identify the astakos account the shibboleth user is associated to, every time
the user logs in from an affiliate shibboleth IdP. 

The shibboleth attribute which gets mapped to the ``REMOTE_USER`` header can be
changed in ``/etc/shibboleth/shibboleth2.xml`` configuration file.

.. code-block:: xml

    <!-- The ApplicationDefaults element is where most of Shibboleth's SAML bits are defined. -->
        <ApplicationDefaults entityID="" 
         REMOTE_USER="eppn persistent-id targeted-id">

.. warning::

 Changing ``mod_shib2`` ``REMOTE_USER`` to map to different shibboleth
 attributes will probably invalidate any existing shibboleth enabled users in
 astakos database. Those users won't be able to login to their existing accounts.

113 114 115
Finally, add 'shibboleth' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
Twitter Authentication
Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
118 119 120 121 122 123 124 125 126 127 128 129 130 131

To enable twitter authentication while signed in under a Twitter account,

Click Create an application.

Fill the necessary information and for callback URL give::

Finally, add 'twitter' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Google Authentication
Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148

To enable google authentication while signed in under a Google account,

Under API Access select Create another client ID, select Web application,
expand more options in Your site or hostname section and in Authorized
Redirect URIs add:

Fill the necessary information and for callback URL give::

Finally, add 'google' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

149 150 151 152

Working with Astakos

153 154
User registration

When a new user signs up, he/she is not directly marked as active. You can see
his/her state by running (on the machine that runs the Astakos app):
158 159 160

.. code-block:: console

   $ snf-manage user-list

More detailed user status is provided in the `status` field of the `user-show`

.. code-block:: console

  $ snf-manage user-show <user-id>

170 171 172 173 174
  id                  : 6
  uuid                : 78661411-5eed-412f-a9ea-2de24f542c2e
  status              : Accepted/Active (accepted policy: manual)
  email               :

176 177 178
Based on the `astakos-app` configuration, there are several ways for a user to
get verified and activated in order to be able to login. We discuss the user
verification and activation flow in the following section.
179 180

User activation flow
181 182

183 184 185 186 187 188 189 190 191 192
A user can register for an account using the astakos signup form. Once the form
is submited successfully a user entry is created in astakos database. That entry
is passed through the astakos activation backend which handles whether the user
should be automatically verified and activated.

Email verification

The verification process takes place in order to ensure that the user owns the
email provided during the signup process. By default, after each successful
signup astakos notifies user with an verification url via email.
194 195 196

At this stage:

    * subsequent registrations invalidate and delete the previous registrations
198 199 200 201 202 203 204 205 206 207 208 209 210
      of the same email address.

    * in case user misses the initial notification, additional emails can be
      send either via the url which is prompted to the user if he tries to
      login, or by the administrator using the ``snf-manage user-activation-send
      <userid>`` command.

    * administrator may also enforce a user to get verified using the
      ``snf-manage user-modify --verify <userid>`` command.

Account activation

Once the user gets verified, it is time for Astakos to decide whether or not to
proceed through user activation process. If ``ASTAKOS_MODERATION_ENABLED``
setting is set to ``False`` (default value) user gets activated automatically.

In case the moderation is enabled Astakos may still automatically activate the
216 217 218 219 220
user in the following cases:

    * User email matches any of the regular expressions defined in
      ``ASTAKOS_RE_USER_EMAIL_PATTERNS`` (defaults to ``[]``)
    * User used a signup method (e.g. ``shibboleth``) for which automatic
      activation is enabled (see
222 223
      :ref:`authentication methods policies <auth_methods_policies>`).

224 225 226 227 228 229
If all of the above fail to trigger automatic activation, an email is sent to
the persons listed in ``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings,
notifing that there is a new user pending for moderation and that it's up to
the administrator to decide if the user should be activated. The UI also shows
a corresponding 'pending moderation' message to the user. The administrator can
activate a user using the ``snf-manage user-modify`` command:
230 231 232 233 234 235 236 237 238

.. code-block:: console

    # command to activate a pending user
    $ snf-manage user-modify --accept <userid>

    # command to reject a pending user
    $ snf-manage user-modify --reject --reject-reason="spammer" <userid>

239 240 241 242
Once the activation process finishes, a greeting message is sent to the user
email address and a notification for the activation to the persons listed in
``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings. Once activated the user is
able to login and access the Synnefo services.
243 244 245 246 247

Additional authentication methods

Astakos supports third party logins from external identity providers. This
can be usefull since it allows users to use their existing credentials to
249 250 251 252 253 254 255 256 257 258 259 260 261
login to astakos service.

Currently astakos supports the following identity providers:

    * `Shibboleth <>`_ (module name
    * `Google <>`_ (module
      name ``google``)
    * `Twitter <>`_ (module name ``twitter``)
    * `LinkedIn <>`_
      (module name ``linkedin``)

To enable any of the above modules (by default only ``local`` accounts are
allowed), retrieve and set the required provider settings and append the
263 264 265 266 267 268 269
module name in ``ASTAKOS_IM_MODULES``.

.. code-block:: python

    # settings from

271 272 273 274 275 276 277 278 279
    # let users signup and login using their google account
    ASTAKOS_IM_MODULES = ['local', 'google']

.. _auth_methods_policies:

Authentication method policies

280 281
Astakos allows you to override the default policies for each enabled provider
separately by adding the approriate settings in your ``.conf`` files in the
282 283 284 285 286 287
following format:


Available policies are:

288 289
    * **CREATE** Users can signup using that provider (default: ``True``)
    * **REMOVE/ADD** Users can remove/add login method from their profile
290 291 292 293 294 295
      (default: ``True``)
    * **AUTOMODERATE** Automatically activate users that signup using that
      provider (default: ``False``)
    * **LOGIN** Whether or not users can use the provider to login (default:

e.g. to enable automatic activation for your academic users, while keeping
297 298 299 300 301 302 303
locally signed up users under moderation you can apply the following settings.

.. code-block:: python


304 305 306 307 308 309 310 311 312 313 314 315 316 317
User login

During the logging procedure, the user is authenticated by the respective
identity provider.

If ``ASTAKOS_RECAPTCHA_ENABLED`` is set and the user fails several times
(``ASTAKOS_RATELIMIT_RETRIES_ALLOWED`` setting) to provide the correct
credentials for a local account, he/she is then prompted to solve a captcha

Upon success, the system renews the token (if it has expired), logins the user
and sets the cookie, before redirecting the user to the ``next`` parameter

319 320
Projects and quota

322 323 324 325 326 327
Synnefo supports granting resources and controling their quota through the
mechanism of *projects*. A project is considered as a pool of finite
resources. Every actual resources allocated by a user (e.g. a Cyclades VM or
a Pithos container) is also assigned to a project where the user is a
member to. For each resource a project specifies the maximum amount that can
be assigned to it and the maximum amount that a single member can assign to it.

329 330
Default quota

332 333 334
Upon user creation, a special purpose user-specific project is automatically
created in order to hold the base quota provided by the system. These *base*
projects are identified with the same UUID as the user.

336 337
To inspect the quota that future users will receive by default through their
base projects, check column ``base_default`` in::

   # snf-manage resource-list

You can modify the default base quota limit for all future users with::

   # snf-manage resource-modify <resource_name> --base-default <value>

345 346
Grant extra quota through projects

348 349 350
A user can apply for a new project through the web interface or the API.
Once it is approved by the administrators, the applicant can join the
project and let other users in too.

352 353
A project member can make use of the quota granted by the project by
specifying this particular project when creating a new quotable entity.

355 356 357 358 359
Note that quota are not accumulative: in order to allocate a 100GB disk,
one must be in a project that grants at least 100GB; it is not possible to
add up quota from different projects. Note also that if allocating an entity
requires multiple resources (e.g. cpu and ram for a Cyclades VM) these must
be all assigned to a single project.

361 362
Control projects
363 364 365 366 367 368 369 370 371 372 373 374 375 376

To list pending project applications in astakos::

    # snf-manage project-list --pending

Note the last column, the application id. To approve it::

    # <app id> from the last column of project-list
    # snf-manage project-control --approve <app id>

To deny an application::

    # snf-manage project-control --deny <app id>

377 378 379 380 381 382 383 384 385
Before taking an action, on can inspect project status, settings and quota
limits with::

   # snf-manage project-show <project-uuid>

For an initialized project, option ``--quota`` also reports the resource

Users designated as *project admins* can approve or deny
386 387 388 389 390
an application through the web interface. In
``20-snf-astakos-app-settings.conf`` set::

    # UUIDs of users that can approve or deny project applications from the web.
    ASTAKOS_PROJECT_ADMINS = [<uuid>, ...]

392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409
Set quota limits

One can change the quota limits of an initialized project with::

   # snf-manage project-modify <project-uuid> --limit <resource_name> <member_limit> <project_limit>

One can set base quota for all accepted users (that is, set limits for base
project), with possible exceptions, with::

   # snf-manage project-modify --all-base-projects --exclude <uuid1>,<uuid2> --limit ...

Quota for a given resource are reported for all projects that the user is
member in with::

   # snf-manage user-show <user-uuid> --quota

With option ``--projects``, owned projects and memberships are also reported.

411 412 413 414 415 416 417 418 419 420 421 422
Astakos advanced operations

Adding "Terms of Use"

Astakos supports versioned terms-of-use. First of all you need to create an
html file that will contain your terms. For example, create the file
``/usr/share/synnefo/sample-terms.html``, which contains the following:

.. code-block:: console

   <h1>My cloud service terms</h1>

   These are the example terms for my cloud service
426 427 428 429 430 431 432 433 434 435

Then, add those terms-of-use with the snf-manage command:

.. code-block:: console

   $ snf-manage term-add /usr/share/synnefo/sample-terms.html

Your terms have been successfully added and you will see the corresponding link
appearing in the Astakos web pages' footer.

436 437 438 439 440 441 442 443 444
During the account registration, if there are approval terms, the user is
presented with an "I agree with the Terms" checkbox that needs to get checked
in order to proceed.

In case there are new approval terms that the user has not signed yet, the
``signed_terms_required`` view decorator redirects to the ``approval_terms``
view, so the user will be presented with the new terms the next time he/she

445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470
Enabling reCAPTCHA

Astakos supports the `reCAPTCHA <>`_ feature.
If enabled, it protects the Astakos forms from bots. To enable the feature, go
to and create your own reCAPTCHA
key pair. Then edit ``/etc/synnefo/20-snf-astakos-app-settings.conf`` and set
the corresponding variables to reflect your newly created key pair. Finally, set
the ``ASTAKOS_RECAPTCHA_ENABLED`` variable to ``True``:

.. code-block:: console

   ASTAKOS_RECAPTCHA_PUBLIC_KEY = 'example_recaptcha_public_key!@#$%^&*('
   ASTAKOS_RECAPTCHA_PRIVATE_KEY = 'example_recaptcha_private_key!@#$%^&*('


Restart the service on the Astakos node(s) and you are ready:

.. code-block:: console

   # /etc/init.d/gunicorn restart

Checkout your new Sign up page. If you see the reCAPTCHA box, you have setup
everything correctly.


472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538
Astakos internals


Alice requests a specific resource from a cloud service e.g.: Pithos. In the
request she supplies the `X-Auth-Token` to identify whether she is eligible to
perform the specific task. The service contacts Astakos through its
``/account/v1.0/authenticate`` api call (see :ref:`authenticate-api-label`)
providing the specific ``X-Auth-Token``. Astakos checkes whether the token
belongs to an active user and it has not expired and returns a dictionary
containing user related information. Finally the service uses the ``uniq``
field included in the dictionary as the account string to identify the user
accessible resources.

.. _authentication-label:

Django Auth methods and Backends

Astakos incorporates Django user authentication system and extends its User model.

Since username field of django User model has a limitation of 30 characters,
AstakosUser is **uniquely** identified by the ``email`` instead. Therefore,
```` is served to authenticate a
user using email if the first argument is actually an email, otherwise tries
the username.

A new AstakosUser instance is assigned with a uui as username and also with a
``auth_token`` used by the cloud services to authenticate the user.
```` is also specified in order
to authenticate the user using the email and the token fields.

Logged on users can perform a number of actions:

 * access and edit their profile via: ``/im/profile``.
 * change their password via: ``/im/password``
 * send feedback for grnet services via: ``/im/send_feedback``
 * logout (and delete cookie) via: ``/im/logout``

Internal Astakos requests are handled using cookie-based Django user sessions.

External systems should forward to the ``/login`` URI. The server,
depending on its configuration will redirect to the appropriate login page.
When done with logging in, the service's login URI should redirect to the URI
provided with next, adding user and token parameters, which contain the email
and token fields respectively.

The login URI accepts the following parameters:

======================  =========================
Request Parameter Name  Value
======================  =========================
next                    The URI to redirect to when the process is finished
renew                   Force token renewal (no value parameter)
force                   Force logout current user (no value parameter)
======================  =========================

External systems inside the ``ASTAKOS_COOKIE_DOMAIN`` scope can acquire the
user information by the cookie identified by ``ASTAKOS_COOKIE_NAME`` setting
(set during the login procedure).

Finally, backend systems having acquired a token can use the
:ref:`authenticate-api-label` API call from a private network or through HTTPS.

539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654
File/Object Storage Service (Pithos+)

Pithos+ is the Synnefo component that implements a storage service and exposes
the associated OpenStack REST APIs with custom extensions.

Pithos+ advanced operations

Enable separate domain for serving user content

Since Synnefo v0.15, there is a possibility to serve untrusted user content
in an isolated domain.

Enabling this feature consists of the following steps:

#. **Declare new domain in apache server**

   In order to enable the apache server to serve several domains it is required
   to setup several virtual hosts.
   Therefore, for adding the new domain e.g. "", append
   the following in ``/etc/apache2/sites-available/synnefo-ssl``:

    .. code-block:: console

        <VirtualHost _default_:443>

            Alias /static "/usr/share/synnefo/static"

            #  SetEnv no-gzip
            #  SetEnv dont-vary

           AllowEncodedSlashes On

           RequestHeader set X-Forwarded-Protocol "https"

        <Proxy * >
            Order allow,deny
            Allow from all

            SetEnv                proxy-sendchunked
            SSLProxyEngine        off
            ProxyErrorOverride    off

            ProxyPass        /static !
            ProxyPass        / http://localhost:8080/ retry=0
            ProxyPassReverse / http://localhost:8080/

            RewriteEngine On
            RewriteCond %{THE_REQUEST} ^.*(\\r|\\n|%0A|%0D).* [NC]
            RewriteRule ^(.*)$ - [F,L]

            SSLEngine on
            SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
            SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

    .. note:: Consider also to purchase and install a certificate for the new

    Finally, restart the apache server::

        pithos-host$ /etc/init.d/apache2 restart

#. **Register Pithos+ as an OAuth2 client in Astakos**

   Starting from synnefo version 0.15, in order to view the content of a
   protected resource, Pithos+ (on behalf of the user) has to be granted
   authorization for the specific resource by Astakos.

   During the authorization grant procedure, Pithos+ has to authenticate
   itself with Astakos since the latter has to prevent serving requests by
   unknown/unauthorized clients.

   Therefore, in the installation guide you were guided to register Pithos+
   as an OAuth2 client in Astakos.

   .. note:: You can see the registered clients by running::
    astakos-host$ snf-manage oauth2-client-list -o identifier,redirect_urls,is_trusted

   However, requests originated from the new domain will be rejected since
   Astakos is ignorant about the new domain.

   Therefore, you need to register a new client pointing to the unsafe domain.
   To do so, use the following command::

        astakos-host$ snf-manage oauth2-client-add pithos-unsafe-domain --secret=<secret> --is-trusted --url

   .. note:: You can also unregister the client pointing to the safe domain,
       since it will no longer be useful.
       To do so, run the following::

        astakos-host$ snf-manage oauth2-client-remove pithos-view

#. **Update Pithos+ configuration**

   Respectively, the ``PITHOS_OAUTH2_CLIENT_CREDENTIALS`` setting should be
   updated to contain the credentials of the client registered in the previous

   Furthermore, you need to restrict all the requests for user content
   to be served exclusively by the unsafe domain.

   To enable this, set the ``PITHOS_UNSAFE_DOMAIN`` setting to the value
   of the new domain e.g. ""

   Finally, restart the gunicorn server::

        pithos-host$ /etc/init.d/gunicorn restart

655 656 657
Compute/Network/Image Service (Cyclades)

658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681

Cyclades is the Synnefo component that implements Compute, Network and Image
services and exposes the associated OpenStack REST APIs. By running Cyclades
you can provide a cloud that can handle thousands of virtual servers and

Cyclades does not include any virtualization software and knows nothing about
the low-level VM management operations, e.g. handling of VM creation or
migrations among physical nodes. Instead, Cyclades is the component that
handles multiple Ganeti backends and exposes the REST APIs. The administrator
can expand the infrastructure dynamically either by adding more Ganeti nodes
or by adding new Ganeti clusters. Cyclades issue VM control commands to Ganeti
via Ganeti's remote API and receive asynchronous notifications from Ganeti
backends whenever the state of a VM changes, due to Synnefo- or
administrator-initiated commands.

Cyclades is the action orchestrator and the API layer on top of multiple Ganeti
clusters. By this decoupled design, Ganeti cluster are self-contained and
the administrator has complete control on them without Cyclades knowing about
it. For example a VM migration to a different physical node is transparent
to Cyclades.

682 683 684
Working with Cyclades

685 686 687 688 689 690 691 692 693 694

When creating a VM, the user must specify the `flavor` of the virtual server.
Flavors are the virtual hardware templates, and provide a description about
the number of CPUs, the amount of RAM, and the size of the disk of the VM.
Besides the size of the disk, Cyclades flavors describe the storage backend
that will be used for the virtual server.

Flavors are created by the administrator and the user can select one of the
695 696
available flavors. After VM creation, the user can resize his VM, by
adding/removing CPU and RAM.
697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720

Cyclades support different storage backends that are described by the disk
template of the flavor, which is mapped to Ganeti's instance `disk template`.
Currently the available disk templates are the following:

* `file`: regulars file
* `sharedfile`: regular files on a shared directory, e.g. NFS
* `plain`: logical volumes
* `drbd`: drbd on top of lvm volumes
* `rbd`: rbd volumes residing inside a RADOS cluster
* `ext`: disks provided by an external shared storage.

  - `ext_archipelago`: External shared storage provided by
    `Archipelago <>`_.

Flavors are created by the administrator using `snf-manage flavor-create`
command. The command takes as argument number of CPUs, amount of RAM, the size
of the disks and the disk templates and create the flavors that belong to the
cartesian product of the specified arguments. For example, the following
command will create two flavors of `40G` disk size with `drbd` disk template,
`4G` RAM and `2` or `4` CPUs.

.. code-block:: console

  $ snf-manage flavor-create 2,4 4096 40 drbd

723 724
To see the available flavors, run `snf-manage flavor-list` command. The
administrator can delete a flavor by using `flavor-modify` command:
725 726 727

.. code-block:: console

  $ snf-manage flavor-modify --deleted=True <flavor_id>

730 731 732 733 734 735 736 737 738 739 740
Finally, the administrator can set if new servers can be created from a flavor
or not, by setting the `allow_create` attribute:

.. code-block:: console

  $ snf-manage flavor-modify --allow-create=False <flavor_id>

Flavors that are marked with `allow_create=False` cannot be used by users to
create new servers. However, they can still be used to resize existing VMs.

741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756

When creating a VM the user must also specify the `image` of the virtual
server. Images are the static templates from which VM instances are
initiated. Cyclades uses Pithos to store system and user-provided images,
taking advantage of all Pithos features, like deduplication and syncing
protocol. An image is a file stored to Pithos with additional metadata that
are describing the image, e.g. the OS family or the root partition. To create
a new image, the administrator or the user has to upload it a file to Pithos,
and then register it as an Image with Cyclades. Then the user can use this
image to spawn new VMs from it.

Images can be private, public or shared between users, exactly like Pithos
files. Since user-provided public images can be untrusted, the administrator
can denote which users are trusted by adding them to the
``UI_SYSTEM_IMAGES_OWNERS`` setting in the
758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779
`/etc/synnefo/20-snf-cyclades-app-ui.conf` file. Images of those users are
properly displayed in the UI.

When creating a new VM, Cyclades pass the location of the image and it's
metadata to Ganeti. After Ganeti creates the instance's disk, `snf-image`
will copy the image to the new disk and perform the image customization
phase. During the phase, `snf-image` sends notifications to Cyclades about
the progress of the image deployment and customization. Customization includes
resizing the root file system, file injection (e.g. SSH keys) and setting
a custom hostname. For better understanding of `snf-image` read the
corresponding `documentation

For passing sensitive data about the image to Ganeti, like the VMs password,
Cyclades keeps all sensitive data in memory caches (memcache) and never allows
them to hit the disk. The data are exposed to `snf-image` via an one-time URL
that is exposed from the `vmapi` application. So, instead of passing sensitive
data to `snf-image` via Ganeti, Cyclades pass an one-time configuration URL
that contains a random UUID. After `snf-image` gets the sensitive data, the
URL is invalidated so no one else can access them.

The administrator can register images, exactly like users, using a system user
(a user that is defined in the ``UI_SYSTEM_IMAGES_OWNERS`` setting). For
781 782 783 784 785 786
example, the following command will register the
`pithos://u53r-un1qu3-1d/images/debian_base-6.0-7-x86_64.diskdump` as an
image to Cyclades:

.. code-block:: console

 $ kamaki image register --name="Debian Base" \
        --location=pithos://u53r-un1qu3-1d/images/debian_base-6.0-7-x86_64.diskdump \
789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827
        --public \
        --disk-format=diskdump \
        --property OSFAMILY=linux --property ROOT_PARTITION=1 \
        --property description="Debian Squeeze Base System" \
        --property size=451 --property kernel=2.6.32 --property GUI="No GUI" \
        --property sortorder=1 --property USERS=root --property OS=debian

Deletion of an image is done via `kamaki image unregister` command, which will
delete the Cyclades Images but will leave the Pithos file as is (unregister).

Apart from using `kamaki` to see and hangle the available images, the
administrator can use `snf-manage image-list` and `snf-manage image-show`
commands to list and inspect the available public images. Also, the `--user-id`
option can be used the see the images of a specific user.

Virtual Servers

As mentioned, Cyclades uses Ganeti for management of VMs. The administrator can
handle Cyclades VMs just like any other Ganeti instance, via `gnt-instance`
commands. All Ganeti instances that belong to Synnefo, are separated from
others, by a prefix in their names. This prefix is defined in
``BACKEND_PREFIX_ID`` setting in

Apart from handling Cyclades VM at the Ganeti level, the administrator can
also use the `snf-manage server-*` commands. These command cover the most
common tasks that are relative with VM handling. Below we describe come
of them, but for more information you can use the `--help` option of all
`snf-manage server-* commands`. These command cover the most

The `snf-manage server-create` command can be used to create a new VM for some
user. This command can be useful when the administrator wants to test Cyclades
functionality without starting the API service, e.g. after an upgrade. Also, by
using `--backend-id` option, the VM will be created in the specified backend,
bypassing automatic VM allocation.

.. code-block:: console

 $ snf-manage server-create --flavor-id=1 --image-id=fc0f6858-f962-42ce-bf9a-1345f89b3d5e \
829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846
    --user-id=7cf4d078-67bf-424d-8ff2-8669eb4841ea --backend-id=2 \
    --password='example_passw0rd' --name='test_vm'

The above commnd will create a new VM for user
`7cf4d078-67bf-424d-8ff2-8669eb4841ea` in the Ganeti backend with ID 2. By
default this command will issue a Ganeti job to create the VM
(`OP_INSTANCE_CREATE`) and return. As in other commands, the `--wait=True`
option can be used in order to wait for the successful completion of the job.

`snf-manage server-list` command can be used to list all the available servers.
The command supports some useful options, like listing servers of a user,
listing servers that exist in a Ganeti backend and listing deleted servers.
Also, as in most of `*-list` commands, the `--filter-by` option can be used to
filter the results. For example, the following command will only display the
started servers of a specific flavor:

.. code-block:: console

 $ snf-manage server-list --filter-by="operstate=STARTED,flavor=<flavor_id>"
848 849 850 851 852 853 854 855 856 857 858

Another very useful command is the `server-inspect` command which will display
all available information about the state of the server in DB and the state
of the server in the Ganeti backend. The output will give you an easy overview
about the state of the VM which can be useful for debugging.

Also the administrator can `suspend` a user's VM, using the `server-modify`

.. code-block:: console

 $ snf-manage server-modify --suspended=True <server_id>
860 861 862 863 864 865

The user is forbidden to do any action on an administratively suspended VM,
which is useful for abuse cases.

Ganeti backends

867 868 869
Since v0.11, Synnefo is able to manage multiple Ganeti clusters (backends)
making it capable to scale linearly to tens of thousands of VMs. Backends
can be dynamically added or removed via `snf-manage` commands.

871 872 873 874
Each newly created VM is allocated to a Ganeti backend by the Cyclades backend
allocator. The VM is "pinned" to this backend, and can not change through its
lifetime. The backend allocator decides in which backend to spawn the VM based
on the available resources of each backend, trying to balance the load between
875 876
them. Also, Networks are created to all Ganeti backends, in order to ensure
that VMs residing on different backends can be connected to the same networks.

878 879 880 881 882 883 884
A backend can be marked as `drained` in order to be excluded from automatic
servers allocation and not receive new servers. Also, a backend can be marked
as `offline` to denote that the backend is not healthy (e.g. broken master)
and avoid the penalty of connection timeouts.

Finally, Cyclades is able to manage Ganeti backends with different enabled
hypervisors (`kvm`, `xen`), and different enabled disk templates.

886 887 888
Listing existing backends
To list all the Ganeti backends known to Synnefo, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed

890 891 892 893 894
.. code-block:: console

   $ snf-manage backend-list

Adding a new Ganeti backend
Christos Stavrakakis's avatar
Christos Stavrakakis committed
896 897
Backends are dynamically added under the control of Synnefo with `snf-manage
backend-add` command. In this section it is assumed that a Ganeti cluster,
898 899
named ```` is already up and running and configured to be
able to host Synnefo VMs.

To add this Ganeti cluster, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed

903 904 905 906
.. code-block:: console

   $ snf-manage backend-add --user="synnefo_user" --pass="synnefo_pass"

907 908
where ``clustername`` is the Cluster hostname of the Ganeti cluster, and
``user`` and ``pass`` are the credentials for the `Ganeti RAPI user
<>`_.  All
910 911
backend attributes can be also changed dynamically using the `snf-manage
backend-modify` command.

``snf-manage backend-add`` will also create all existing public networks to
914 915
the new backend. You can verify that the backend is added, by running
`snf-manage backend-list`.

Note that no VMs will be spawned to this backend, since by default it is in a
918 919
``drained`` state after addition in order to manually verify the state of the

921 922
So, after making sure everything works as expected, make the new backend active
by un-setting the ``drained`` flag. You can do this by running:
923 924 925

.. code-block:: console

   $ snf-manage backend-modify --drained=False <backend_id>

928 929 930 931 932 933
Allocation of VMs in Ganeti backends
As already mentioned, the Cyclades backend allocator is responsible for
allocating new VMs to backends. This allocator does not choose the exact Ganeti
node that will host the VM but just the Ganeti backend. The exact node is
chosen by the Ganeti cluster's allocator (hail).
934 935 936 937

The decision about which backend will host a VM is based on the available
resources. The allocator computes a score for each backend, that shows its load
factor, and the one with the minimum score is chosen. The admin can exclude
backends from the allocation phase by marking them as ``drained`` by running:
Christos Stavrakakis's avatar
Christos Stavrakakis committed

940 941
.. code-block:: console

   $ snf-manage backend-modify --drained=True <backend_id>
943 944

The backend resources are periodically updated, at a period defined by
945 946 947 948
the ``BACKEND_REFRESH_MIN`` setting, or by running `snf-manage
backend-update-status` command. It is advised to have a cron job running this
command at a smaller interval than ``BACKEND_REFRESH_MIN`` in order to remove
the load of refreshing the backends stats from the VM creation phase.

950 951 952
Finally, the admin can decide to have a user's VMs being allocated to a
specific backend, with the ``BACKEND_PER_USER`` setting. This is a mapping
between users and backends. If the user is found in ``BACKEND_PER_USER``, then
953 954
Synnefo allocates all his/hers VMs to the specific backend in the variable,
even if is marked as drained (useful for testing).

956 957
.. _alloc_disk_templates:

958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983
Allocation based on disk-templates

Besides the available resources of each Ganeti backend, the allocator takes
into consideration the disk template of the instance when trying to allocate it
to a Ganeti backend. Specifically, the allocator checks if the flavor of the
instance belongs to the available disk templates of each Ganeti backend.

A Ganeti cluster has a list of enabled disk templates
(`--enabled-disk-templates`) and a list of allowed disk templates for new
instances (`--ipolicy-disk-templates`). See the `gnt-cluster` manpage for more
details about these options.

When Synnefo allocates an instance, it checks whether the disk template of the
new instance belongs both in the enabled and ipolicy disk templates. You can
see the list of the available disk-templates by running `snf-manage
backend-list`. This list should be updated automatically after changing
these options in Ganeti and it can also be updated by running `snf-manage

So the administrator, can route instances on different backends based on their
flavor disk template, by modifying the enabled or ipolicy disk templates of
each backend.  Also, the administrator can route instances between different
nodes of the same Ganeti backend, by modifying the same options at the
nodegroup level (see `gnt-group` manpage for mor details).

984 985 986 987 988 989 990
Removing an existing Ganeti backend
In order to remove an existing backend from Synnefo, you must first make
sure that there are not active servers in the backend, and then run:

.. code-block:: console

   $ snf-manage backend-remove <backend_id>
992 993

994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034
Virtual Networks

Cyclades also implements the Network service and exposes the Quantum Openstack
API. Cyclades supports full IPv4 and IPv6 connectivity to the public internet
for it's VMs. Also, Cyclades provides L2 and L3 virtual private networks,
giving the user freedom to create arbitraty network topologies of
interconnected VMs.

Public networking is desployment specific and must be customized based on the
specific needs of the system administrator. Private virtual networks can be
provided by different network technologies which are exposed as different
network flavors. For better understanding of networking please refer to the
:ref:`Network <networks>` section.

A Cyclades virtual network is an isolated Layer-2 broadcast domain. A network
can also have an associated IPv4 and IPv6 subnet representing the Layer-3
characteristics of the network. Each subnet represents an IP address block
that is used in order to assign addresses to VMs.

To connect a VM to a network, a port must be created, which represent a virtual
port on a network switch. VMs are connected to networks by attaching a virtual
interface to a port.

Cyclades also supports `floating IPs`, which are public IPv4 addresses that
can dynamically(hotplug-able) be added and removed to VMs. Floating IPs are
a quotable resource that is allocated to each user. Unlike other cloud
platforms, floating IPs are not implemented using 1-1 NAT to a ports private
IP. Instead, floating IPs are directly assigned to virtual interfaces of VMs.

Exactly like VMS, networks can be handled as Ganeti networks via `gnt-network`
commands. All Ganeti networks that belong to Synnefo are named with the prefix
`${BACKEND_PREFIX_ID}-net-`. Also, there are a number of `snf-manage` commands
for handling of `networks`, `subnets`, `ports` and `floating IPs`. Below
we will present a use case scenario using some of these commands. For better
understanding of these commands, refer to their help messages.

Create a virtual private network for user
`7cf4d078-67bf-424d-8ff2-8669eb4841ea` using the `PHYSICAL_VLAN` flavor, which
means that the network will be uniquely assigned a phsyical VLAN. The network
is assigned an IPv4 subnet, described by it's CIDR and gateway. Also,
the `--dhcp=True` option is used, to make `nfdhcpd` response to DHCP queries
from VMs.

.. code-block:: console

 $ snf-manage network-create --owner=7cf4d078-67bf-424d-8ff2-8669eb4841ea --name=prv_net-1 \
    --subnet= --gateway= --dhcp=True --flavor=PHYSICAL_VLAN

Inspect the state of the network in Cyclades DB and in all the Ganeti backends:

.. code-block:: console

  $ snf-manage network-inspect <network_id>

1049 1050 1051 1052 1053
Inspect the state of the network's subnet, containg an overview of the
subnet's IPv4 address allocation pool:

.. code-block:: console

  $ snf-manage subnet-inspect <subnet_id>
1055 1056 1057 1058 1059

Connect a VM to the created private network. The port will be automatically
be assigned an IPv4 address from one of the network's available IPs. This
command will result in sending an `OP_INSTANCE_MODIFY` Ganeti command and
attaching a NIC to the specified Ganeti instance.

1061 1062
.. code-block:: console

 $ snf-manage port-create --network=<network_id> --server=<server_id>

Inspect the state of the the port in Cyclades DB and in the Ganeti backend:

.. code-block:: console

 $ snf-manage port-inspect <port_id>
1070 1071 1072 1073 1074

Disconnect the VM from the network and delete the network:

.. code-block:: console

1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184
 $ snf-manage port-remove <port_id>
 $ snf-manage network-remove <network_id>

Enabling DHCP

When connecting a VM to a network, Cyclades will automatically assign an IPv4
address from the IPv4 or/and IPv6 subnets of the network. If the network has
no subnets, then it will not be assigned any IP address.

If the network has DHCP enabled, then `nfdhcpd` daemon, which must be running
on all Ganeti nodes, will respond to DHCP queries from VMs and assign to them
the IP address that was allocated by Cyclades. DCHP can be enabled/disabled
using the `--dhcp` option of `network-create` command.

Public network connectivity

Since v0.14, users are able to dynamically connect and disconnect their VMs
from public networks. In order to do that, they have to use a `floating IP`.
Floating IPs are basically public IPv4 addresses that can be dynamically
attached and detached from VMs. The user creates a floating IP address from a
network that has set the `floating_ip_pool` attribute. The floating IP is
accounted to the user, who can then connect his VMs to public networks by
creating ports that they are using this floating IP. Performing this work-flow
from `snf-manage` would look like this:

.. code-block:: console

 $ snf-manage network-list --filter-by="floating_ip_pool=True"
 id      name  user.uuid   state  public  subnet.ipv4  gateway.ipv4  drained  floating_ip_pool
  1  Internet       None  ACTIVE    True    False              True

 $ snf-manage floating-ip-create --owner=7cf4d078-67bf-424d-8ff2-8669eb4841ea --network=1

 $ snf-manage floating-ip-list --user=7cf4d078-67bf-424d-8ff2-8669eb4841ea
 id   address       network                             user.uuid  server
 38             1  7cf4d078-67bf-424d-8ff2-8669eb4841ea      42

 $ snf-manage port-create --owner=7cf4d078-67bf-424d-8ff2-8669eb4841ea --network=1 \
                          --ipv4-address= --floating-ip=38

 $ snf-manage port-list --user=7cf4d078-67bf-424d-8ff2-8669eb4841ea
 id                            user.uuid        mac_address  network  server_id  fixed_ips   state
 163 7cf4d078-67bf-424d-8ff2-8669eb4841ea  aa:00:00:45:13:98       1         77  ACTIVE

 $ snf-manage port-remove 163
 $ snf-manage floating-ip-remove 38

Users do not have permission to connect and disconnect VMs from public
networks without using a floating IP address. However, the administrator
have the ability to perform this tasks, using `port-create` and `port-remove`

Network connectivity for newly created servers

When creating a virtual server, the user can specify the networks that the
newly created server will be connected to. Beyond this, the administrator can
define a list of networks that every new server will be forced to connect to.
For example, you can enforce all VMs to be connected to a public network
containing a metadata server. The networks must be specified in the
``/etc/synnefo/20-snf-cyclades-app-api.conf``. For the networks in this
setting, no access control or quota policy are enforced!

Finally, the administrator can define a list of networks that new servers will
be connected, *if the user has not* specified networks in the request to create
the server. Access control and quota policy are enforced, just as if the user
had specified these networks. The list of these networks is defined in the
``/etc/synnefo/20-snf-cyclades-app-api.conf``. This setting should only
be used if Cyclades are being accessed by external clients that are
unaware of the `Neutron API extensions` in the `Compute API`.

Each member of the above mentioned settings can be:

* a network UUID
* a tuple of network UUIDs: the server will be connected to only one of these
  networks, e.g. one that has a free IPv4 address
* `SNF:ANY_PUBLIC_IPV4`: the server will be connected to any network with
  an IPv4 subnet defined
* `SNF:ANY_PUBLIC_IPV6`: the server will be connected to any network with
  only an IPv6 subnet defined.
* `SNF:ANY_PUBLIC`: the server will be connected to any public network.

Public IP accounting

There are many use cases, e.g. abuse ports, where you need to find which user
or which server had a public IP address. For this reason, Cyclades keeps track
usage of public IPv4/IPv6 addresses. Specifically, it keeps the date and time
that each public IP address was allocated and released from a virtual server.
This information can be found using `ip-list` command:

.. code-block:: console

 $ snf-manage ip-list

 Show usage of a specific address:
 $ snf-manage ip-list --address=

 Show public IPs of a specific server:
 $ snf-manage ip-list --server=<server_id>


Managing Network Resources

1189 1190 1191
Proper operation of the Cyclades Network Service depends on the unique
assignment of specific resources to each type of virtual network. Specifically,
these resources are:

1193 1194
* IP addresses. Cyclades creates a Pool of IPs for each Network, and assigns a
  unique IP address to each VM, thus connecting it to this Network. You can see
1195 1196
  the IP pool of each network by running `snf-manage subnet-inspect
  <subnet_ID>`. IP pools are automatically created and managed by Cyclades,
  depending on the subnet of the Network.
* Bridges corresponding to physical VLANs, which are required for networks of
1199 1200 1201
* One Bridge corresponding to one physical VLAN which is required for networks of

1203 1204 1205 1206
IPv4 addresses

An allocation pool of IPv4 addresses is automatically created for every network
1207 1208 1209 1210 1211
with an IPv4 subnet. By default, the allocation pool contains the range of IP
addresses that are included in the subnet, except from the gateway and the
broadcast address of the network. The range of IP addresses can be restricted
using the `--allocation-pool` option of `snf-manage network-create` command.
The admin can externally reserve IP addresses to exclude them from automatic
allocation with the `--add-reserved-ips` option of `snf-manage network-modify`
1213 1214
command. For example the following command will reserve two IP addresses from
network with ID `42`:

1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231
.. code-block:: console

 snf-manage network-modify --add-reserved-ips=, 42

.. warning:: Externally reserving IP addresses is also available at the Ganeti.
 However, when using Cyclades with multiple Ganeti backends, the handling of
 IP pools must be performed from Cyclades!


As already mentioned Cyclades use a pool of Bridges that must correspond
to Physical VLAN at the Ganeti level. A bridge from the pool is assigned to
each network of flavor `PHYSICAL_VLAN`. Creation of this pool is done
using `snf-manage pool-create` command. For example the following command
will create a pool containing the brdiges from `prv1` to `prv21`.
Christos Stavrakakis's avatar
Christos Stavrakakis committed

1233 1234 1235 1236 1237
.. code-block:: console

   # snf-manage pool-create --type=bridge --base=prv --size=20

You can verify the creation of the pool, and check its contents by running:
Christos Stavrakakis's avatar
Christos Stavrakakis committed

1239 1240 1241 1242 1243
.. code-block:: console

   # snf-manage pool-list
   # snf-manage pool-show --type=bridge 1

1244 1245 1246 1247 1248 1249 1250 1251 1252
Finally you can use the `pool-modify` management command in order to externally
reserve the values from pool, extend or shrink the pool if possible.

MAC Prefixes

Cyclades also use a pool of MAC prefixes to assign to networks of flavor
`MAC_FILTERED`. Handling of this pool is done exactly as with pool of bridges,
except that the type option must be set to mac-prefix:
Christos Stavrakakis's avatar
Christos Stavrakakis committed

1254 1255 1256 1257
.. code-block:: console

   # snf-manage pool-create --type=mac-prefix --base=aa:00:0 --size=65536

1258 1259 1260 1261 1262
The above command will create a pool of MAC prefixes from ``aa:00:1`` to
``b9:ff:f``. The MAC prefix pool is responsible for providing only unicast and
locally administered MAC addresses, so many of these prefixes will be
externally reserved, to exclude from allocation.

1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273

Handling of quotas for Cyclades resources is powered by Astakos quota
mechanism. During registration of Cyclades service to Astakos, the Cyclades
resources are also imported to Astakos for accounting and presentation.

Upon a request that will result in a resource creation or removal, Cyclades
will communicate with Astakos to ensure that user quotas are within limits and
update the corresponding usage. If a limit is reached, the request will be
denied with an `overLimit(413)` fault.

The resources that are exported by Cyclades are the following:

1277 1278 1279 1280 1281 1282 1283 1284
* `cyclades.vm`: Number of virtual machines
* `cyclades.total_cpu`: Number of virtual machine processors
* `cyclades.cpu`: Number of virtual machine processors of running VMs
* `cyclades.total_ram`: Virtual machine memory size
* `cyclades.ram`: Virtual machine memory size of running VMs
* `cyclades.disk`: Virtual machine disk size
* `cyclades.floating_ip`: Number of floating IP addresses
* ``: Number of private virtual networks

1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332
Enforcing quotas

User quota can get overlimit, for example when a user is removed from a
project granting Cyclades resources. However, no action is automatically
taken to restrict users to their new limits. There is a special tool for
quota enforcement:

.. code-block:: console

  # snf-manage enforce-resources-cyclades

This command will check and report which users are overlimit on their
Cyclades quota; it will also suggest actions to be taken in order to enforce
quota limits, dependent on the overlimit resource:

* `cyclades.vm`: Delete VMs
* `cyclades.total_cpu`: Delete VMs
* `cyclades.cpu`: Shutdown VMs
* `cyclades.total_ram`: Delete VMs
* `cyclades.ram`: Shutdown VMs
* `cyclades.disk`: Delete VMs
* `cyclades.floating_ip`: Detach and remove IPs

VMs to be deleted/shutdown are chosen first by state in the following order:
ERROR, BUILD, STOPPED, STARTED or RESIZE and then by decreasing ID. When
needing to remove IPs, we first choose IPs that are free, then those
attached to VMs, using the same VM ordering.

By default, the command checks only the following resources: `cyclades.cpu`,
`cyclades.ram`, and `cyclades.floating_ip`; that is, the less dangerous
ones, those that do not result in *deleting* any VM. One can change the
default behavior by specifying the desired resources with option
``--resources``. It is also possible to specify users to be checked or

Actual enforcement is done with option ``--fix``. In order to control the
load that quota enforcement may cause on Cyclades, one can limit the number
of operations per backend. For example,

.. code-block:: console

  # snf-manage enforce-resources-cyclades --fix --max-operations 10

will apply only the first 10 listed actions per backend. One can repeat the
operation, until nothing is left to be done.

1333 1334 1335 1336
To control load a timeout can also be set for shutting down VMs (using
option ``--shutdown-timeout <sec>``). This may be needed to avoid
expensive operations triggered by shutdown, such as Windows updates.

1337 1338 1339 1340
The command outputs the list of applied actions and reports whether each
action succeeded or not. Failure is reported if for any reason cyclades
failed to process the job and submit it to the backend.

1341 1342 1343
Cyclades advanced operations

1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404
Reconciliation mechanism

Cyclades - Ganeti reconciliation

On certain occasions, such as a Ganeti or RabbitMQ failure, the state of
Cyclades database may differ from the real state of VMs and networks in the
Ganeti backends. The reconciliation process is designed to synchronize the
state of the Cyclades DB with Ganeti. There are two management commands for
reconciling VMs and Networks that will detect stale, orphans and out-of-sync
VMs and networks. To fix detected inconsistencies, use the `--fix-all`.

.. code-block:: console

  $ snf-manage reconcile-servers
  $ snf-manage reconcile-servers --fix-all

  $ snf-manage reconcile-networks
  $ snf-manage reconcile-networks --fix-all

Please see ``snf-manage reconcile-servers --help`` and ``snf-manage
reconcile--networks --help`` for all the details.

Cyclades - Astakos reconciliation

As already mentioned, Cyclades communicates with Astakos for resource
accounting and quota enforcement. In rare cases, e.g. unexpected
failures, the two services may get unsynchronized. For this reason there
are the `reconcile-commissions-cyclades` and `reconcile-resources-cyclades`
command that will synchronize the state of the two services. The first
command will detect any pending commissions, while the second command will
detect that the usage that is reported by Astakos is correct.
To fix detected inconsistencies, use the `--fix` option.

.. code-block:: console

  $ snf-manage reconcile-commissions-cyclades
  $ snf-manage reconcile-commissions-cyclades --fix

  $ snf-manage reconcile-resources-cyclades
  $ snf-manage reconcile-resources-cyclades --fix

Cyclades resources reconciliation

Reconciliation of pools will check the consistency of available pools by
checking that the values from each pool are not used more than once, and also
that the only reserved values in a pool are the ones used. Pool reconciliation
will check pools of bridges, MAC prefixes, and IPv4 addresses for all networks.
To fix detected inconsistencies, use the `--fix` option.

.. code-block:: console

  $ snf-manage reconcile-pools
  $ snf-manage reconcile-pools --fix

1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523
.. _admin-guide-stats:

VM stats collecting

snf-cyclades-gtools comes with a collectd plugin to collect CPU and network
stats for Ganeti VMs and an example collectd configuration. snf-stats-app is a
Django (snf-webproject) app that serves the VM stats graphs by reading the VM
stats (from RRD files) and serves graphs.

The snf-stats-app was originally written by `GRNET NOC <>`_
as a WSGI Python app and was ported to a Synnefo (snf-webproject) app.

snf-stats-app configuration

The snf-stats-app node should have collectd installed. The collectd
configuration should enable the network plugin, assuming the server role, and
the RRD plugin / backend, to store the incoming stats. Your
``/etc/collectd/collectd.conf`` should look like:

.. code-block:: console

    FQDNLookup true
    LoadPlugin syslog
    <Plugin syslog>
        LogLevel info

    LoadPlugin network
    LoadPlugin rrdtool
    <Plugin network>
        TimeToLive 128
        <Listen "" "25826">
            SecurityLevel "Sign"
            AuthFile "/etc/collectd/passwd"

        ReportStats false
        MaxPacketSize 65535

    <Plugin rrdtool>
        DataDir "/var/lib/collectd/rrd"
        CacheTimeout 120
        CacheFlush 900
        WritesPerSecond 30
        RandomTimeout 0

    Include "/etc/collectd/filters.conf"
    Include "/etc/collectd/thresholds.conf"

An example collectd config file is provided in

The recommended deployment is to run snf-stats-app using gunicorn with an
Apache2 or nginx reverse proxy (using the same configuration as the other
Synnefo services / apps). An example gunicorn config file is provided in

Make sure to edit the settings under
``/etc/synnefo/20-snf-stats-app-settings.conf`` to match your deployment.
More specifically, you should change the ``STATS_BASE_URL`` setting (refer
to previous documentation on the BASE_URL settings used by the other Synnefo
services / apps) and the ``RRD_PREFIX`` and ``GRAPH_PREFIX`` settings.

You should also set the ``STATS_SECRET_KEY`` to a random string and make sure
it's the same at the ``CYCLADES_STATS_SECRET_KEY`` on the Cyclades host (see

``RRD_PREFIX`` is the directory where collectd stores the RRD files. The
default setting matches the default RRD directory for the collectd RRDtool
plugin. In a more complex setup, the collectd daemon could run on a separate
host and export the RRD directory to the snf-stats-app node via e.g. NFS.

``GRAPH_PREFIX`` is the directory where collectd stores the resulting
stats graphs. You should create it manually, in case it doesn't exist.

.. code-block::

    # mkdir /var/cache/snf-stats-app/
    # chown www-data:wwwdata /var/cache/snf-stats-app/

The snf-stats-app will typically run as the ``www-data`` user. In that case,
make sure that the ``www-data`` user should have read access to the
``RRD_PREFIX`` directory and read / write access to the ``GRAPH_PREFIX``

snf-stats-app, based on the ``STATS_BASE_URL`` setting will export the
following URL 'endpoints`:
 * CPU stats bar: ``STATS_BASE_URL``/v1.0/cpu-bar/<encrypted VM hostname>
 * Network stats bar: ``STATS_BASE_URL``/v1.0/net-bar/<encrypted VM hostname>
 * CPU stats daily graph: ``STATS_BASE_URL``/v1.0/cpu-ts/<encrypted VM hostname>
 * Network stats daily graph: ``STATS_BASE_URL``/v1.0/net-ts/<encrypted VM hostname>
 * CPU stats weekly graph: ``STATS_BASE_URL``/v1.0/cpu-ts-w/<encrypted VM hostname>
 * Network stats weekly graph: ``STATS_BASE_URL``/v1.0/net