admin-guide.rst 66.2 KB
Newer Older
1
2
3
4
5
6
7
8
.. _admin-guide:

Synnefo Administrator's Guide
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is the complete Synnefo Administrator's Guide.


9
.. _syn+archip:
10

11
12
13
General Synnefo Architecture
============================

14
15
16
17
The following figure shows a detailed view of the whole Synnefo architecture
and how it interacts with multiple Ganeti clusters. We hope that after reading
the Administrator's Guide you will be able to understand every component and
all the interactions between them.
18

19
.. image:: images/synnefo-arch2.png
20
   :width: 100%
21
   :target: _images/synnefo-arch2.png
22

23
24
25
Synnefo also supports RADOS as an alternative storage backend for
Files/Images/VM disks. You will find the :ref:`corresponding figure
<syn+archip+rados>` later in this guide.
26

27

28
29
Identity Service (Astakos)
==========================
30
31


32
Authentication methods
33
----------------------
34

35
Astakos supports multiple authentication methods:
36

37
38
39
40
41
42
 * local username/password
 * LDAP / Active Directory
 * SAML 2.0 (Shibboleth) federated logins
 * Google
 * Twitter
 * LinkedIn
43
44
45
46

.. _shibboleth-auth:

Shibboleth Authentication
47
~~~~~~~~~~~~~~~~~~~~~~~~~
48
49
50
51
52
53
54
55
56
57
58
59
60
61

Astakos can delegate user authentication to a Shibboleth federation.

To setup shibboleth, install package::

  apt-get install libapache2-mod-shib2

Change appropriately the configuration files in ``/etc/shibboleth``.

Add in ``/etc/apache2/sites-available/synnefo-ssl``::

  ShibConfig /etc/shibboleth/shibboleth2.xml
  Alias      /shibboleth-sp /usr/share/shibboleth

62
  <Location /ui/login/shibboleth>
63
64
65
66
67
68
69
70
71
72
73
74
75
    AuthType shibboleth
    ShibRequireSession On
    ShibUseHeaders On
    require valid-user
  </Location>

and before the line containing::

  ProxyPass        / http://localhost:8080/ retry=0

add::

  ProxyPass /Shibboleth.sso !
76

77
78
79
Then, enable the shibboleth module::

  a2enmod shib2
80

81
82
83
84
85
86
87
88
89
90
91
92
93
94
After passing through the apache module, the following tokens should be
available at the destination::

  eppn # eduPersonPrincipalName
  Shib-InetOrgPerson-givenName
  Shib-Person-surname
  Shib-Person-commonName
  Shib-InetOrgPerson-displayName
  Shib-EP-Affiliation
  Shib-Session-ID

Finally, add 'shibboleth' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
95
Twitter Authentication
96
~~~~~~~~~~~~~~~~~~~~~~
Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
97
98
99
100
101
102
103
104
105
106
107
108
109
110

To enable twitter authentication while signed in under a Twitter account,
visit dev.twitter.com/apps.

Click Create an application.

Fill the necessary information and for callback URL give::

    https://node1.example.com/ui/login/twitter/authenticated

Finally, add 'twitter' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Google Authentication
111
~~~~~~~~~~~~~~~~~~~~~
Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127

To enable google authentication while signed in under a Google account,
visit https://code.google.com/apis/console/.

Under API Access select Create another client ID, select Web application,
expand more options in Your site or hostname section and in Authorized
Redirect URIs add:


Fill the necessary information and for callback URL give::

    https://node1.example.com/ui/login/google/authenticated

Finally, add 'google' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

128
129
130
131

Working with Astakos
--------------------

132
133
User registration
~~~~~~~~~~~~~~~~~
134

135
136
When a new user signs up, he/she is not directly marked as active. You can see 
his/her state by running (on the machine that runs the Astakos app):
137
138
139

.. code-block:: console

140
   $ snf-manage user-list
141

142
143
More detailed user status is provided in the `status` field of the `user-show` 
command:
144

145
.. code-block:: console
146

147
  $ snf-manage user-show <user-id>
148

149
150
151
152
153
  id                  : 6
  uuid                : 78661411-5eed-412f-a9ea-2de24f542c2e
  status              : Accepted/Active (accepted policy: manual)
  email               : user@synnefo.org
  ....
154

155
156
157
Based on the `astakos-app` configuration, there are several ways for a user to
get verified and activated in order to be able to login. We discuss the user
verification and activation flow in the following section.
158
159

User activation flow
160
161
````````````````````

162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
A user can register for an account using the astakos signup form. Once the form
is submited successfully a user entry is created in astakos database. That entry
is passed through the astakos activation backend which handles whether the user
should be automatically verified and activated.

Email verification
``````````````````

The verification process takes place in order to ensure that the user owns the
email provided during the signup process. By default, after each successful
signup astakos notifies user with an verification url via email. 

At this stage:

    * subsequent registrations invalidate and delete the previous registrations 
      of the same email address.

    * in case user misses the initial notification, additional emails can be
      send either via the url which is prompted to the user if he tries to
      login, or by the administrator using the ``snf-manage user-activation-send
      <userid>`` command.

    * administrator may also enforce a user to get verified using the
      ``snf-manage user-modify --verify <userid>`` command.

Account activation
``````````````````

190
Once the user gets verified, it is time for Astakos to decide whether or not to
191
192
193
proceed through user activation process. If ``ASTAKOS_MODERATION_ENABLED``
setting is set to ``False`` (default value) user gets activated automatically. 

194
In case the moderation is enabled Astakos may still automatically activate the
195
196
197
198
199
200
201
202
user in the following cases:

    * User email matches any of the regular expressions defined in
      ``ASTAKOS_RE_USER_EMAIL_PATTERNS`` (defaults to ``[]``)
    * User used a signup method (e.g. ``shibboleth``) for which automatic
      activation is enabled (see 
      :ref:`authentication methods policies <auth_methods_policies>`).

203
204
205
206
207
208
If all of the above fail to trigger automatic activation, an email is sent to
the persons listed in ``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings,
notifing that there is a new user pending for moderation and that it's up to
the administrator to decide if the user should be activated. The UI also shows
a corresponding 'pending moderation' message to the user. The administrator can
activate a user using the ``snf-manage user-modify`` command:
209
210
211
212
213
214
215
216
217

.. code-block:: console

    # command to activate a pending user
    $ snf-manage user-modify --accept <userid>

    # command to reject a pending user
    $ snf-manage user-modify --reject --reject-reason="spammer" <userid>

218
219
220
221
Once the activation process finishes, a greeting message is sent to the user
email address and a notification for the activation to the persons listed in
``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings. Once activated the user is
able to login and access the Synnefo services.
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282

Additional authentication methods
`````````````````````````````````

Astakos supports third party logins from external identity providers. This
can be usefull since it allows users to use their existing credentials to 
login to astakos service.

Currently astakos supports the following identity providers:

    * `Shibboleth <http://www.internet2.edu/shibboleth>`_ (module name
      ``shibboleth``)
    * `Google <https://developers.google.com/accounts/docs/OAuth2>`_ (module
      name ``google``)
    * `Twitter <https://dev.twitter.com/docs/auth>`_ (module name ``twitter``)
    * `LinkedIn <http://developer.linkedin.com/documents/authentication>`_
      (module name ``linkedin``)

To enable any of the above modules (by default only ``local`` accounts are
allowed), retrieve and set the required provider settings and append the 
module name in ``ASTAKOS_IM_MODULES``.

.. code-block:: python

    # settings from https://code.google.com/apis/console/
    ASTAKOS_GOOGLE_CLIENT_ID = '1111111111-epi60tvimgha63qqnjo40cljkojcann3.apps.googleusercontent.com'
    ASTAKOS_GOOGLE_SECRET = 'tNDQqTDKlTf7_LaeUcWTWwZM'
    
    # let users signup and login using their google account
    ASTAKOS_IM_MODULES = ['local', 'google']


.. _auth_methods_policies:

Authentication method policies
``````````````````````````````

Astakos allows you to override the default policies for each enabled provider 
separately by adding the approriate settings in your ``.conf`` files in the 
following format:

**ASTAKOS_AUTH_PROVIDER_<module>_<policy>_POLICY**

Available policies are:

    * **CREATE** Users can signup using that provider (default: ``True``) 
    * **REMOVE/ADD** Users can remove/add login method from their profile 
      (default: ``True``)
    * **AUTOMODERATE** Automatically activate users that signup using that
      provider (default: ``False``)
    * **LOGIN** Whether or not users can use the provider to login (default:
      ``True``).

e.g. to enable automatic activation for your academic users, while keeping 
locally signed up users under moderation you can apply the following settings.

.. code-block:: python

    ASTAKOS_AUTH_PROVIDER_SHIBBOLETH_AUTOMODERATE_POLICY = True
    ASTAKOS_AUTH_PROVIDER_SHIBBOLETH_REMOVE_POLICY = False

283
284
285
286
287
288
289
290
291
292
293
294
295
296
User login
~~~~~~~~~~

During the logging procedure, the user is authenticated by the respective
identity provider.

If ``ASTAKOS_RECAPTCHA_ENABLED`` is set and the user fails several times
(``ASTAKOS_RATELIMIT_RETRIES_ALLOWED`` setting) to provide the correct
credentials for a local account, he/she is then prompted to solve a captcha
challenge.

Upon success, the system renews the token (if it has expired), logins the user
and sets the cookie, before redirecting the user to the ``next`` parameter
value.
297

298
299
300
Setting quota limits
~~~~~~~~~~~~~~~~~~~~

301
302
Set default quota
`````````````````
303
To inspect current default base quota limits, run::
304

305
   # snf-manage resource-list
306

307
You can modify the default base quota limit for all future users with::
308

309
   # snf-manage resource-modify <resource_name> --default-quota <value>
310

311
312
Set base quota for individual users
```````````````````````````````````
313

314
For individual users that need different quota than the default
315
316
you can set it for each resource like this::

317
318
    # use this to display quota / uuid
    # snf-manage user-show 'uuid or email' --quota
319

320
    # snf-manage user-modify 'user-uuid' --set-base-quota 'cyclades.vm' 10
321
322
323
324
325
326
327
328
329
330
331
332


Enable the Projects feature
~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you want to enable the projects feature so that users may apply
on their own for resources by creating and joining projects,
in ``20-snf-astakos-app-settings.conf`` set::

    # this will make the 'projects' page visible in the dashboard
    ASTAKOS_PROJECTS_VISIBLE = True

333
334
You can change the maximum allowed number of pending project applications
per user with::
335

336
    # snf-manage resource-modify astakos.pending_app --default-quota <number>
337
338
339

You can also set a user-specific limit with::

340
    # snf-manage user-modify 'user-uuid' --set-base-quota 'astakos.pending_app' 5
341

342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
When users apply for projects they are not automatically granted
the resources. They must first be approved by the administrator.

To list pending project applications in astakos::

    # snf-manage project-list --pending

Note the last column, the application id. To approve it::

    # <app id> from the last column of project-list
    # snf-manage project-control --approve <app id>

To deny an application::

    # snf-manage project-control --deny <app id>

358
359
360
361
362
363
Users designated as *project admins* can approve, deny, or modify
an application through the web interface. In
``20-snf-astakos-app-settings.conf`` set::

    # UUIDs of users that can approve or deny project applications from the web.
    ASTAKOS_PROJECT_ADMINS = [<uuid>, ...]
364
365


366
367
368
369
370
371
372
373
374
375
376
377
Astakos advanced operations
---------------------------

Adding "Terms of Use"
~~~~~~~~~~~~~~~~~~~~~

Astakos supports versioned terms-of-use. First of all you need to create an
html file that will contain your terms. For example, create the file
``/usr/share/synnefo/sample-terms.html``, which contains the following:

.. code-block:: console

378
   <h1>My cloud service terms</h1>
379

380
   These are the example terms for my cloud service
381
382
383
384
385
386
387
388
389
390

Then, add those terms-of-use with the snf-manage command:

.. code-block:: console

   $ snf-manage term-add /usr/share/synnefo/sample-terms.html

Your terms have been successfully added and you will see the corresponding link
appearing in the Astakos web pages' footer.

391
392
393
394
395
396
397
398
399
During the account registration, if there are approval terms, the user is
presented with an "I agree with the Terms" checkbox that needs to get checked
in order to proceed.

In case there are new approval terms that the user has not signed yet, the
``signed_terms_required`` view decorator redirects to the ``approval_terms``
view, so the user will be presented with the new terms the next time he/she
logins.

400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
Enabling reCAPTCHA
~~~~~~~~~~~~~~~~~~

Astakos supports the `reCAPTCHA <http://www.google.com/recaptcha>`_ feature.
If enabled, it protects the Astakos forms from bots. To enable the feature, go
to https://www.google.com/recaptcha/admin/create and create your own reCAPTCHA
key pair. Then edit ``/etc/synnefo/20-snf-astakos-app-settings.conf`` and set
the corresponding variables to reflect your newly created key pair. Finally, set
the ``ASTAKOS_RECAPTCHA_ENABLED`` variable to ``True``:

.. code-block:: console

   ASTAKOS_RECAPTCHA_PUBLIC_KEY = 'example_recaptcha_public_key!@#$%^&*('
   ASTAKOS_RECAPTCHA_PRIVATE_KEY = 'example_recaptcha_private_key!@#$%^&*('

   ASTAKOS_RECAPTCHA_ENABLED = True

Restart the service on the Astakos node(s) and you are ready:

.. code-block:: console

   # /etc/init.d/gunicorn restart

Checkout your new Sign up page. If you see the reCAPTCHA box, you have setup
everything correctly.

426

427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
Astakos internals
-----------------

X-Auth-Token
~~~~~~~~~~~~

Alice requests a specific resource from a cloud service e.g.: Pithos. In the
request she supplies the `X-Auth-Token` to identify whether she is eligible to
perform the specific task. The service contacts Astakos through its
``/account/v1.0/authenticate`` api call (see :ref:`authenticate-api-label`)
providing the specific ``X-Auth-Token``. Astakos checkes whether the token
belongs to an active user and it has not expired and returns a dictionary
containing user related information. Finally the service uses the ``uniq``
field included in the dictionary as the account string to identify the user
accessible resources.

.. _authentication-label:

Django Auth methods and Backends
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Astakos incorporates Django user authentication system and extends its User model.

Since username field of django User model has a limitation of 30 characters,
AstakosUser is **uniquely** identified by the ``email`` instead. Therefore,
``astakos.im.authentication_backends.EmailBackend`` is served to authenticate a
user using email if the first argument is actually an email, otherwise tries
the username.

A new AstakosUser instance is assigned with a uui as username and also with a
``auth_token`` used by the cloud services to authenticate the user.
``astakos.im.authentication_backends.TokenBackend`` is also specified in order
to authenticate the user using the email and the token fields.

Logged on users can perform a number of actions:

 * access and edit their profile via: ``/im/profile``.
 * change their password via: ``/im/password``
 * send feedback for grnet services via: ``/im/send_feedback``
 * logout (and delete cookie) via: ``/im/logout``

Internal Astakos requests are handled using cookie-based Django user sessions.

External systems should forward to the ``/login`` URI. The server,
depending on its configuration will redirect to the appropriate login page.
When done with logging in, the service's login URI should redirect to the URI
provided with next, adding user and token parameters, which contain the email
and token fields respectively.

The login URI accepts the following parameters:

======================  =========================
Request Parameter Name  Value
======================  =========================
next                    The URI to redirect to when the process is finished
renew                   Force token renewal (no value parameter)
force                   Force logout current user (no value parameter)
======================  =========================

External systems inside the ``ASTAKOS_COOKIE_DOMAIN`` scope can acquire the
user information by the cookie identified by ``ASTAKOS_COOKIE_NAME`` setting
(set during the login procedure).

Finally, backend systems having acquired a token can use the
:ref:`authenticate-api-label` API call from a private network or through HTTPS.


494
495
496
497
498
499
Compute/Network/Image Service (Cyclades)
========================================

Working with Cyclades
---------------------

500
Managing Ganeti Backends
Christos Stavrakakis's avatar
Christos Stavrakakis committed
501
~~~~~~~~~~~~~~~~~~~~~~~~
502

503
504
505
Since v0.11, Synnefo is able to manage multiple Ganeti clusters (backends)
making it capable to scale linearly to tens of thousands of VMs. Backends
can be dynamically added or removed via `snf-manage` commands.
506

507
508
509
510
511
Each newly created VM is allocated to a Ganeti backend by the Cyclades backend
allocator. The VM is "pinned" to this backend, and can not change through its
lifetime. The backend allocator decides in which backend to spawn the VM based
on the available resources of each backend, trying to balance the load between
them.
512
513

Handling of Networks, as far as backends are concerned, is based on whether the
514
515
516
517
network is public or not. Public networks are created through the `snf-manage
network-create` command, and are only created on one backend. Private networks
are created on all backends, in order to ensure that VMs residing on different
backends can be connected to the same private network.
518

519
520
521
Listing existing backends
`````````````````````````
To list all the Ganeti backends known to Synnefo, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
522

523
524
525
526
527
.. code-block:: console

   $ snf-manage backend-list

Adding a new Ganeti backend
Christos Stavrakakis's avatar
Christos Stavrakakis committed
528
```````````````````````````
529
530
Backends are dynamically added under the control of Synnefo with `snf-manage
backend-add` command. In this section it is assumed that a Ganeti cluster,
531
532
named ``cluster.example.com`` is already up and running and configured to be
able to host Synnefo VMs.
533

534
To add this Ganeti cluster, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
535

536
537
538
539
.. code-block:: console

   $ snf-manage backend-add --clustername=cluster.example.com --user="synnefo_user" --pass="synnefo_pass"

540
541
542
543
544
where ``clustername`` is the Cluster hostname of the Ganeti cluster, and
``user`` and ``pass`` are the credentials for the `Ganeti RAPI user
<http://docs.ganeti.org/ganeti/2.2/html/rapi.html#users-and-passwords>`_.  All
backend attributes can be also changed dynamically using the `snf-manage
backend-modify` command.
545

546
547
548
``snf-manage backend-add`` will also create all existing private networks to
the new backend. You can verify that the backend is added, by running
`snf-manage backend-list`.
549

550
551
552
553
554
555
556
557
558
559
Note that no VMs will be spawned to this backend, since by default it is in a
``drained`` state after addition and also it has no public network assigned to
it.

So, first you need to create its public network, make sure everything works as
expected and finally make it active by un-setting the ``drained`` flag. You can
do this by running:

.. code-block:: console

560
   $ snf-manage backend-modify --drained=False <backend_id>
561
562

Removing an existing Ganeti backend
Christos Stavrakakis's avatar
Christos Stavrakakis committed
563
```````````````````````````````````
564
In order to remove an existing backend from Synnefo, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
565

566
567
.. code-block:: console

568
   # snf-manage backend-remove <backend_id>
569

570
571
572
This command will fail if there are active VMs on the backend. Also, the
backend is not cleaned before removal, so all the Synnefo private networks
will be left on the Ganeti nodes. You need to remove them manually.
573

574
575
576
577
578
579
Allocation of VMs in Ganeti backends
````````````````````````````````````
As already mentioned, the Cyclades backend allocator is responsible for
allocating new VMs to backends. This allocator does not choose the exact Ganeti
node that will host the VM but just the Ganeti backend. The exact node is
chosen by the Ganeti cluster's allocator (hail).
580
581
582
583

The decision about which backend will host a VM is based on the available
resources. The allocator computes a score for each backend, that shows its load
factor, and the one with the minimum score is chosen. The admin can exclude
584
backends from the allocation phase by marking them as ``drained`` by running:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
585

586
587
.. code-block:: console

588
   $ snf-manage backend-modify --drained=True <backend_id>
589
590

The backend resources are periodically updated, at a period defined by
591
the ``BACKEND_REFRESH_MIN`` setting, or by running `snf-manage backend-update-status`
592
command. It is advised to have a cron job running this command at a smaller
593
interval than ``BACKEND_REFRESH_MIN`` in order to remove the load of refreshing
594
595
the backends stats from the VM creation phase.

596
597
598
Finally, the admin can decide to have a user's VMs being allocated to a
specific backend, with the ``BACKEND_PER_USER`` setting. This is a mapping
between users and backends. If the user is found in ``BACKEND_PER_USER``, then
599
600
Synnefo allocates all his/hers VMs to the specific backend in the variable,
even if is marked as drained (useful for testing).
601

602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
Allocation based on disk-templates
**********************************

Besides the available resources of each Ganeti backend, the allocator takes
into consideration the disk template of the instance when trying to allocate it
to a Ganeti backend. Specifically, the allocator checks if the flavor of the
instance belongs to the available disk templates of each Ganeti backend.

A Ganeti cluster has a list of enabled disk templates
(`--enabled-disk-templates`) and a list of allowed disk templates for new
instances (`--ipolicy-disk-templates`). See the `gnt-cluster` manpage for more
details about these options.

When Synnefo allocates an instance, it checks whether the disk template of the
new instance belongs both in the enabled and ipolicy disk templates. You can
see the list of the available disk-templates by running `snf-manage
backend-list`. This list should be updated automatically after changing
these options in Ganeti and it can also be updated by running `snf-manage
backend-update-status`.

So the administrator, can route instances on different backends based on their
flavor disk template, by modifying the enabled or ipolicy disk templates of
each backend.  Also, the administrator can route instances between different
nodes of the same Ganeti backend, by modifying the same options at the
nodegroup level (see `gnt-group` manpage for mor details).


629
630
631
Managing Virtual Machines
~~~~~~~~~~~~~~~~~~~~~~~~~

632
633
634
635
636
637
As mentioned, Cyclades uses Ganeti for management of VMs. The administrator can
handle Cyclades VMs just like any other Ganeti instance, via `gnt-instance`
commands. All Ganeti instances that belong to Synnefo, are separated from
others, by a prefix in their names. This prefix is defined in
``BACKEND_PREFIX_ID`` setting in
``/etc/synnefo/20-snf-cyclades-app-backend.conf``.
638

639
Apart from handling instances directly in the Ganeti level, a number of `snf-manage`
640
641
commands are available:

642
643
644
645
646
647
* ``snf-manage server-list``: List servers
* ``snf-manage server-show``: Show information about a server in the Cyclades DB
* ``snf-manage server-inspect``: Inspect the state of a server both in DB and Ganeti
* ``snf-manage server-modify``: Modify the state of a server in the Cycldes DB
* ``snf-manage server-create``: Create a new server
* ``snf-manage server-import``: Import an existing Ganeti instance to Cyclades
648
649
650
651
652


Managing Virtual Networks
~~~~~~~~~~~~~~~~~~~~~~~~~

653
654
655
656
Cyclades is able to create and manage Virtual Networks. Networking is
desployment specific and must be customized based on the specific needs of the
system administrator. For better understanding of networking please refer to
the :ref:`Network <networks>` section.
657
658
659

Exactly as Cyclades VMs can be handled like Ganeti instances, Cyclades Networks
can also by handled as Ganeti networks, via `gnt-network commands`. All Ganeti
660
networks that belong to Synnefo are named with the prefix
661
662
`${BACKEND_PREFIX_ID}-net-`.

663
There are also the following `snf-manage` commands for managing networks:
664

665
666
667
668
669
670
* ``snf-manage network-list``: List networks
* ``snf-manage network-show``: Show information about a network in the Cyclades DB
* ``snf-manage network-inspect``: Inspect the state of the network in DB and Ganeti backends
* ``snf-manage network-modify``: Modify the state of a network in the Cycldes DB
* ``snf-manage network-create``: Create a new network
* ``snf-manage network-remove``: Remove an existing network
671

672
Managing Network Resources
673
``````````````````````````
674

675
676
677
Proper operation of the Cyclades Network Service depends on the unique
assignment of specific resources to each type of virtual network. Specifically,
these resources are:
678

679
680
681
682
683
* IP addresses. Cyclades creates a Pool of IPs for each Network, and assigns a
  unique IP address to each VM, thus connecting it to this Network. You can see
  the IP pool of each network by running `snf-manage network-inspect
  <network_ID>`. IP pools are automatically created and managed by Cyclades,
  depending on the subnet of the Network.
684
* Bridges corresponding to physical VLANs, which are required for networks of
685
686
687
  type `PRIVATE_PHYSICAL_VLAN`.
* One Bridge corresponding to one physical VLAN which is required for networks of
  type `PRIVATE_MAC_PREFIX`.
688

689
690
691
692
693
694
695
696
697
698
699
IPv4 addresses
**************

An allocation pool of IPv4 addresses is automatically created for every network
that has the attribute `dhcp` set to True. The allocation pool contains the
range of IP addresses that are included in the subnet. The gateway and the
broadcast address of the network are excluded from the allocation pool. The
admin can externally reserve IP addresses to exclude them from automatic
allocation with the `--add-reserved-ips` option of `snf-manage network-modify`
command. For example the following command will reserve two IP addresses
from network with ID `42`:
700

701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
.. code-block:: console

 snf-manage network-modify --add-reserved-ips=10.0.0.21,10.0.0.22 42

.. warning:: Externally reserving IP addresses is also available at the Ganeti.
 However, when using Cyclades with multiple Ganeti backends, the handling of
 IP pools must be performed from Cyclades!

Bridges
*******

As already mentioned Cyclades use a pool of Bridges that must correspond
to Physical VLAN at the Ganeti level. A bridge from the pool is assigned to
each network of flavor `PHYSICAL_VLAN`. Creation of this pool is done
using `snf-manage pool-create` command. For example the following command
will create a pool containing the brdiges from `prv1` to `prv21`.
Christos Stavrakakis's avatar
Christos Stavrakakis committed
717

718
719
720
721
722
.. code-block:: console

   # snf-manage pool-create --type=bridge --base=prv --size=20

You can verify the creation of the pool, and check its contents by running:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
723

724
725
726
727
728
.. code-block:: console

   # snf-manage pool-list
   # snf-manage pool-show --type=bridge 1

729
730
731
732
733
734
735
736
737
Finally you can use the `pool-modify` management command in order to externally
reserve the values from pool, extend or shrink the pool if possible.

MAC Prefixes
************

Cyclades also use a pool of MAC prefixes to assign to networks of flavor
`MAC_FILTERED`. Handling of this pool is done exactly as with pool of bridges,
except that the type option must be set to mac-prefix:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
738

739
740
741
742
.. code-block:: console

   # snf-manage pool-create --type=mac-prefix --base=aa:00:0 --size=65536

743
744
745
746
747
748
749
750
751
752
753
The above command will create a pool of MAC prefixes from ``aa:00:1`` to
``b9:ff:f``. The MAC prefix pool is responsible for providing only unicast and
locally administered MAC addresses, so many of these prefixes will be
externally reserved, to exclude from allocation.

Pool reconciliation
*******************

The management command `snf-manage reconcile-pools` can be used that all the
above mentioned pools are consistent and that all values that come from the
pool are not used more than once.
754

755

756
757
758
759
760
Cyclades advanced operations
----------------------------

Reconciliation mechanism
~~~~~~~~~~~~~~~~~~~~~~~~
761

762
763
764
765
766
767
On certain occasions, such as a Ganeti or RabbitMQ failure, the state of
Cyclades database may differ from the real state of VMs and networks in the
Ganeti backends. The reconciliation process is designed to synchronize
the state of the Cyclades DB with Ganeti. There are two management commands
for reconciling VMs and Networks

768
769
770
Reconciling Virtual Machines
````````````````````````````

771
Reconciliation of VMs detects the following conditions:
772

773
774
 * Stale DB servers without corresponding Ganeti instances
 * Orphan Ganeti instances, without corresponding DB entries
775
 * Out-of-sync state for DB entries wrt to Ganeti instances
776

777
To detect all inconsistencies you can just run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
778

779
.. code-block:: console
780
781

  $ snf-manage reconcile-servers
782

783
Adding the `--fix-all` option, will do the actual synchronization:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
784

785
.. code-block:: console
786

787
  $ snf-manage reconcile-servers --fix-all
788

789
Please see ``snf-manage reconcile-servers --help`` for all the details.
790

791
Reconciling Networks
Christos Stavrakakis's avatar
Christos Stavrakakis committed
792
````````````````````
793

794
Reconciliation of Networks detects the following conditions:
795

796
797
798
799
800
801
  * Stale DB networks without corresponding Ganeti networks
  * Orphan Ganeti networks, without corresponding DB entries
  * Private networks that are not created to all Ganeti backends
  * Unsynchronized IP pools

To detect all inconsistencies you can just run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
802

803
.. code-block:: console
804

805
806
807
  $ snf-manage reconcile-networks

Adding the `--fix-all` option, will do the actual synchronization:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
808

809
.. code-block:: console
810

811
812
813
  $ snf-manage reconcile-networks --fix-all

Please see ``snf-manage reconcile-networks --help`` for all the details.
814
815


816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
Cyclades internals
------------------

Asynchronous communication with Ganeti backends
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Synnefo uses Google Ganeti backends for VM cluster management. In order for
Cyclades to be able to handle thousands of user requests, Cyclades and Ganeti
communicate asynchronously. Briefly, requests are submitted to Ganeti through
Ganeti's RAPI/HTTP interface, and then asynchronous notifications about the
progress of Ganeti jobs are being created and pushed upwards to Cyclades. The
architecture and communication with a Ganeti backend is shown in the graph
below:

.. image:: images/cyclades-ganeti-communication.png
   :width: 50%
   :target: _images/cyclades-ganeti-communication.png

The Cyclades API server is responsible for handling user requests. Read-only
requests are directly served by looking up the Cyclades DB. If the request
needs an action in the Ganeti backend, Cyclades submit jobs to the Ganeti
master using the `Ganeti RAPI interface
<http://docs.ganeti.org/ganeti/2.2/html/rapi.html>`_.

While Ganeti executes the job, `snf-ganeti-eventd`, `snf-ganeti-hook` and
`snf-progress-monitor` are monitoring the progress of the job and send
corresponding messages to the RabbitMQ servers. These components are part
of `snf-cyclades-gtools` and must be installed on all Ganeti nodes. Specially:

* *snf-ganeti-eventd* sends messages about operations affecting the operating
  state of instances and networks. Works by monitoring the Ganeti job queue.
* *snf-ganeti_hook* sends messages about the NICs of instances. It includes a
  number of `Ganeti hooks <http://docs.ganeti.org/ganeti/2.2/html/hooks.html>`_
  for customisation of operations.
* *snf-progress_monitor* sends messages about the progress of the Image deployment
  phase which is done by the Ganeti OS Definition `snf-image`.

Finally, `snf-dispatcher` consumes messages from the RabbitMQ queues, processes
these messages and properly updates the state of the Cyclades DB. Subsequent
requests to the Cyclades API, will retrieve the updated state from the DB.


857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
Synnefo management commands ("snf-manage")
==========================================

Each Synnefo service, Astakos, Pithos and Cyclades are controlled by the
administrator using the "snf-manage" admin tool. This tool is an extension of
the Django command-line management utility. It is run on the host that runs
each service and provides different types of commands depending the services
running on the host. If you are running more than one service on the same host
"snf-manage" adds all the corresponding commands for each service dynamically,
providing a unified admin environment.

To run "snf-manage" you just type:

.. code-block:: console

   # snf-manage <command> [arguments]

on the corresponding host that runs the service. For example, if you have all
services running on different physical hosts you would do:

.. code-block:: console

   root@astakos-host # snf-manage <astakos-command> [argument]
   root@pithos-host # snf-manage <pithos-command> [argument]
   root@cyclades-host # snf-manage <cyclades-command> [argument]

If you have all services running on the same host you would do:

.. code-block:: console

   root@synnefo-host # snf-manage <{astakos,pithos,cyclades}-command> [argument]

Note that you cannot execute a service's command on a host that is not running
this service. For example, the following will return an error if Astakos and
Cyclades are installed on different physical hosts:

.. code-block:: console

   root@astakos-host # snf-manage <cyclades-command> [argument]
   Unknown command: 'cyclades-command'
   Type 'snf-manage help' for usage.

This is the complete list of "snf-manage" commands for each service.

Astakos snf-manage commands
---------------------------

============================  ===========================
Name                          Description
============================  ===========================
fix-superusers                Transform superusers created by syncdb into AstakosUser instances
908
cleanup-full                  Cleanup sessions and session catalog
909
910
commission-list               List pending commissions
commission-show               Show details for a pending commission
911
912
913
component-add                 Register a component
component-list                List components
component-modify              Modify component attributes
914
component-show                Show component details
915
916
917
project-control               Manage projects and applications
project-list                  List projects
project-show                  Show project details
918
919
quota-list                    List user quota
quota-verify                  Check the integrity of user quota
920
921
reconcile-resources-astakos   Reconcile resource usage of Quotaholder with Astakos DB
resource-list                 List resources
922
resource-modify               Modify a resource's default base quota and boolean flags
923
service-export-astakos        Export Astakos services and resources in JSON format
924
service-import                Register services
925
service-list                  List services
926
service-show                  Show service details
927
928
929
term-add                      Add approval terms
user-activation-send          Send user activation
user-add                      Add user
930
931
932
933
934
935
936
authpolicy-add                Create a new authentication provider policy profile
authpolicy-list               List existing authentication provider policy profiles
authpolicy-remove             Remove an authentication provider policy
authpolicy-set                Assign an existing authentication provider policy profile to a user or group
authpolicy-show               Show authentication provider profile details
group-add                     Create a group with the given name
group-list                    List available groups
937
938
939
940
941
942
943
944
945
946
947
user-list                     List users
user-modify                   Modify user
user-show                     Show user details
============================  ===========================

Pithos snf-manage commands
--------------------------

============================  ===========================
Name                          Description
============================  ===========================
948
reconcile-commissions-pithos  Display unresolved commissions and trigger their recovery
949
service-export-pithos         Export Pithos services and resources in JSON format
950
reconcile-resources-pithos    Detect unsynchronized usage between Astakos and Pithos DB resources and synchronize them if specified so.
951
952
953
954
955
============================  ===========================

Cyclades snf-manage commands
----------------------------

956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
============================== ===========================
Name                           Description
============================== ===========================
backend-add                    Add a new Ganeti backend
backend-list                   List backends
backend-modify                 Modify a backend
backend-update-status          Update backend statistics for instance allocation
backend-remove                 Remove a Ganeti backend
server-create                  Create a new server
server-show                    Show server details
server-list                    List servers
server-modify                  Modify a server
server-import                  Import an existing Ganeti VM into synnefo
server-inspect                 Inspect a server in DB and Ganeti
network-create                 Create a new network
network-list                   List networks
network-modify                 Modify a network
network-inspect                Inspect network state in DB and Ganeti
network-remove                 Delete a network
flavor-create                  Create a new flavor
flavor-list                    List flavors
flavor-modify                  Modify a flavor
image-list                     List images
image-show                     Show image details
pool-create                    Create a bridge or mac-prefix pool
pool-show                      Show pool details
pool-list                      List pools
pool-modify                    Modify a pool
pool-remove                    Delete a pool
985
986
987
988
989
990
991
992
993
port-create                    Create a port connecting a server to a network
port-inspect                   Inspect the state of a port in DB and Ganeti
port-list                      List ports
port-remove                    Delete a port
floating-ip-create             Create a new floating IP
floating-ip-attach             Attach a floating IP to a server
floating-ip-dettach            Dettach a flotaing IP from a server
floating-ip-list               List floating IPs
floating-ip-remove             Delete a floating IP
994
995
queue-inspect                  Inspect the messages of a RabbitMQ queue
queue-retry                    Resend messages from Dead Letter queues to original exchanges
996
service-export-cyclades        Export Cyclades services and resources in JSON format
997
998
999
1000
subnet-create                  Create a subnet
subnet-inspect                 Inspect a subnet in DB
subnet-list                    List subnets
subnet-modify                  Modify a subnet
1001
1002
1003
1004
1005
1006
reconcile-servers              Reconcile servers of Synnefo DB with state of Ganeti backend
reconcile-networks             Reconcile networks of Synnefo DB with state of Ganeti backend
reconcile-pools                Check consistency of pool resources
reconcile-commissions-cyclades Detect and resolve pending commissions to Quotaholder
reconcile-resources-cyclades   Reconcile resource usage of Astakos with Cyclades DB.
============================== ===========================
1007

1008

1009
1010
1011
1012
1013
1014
1015
1016
Astakos helper scripts
======================

Astakos includes two scripts to facilitate the installation procedure.
Running:

.. code-block:: console

1017
   snf-component-register [<component_name>]
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028

automates the registration of the standard Synnefo components (astakos,
cyclades, and pithos) in astakos database. It internally uses the script:

.. code-block:: console

   snf-service-export <component_name> <base_url>

which simulates the export of service and resource definitions of the
standard Synnefo components.

1029

1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
Pithos managing accounts
========================

Pithos provides a utility tool for managing accounts.
To run you just type:

.. code-block:: console

   # pithos-manage-accounts <command> [arguments]

This is the list of the available commands:

============================  ===========================
Name                          Description
============================  ===========================
delete                        Remove an account from the Pithos DB
export-quota                  Export account quota in a file
list                          List existing/dublicate accounts
merge                         Move an account contents in another account
set-container-quota           Set container quota for all or a specific account
============================  ===========================

1052

1053
The "kamaki" API client
1054
=======================
1055
1056
1057

To upload, register or modify an image you will need the **kamaki** tool.
Before proceeding make sure that it is configured properly. Verify that
1058
*image.url*, *file.url*, *user.url* and *token* are set as needed:
1059
1060
1061
1062
1063

.. code-block:: console

   $ kamaki config list

1064
To change a setting use ``kamaki config set``:
1065
1066
1067

.. code-block:: console

1068
   $ kamaki config set image.url https://cyclades.example.com/image
1069
1070
   $ kamaki config set file.url https://pithos.example.com/v1
   $ kamaki config set user.url https://accounts.example.com
1071
1072
   $ kamaki config set token ...

1073
1074
To test that everything works, try authenticating the current account with
kamaki:
1075
1076
1077
1078
1079

.. code-block:: console

  $ kamaki user authenticate

1080
This will output user information.
1081

1082
1083
1084
Upload Image
------------

1085
1086
By convention, images are stored in a container called ``images``. Check if the
container exists, by listing all containers in your account:
1087
1088
1089

.. code-block:: console

1090
   $ kamaki file list
1091

1092
If the container ``images`` does not exist, create it:
1093
1094
1095

.. code-block:: console

1096
  $ kamaki file create images
1097

1098
You are now ready to upload an image to container ``images``. You can upload it
1099
with a Pithos client, or use kamaki directly:
1100
1101
1102

.. code-block:: console

1103
   $ kamaki file upload ubuntu.iso images
1104

1105
You can use any Pithos client to verify that the image was uploaded correctly,
1106
or you can list the contents of the container with kamaki:
1107

1108
1109
1110
.. code-block:: console

  $ kamaki file list images
1111
1112

The full Pithos URL for the previous example will be
1113
1114
``pithos://u53r-un1qu3-1d/images/ubuntu.iso`` where ``u53r-un1qu3-1d`` is the
unique user id (uuid).
1115
1116
1117
1118

Register Image
--------------

1119
To register an image you will need to use the full Pithos URL. To register as
1120
1121
1122
1123
a public image the one from the previous example use:

.. code-block:: console

1124
   $ kamaki image register Ubuntu pithos://u53r-un1qu3-1d/images/ubuntu.iso --public
1125
1126

The ``--public`` flag is important, if missing the registered image will not
1127
be listed by ``kamaki image list``.
1128

1129
Use ``kamaki image register`` with no arguments to see a list of available
1130
1131
1132
1133
options. A more complete example would be the following:

.. code-block:: console

1134
   $ kamaki image register Ubuntu pithos://u53r-un1qu3-1d/images/ubuntu.iso \
1135
1136
1137
1138
1139
1140
            --public --disk-format diskdump --property kernel=3.1.2

To verify that the image was registered successfully use:

.. code-block:: console

1141
   $ kamaki image list --name-like=ubuntu
1142

1143
1144
1145
1146

Miscellaneous
=============

1147
.. _branding:
Olga Brani's avatar
Olga Brani committed
1148

1149
Branding
1150
--------
Olga Brani's avatar
Olga Brani committed
1151

1152
1153
1154
1155
Since Synnefo v0.14, you are able to adapt the Astakos, Pithos and Cyclades Web
UI to your company’s visual identity. This is possible using the snf-branding
component, which is automatically installed on the nodes running the API
servers for Astakos, Pithos and Cyclades. 
Olga Brani's avatar
Olga Brani committed
1156
1157
1158
1159

Configuration
~~~~~~~~~~~~~

1160
1161
1162
1163
This can be done by modifing the settings provided by the snf-branding component
to match your service identity. The settings for the snf-branding application
can be found inside the configuration file ``/etc/synnefo/15-snf-branding.conf``
on the nodes that have Astakos, Pithos and Cyclades installed.
Olga Brani's avatar
Olga Brani committed
1164

1165
1166
1167
By default, the global service name is "Synnefo" and the company name is
"GRNET". These names and their respective logos and URLs are used throughout
the Astakos, Pithos and Cyclades UI.
Olga Brani's avatar
Olga Brani committed
1168

1169
**Names and URLs:**
Olga Brani's avatar
Olga Brani committed
1170

1171
1172
The first group of branding customization refers to the service's and company's
information.
Olga Brani's avatar
Olga Brani committed
1173

1174
1175
You can overwrite the company and the service name and URL respectively by
uncommenting and setting the following:
Olga Brani's avatar
Olga Brani committed
1176

1177
1178
1179
1180
1181
.. code-block:: python
  
  # setting used in Astakos Dashboard/Projects pages
  BRANDING_SERVICE_NAME = 'My cloud'
  BRANDING_SERVICE_URL = 'http://www.mycloud.synnefo.org/'
Olga Brani's avatar
Olga Brani committed
1182

1183
1184
1185
1186
1187
  # settings used in Astakos, Pithos, Cyclades footer only if 
  # BRANDING_SHOW_COPYRIGHT is set to True
  BRANDING_SHOW_COPYRIGHT = True
  BRANDING_COMPANY_NAME = 'Company LTD'
  BRANDING_COMPANY_URL = 'https://www.company-ltd.synnefo.org/'
Olga Brani's avatar
Olga Brani committed
1188
1189


Olga Brani's avatar
Olga Brani committed
1190
**Copyright and footer options:**
Olga Brani's avatar
Olga Brani committed
1191

1192
1193
By default, no Copyright message is shown in the UI footer. If you want to make
it visible in the footer of Astakos, Pithos and Cyclades UI, you can uncomment
1194
and set to ``True`` the ``BRANDING_SHOW_COPYRIGHT`` setting:
Olga Brani's avatar
Olga Brani committed
1195

1196
.. code-block:: python
Olga Brani's avatar
Olga Brani committed
1197
1198
1199

  #BRANDING_SHOW_COPYRIGHT = False

1200
Copyright message defaults to 'Copyright (c) 2011-<current_year>
1201
1202
<BRANDING_COMPANY_NAME>.' but you can overwrite it to a completely custom one by
setting the following option:
Olga Brani's avatar
Olga Brani committed
1203

1204
1205
1206
.. code-block:: python

  BRANDING_COPYRIGHT_MESSAGE = 'Copyright (c) 2011-2013 GRNET'
Olga Brani's avatar
Olga Brani committed
1207

Olga Brani's avatar
Olga Brani committed
1208
1209
1210
1211
1212
1213
1214
1215
1216
If you want to include a custom message in the footer, you can uncomment and 
set the ``BRANDING_FOOTER_EXTRA_MESSAGE`` setting. You can use html markup. 
Your custom message will appear  above Copyright message at the Compute 
templates and the Dashboard UI.

.. code-block:: python

  #BRANDING_FOOTER_EXTRA_MESSAGE = ''

Olga Brani's avatar
Olga Brani committed
1217
1218
1219

**Images:**

1220
1221
The Astakos, Pithos and Cyclades Web UI has some logos and images.
 
Olga Brani's avatar
Olga Brani committed
1222
1223
1224
1225
1226
1227
1228
1229
The branding-related images are presented in  the following table:

===============  ============================  =========
Image            Name/extension  convention    Usage
===============  ============================  =========
Favicon          favicon.ico                   Favicon for all services
Dashboard logo   dashboard_logo.png            Visible in all Astakos UI pages
Compute logo     compute_logo.png              Visible in all Cyclades UI pages
1230
1231
Console logo     console_logo.png              Visible in the Cyclades Console Window
Storage logo     storage_logo.png              Visible in all Pithos UI pages
Olga Brani's avatar
Olga Brani committed
1232
1233
===============  ============================  =========

1234
There are two methods  available for replacing all, or individual, 
1235
branding-related images:
Olga Brani's avatar
Olga Brani committed
1236

1237
1238
1. Create a new directory inside ``/usr/share/synnefo/static/`` (e.g.
   ``mybranding``) and place there some or all of your images.
Olga Brani's avatar
Olga Brani committed
1239

1240
   If you want to replace all of your images, keep the name/extension
1241
1242
   conventions as indicated in the above table and change the
   ``BRANDING_IMAGE_MEDIA_URL`` setting accordingly:
Olga Brani's avatar
Olga Brani committed
1243

1244
1245
1246
1247
   .. code-block:: python
        
      # using relative path
      BRANDING_IMAGE_MEDIA_URL= '/static/mybranding/images/' 
Olga Brani's avatar
Olga Brani committed
1248

1249
1250
      # or if you already host them in a separate domain (e.g. cdn)
      BRANDING_IMAGE_MEDIA_URL= 'https://cdn.synnefo.org/branding/images/'
Olga Brani's avatar
Olga Brani committed
1251
1252


1253
1254
1255
1256
   If you wish to replace individual images, **do not uncomment**
   ``BRANDING_IMAGE_MEDIA_URL``, but instead provide a relative path, pointing to
   the file inside your directory for each ``BRANDING_<image>_URL`` that you wish
   to replace.
Olga Brani's avatar
Olga Brani committed
1257

1258
2. Upload some or all of your images to a server and replace each 
1259
   ``BRANDING_<image>_URL`` with the absolute url of the image (i.e.
1260
   ``BRANDING_DASHBOARD_URL = 'https://www.synnefo.com/images/my_dashboard.jpg'``).
Olga Brani's avatar
Olga Brani committed
1261

1262
   Note that the alternative text  for each image tag inside html documents is 
1263
   alt=“BRANDING_SERVICE_NAME {Dashboard, Compute. Console, Storage}” respectively.
Olga Brani's avatar
Olga Brani committed
1264

1265
.. note:: Retina optimized images:
Olga Brani's avatar
Olga Brani committed
1266

1267
1268
   Synnefo UI is optimized for Retina displays. As far as images are concerned,  
   `retina.js <http://retinajs.com/>`_ is used.
Olga Brani's avatar
Olga Brani committed
1269

1270
   Retina.js checks each image on a page to see if there is a high-resolution 
1271
1272
   version of that image on your server. If a high-resolution variant exists, 
   the script will swap in that image in-place.
Olga Brani's avatar
Olga Brani committed
1273

1274
1275
1276
1277
1278
   The script assumes you use  `Apple's prescribed high-resolution modifier (@2x)
   <http://developer.apple.com/library/ios/#documentation/2DDrawing/Conceptual/
   DrawingPrintingiOS/SupportingHiResScreensInViews/SupportingHiResScreensInViews
   .html#//apple_ref/doc/uid/TP40010156-CH15-SW1>`_ to denote high-resolution 
   image variants on your server.
Olga Brani's avatar
Olga Brani committed
1279

1280
1281
1282
1283
1284
   For each of the images that you wish the script to  replace, you must have a 
   high-resolution variant in the same folder  named correctly and it will be 
   detected automatically. For example if your image is in <my_directory> and is 
   named "my_image.jpg" the script will look in the same directory for an image 
   named "my_image@2x.jpg".
Olga Brani's avatar
Olga Brani committed
1285

1286
1287
   In case that you don’t want to use a high-resolution image, the 
   normal-resolution image will be visible.
Olga Brani's avatar
Olga Brani committed
1288

1289
1290
More branding
~~~~~~~~~~~~~
Olga Brani's avatar
Olga Brani committed
1291

1292
1293
Although, it is not 100% branding-related, further verbal customization is
feasible. 
Olga Brani's avatar
Olga Brani committed
1294
1295
1296

**EMAILS**

1297
1298
1299
The output of all email `*`.txt files will be already customized to contain your
company and service names but you can further alter their content if you feel it
best fits your needs as simple as creasynnefo template.    
1300

1301
1302
In order to overwrite one or more email-templates you need to place your 
modified <email-file>.txt files respecting the following structure:
1303
  
1304
1305
  **/etc/synnefo/templates/**
      **im/**
1306
1307
1308
1309
1310
          | activation_email.txt
          | email.txt
          | invitation.txt
          | switch_accounts_email.txt
          | welcome_email.txt
1311
          **projects/**
1312
1313
1314
1315
1316
1317
1318
1319
              | project_approval_notification.txt
              | project_denial_notification.txt    
              | project_membership_change_notification.txt
              | project_membership_enroll_notification.txt
              | project_membership_leave_request_notification.txt
              | project_membership_request_notification.txt
              | project_suspension_notification.txt
              | project_termination_notification.txt
1320
      **registration/**
1321
1322
1323
1324
1325
          | email_change_email.txt
          | password_email.txt

Feel free to omit any of the above files you do not wish to overwrite.

Olga Brani's avatar
Olga Brani committed
1326
1327
1328
Below is a list of all emails sent by Synnefo to users along with a short 
description and a link to their content:

1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
* ``snf-astakos-app/astakos/im/templates/im/email.txt``
  Base email template. Contains a contact email and a “thank you” message.
  (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/activation_email.txt`` Email sent to
  user that prompts  him/her to click on a link provided to activate the account.
  Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/activation_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/invitation.txt`` Email sent to an
  invited user. He/she has to click on a link provided to activate the account.
  Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/invitation.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/switch_accounts_email.txt`` Email
  sent to user upon his/her request to associate this email address with a
  shibboleth account. He/she has to click on a link provided to activate the
  association. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/switch_accounts_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/welcome_email.txt`` Email sent to
  inform the user that his/ her account has been activated. Extends “email.txt”
  (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/welcome_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/registration/email_change_email.txt``
  Email sent to user when he/she has requested new email address assignment. The
  user has to click on a link provided to validate this action. Extends
  “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/registration/email_change_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/registration/password_email.txt`` Email
  sent for resetting password purpose. The user has to click on a link provided
  to validate this action. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/registration/password_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_approval_notification.txt``
  Informs  the project owner that his/her project has been approved. Extends
  “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_approval_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_denial_notification.txt``
  Informs the project owner that his/her  project application has been denied
  explaining the reasons. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_denial_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_change_notification.txt``
  An email is sent to a user containing information about his project membership
  (whether he has been accepted, rejected or removed). Extends “email.txt” (`Link
  <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_change_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_enroll_notification.txt``
  Informs a user that he/she  has been enrolled to a project. Extends
  “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_enroll_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_leave_request_notification.txt``
  An email is sent to the project owner to make him aware of a  user having
  requested to leave his project. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_leave_request_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_request_notification.txt``
  An email is sent to the project owner to make him/her aware of a user having
  requested to join  his project. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_request_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_suspension_notification.txt``
  An email is sent to the project owner to make him/her aware of his/her project
  having been suspended. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_suspension_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_termination_notification.txt``
  An email is sent to the project owner to make him/her aware of his/her project
  having been terminated. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_termination_notification.txt>`_)
Olga Brani's avatar
Olga Brani committed
1377
1378
1379
1380

.. warning:: Django templates language:

  If you choose to  overwrite these email templates, be mindful of the necessary 
1381
1382
1383
  information contained in django template variables that must not be omitted, 
  such as the activation link for activating one’s account and many more. 
  These variables are contained into {{}} inside the templates.
Olga Brani's avatar
Olga Brani committed
1384
1385


1386
.. RabbitMQ
1387

1388
1389
1390
1391
1392
1393
RabbitMQ Broker
---------------

Queue nodes run the RabbitMQ sofware, which provides AMQP functionality. To
guarantee high-availability, more than one Queue nodes should be deployed, each
of them belonging to the same `RabbitMQ cluster
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
<http://www.rabbitmq.com/clustering.html>`_. Synnefo uses the RabbitMQ
active/active `High Available Queues <http://www.rabbitmq.com/ha.html>`_ which
are mirrored between two nodes within a RabbitMQ cluster.

The RabbitMQ nodes that form the cluster, are declared to Synnefo through the
`AMQP_HOSTS` setting. Each time a Synnefo component needs to connect to
RabbitMQ, one of these nodes is chosen in a random way. The client that Synnefo
uses to connect to RabbitMQ, handles connection failures transparently and
tries to reconnect to a different node. As long as one of these nodes are up
and running, functionality of Synnefo should not be downgraded by the RabbitMQ
node failures.
1405
1406

All the queues that are being used are declared as durable, meaning that
1407
1408
messages are persistently stored to RabbitMQ, until they get successfully
processed by a client.
1409
1410
1411

Currently, RabbitMQ is used by the following components:

1412
* `snf-ganeti-eventd`, `snf-ganeti-hook` and `snf-progress-monitor`:
1413
1414
  These components send messages concerning the status and progress of
  jobs in the Ganeti backend.
1415
1416
* `snf-dispatcher`: This daemon, consumes the messages that are sent from
  the above components, and updates the Cyclades DB accordingly.
1417

Christos Stavrakakis's avatar
Christos Stavrakakis committed
1418

1419
Installation
1420
1421
~~~~~~~~~~~~

1422
1423
1424
1425
1426
1427
Please check the RabbitMQ documentation which covers extensively the
`installation of RabbitMQ server <http://www.rabbitmq.com/download.html>`_ and
the setup of a `RabbitMQ cluster <http://www.rabbitmq.com/clustering.html>`_.
Also, check out the `web management plugin
<http://www.rabbitmq.com/management.html>`_ that can be useful for managing and
monitoring RabbitMQ.
1428
1429

For a basic installation of RabbitMQ on two nodes (node1 and node2) you can do
1430
the following:
1431

1432
On both nodes, install rabbitmq-server and create a Synnefo user:
1433
1434
1435
1436
1437
1438
1439

.. code-block:: console

  $ apt-get install rabbitmq-server
  $ rabbitmqctl add_user synnefo "example_pass"
  $ rabbitmqctl set_permissions synnefo  ".*" ".*" ".*"

1440
Also guarantee that both nodes share the same cookie, by running:
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468

.. code-block:: console

  $ scp node1:/var/lib/rabbitmq/.erlang.cookie node2:/var/lib/rabbitmq/.erlang.cookie

and restart the nodes:

.. code-block:: console

  $ /etc/init.d/rabbitmq-server restart


To setup the RabbitMQ cluster run:

.. code-block:: console

  root@node2: rabbitmqctl stop_app
  root@node2: rabbitmqctl reset
  root@node2: rabbitmqctl cluster rabbit@node1 rabbit@node2
  root@node2: rabbitmqctl start_app

You can verify that the cluster is set up correctly by running:

.. code-block:: console

  root@node2: rabbitmqctl cluster_status


1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
Logging
-------

Logging in Synnefo is using Python's logging module. The module is configured
using dictionary configuration, whose format is described here:

http://docs.python.org/release/2.7.1/library/logging.html#logging-config-dictschema

Note that this is a feature of Python 2.7 that we have backported for use in
Python 2.6.

1480 <