admin-guide.rst 66.3 KB
Newer Older
1
2
3
4
5
6
7
8
.. _admin-guide:

Synnefo Administrator's Guide
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is the complete Synnefo Administrator's Guide.


9
.. _syn+archip:
10

11
12
13
General Synnefo Architecture
============================

14
15
16
17
The following figure shows a detailed view of the whole Synnefo architecture
and how it interacts with multiple Ganeti clusters. We hope that after reading
the Administrator's Guide you will be able to understand every component and
all the interactions between them.
18

19
.. image:: images/synnefo-arch2.png
20
   :width: 100%
21
   :target: _images/synnefo-arch2.png
22

23
24
25
Synnefo also supports RADOS as an alternative storage backend for
Files/Images/VM disks. You will find the :ref:`corresponding figure
<syn+archip+rados>` later in this guide.
26

27

28
29
Identity Service (Astakos)
==========================
30
31


32
Authentication methods
33
----------------------
34

35
Astakos supports multiple authentication methods:
36

37
38
39
40
41
42
 * local username/password
 * LDAP / Active Directory
 * SAML 2.0 (Shibboleth) federated logins
 * Google
 * Twitter
 * LinkedIn
43
44
45
46

.. _shibboleth-auth:

Shibboleth Authentication
47
~~~~~~~~~~~~~~~~~~~~~~~~~
48
49
50
51
52
53
54
55
56
57
58
59
60
61

Astakos can delegate user authentication to a Shibboleth federation.

To setup shibboleth, install package::

  apt-get install libapache2-mod-shib2

Change appropriately the configuration files in ``/etc/shibboleth``.

Add in ``/etc/apache2/sites-available/synnefo-ssl``::

  ShibConfig /etc/shibboleth/shibboleth2.xml
  Alias      /shibboleth-sp /usr/share/shibboleth

62
  <Location /ui/login/shibboleth>
63
64
65
66
67
68
69
70
71
72
73
74
75
    AuthType shibboleth
    ShibRequireSession On
    ShibUseHeaders On
    require valid-user
  </Location>

and before the line containing::

  ProxyPass        / http://localhost:8080/ retry=0

add::

  ProxyPass /Shibboleth.sso !
76

77
78
79
Then, enable the shibboleth module::

  a2enmod shib2
80

81
82
83
84
85
86
87
88
89
90
91
92
93
94
After passing through the apache module, the following tokens should be
available at the destination::

  eppn # eduPersonPrincipalName
  Shib-InetOrgPerson-givenName
  Shib-Person-surname
  Shib-Person-commonName
  Shib-InetOrgPerson-displayName
  Shib-EP-Affiliation
  Shib-Session-ID

Finally, add 'shibboleth' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
95
Twitter Authentication
96
~~~~~~~~~~~~~~~~~~~~~~
Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
97
98
99
100
101
102
103
104
105
106
107
108
109
110

To enable twitter authentication while signed in under a Twitter account,
visit dev.twitter.com/apps.

Click Create an application.

Fill the necessary information and for callback URL give::

    https://node1.example.com/ui/login/twitter/authenticated

Finally, add 'twitter' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Google Authentication
111
~~~~~~~~~~~~~~~~~~~~~
Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127

To enable google authentication while signed in under a Google account,
visit https://code.google.com/apis/console/.

Under API Access select Create another client ID, select Web application,
expand more options in Your site or hostname section and in Authorized
Redirect URIs add:


Fill the necessary information and for callback URL give::

    https://node1.example.com/ui/login/google/authenticated

Finally, add 'google' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

128
129
130
131

Working with Astakos
--------------------

132
133
User registration
~~~~~~~~~~~~~~~~~
134

135
136
When a new user signs up, he/she is not directly marked as active. You can see 
his/her state by running (on the machine that runs the Astakos app):
137
138
139

.. code-block:: console

140
   $ snf-manage user-list
141

142
143
More detailed user status is provided in the `status` field of the `user-show` 
command:
144

145
.. code-block:: console
146

147
  $ snf-manage user-show <user-id>
148

149
150
151
152
153
  id                  : 6
  uuid                : 78661411-5eed-412f-a9ea-2de24f542c2e
  status              : Accepted/Active (accepted policy: manual)
  email               : user@synnefo.org
  ....
154

155
156
157
Based on the `astakos-app` configuration, there are several ways for a user to
get verified and activated in order to be able to login. We discuss the user
verification and activation flow in the following section.
158
159

User activation flow
160
161
````````````````````

162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
A user can register for an account using the astakos signup form. Once the form
is submited successfully a user entry is created in astakos database. That entry
is passed through the astakos activation backend which handles whether the user
should be automatically verified and activated.

Email verification
``````````````````

The verification process takes place in order to ensure that the user owns the
email provided during the signup process. By default, after each successful
signup astakos notifies user with an verification url via email. 

At this stage:

    * subsequent registrations invalidate and delete the previous registrations 
      of the same email address.

    * in case user misses the initial notification, additional emails can be
      send either via the url which is prompted to the user if he tries to
      login, or by the administrator using the ``snf-manage user-activation-send
      <userid>`` command.

    * administrator may also enforce a user to get verified using the
      ``snf-manage user-modify --verify <userid>`` command.

Account activation
``````````````````

190
Once the user gets verified, it is time for Astakos to decide whether or not to
191
192
193
proceed through user activation process. If ``ASTAKOS_MODERATION_ENABLED``
setting is set to ``False`` (default value) user gets activated automatically. 

194
In case the moderation is enabled Astakos may still automatically activate the
195
196
197
198
199
200
201
202
user in the following cases:

    * User email matches any of the regular expressions defined in
      ``ASTAKOS_RE_USER_EMAIL_PATTERNS`` (defaults to ``[]``)
    * User used a signup method (e.g. ``shibboleth``) for which automatic
      activation is enabled (see 
      :ref:`authentication methods policies <auth_methods_policies>`).

203
204
205
206
207
208
If all of the above fail to trigger automatic activation, an email is sent to
the persons listed in ``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings,
notifing that there is a new user pending for moderation and that it's up to
the administrator to decide if the user should be activated. The UI also shows
a corresponding 'pending moderation' message to the user. The administrator can
activate a user using the ``snf-manage user-modify`` command:
209
210
211
212
213
214
215
216
217

.. code-block:: console

    # command to activate a pending user
    $ snf-manage user-modify --accept <userid>

    # command to reject a pending user
    $ snf-manage user-modify --reject --reject-reason="spammer" <userid>

218
219
220
221
Once the activation process finishes, a greeting message is sent to the user
email address and a notification for the activation to the persons listed in
``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings. Once activated the user is
able to login and access the Synnefo services.
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282

Additional authentication methods
`````````````````````````````````

Astakos supports third party logins from external identity providers. This
can be usefull since it allows users to use their existing credentials to 
login to astakos service.

Currently astakos supports the following identity providers:

    * `Shibboleth <http://www.internet2.edu/shibboleth>`_ (module name
      ``shibboleth``)
    * `Google <https://developers.google.com/accounts/docs/OAuth2>`_ (module
      name ``google``)
    * `Twitter <https://dev.twitter.com/docs/auth>`_ (module name ``twitter``)
    * `LinkedIn <http://developer.linkedin.com/documents/authentication>`_
      (module name ``linkedin``)

To enable any of the above modules (by default only ``local`` accounts are
allowed), retrieve and set the required provider settings and append the 
module name in ``ASTAKOS_IM_MODULES``.

.. code-block:: python

    # settings from https://code.google.com/apis/console/
    ASTAKOS_GOOGLE_CLIENT_ID = '1111111111-epi60tvimgha63qqnjo40cljkojcann3.apps.googleusercontent.com'
    ASTAKOS_GOOGLE_SECRET = 'tNDQqTDKlTf7_LaeUcWTWwZM'
    
    # let users signup and login using their google account
    ASTAKOS_IM_MODULES = ['local', 'google']


.. _auth_methods_policies:

Authentication method policies
``````````````````````````````

Astakos allows you to override the default policies for each enabled provider 
separately by adding the approriate settings in your ``.conf`` files in the 
following format:

**ASTAKOS_AUTH_PROVIDER_<module>_<policy>_POLICY**

Available policies are:

    * **CREATE** Users can signup using that provider (default: ``True``) 
    * **REMOVE/ADD** Users can remove/add login method from their profile 
      (default: ``True``)
    * **AUTOMODERATE** Automatically activate users that signup using that
      provider (default: ``False``)
    * **LOGIN** Whether or not users can use the provider to login (default:
      ``True``).

e.g. to enable automatic activation for your academic users, while keeping 
locally signed up users under moderation you can apply the following settings.

.. code-block:: python

    ASTAKOS_AUTH_PROVIDER_SHIBBOLETH_AUTOMODERATE_POLICY = True
    ASTAKOS_AUTH_PROVIDER_SHIBBOLETH_REMOVE_POLICY = False

283
284
285
286
287
288
289
290
291
292
293
294
295
296
User login
~~~~~~~~~~

During the logging procedure, the user is authenticated by the respective
identity provider.

If ``ASTAKOS_RECAPTCHA_ENABLED`` is set and the user fails several times
(``ASTAKOS_RATELIMIT_RETRIES_ALLOWED`` setting) to provide the correct
credentials for a local account, he/she is then prompted to solve a captcha
challenge.

Upon success, the system renews the token (if it has expired), logins the user
and sets the cookie, before redirecting the user to the ``next`` parameter
value.
297

298
299
300
Setting quota limits
~~~~~~~~~~~~~~~~~~~~

301
302
Set default quota
`````````````````
303
304
305
306

In 20-snf-astakos-app-settings.conf, 
uncomment the default setting ``ASTAKOS_SERVICES``
and customize the ``'uplimit'`` values.
307
These are the default base quota for all users.
308

309
You can modify the default base quota limit for all future users with::
310

311
   # snf-manage resource-modify <resource_name> --default-quota <value>
312

313
314
Set base quota for individual users
```````````````````````````````````
315

316
For individual users that need different quota than the default
317
318
you can set it for each resource like this::

319
320
    # use this to display quota / uuid
    # snf-manage user-show 'uuid or email' --quota
321

322
    # snf-manage user-modify 'user-uuid' --set-base-quota 'cyclades.vm' 10
323
324
325
326
327
328
329
330
331
332
333
334


Enable the Projects feature
~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you want to enable the projects feature so that users may apply
on their own for resources by creating and joining projects,
in ``20-snf-astakos-app-settings.conf`` set::

    # this will make the 'projects' page visible in the dashboard
    ASTAKOS_PROJECTS_VISIBLE = True

335
336
You can change the maximum allowed number of pending project applications
per user with::
337

338
    # snf-manage resource-modify astakos.pending_app --default-quota <number>
339
340
341

You can also set a user-specific limit with::

342
    # snf-manage user-modify 'user-uuid' --set-base-quota 'astakos.pending_app' 5
343

344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
When users apply for projects they are not automatically granted
the resources. They must first be approved by the administrator.

To list pending project applications in astakos::

    # snf-manage project-list --pending

Note the last column, the application id. To approve it::

    # <app id> from the last column of project-list
    # snf-manage project-control --approve <app id>

To deny an application::

    # snf-manage project-control --deny <app id>

360
361
362
363
364
365
Users designated as *project admins* can approve, deny, or modify
an application through the web interface. In
``20-snf-astakos-app-settings.conf`` set::

    # UUIDs of users that can approve or deny project applications from the web.
    ASTAKOS_PROJECT_ADMINS = [<uuid>, ...]
366
367


368
369
370
371
372
373
374
375
376
377
378
379
Astakos advanced operations
---------------------------

Adding "Terms of Use"
~~~~~~~~~~~~~~~~~~~~~

Astakos supports versioned terms-of-use. First of all you need to create an
html file that will contain your terms. For example, create the file
``/usr/share/synnefo/sample-terms.html``, which contains the following:

.. code-block:: console

380
   <h1>My cloud service terms</h1>
381

382
   These are the example terms for my cloud service
383
384
385
386
387
388
389
390
391
392

Then, add those terms-of-use with the snf-manage command:

.. code-block:: console

   $ snf-manage term-add /usr/share/synnefo/sample-terms.html

Your terms have been successfully added and you will see the corresponding link
appearing in the Astakos web pages' footer.

393
394
395
396
397
398
399
400
401
During the account registration, if there are approval terms, the user is
presented with an "I agree with the Terms" checkbox that needs to get checked
in order to proceed.

In case there are new approval terms that the user has not signed yet, the
``signed_terms_required`` view decorator redirects to the ``approval_terms``
view, so the user will be presented with the new terms the next time he/she
logins.

402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
Enabling reCAPTCHA
~~~~~~~~~~~~~~~~~~

Astakos supports the `reCAPTCHA <http://www.google.com/recaptcha>`_ feature.
If enabled, it protects the Astakos forms from bots. To enable the feature, go
to https://www.google.com/recaptcha/admin/create and create your own reCAPTCHA
key pair. Then edit ``/etc/synnefo/20-snf-astakos-app-settings.conf`` and set
the corresponding variables to reflect your newly created key pair. Finally, set
the ``ASTAKOS_RECAPTCHA_ENABLED`` variable to ``True``:

.. code-block:: console

   ASTAKOS_RECAPTCHA_PUBLIC_KEY = 'example_recaptcha_public_key!@#$%^&*('
   ASTAKOS_RECAPTCHA_PRIVATE_KEY = 'example_recaptcha_private_key!@#$%^&*('

   ASTAKOS_RECAPTCHA_ENABLED = True

Restart the service on the Astakos node(s) and you are ready:

.. code-block:: console

   # /etc/init.d/gunicorn restart

Checkout your new Sign up page. If you see the reCAPTCHA box, you have setup
everything correctly.

428

429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
Astakos internals
-----------------

X-Auth-Token
~~~~~~~~~~~~

Alice requests a specific resource from a cloud service e.g.: Pithos. In the
request she supplies the `X-Auth-Token` to identify whether she is eligible to
perform the specific task. The service contacts Astakos through its
``/account/v1.0/authenticate`` api call (see :ref:`authenticate-api-label`)
providing the specific ``X-Auth-Token``. Astakos checkes whether the token
belongs to an active user and it has not expired and returns a dictionary
containing user related information. Finally the service uses the ``uniq``
field included in the dictionary as the account string to identify the user
accessible resources.

.. _authentication-label:

Django Auth methods and Backends
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Astakos incorporates Django user authentication system and extends its User model.

Since username field of django User model has a limitation of 30 characters,
AstakosUser is **uniquely** identified by the ``email`` instead. Therefore,
``astakos.im.authentication_backends.EmailBackend`` is served to authenticate a
user using email if the first argument is actually an email, otherwise tries
the username.

A new AstakosUser instance is assigned with a uui as username and also with a
``auth_token`` used by the cloud services to authenticate the user.
``astakos.im.authentication_backends.TokenBackend`` is also specified in order
to authenticate the user using the email and the token fields.

Logged on users can perform a number of actions:

 * access and edit their profile via: ``/im/profile``.
 * change their password via: ``/im/password``
 * send feedback for grnet services via: ``/im/send_feedback``
 * logout (and delete cookie) via: ``/im/logout``

Internal Astakos requests are handled using cookie-based Django user sessions.

External systems should forward to the ``/login`` URI. The server,
depending on its configuration will redirect to the appropriate login page.
When done with logging in, the service's login URI should redirect to the URI
provided with next, adding user and token parameters, which contain the email
and token fields respectively.

The login URI accepts the following parameters:

======================  =========================
Request Parameter Name  Value
======================  =========================
next                    The URI to redirect to when the process is finished
renew                   Force token renewal (no value parameter)
force                   Force logout current user (no value parameter)
======================  =========================

External systems inside the ``ASTAKOS_COOKIE_DOMAIN`` scope can acquire the
user information by the cookie identified by ``ASTAKOS_COOKIE_NAME`` setting
(set during the login procedure).

Finally, backend systems having acquired a token can use the
:ref:`authenticate-api-label` API call from a private network or through HTTPS.


496
497
498
499
500
501
Compute/Network/Image Service (Cyclades)
========================================

Working with Cyclades
---------------------

502
Managing Ganeti Backends
Christos Stavrakakis's avatar
Christos Stavrakakis committed
503
~~~~~~~~~~~~~~~~~~~~~~~~
504

505
506
507
Since v0.11, Synnefo is able to manage multiple Ganeti clusters (backends)
making it capable to scale linearly to tens of thousands of VMs. Backends
can be dynamically added or removed via `snf-manage` commands.
508

509
510
511
512
513
Each newly created VM is allocated to a Ganeti backend by the Cyclades backend
allocator. The VM is "pinned" to this backend, and can not change through its
lifetime. The backend allocator decides in which backend to spawn the VM based
on the available resources of each backend, trying to balance the load between
them.
514
515

Handling of Networks, as far as backends are concerned, is based on whether the
516
517
518
519
network is public or not. Public networks are created through the `snf-manage
network-create` command, and are only created on one backend. Private networks
are created on all backends, in order to ensure that VMs residing on different
backends can be connected to the same private network.
520

521
522
523
Listing existing backends
`````````````````````````
To list all the Ganeti backends known to Synnefo, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
524

525
526
527
528
529
.. code-block:: console

   $ snf-manage backend-list

Adding a new Ganeti backend
Christos Stavrakakis's avatar
Christos Stavrakakis committed
530
```````````````````````````
531
532
Backends are dynamically added under the control of Synnefo with `snf-manage
backend-add` command. In this section it is assumed that a Ganeti cluster,
533
534
named ``cluster.example.com`` is already up and running and configured to be
able to host Synnefo VMs.
535

536
To add this Ganeti cluster, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
537

538
539
540
541
.. code-block:: console

   $ snf-manage backend-add --clustername=cluster.example.com --user="synnefo_user" --pass="synnefo_pass"

542
543
544
545
546
where ``clustername`` is the Cluster hostname of the Ganeti cluster, and
``user`` and ``pass`` are the credentials for the `Ganeti RAPI user
<http://docs.ganeti.org/ganeti/2.2/html/rapi.html#users-and-passwords>`_.  All
backend attributes can be also changed dynamically using the `snf-manage
backend-modify` command.
547

548
549
550
``snf-manage backend-add`` will also create all existing private networks to
the new backend. You can verify that the backend is added, by running
`snf-manage backend-list`.
551

552
553
554
555
556
557
558
559
560
561
Note that no VMs will be spawned to this backend, since by default it is in a
``drained`` state after addition and also it has no public network assigned to
it.

So, first you need to create its public network, make sure everything works as
expected and finally make it active by un-setting the ``drained`` flag. You can
do this by running:

.. code-block:: console

562
   $ snf-manage backend-modify --drained=False <backend_id>
563
564

Removing an existing Ganeti backend
Christos Stavrakakis's avatar
Christos Stavrakakis committed
565
```````````````````````````````````
566
In order to remove an existing backend from Synnefo, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
567

568
569
.. code-block:: console

570
   # snf-manage backend-remove <backend_id>
571

572
573
574
This command will fail if there are active VMs on the backend. Also, the
backend is not cleaned before removal, so all the Synnefo private networks
will be left on the Ganeti nodes. You need to remove them manually.
575

576
577
578
579
580
581
Allocation of VMs in Ganeti backends
````````````````````````````````````
As already mentioned, the Cyclades backend allocator is responsible for
allocating new VMs to backends. This allocator does not choose the exact Ganeti
node that will host the VM but just the Ganeti backend. The exact node is
chosen by the Ganeti cluster's allocator (hail).
582
583
584
585

The decision about which backend will host a VM is based on the available
resources. The allocator computes a score for each backend, that shows its load
factor, and the one with the minimum score is chosen. The admin can exclude
586
backends from the allocation phase by marking them as ``drained`` by running:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
587

588
589
.. code-block:: console

590
   $ snf-manage backend-modify --drained=True <backend_id>
591
592

The backend resources are periodically updated, at a period defined by
593
the ``BACKEND_REFRESH_MIN`` setting, or by running `snf-manage backend-update-status`
594
command. It is advised to have a cron job running this command at a smaller
595
interval than ``BACKEND_REFRESH_MIN`` in order to remove the load of refreshing
596
597
the backends stats from the VM creation phase.

598
599
600
Finally, the admin can decide to have a user's VMs being allocated to a
specific backend, with the ``BACKEND_PER_USER`` setting. This is a mapping
between users and backends. If the user is found in ``BACKEND_PER_USER``, then
601
602
Synnefo allocates all his/hers VMs to the specific backend in the variable,
even if is marked as drained (useful for testing).
603

604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
Allocation based on disk-templates
**********************************

Besides the available resources of each Ganeti backend, the allocator takes
into consideration the disk template of the instance when trying to allocate it
to a Ganeti backend. Specifically, the allocator checks if the flavor of the
instance belongs to the available disk templates of each Ganeti backend.

A Ganeti cluster has a list of enabled disk templates
(`--enabled-disk-templates`) and a list of allowed disk templates for new
instances (`--ipolicy-disk-templates`). See the `gnt-cluster` manpage for more
details about these options.

When Synnefo allocates an instance, it checks whether the disk template of the
new instance belongs both in the enabled and ipolicy disk templates. You can
see the list of the available disk-templates by running `snf-manage
backend-list`. This list should be updated automatically after changing
these options in Ganeti and it can also be updated by running `snf-manage
backend-update-status`.

So the administrator, can route instances on different backends based on their
flavor disk template, by modifying the enabled or ipolicy disk templates of
each backend.  Also, the administrator can route instances between different
nodes of the same Ganeti backend, by modifying the same options at the
nodegroup level (see `gnt-group` manpage for mor details).


631
632
633
Managing Virtual Machines
~~~~~~~~~~~~~~~~~~~~~~~~~

634
635
636
637
638
639
As mentioned, Cyclades uses Ganeti for management of VMs. The administrator can
handle Cyclades VMs just like any other Ganeti instance, via `gnt-instance`
commands. All Ganeti instances that belong to Synnefo, are separated from
others, by a prefix in their names. This prefix is defined in
``BACKEND_PREFIX_ID`` setting in
``/etc/synnefo/20-snf-cyclades-app-backend.conf``.
640

641
Apart from handling instances directly in the Ganeti level, a number of `snf-manage`
642
643
commands are available:

644
645
646
647
648
649
* ``snf-manage server-list``: List servers
* ``snf-manage server-show``: Show information about a server in the Cyclades DB
* ``snf-manage server-inspect``: Inspect the state of a server both in DB and Ganeti
* ``snf-manage server-modify``: Modify the state of a server in the Cycldes DB
* ``snf-manage server-create``: Create a new server
* ``snf-manage server-import``: Import an existing Ganeti instance to Cyclades
650
651
652
653
654


Managing Virtual Networks
~~~~~~~~~~~~~~~~~~~~~~~~~

655
656
657
658
Cyclades is able to create and manage Virtual Networks. Networking is
desployment specific and must be customized based on the specific needs of the
system administrator. For better understanding of networking please refer to
the :ref:`Network <networks>` section.
659
660
661

Exactly as Cyclades VMs can be handled like Ganeti instances, Cyclades Networks
can also by handled as Ganeti networks, via `gnt-network commands`. All Ganeti
662
networks that belong to Synnefo are named with the prefix
663
664
`${BACKEND_PREFIX_ID}-net-`.

665
There are also the following `snf-manage` commands for managing networks:
666

667
668
669
670
671
672
* ``snf-manage network-list``: List networks
* ``snf-manage network-show``: Show information about a network in the Cyclades DB
* ``snf-manage network-inspect``: Inspect the state of the network in DB and Ganeti backends
* ``snf-manage network-modify``: Modify the state of a network in the Cycldes DB
* ``snf-manage network-create``: Create a new network
* ``snf-manage network-remove``: Remove an existing network
673

674
Managing Network Resources
675
``````````````````````````
676

677
678
679
Proper operation of the Cyclades Network Service depends on the unique
assignment of specific resources to each type of virtual network. Specifically,
these resources are:
680

681
682
683
684
685
* IP addresses. Cyclades creates a Pool of IPs for each Network, and assigns a
  unique IP address to each VM, thus connecting it to this Network. You can see
  the IP pool of each network by running `snf-manage network-inspect
  <network_ID>`. IP pools are automatically created and managed by Cyclades,
  depending on the subnet of the Network.
686
* Bridges corresponding to physical VLANs, which are required for networks of
687
688
689
  type `PRIVATE_PHYSICAL_VLAN`.
* One Bridge corresponding to one physical VLAN which is required for networks of
  type `PRIVATE_MAC_PREFIX`.
690

691
692
693
694
695
696
697
698
699
700
701
IPv4 addresses
**************

An allocation pool of IPv4 addresses is automatically created for every network
that has the attribute `dhcp` set to True. The allocation pool contains the
range of IP addresses that are included in the subnet. The gateway and the
broadcast address of the network are excluded from the allocation pool. The
admin can externally reserve IP addresses to exclude them from automatic
allocation with the `--add-reserved-ips` option of `snf-manage network-modify`
command. For example the following command will reserve two IP addresses
from network with ID `42`:
702

703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
.. code-block:: console

 snf-manage network-modify --add-reserved-ips=10.0.0.21,10.0.0.22 42

.. warning:: Externally reserving IP addresses is also available at the Ganeti.
 However, when using Cyclades with multiple Ganeti backends, the handling of
 IP pools must be performed from Cyclades!

Bridges
*******

As already mentioned Cyclades use a pool of Bridges that must correspond
to Physical VLAN at the Ganeti level. A bridge from the pool is assigned to
each network of flavor `PHYSICAL_VLAN`. Creation of this pool is done
using `snf-manage pool-create` command. For example the following command
will create a pool containing the brdiges from `prv1` to `prv21`.
Christos Stavrakakis's avatar
Christos Stavrakakis committed
719

720
721
722
723
724
.. code-block:: console

   # snf-manage pool-create --type=bridge --base=prv --size=20

You can verify the creation of the pool, and check its contents by running:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
725

726
727
728
729
730
.. code-block:: console

   # snf-manage pool-list
   # snf-manage pool-show --type=bridge 1

731
732
733
734
735
736
737
738
739
Finally you can use the `pool-modify` management command in order to externally
reserve the values from pool, extend or shrink the pool if possible.

MAC Prefixes
************

Cyclades also use a pool of MAC prefixes to assign to networks of flavor
`MAC_FILTERED`. Handling of this pool is done exactly as with pool of bridges,
except that the type option must be set to mac-prefix:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
740

741
742
743
744
.. code-block:: console

   # snf-manage pool-create --type=mac-prefix --base=aa:00:0 --size=65536

745
746
747
748
749
750
751
752
753
754
755
The above command will create a pool of MAC prefixes from ``aa:00:1`` to
``b9:ff:f``. The MAC prefix pool is responsible for providing only unicast and
locally administered MAC addresses, so many of these prefixes will be
externally reserved, to exclude from allocation.

Pool reconciliation
*******************

The management command `snf-manage reconcile-pools` can be used that all the
above mentioned pools are consistent and that all values that come from the
pool are not used more than once.
756

757

758
759
760
761
762
Cyclades advanced operations
----------------------------

Reconciliation mechanism
~~~~~~~~~~~~~~~~~~~~~~~~
763

764
765
766
767
768
769
On certain occasions, such as a Ganeti or RabbitMQ failure, the state of
Cyclades database may differ from the real state of VMs and networks in the
Ganeti backends. The reconciliation process is designed to synchronize
the state of the Cyclades DB with Ganeti. There are two management commands
for reconciling VMs and Networks

770
771
772
Reconciling Virtual Machines
````````````````````````````

773
Reconciliation of VMs detects the following conditions:
774

775
776
 * Stale DB servers without corresponding Ganeti instances
 * Orphan Ganeti instances, without corresponding DB entries
777
 * Out-of-sync state for DB entries wrt to Ganeti instances
778

779
To detect all inconsistencies you can just run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
780

781
.. code-block:: console
782
783

  $ snf-manage reconcile-servers
784

785
Adding the `--fix-all` option, will do the actual synchronization:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
786

787
.. code-block:: console
788

789
  $ snf-manage reconcile-servers --fix-all
790

791
Please see ``snf-manage reconcile-servers --help`` for all the details.
792

793
Reconciling Networks
Christos Stavrakakis's avatar
Christos Stavrakakis committed
794
````````````````````
795

796
Reconciliation of Networks detects the following conditions:
797

798
799
800
801
802
803
  * Stale DB networks without corresponding Ganeti networks
  * Orphan Ganeti networks, without corresponding DB entries
  * Private networks that are not created to all Ganeti backends
  * Unsynchronized IP pools

To detect all inconsistencies you can just run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
804

805
.. code-block:: console
806

807
808
809
  $ snf-manage reconcile-networks

Adding the `--fix-all` option, will do the actual synchronization:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
810

811
.. code-block:: console
812

813
814
815
  $ snf-manage reconcile-networks --fix-all

Please see ``snf-manage reconcile-networks --help`` for all the details.
816
817


818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
Cyclades internals
------------------

Asynchronous communication with Ganeti backends
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Synnefo uses Google Ganeti backends for VM cluster management. In order for
Cyclades to be able to handle thousands of user requests, Cyclades and Ganeti
communicate asynchronously. Briefly, requests are submitted to Ganeti through
Ganeti's RAPI/HTTP interface, and then asynchronous notifications about the
progress of Ganeti jobs are being created and pushed upwards to Cyclades. The
architecture and communication with a Ganeti backend is shown in the graph
below:

.. image:: images/cyclades-ganeti-communication.png
   :width: 50%
   :target: _images/cyclades-ganeti-communication.png

The Cyclades API server is responsible for handling user requests. Read-only
requests are directly served by looking up the Cyclades DB. If the request
needs an action in the Ganeti backend, Cyclades submit jobs to the Ganeti
master using the `Ganeti RAPI interface
<http://docs.ganeti.org/ganeti/2.2/html/rapi.html>`_.

While Ganeti executes the job, `snf-ganeti-eventd`, `snf-ganeti-hook` and
`snf-progress-monitor` are monitoring the progress of the job and send
corresponding messages to the RabbitMQ servers. These components are part
of `snf-cyclades-gtools` and must be installed on all Ganeti nodes. Specially:

* *snf-ganeti-eventd* sends messages about operations affecting the operating
  state of instances and networks. Works by monitoring the Ganeti job queue.
* *snf-ganeti_hook* sends messages about the NICs of instances. It includes a
  number of `Ganeti hooks <http://docs.ganeti.org/ganeti/2.2/html/hooks.html>`_
  for customisation of operations.
* *snf-progress_monitor* sends messages about the progress of the Image deployment
  phase which is done by the Ganeti OS Definition `snf-image`.

Finally, `snf-dispatcher` consumes messages from the RabbitMQ queues, processes
these messages and properly updates the state of the Cyclades DB. Subsequent
requests to the Cyclades API, will retrieve the updated state from the DB.


859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
Synnefo management commands ("snf-manage")
==========================================

Each Synnefo service, Astakos, Pithos and Cyclades are controlled by the
administrator using the "snf-manage" admin tool. This tool is an extension of
the Django command-line management utility. It is run on the host that runs
each service and provides different types of commands depending the services
running on the host. If you are running more than one service on the same host
"snf-manage" adds all the corresponding commands for each service dynamically,
providing a unified admin environment.

To run "snf-manage" you just type:

.. code-block:: console

   # snf-manage <command> [arguments]

on the corresponding host that runs the service. For example, if you have all
services running on different physical hosts you would do:

.. code-block:: console

   root@astakos-host # snf-manage <astakos-command> [argument]
   root@pithos-host # snf-manage <pithos-command> [argument]
   root@cyclades-host # snf-manage <cyclades-command> [argument]

If you have all services running on the same host you would do:

.. code-block:: console

   root@synnefo-host # snf-manage <{astakos,pithos,cyclades}-command> [argument]

Note that you cannot execute a service's command on a host that is not running
this service. For example, the following will return an error if Astakos and
Cyclades are installed on different physical hosts:

.. code-block:: console

   root@astakos-host # snf-manage <cyclades-command> [argument]
   Unknown command: 'cyclades-command'
   Type 'snf-manage help' for usage.

This is the complete list of "snf-manage" commands for each service.

Astakos snf-manage commands
---------------------------

============================  ===========================
Name                          Description
============================  ===========================
fix-superusers                Transform superusers created by syncdb into AstakosUser instances
910
cleanup-full                  Cleanup sessions and session catalog
911
912
commission-list               List pending commissions
commission-show               Show details for a pending commission
913
914
915
component-add                 Register a component
component-list                List components
component-modify              Modify component attributes
916
component-show                Show component details
917
918
919
920
921
922
project-control               Manage projects and applications
project-list                  List projects
project-show                  Show project details
quota                         List and check the integrity of user quota
reconcile-resources-astakos   Reconcile resource usage of Quotaholder with Astakos DB
resource-list                 List resources
923
resource-modify               Modify a resource's default base quota and boolean flags
924
service-export-astakos        Export Astakos services and resources in JSON format
925
service-import                Register services
926
service-list                  List services
927
service-show                  Show service details
928
929
930
term-add                      Add approval terms
user-activation-send          Send user activation
user-add                      Add user
931
932
933
934
935
936
937
authpolicy-add                Create a new authentication provider policy profile
authpolicy-list               List existing authentication provider policy profiles
authpolicy-remove             Remove an authentication provider policy
authpolicy-set                Assign an existing authentication provider policy profile to a user or group
authpolicy-show               Show authentication provider profile details
group-add                     Create a group with the given name
group-list                    List available groups
938
939
940
941
942
943
944
945
946
947
948
user-list                     List users
user-modify                   Modify user
user-show                     Show user details
============================  ===========================

Pithos snf-manage commands
--------------------------

============================  ===========================
Name                          Description
============================  ===========================
949
reconcile-commissions-pithos  Display unresolved commissions and trigger their recovery
950
service-export-pithos         Export Pithos services and resources in JSON format
951
reconcile-resources-pithos    Detect unsynchronized usage between Astakos and Pithos DB resources and synchronize them if specified so.
952
953
954
955
956
============================  ===========================

Cyclades snf-manage commands
----------------------------

957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
============================== ===========================
Name                           Description
============================== ===========================
backend-add                    Add a new Ganeti backend
backend-list                   List backends
backend-modify                 Modify a backend
backend-update-status          Update backend statistics for instance allocation
backend-remove                 Remove a Ganeti backend
server-create                  Create a new server
server-show                    Show server details
server-list                    List servers
server-modify                  Modify a server
server-import                  Import an existing Ganeti VM into synnefo
server-inspect                 Inspect a server in DB and Ganeti
network-create                 Create a new network
network-list                   List networks
network-modify                 Modify a network
network-inspect                Inspect network state in DB and Ganeti
network-remove                 Delete a network
flavor-create                  Create a new flavor
flavor-list                    List flavors
flavor-modify                  Modify a flavor
image-list                     List images
image-show                     Show image details
pool-create                    Create a bridge or mac-prefix pool
pool-show                      Show pool details
pool-list                      List pools
pool-modify                    Modify a pool
pool-remove                    Delete a pool
986
987
988
989
990
991
992
993
994
port-create                    Create a port connecting a server to a network
port-inspect                   Inspect the state of a port in DB and Ganeti
port-list                      List ports
port-remove                    Delete a port
floating-ip-create             Create a new floating IP
floating-ip-attach             Attach a floating IP to a server
floating-ip-dettach            Dettach a flotaing IP from a server
floating-ip-list               List floating IPs
floating-ip-remove             Delete a floating IP
995
996
queue-inspect                  Inspect the messages of a RabbitMQ queue
queue-retry                    Resend messages from Dead Letter queues to original exchanges
997
service-export-cyclades        Export Cyclades services and resources in JSON format
998
999
1000
1001
subnet-create                  Create a subnet
subnet-inspect                 Inspect a subnet in DB
subnet-list                    List subnets
subnet-modify                  Modify a subnet
1002
1003
1004
1005
1006
1007
reconcile-servers              Reconcile servers of Synnefo DB with state of Ganeti backend
reconcile-networks             Reconcile networks of Synnefo DB with state of Ganeti backend
reconcile-pools                Check consistency of pool resources
reconcile-commissions-cyclades Detect and resolve pending commissions to Quotaholder
reconcile-resources-cyclades   Reconcile resource usage of Astakos with Cyclades DB.
============================== ===========================
1008

1009

1010
1011
1012
1013
1014
1015
1016
1017
Astakos helper scripts
======================

Astakos includes two scripts to facilitate the installation procedure.
Running:

.. code-block:: console

1018
   snf-component-register [<component_name>]
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029

automates the registration of the standard Synnefo components (astakos,
cyclades, and pithos) in astakos database. It internally uses the script:

.. code-block:: console

   snf-service-export <component_name> <base_url>

which simulates the export of service and resource definitions of the
standard Synnefo components.

1030

1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
Pithos managing accounts
========================

Pithos provides a utility tool for managing accounts.
To run you just type:

.. code-block:: console

   # pithos-manage-accounts <command> [arguments]

This is the list of the available commands:

============================  ===========================
Name                          Description
============================  ===========================
delete                        Remove an account from the Pithos DB
export-quota                  Export account quota in a file
list                          List existing/dublicate accounts
merge                         Move an account contents in another account
set-container-quota           Set container quota for all or a specific account
============================  ===========================

1053

1054
The "kamaki" API client
1055
=======================
1056
1057
1058

To upload, register or modify an image you will need the **kamaki** tool.
Before proceeding make sure that it is configured properly. Verify that
1059
*image.url*, *file.url*, *user.url* and *token* are set as needed:
1060
1061
1062
1063
1064

.. code-block:: console

   $ kamaki config list

1065
To change a setting use ``kamaki config set``:
1066
1067
1068

.. code-block:: console

1069
   $ kamaki config set image.url https://cyclades.example.com/image
1070
1071
   $ kamaki config set file.url https://pithos.example.com/v1
   $ kamaki config set user.url https://accounts.example.com
1072
1073
   $ kamaki config set token ...

1074
1075
To test that everything works, try authenticating the current account with
kamaki:
1076
1077
1078
1079
1080

.. code-block:: console

  $ kamaki user authenticate

1081
This will output user information.
1082

1083
1084
1085
Upload Image
------------

1086
1087
By convention, images are stored in a container called ``images``. Check if the
container exists, by listing all containers in your account:
1088
1089
1090

.. code-block:: console

1091
   $ kamaki file list
1092

1093
If the container ``images`` does not exist, create it:
1094
1095
1096

.. code-block:: console

1097
  $ kamaki file create images
1098

1099
You are now ready to upload an image to container ``images``. You can upload it
1100
with a Pithos client, or use kamaki directly:
1101
1102
1103

.. code-block:: console

1104
   $ kamaki file upload ubuntu.iso images
1105

1106
You can use any Pithos client to verify that the image was uploaded correctly,
1107
or you can list the contents of the container with kamaki:
1108

1109
1110
1111
.. code-block:: console

  $ kamaki file list images
1112
1113

The full Pithos URL for the previous example will be
1114
1115
``pithos://u53r-un1qu3-1d/images/ubuntu.iso`` where ``u53r-un1qu3-1d`` is the
unique user id (uuid).
1116
1117
1118
1119

Register Image
--------------

1120
To register an image you will need to use the full Pithos URL. To register as
1121
1122
1123
1124
a public image the one from the previous example use:

.. code-block:: console

1125
   $ kamaki image register Ubuntu pithos://u53r-un1qu3-1d/images/ubuntu.iso --public
1126
1127

The ``--public`` flag is important, if missing the registered image will not
1128
be listed by ``kamaki image list``.
1129

1130
Use ``kamaki image register`` with no arguments to see a list of available
1131
1132
1133
1134
options. A more complete example would be the following:

.. code-block:: console

1135
   $ kamaki image register Ubuntu pithos://u53r-un1qu3-1d/images/ubuntu.iso \
1136
1137
1138
1139
1140
1141
            --public --disk-format diskdump --property kernel=3.1.2

To verify that the image was registered successfully use:

.. code-block:: console

1142
   $ kamaki image list --name-like=ubuntu
1143

1144
1145
1146
1147

Miscellaneous
=============

1148
.. _branding:
Olga Brani's avatar
Olga Brani committed
1149

1150
Branding
1151
--------
Olga Brani's avatar
Olga Brani committed
1152

1153
1154
1155
1156
Since Synnefo v0.14, you are able to adapt the Astakos, Pithos and Cyclades Web
UI to your company’s visual identity. This is possible using the snf-branding
component, which is automatically installed on the nodes running the API
servers for Astakos, Pithos and Cyclades. 
Olga Brani's avatar
Olga Brani committed
1157
1158
1159
1160

Configuration
~~~~~~~~~~~~~

1161
1162
1163
1164
This can be done by modifing the settings provided by the snf-branding component
to match your service identity. The settings for the snf-branding application
can be found inside the configuration file ``/etc/synnefo/15-snf-branding.conf``
on the nodes that have Astakos, Pithos and Cyclades installed.
Olga Brani's avatar
Olga Brani committed
1165

1166
1167
1168
By default, the global service name is "Synnefo" and the company name is
"GRNET". These names and their respective logos and URLs are used throughout
the Astakos, Pithos and Cyclades UI.
Olga Brani's avatar
Olga Brani committed
1169

1170
**Names and URLs:**
Olga Brani's avatar
Olga Brani committed
1171

1172
1173
The first group of branding customization refers to the service's and company's
information.
Olga Brani's avatar
Olga Brani committed
1174

1175
1176
You can overwrite the company and the service name and URL respectively by
uncommenting and setting the following:
Olga Brani's avatar
Olga Brani committed
1177

1178
1179
1180
1181
1182
.. code-block:: python
  
  # setting used in Astakos Dashboard/Projects pages
  BRANDING_SERVICE_NAME = 'My cloud'
  BRANDING_SERVICE_URL = 'http://www.mycloud.synnefo.org/'
Olga Brani's avatar
Olga Brani committed
1183

1184
1185
1186
1187
1188
  # settings used in Astakos, Pithos, Cyclades footer only if 
  # BRANDING_SHOW_COPYRIGHT is set to True
  BRANDING_SHOW_COPYRIGHT = True
  BRANDING_COMPANY_NAME = 'Company LTD'
  BRANDING_COMPANY_URL = 'https://www.company-ltd.synnefo.org/'
Olga Brani's avatar
Olga Brani committed
1189
1190


Olga Brani's avatar
Olga Brani committed
1191
**Copyright and footer options:**
Olga Brani's avatar
Olga Brani committed
1192

1193
1194
By default, no Copyright message is shown in the UI footer. If you want to make
it visible in the footer of Astakos, Pithos and Cyclades UI, you can uncomment
1195
and set to ``True`` the ``BRANDING_SHOW_COPYRIGHT`` setting:
Olga Brani's avatar
Olga Brani committed
1196

1197
.. code-block:: python
Olga Brani's avatar
Olga Brani committed
1198
1199
1200

  #BRANDING_SHOW_COPYRIGHT = False

1201
Copyright message defaults to 'Copyright (c) 2011-<current_year>
1202
1203
<BRANDING_COMPANY_NAME>.' but you can overwrite it to a completely custom one by
setting the following option:
Olga Brani's avatar
Olga Brani committed
1204

1205
1206
1207
.. code-block:: python

  BRANDING_COPYRIGHT_MESSAGE = 'Copyright (c) 2011-2013 GRNET'
Olga Brani's avatar
Olga Brani committed
1208

Olga Brani's avatar
Olga Brani committed
1209
1210
1211
1212
1213
1214
1215
1216
1217
If you want to include a custom message in the footer, you can uncomment and 
set the ``BRANDING_FOOTER_EXTRA_MESSAGE`` setting. You can use html markup. 
Your custom message will appear  above Copyright message at the Compute 
templates and the Dashboard UI.

.. code-block:: python

  #BRANDING_FOOTER_EXTRA_MESSAGE = ''

Olga Brani's avatar
Olga Brani committed
1218
1219
1220

**Images:**

1221
1222
The Astakos, Pithos and Cyclades Web UI has some logos and images.
 
Olga Brani's avatar
Olga Brani committed
1223
1224
1225
1226
1227
1228
1229
1230
The branding-related images are presented in  the following table:

===============  ============================  =========
Image            Name/extension  convention    Usage
===============  ============================  =========
Favicon          favicon.ico                   Favicon for all services
Dashboard logo   dashboard_logo.png            Visible in all Astakos UI pages
Compute logo     compute_logo.png              Visible in all Cyclades UI pages
1231
1232
Console logo     console_logo.png              Visible in the Cyclades Console Window
Storage logo     storage_logo.png              Visible in all Pithos UI pages
Olga Brani's avatar
Olga Brani committed
1233
1234
===============  ============================  =========

1235
There are two methods  available for replacing all, or individual, 
1236
branding-related images:
Olga Brani's avatar
Olga Brani committed
1237

1238
1239
1. Create a new directory inside ``/usr/share/synnefo/static/`` (e.g.
   ``mybranding``) and place there some or all of your images.
Olga Brani's avatar
Olga Brani committed
1240

1241
   If you want to replace all of your images, keep the name/extension
1242
1243
   conventions as indicated in the above table and change the
   ``BRANDING_IMAGE_MEDIA_URL`` setting accordingly:
Olga Brani's avatar
Olga Brani committed
1244

1245
1246
1247
1248
   .. code-block:: python
        
      # using relative path
      BRANDING_IMAGE_MEDIA_URL= '/static/mybranding/images/' 
Olga Brani's avatar
Olga Brani committed
1249

1250
1251
      # or if you already host them in a separate domain (e.g. cdn)
      BRANDING_IMAGE_MEDIA_URL= 'https://cdn.synnefo.org/branding/images/'
Olga Brani's avatar
Olga Brani committed
1252
1253


1254
1255
1256
1257
   If you wish to replace individual images, **do not uncomment**
   ``BRANDING_IMAGE_MEDIA_URL``, but instead provide a relative path, pointing to
   the file inside your directory for each ``BRANDING_<image>_URL`` that you wish
   to replace.
Olga Brani's avatar
Olga Brani committed
1258

1259
2. Upload some or all of your images to a server and replace each 
1260
   ``BRANDING_<image>_URL`` with the absolute url of the image (i.e.
1261
   ``BRANDING_DASHBOARD_URL = 'https://www.synnefo.com/images/my_dashboard.jpg'``).
Olga Brani's avatar
Olga Brani committed
1262

1263
   Note that the alternative text  for each image tag inside html documents is 
1264
   alt=“BRANDING_SERVICE_NAME {Dashboard, Compute. Console, Storage}” respectively.
Olga Brani's avatar
Olga Brani committed
1265

1266
.. note:: Retina optimized images:
Olga Brani's avatar
Olga Brani committed
1267

1268
1269
   Synnefo UI is optimized for Retina displays. As far as images are concerned,  
   `retina.js <http://retinajs.com/>`_ is used.
Olga Brani's avatar
Olga Brani committed
1270

1271
   Retina.js checks each image on a page to see if there is a high-resolution 
1272
1273
   version of that image on your server. If a high-resolution variant exists, 
   the script will swap in that image in-place.
Olga Brani's avatar
Olga Brani committed
1274

1275
1276
1277
1278
1279
   The script assumes you use  `Apple's prescribed high-resolution modifier (@2x)
   <http://developer.apple.com/library/ios/#documentation/2DDrawing/Conceptual/
   DrawingPrintingiOS/SupportingHiResScreensInViews/SupportingHiResScreensInViews
   .html#//apple_ref/doc/uid/TP40010156-CH15-SW1>`_ to denote high-resolution 
   image variants on your server.
Olga Brani's avatar
Olga Brani committed
1280

1281
1282
1283
1284
1285
   For each of the images that you wish the script to  replace, you must have a 
   high-resolution variant in the same folder  named correctly and it will be 
   detected automatically. For example if your image is in <my_directory> and is 
   named "my_image.jpg" the script will look in the same directory for an image 
   named "my_image@2x.jpg".
Olga Brani's avatar
Olga Brani committed
1286

1287
1288
   In case that you don’t want to use a high-resolution image, the 
   normal-resolution image will be visible.
Olga Brani's avatar
Olga Brani committed
1289

1290
1291
More branding
~~~~~~~~~~~~~
Olga Brani's avatar
Olga Brani committed
1292

1293
1294
Although, it is not 100% branding-related, further verbal customization is
feasible. 
Olga Brani's avatar
Olga Brani committed
1295
1296
1297

**EMAILS**

1298
1299
1300
The output of all email `*`.txt files will be already customized to contain your
company and service names but you can further alter their content if you feel it
best fits your needs as simple as creasynnefo template.    
1301

1302
1303
In order to overwrite one or more email-templates you need to place your 
modified <email-file>.txt files respecting the following structure:
1304
  
1305
1306
  **/etc/synnefo/templates/**
      **im/**
1307
1308
1309
1310
1311
          | activation_email.txt
          | email.txt
          | invitation.txt
          | switch_accounts_email.txt
          | welcome_email.txt
1312
          **projects/**
1313
1314
1315
1316
1317
1318
1319
1320
              | project_approval_notification.txt
              | project_denial_notification.txt    
              | project_membership_change_notification.txt
              | project_membership_enroll_notification.txt
              | project_membership_leave_request_notification.txt
              | project_membership_request_notification.txt
              | project_suspension_notification.txt
              | project_termination_notification.txt
1321
      **registration/**
1322
1323
1324
1325
1326
          | email_change_email.txt
          | password_email.txt

Feel free to omit any of the above files you do not wish to overwrite.

Olga Brani's avatar
Olga Brani committed
1327
1328
1329
Below is a list of all emails sent by Synnefo to users along with a short 
description and a link to their content:

1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
* ``snf-astakos-app/astakos/im/templates/im/email.txt``
  Base email template. Contains a contact email and a “thank you” message.
  (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/activation_email.txt`` Email sent to
  user that prompts  him/her to click on a link provided to activate the account.
  Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/activation_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/invitation.txt`` Email sent to an
  invited user. He/she has to click on a link provided to activate the account.
  Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/invitation.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/switch_accounts_email.txt`` Email
  sent to user upon his/her request to associate this email address with a
  shibboleth account. He/she has to click on a link provided to activate the
  association. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/switch_accounts_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/welcome_email.txt`` Email sent to
  inform the user that his/ her account has been activated. Extends “email.txt”
  (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/welcome_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/registration/email_change_email.txt``
  Email sent to user when he/she has requested new email address assignment. The
  user has to click on a link provided to validate this action. Extends
  “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/registration/email_change_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/registration/password_email.txt`` Email
  sent for resetting password purpose. The user has to click on a link provided
  to validate this action. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/registration/password_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_approval_notification.txt``
  Informs  the project owner that his/her project has been approved. Extends
  “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_approval_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_denial_notification.txt``
  Informs the project owner that his/her  project application has been denied
  explaining the reasons. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_denial_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_change_notification.txt``
  An email is sent to a user containing information about his project membership
  (whether he has been accepted, rejected or removed). Extends “email.txt” (`Link
  <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_change_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_enroll_notification.txt``
  Informs a user that he/she  has been enrolled to a project. Extends
  “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_enroll_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_leave_request_notification.txt``
  An email is sent to the project owner to make him aware of a  user having
  requested to leave his project. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_leave_request_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_request_notification.txt``
  An email is sent to the project owner to make him/her aware of a user having
  requested to join  his project. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_request_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_suspension_notification.txt``
  An email is sent to the project owner to make him/her aware of his/her project
  having been suspended. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_suspension_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_termination_notification.txt``
  An email is sent to the project owner to make him/her aware of his/her project
  having been terminated. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_termination_notification.txt>`_)
Olga Brani's avatar
Olga Brani committed
1378
1379
1380
1381

.. warning:: Django templates language:

  If you choose to  overwrite these email templates, be mindful of the necessary 
1382
1383
1384
  information contained in django template variables that must not be omitted, 
  such as the activation link for activating one’s account and many more. 
  These variables are contained into {{}} inside the templates.
Olga Brani's avatar
Olga Brani committed
1385
1386


1387
.. RabbitMQ
1388

1389
1390
1391
1392
1393
1394
RabbitMQ Broker
---------------

Queue nodes run the RabbitMQ sofware, which provides AMQP functionality. To
guarantee high-availability, more than one Queue nodes should be deployed, each
of them belonging to the same `RabbitMQ cluster
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
<http://www.rabbitmq.com/clustering.html>`_. Synnefo uses the RabbitMQ
active/active `High Available Queues <http://www.rabbitmq.com/ha.html>`_ which
are mirrored between two nodes within a RabbitMQ cluster.

The RabbitMQ nodes that form the cluster, are declared to Synnefo through the
`AMQP_HOSTS` setting. Each time a Synnefo component needs to connect to
RabbitMQ, one of these nodes is chosen in a random way. The client that Synnefo
uses to connect to RabbitMQ, handles connection failures transparently and
tries to reconnect to a different node. As long as one of these nodes are up
and running, functionality of Synnefo should not be downgraded by the RabbitMQ
node failures.
1406
1407

All the queues that are being used are declared as durable, meaning that
1408
1409
messages are persistently stored to RabbitMQ, until they get successfully
processed by a client.
1410
1411
1412

Currently, RabbitMQ is used by the following components:

1413
* `snf-ganeti-eventd`, `snf-ganeti-hook` and `snf-progress-monitor`:
1414
1415
  These components send messages concerning the status and progress of
  jobs in the Ganeti backend.
1416
1417
* `snf-dispatcher`: This daemon, consumes the messages that are sent from
  the above components, and updates the Cyclades DB accordingly.
1418

Christos Stavrakakis's avatar
Christos Stavrakakis committed
1419

1420
Installation
1421
1422
~~~~~~~~~~~~

1423
1424
1425
1426
1427
1428
Please check the RabbitMQ documentation which covers extensively the
`installation of RabbitMQ server <http://www.rabbitmq.com/download.html>`_ and
the setup of a `RabbitMQ cluster <http://www.rabbitmq.com/clustering.html>`_.
Also, check out the `web management plugin
<http://www.rabbitmq.com/management.html>`_ that can be useful for managing and
monitoring RabbitMQ.
1429
1430

For a basic installation of RabbitMQ on two nodes (node1 and node2) you can do
1431
the following:
1432

1433
On both nodes, install rabbitmq-server and create a Synnefo user:
1434
1435
1436
1437
1438
1439
1440

.. code-block:: console

  $ apt-get install rabbitmq-server
  $ rabbitmqctl add_user synnefo "example_pass"
  $ rabbitmqctl set_permissions synnefo  ".*" ".*" ".*"

1441
Also guarantee that both nodes share the same cookie, by running:
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469

.. code-block:: console

  $ scp node1:/var/lib/rabbitmq/.erlang.cookie node2:/var/lib/rabbitmq/.erlang.cookie

and restart the nodes:

.. code-block:: console

  $ /etc/init.d/rabbitmq-server restart


To setup the RabbitMQ cluster run:

.. code-block:: console

  root@node2: rabbitmqctl stop_app
  root@node2: rabbitmqctl reset
  root@node2: rabbitmqctl cluster rabbit@node1 rabbit@node2
  root@node2: rabbitmqctl start_app

You can verify that the cluster is set up correctly by running:

.. code-block:: console

  root@node2: rabbitmqctl cluster_status


1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
Logging
-------

Logging in Synnefo is using Python's logging module. The module is configured
using dictionary configuration, whose format is described here:

http://docs.python.org/release/2.7.1/library/logging.html#logging-config-dictschema

Note that this is a feature of Python 2.7 that we have backported for use in
Python 2.6.

1481
The logging configuration dictionary is defined in
1482
``/etc/synnefo/10-snf-webproject-logging.conf``
1483

Constantinos Venetsanopoulos's avatar