admin-guide.rst 66.2 KB
Newer Older
1
2
3
4
5
6
7
8
.. _admin-guide:

Synnefo Administrator's Guide
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is the complete Synnefo Administrator's Guide.


9
.. _syn+archip:
10

11
12
13
General Synnefo Architecture
============================

14
15
16
17
The following figure shows a detailed view of the whole Synnefo architecture
and how it interacts with multiple Ganeti clusters. We hope that after reading
the Administrator's Guide you will be able to understand every component and
all the interactions between them.
18

19
.. image:: images/synnefo-arch2.png
20
   :width: 100%
21
   :target: _images/synnefo-arch2.png
22

23
24
25
Synnefo also supports RADOS as an alternative storage backend for
Files/Images/VM disks. You will find the :ref:`corresponding figure
<syn+archip+rados>` later in this guide.
26

27

28
29
Identity Service (Astakos)
==========================
30
31


32
Authentication methods
33
----------------------
34

35
Astakos supports multiple authentication methods:
36

37
38
39
40
41
42
 * local username/password
 * LDAP / Active Directory
 * SAML 2.0 (Shibboleth) federated logins
 * Google
 * Twitter
 * LinkedIn
43
44
45
46

.. _shibboleth-auth:

Shibboleth Authentication
47
~~~~~~~~~~~~~~~~~~~~~~~~~
48
49
50
51
52
53
54
55
56
57
58
59
60
61

Astakos can delegate user authentication to a Shibboleth federation.

To setup shibboleth, install package::

  apt-get install libapache2-mod-shib2

Change appropriately the configuration files in ``/etc/shibboleth``.

Add in ``/etc/apache2/sites-available/synnefo-ssl``::

  ShibConfig /etc/shibboleth/shibboleth2.xml
  Alias      /shibboleth-sp /usr/share/shibboleth

62
  <Location /ui/login/shibboleth>
63
64
65
66
67
68
69
70
71
72
73
74
75
    AuthType shibboleth
    ShibRequireSession On
    ShibUseHeaders On
    require valid-user
  </Location>

and before the line containing::

  ProxyPass        / http://localhost:8080/ retry=0

add::

  ProxyPass /Shibboleth.sso !
76

77
78
79
Then, enable the shibboleth module::

  a2enmod shib2
80

81
82
83
84
85
86
87
88
89
90
91
92
93
94
After passing through the apache module, the following tokens should be
available at the destination::

  eppn # eduPersonPrincipalName
  Shib-InetOrgPerson-givenName
  Shib-Person-surname
  Shib-Person-commonName
  Shib-InetOrgPerson-displayName
  Shib-EP-Affiliation
  Shib-Session-ID

Finally, add 'shibboleth' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
95
Twitter Authentication
96
~~~~~~~~~~~~~~~~~~~~~~
Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
97
98
99
100
101
102
103
104
105
106
107
108
109
110

To enable twitter authentication while signed in under a Twitter account,
visit dev.twitter.com/apps.

Click Create an application.

Fill the necessary information and for callback URL give::

    https://node1.example.com/ui/login/twitter/authenticated

Finally, add 'twitter' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Google Authentication
111
~~~~~~~~~~~~~~~~~~~~~
Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127

To enable google authentication while signed in under a Google account,
visit https://code.google.com/apis/console/.

Under API Access select Create another client ID, select Web application,
expand more options in Your site or hostname section and in Authorized
Redirect URIs add:


Fill the necessary information and for callback URL give::

    https://node1.example.com/ui/login/google/authenticated

Finally, add 'google' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

128
129
130
131

Working with Astakos
--------------------

132
133
User registration
~~~~~~~~~~~~~~~~~
134

135
136
When a new user signs up, he/she is not directly marked as active. You can see 
his/her state by running (on the machine that runs the Astakos app):
137
138
139

.. code-block:: console

140
   $ snf-manage user-list
141

142
143
More detailed user status is provided in the `status` field of the `user-show` 
command:
144

145
.. code-block:: console
146

147
  $ snf-manage user-show <user-id>
148

149
150
151
152
153
  id                  : 6
  uuid                : 78661411-5eed-412f-a9ea-2de24f542c2e
  status              : Accepted/Active (accepted policy: manual)
  email               : user@synnefo.org
  ....
154

155
156
157
Based on the `astakos-app` configuration, there are several ways for a user to
get verified and activated in order to be able to login. We discuss the user
verification and activation flow in the following section.
158
159

User activation flow
160
161
````````````````````

162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
A user can register for an account using the astakos signup form. Once the form
is submited successfully a user entry is created in astakos database. That entry
is passed through the astakos activation backend which handles whether the user
should be automatically verified and activated.

Email verification
``````````````````

The verification process takes place in order to ensure that the user owns the
email provided during the signup process. By default, after each successful
signup astakos notifies user with an verification url via email. 

At this stage:

    * subsequent registrations invalidate and delete the previous registrations 
      of the same email address.

    * in case user misses the initial notification, additional emails can be
      send either via the url which is prompted to the user if he tries to
      login, or by the administrator using the ``snf-manage user-activation-send
      <userid>`` command.

    * administrator may also enforce a user to get verified using the
      ``snf-manage user-modify --verify <userid>`` command.

Account activation
``````````````````

190
Once the user gets verified, it is time for Astakos to decide whether or not to
191
192
193
proceed through user activation process. If ``ASTAKOS_MODERATION_ENABLED``
setting is set to ``False`` (default value) user gets activated automatically. 

194
In case the moderation is enabled Astakos may still automatically activate the
195
196
197
198
199
200
201
202
user in the following cases:

    * User email matches any of the regular expressions defined in
      ``ASTAKOS_RE_USER_EMAIL_PATTERNS`` (defaults to ``[]``)
    * User used a signup method (e.g. ``shibboleth``) for which automatic
      activation is enabled (see 
      :ref:`authentication methods policies <auth_methods_policies>`).

203
204
205
206
207
208
If all of the above fail to trigger automatic activation, an email is sent to
the persons listed in ``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings,
notifing that there is a new user pending for moderation and that it's up to
the administrator to decide if the user should be activated. The UI also shows
a corresponding 'pending moderation' message to the user. The administrator can
activate a user using the ``snf-manage user-modify`` command:
209
210
211
212
213
214
215
216
217

.. code-block:: console

    # command to activate a pending user
    $ snf-manage user-modify --accept <userid>

    # command to reject a pending user
    $ snf-manage user-modify --reject --reject-reason="spammer" <userid>

218
219
220
221
Once the activation process finishes, a greeting message is sent to the user
email address and a notification for the activation to the persons listed in
``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings. Once activated the user is
able to login and access the Synnefo services.
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282

Additional authentication methods
`````````````````````````````````

Astakos supports third party logins from external identity providers. This
can be usefull since it allows users to use their existing credentials to 
login to astakos service.

Currently astakos supports the following identity providers:

    * `Shibboleth <http://www.internet2.edu/shibboleth>`_ (module name
      ``shibboleth``)
    * `Google <https://developers.google.com/accounts/docs/OAuth2>`_ (module
      name ``google``)
    * `Twitter <https://dev.twitter.com/docs/auth>`_ (module name ``twitter``)
    * `LinkedIn <http://developer.linkedin.com/documents/authentication>`_
      (module name ``linkedin``)

To enable any of the above modules (by default only ``local`` accounts are
allowed), retrieve and set the required provider settings and append the 
module name in ``ASTAKOS_IM_MODULES``.

.. code-block:: python

    # settings from https://code.google.com/apis/console/
    ASTAKOS_GOOGLE_CLIENT_ID = '1111111111-epi60tvimgha63qqnjo40cljkojcann3.apps.googleusercontent.com'
    ASTAKOS_GOOGLE_SECRET = 'tNDQqTDKlTf7_LaeUcWTWwZM'
    
    # let users signup and login using their google account
    ASTAKOS_IM_MODULES = ['local', 'google']


.. _auth_methods_policies:

Authentication method policies
``````````````````````````````

Astakos allows you to override the default policies for each enabled provider 
separately by adding the approriate settings in your ``.conf`` files in the 
following format:

**ASTAKOS_AUTH_PROVIDER_<module>_<policy>_POLICY**

Available policies are:

    * **CREATE** Users can signup using that provider (default: ``True``) 
    * **REMOVE/ADD** Users can remove/add login method from their profile 
      (default: ``True``)
    * **AUTOMODERATE** Automatically activate users that signup using that
      provider (default: ``False``)
    * **LOGIN** Whether or not users can use the provider to login (default:
      ``True``).

e.g. to enable automatic activation for your academic users, while keeping 
locally signed up users under moderation you can apply the following settings.

.. code-block:: python

    ASTAKOS_AUTH_PROVIDER_SHIBBOLETH_AUTOMODERATE_POLICY = True
    ASTAKOS_AUTH_PROVIDER_SHIBBOLETH_REMOVE_POLICY = False

283
284
285
286
287
288
289
290
291
292
293
294
295
296
User login
~~~~~~~~~~

During the logging procedure, the user is authenticated by the respective
identity provider.

If ``ASTAKOS_RECAPTCHA_ENABLED`` is set and the user fails several times
(``ASTAKOS_RATELIMIT_RETRIES_ALLOWED`` setting) to provide the correct
credentials for a local account, he/she is then prompted to solve a captcha
challenge.

Upon success, the system renews the token (if it has expired), logins the user
and sets the cookie, before redirecting the user to the ``next`` parameter
value.
297

298
299
300
Setting quota limits
~~~~~~~~~~~~~~~~~~~~

301
302
Set default quota
`````````````````
303
304
305
306

In 20-snf-astakos-app-settings.conf, 
uncomment the default setting ``ASTAKOS_SERVICES``
and customize the ``'uplimit'`` values.
307
These are the default base quota for all users.
308
309
310
311

To apply your configuration run::

    # snf-manage astakos-init --load-service-resources
312
    # snf-manage quota --sync
313

314
315
Set base quota for individual users
```````````````````````````````````
316

317
For individual users that need different quota than the default
318
319
you can set it for each resource like this::

320
321
    # use this to display quota / uuid
    # snf-manage user-show 'uuid or email' --quota
322

323
    # snf-manage user-modify 'user-uuid' --set-base-quota 'cyclades.vm' 10
324
325
326
327
328
329
330
331
332
333
334
335


Enable the Projects feature
~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you want to enable the projects feature so that users may apply
on their own for resources by creating and joining projects,
in ``20-snf-astakos-app-settings.conf`` set::

    # this will make the 'projects' page visible in the dashboard
    ASTAKOS_PROJECTS_VISIBLE = True

336
337
You can change the maximum allowed number of pending project applications
per user with::
338

339
340
341
342
    # snf-manage resource-modify astakos.pending_app --limit <number>

You can also set a user-specific limit with::

343
    # snf-manage user-modify 'user-uuid' --set-base-quota 'astakos.pending_app' 5
344

345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
When users apply for projects they are not automatically granted
the resources. They must first be approved by the administrator.

To list pending project applications in astakos::

    # snf-manage project-list --pending

Note the last column, the application id. To approve it::

    # <app id> from the last column of project-list
    # snf-manage project-control --approve <app id>

To deny an application::

    # snf-manage project-control --deny <app id>

361
362
363
364
365
366
Users designated as *project admins* can approve, deny, or modify
an application through the web interface. In
``20-snf-astakos-app-settings.conf`` set::

    # UUIDs of users that can approve or deny project applications from the web.
    ASTAKOS_PROJECT_ADMINS = [<uuid>, ...]
367
368


369
370
371
372
373
374
375
376
377
378
379
380
Astakos advanced operations
---------------------------

Adding "Terms of Use"
~~~~~~~~~~~~~~~~~~~~~

Astakos supports versioned terms-of-use. First of all you need to create an
html file that will contain your terms. For example, create the file
``/usr/share/synnefo/sample-terms.html``, which contains the following:

.. code-block:: console

381
   <h1>My cloud service terms</h1>
382

383
   These are the example terms for my cloud service
384
385
386
387
388
389
390
391
392
393

Then, add those terms-of-use with the snf-manage command:

.. code-block:: console

   $ snf-manage term-add /usr/share/synnefo/sample-terms.html

Your terms have been successfully added and you will see the corresponding link
appearing in the Astakos web pages' footer.

394
395
396
397
398
399
400
401
402
During the account registration, if there are approval terms, the user is
presented with an "I agree with the Terms" checkbox that needs to get checked
in order to proceed.

In case there are new approval terms that the user has not signed yet, the
``signed_terms_required`` view decorator redirects to the ``approval_terms``
view, so the user will be presented with the new terms the next time he/she
logins.

403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
Enabling reCAPTCHA
~~~~~~~~~~~~~~~~~~

Astakos supports the `reCAPTCHA <http://www.google.com/recaptcha>`_ feature.
If enabled, it protects the Astakos forms from bots. To enable the feature, go
to https://www.google.com/recaptcha/admin/create and create your own reCAPTCHA
key pair. Then edit ``/etc/synnefo/20-snf-astakos-app-settings.conf`` and set
the corresponding variables to reflect your newly created key pair. Finally, set
the ``ASTAKOS_RECAPTCHA_ENABLED`` variable to ``True``:

.. code-block:: console

   ASTAKOS_RECAPTCHA_PUBLIC_KEY = 'example_recaptcha_public_key!@#$%^&*('
   ASTAKOS_RECAPTCHA_PRIVATE_KEY = 'example_recaptcha_private_key!@#$%^&*('

   ASTAKOS_RECAPTCHA_ENABLED = True

Restart the service on the Astakos node(s) and you are ready:

.. code-block:: console

   # /etc/init.d/gunicorn restart

Checkout your new Sign up page. If you see the reCAPTCHA box, you have setup
everything correctly.

429

430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
Astakos internals
-----------------

X-Auth-Token
~~~~~~~~~~~~

Alice requests a specific resource from a cloud service e.g.: Pithos. In the
request she supplies the `X-Auth-Token` to identify whether she is eligible to
perform the specific task. The service contacts Astakos through its
``/account/v1.0/authenticate`` api call (see :ref:`authenticate-api-label`)
providing the specific ``X-Auth-Token``. Astakos checkes whether the token
belongs to an active user and it has not expired and returns a dictionary
containing user related information. Finally the service uses the ``uniq``
field included in the dictionary as the account string to identify the user
accessible resources.

.. _authentication-label:

Django Auth methods and Backends
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Astakos incorporates Django user authentication system and extends its User model.

Since username field of django User model has a limitation of 30 characters,
AstakosUser is **uniquely** identified by the ``email`` instead. Therefore,
``astakos.im.authentication_backends.EmailBackend`` is served to authenticate a
user using email if the first argument is actually an email, otherwise tries
the username.

A new AstakosUser instance is assigned with a uui as username and also with a
``auth_token`` used by the cloud services to authenticate the user.
``astakos.im.authentication_backends.TokenBackend`` is also specified in order
to authenticate the user using the email and the token fields.

Logged on users can perform a number of actions:

 * access and edit their profile via: ``/im/profile``.
 * change their password via: ``/im/password``
 * send feedback for grnet services via: ``/im/send_feedback``
 * logout (and delete cookie) via: ``/im/logout``

Internal Astakos requests are handled using cookie-based Django user sessions.

External systems should forward to the ``/login`` URI. The server,
depending on its configuration will redirect to the appropriate login page.
When done with logging in, the service's login URI should redirect to the URI
provided with next, adding user and token parameters, which contain the email
and token fields respectively.

The login URI accepts the following parameters:

======================  =========================
Request Parameter Name  Value
======================  =========================
next                    The URI to redirect to when the process is finished
renew                   Force token renewal (no value parameter)
force                   Force logout current user (no value parameter)
======================  =========================

External systems inside the ``ASTAKOS_COOKIE_DOMAIN`` scope can acquire the
user information by the cookie identified by ``ASTAKOS_COOKIE_NAME`` setting
(set during the login procedure).

Finally, backend systems having acquired a token can use the
:ref:`authenticate-api-label` API call from a private network or through HTTPS.


497
498
499
500
501
502
Compute/Network/Image Service (Cyclades)
========================================

Working with Cyclades
---------------------

503
Managing Ganeti Backends
Christos Stavrakakis's avatar
Christos Stavrakakis committed
504
~~~~~~~~~~~~~~~~~~~~~~~~
505

506
507
508
Since v0.11, Synnefo is able to manage multiple Ganeti clusters (backends)
making it capable to scale linearly to tens of thousands of VMs. Backends
can be dynamically added or removed via `snf-manage` commands.
509

510
511
512
513
514
Each newly created VM is allocated to a Ganeti backend by the Cyclades backend
allocator. The VM is "pinned" to this backend, and can not change through its
lifetime. The backend allocator decides in which backend to spawn the VM based
on the available resources of each backend, trying to balance the load between
them.
515
516

Handling of Networks, as far as backends are concerned, is based on whether the
517
518
519
520
network is public or not. Public networks are created through the `snf-manage
network-create` command, and are only created on one backend. Private networks
are created on all backends, in order to ensure that VMs residing on different
backends can be connected to the same private network.
521

522
523
524
Listing existing backends
`````````````````````````
To list all the Ganeti backends known to Synnefo, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
525

526
527
528
529
530
.. code-block:: console

   $ snf-manage backend-list

Adding a new Ganeti backend
Christos Stavrakakis's avatar
Christos Stavrakakis committed
531
```````````````````````````
532
533
Backends are dynamically added under the control of Synnefo with `snf-manage
backend-add` command. In this section it is assumed that a Ganeti cluster,
534
535
named ``cluster.example.com`` is already up and running and configured to be
able to host Synnefo VMs.
536

537
To add this Ganeti cluster, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
538

539
540
541
542
.. code-block:: console

   $ snf-manage backend-add --clustername=cluster.example.com --user="synnefo_user" --pass="synnefo_pass"

543
544
545
546
547
where ``clustername`` is the Cluster hostname of the Ganeti cluster, and
``user`` and ``pass`` are the credentials for the `Ganeti RAPI user
<http://docs.ganeti.org/ganeti/2.2/html/rapi.html#users-and-passwords>`_.  All
backend attributes can be also changed dynamically using the `snf-manage
backend-modify` command.
548

549
550
551
``snf-manage backend-add`` will also create all existing private networks to
the new backend. You can verify that the backend is added, by running
`snf-manage backend-list`.
552

553
554
555
556
557
558
559
560
561
562
Note that no VMs will be spawned to this backend, since by default it is in a
``drained`` state after addition and also it has no public network assigned to
it.

So, first you need to create its public network, make sure everything works as
expected and finally make it active by un-setting the ``drained`` flag. You can
do this by running:

.. code-block:: console

563
   $ snf-manage backend-modify --drained=False <backend_id>
564
565

Removing an existing Ganeti backend
Christos Stavrakakis's avatar
Christos Stavrakakis committed
566
```````````````````````````````````
567
In order to remove an existing backend from Synnefo, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
568

569
570
.. code-block:: console

571
   # snf-manage backend-remove <backend_id>
572

573
574
575
This command will fail if there are active VMs on the backend. Also, the
backend is not cleaned before removal, so all the Synnefo private networks
will be left on the Ganeti nodes. You need to remove them manually.
576

577
578
579
580
581
582
Allocation of VMs in Ganeti backends
````````````````````````````````````
As already mentioned, the Cyclades backend allocator is responsible for
allocating new VMs to backends. This allocator does not choose the exact Ganeti
node that will host the VM but just the Ganeti backend. The exact node is
chosen by the Ganeti cluster's allocator (hail).
583
584
585
586

The decision about which backend will host a VM is based on the available
resources. The allocator computes a score for each backend, that shows its load
factor, and the one with the minimum score is chosen. The admin can exclude
587
backends from the allocation phase by marking them as ``drained`` by running:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
588

589
590
.. code-block:: console

591
   $ snf-manage backend-modify --drained=True <backend_id>
592
593

The backend resources are periodically updated, at a period defined by
594
the ``BACKEND_REFRESH_MIN`` setting, or by running `snf-manage backend-update-status`
595
command. It is advised to have a cron job running this command at a smaller
596
interval than ``BACKEND_REFRESH_MIN`` in order to remove the load of refreshing
597
598
the backends stats from the VM creation phase.

599
600
601
Finally, the admin can decide to have a user's VMs being allocated to a
specific backend, with the ``BACKEND_PER_USER`` setting. This is a mapping
between users and backends. If the user is found in ``BACKEND_PER_USER``, then
602
603
Synnefo allocates all his/hers VMs to the specific backend in the variable,
even if is marked as drained (useful for testing).
604

605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
Allocation based on disk-templates
**********************************

Besides the available resources of each Ganeti backend, the allocator takes
into consideration the disk template of the instance when trying to allocate it
to a Ganeti backend. Specifically, the allocator checks if the flavor of the
instance belongs to the available disk templates of each Ganeti backend.

A Ganeti cluster has a list of enabled disk templates
(`--enabled-disk-templates`) and a list of allowed disk templates for new
instances (`--ipolicy-disk-templates`). See the `gnt-cluster` manpage for more
details about these options.

When Synnefo allocates an instance, it checks whether the disk template of the
new instance belongs both in the enabled and ipolicy disk templates. You can
see the list of the available disk-templates by running `snf-manage
backend-list`. This list should be updated automatically after changing
these options in Ganeti and it can also be updated by running `snf-manage
backend-update-status`.

So the administrator, can route instances on different backends based on their
flavor disk template, by modifying the enabled or ipolicy disk templates of
each backend.  Also, the administrator can route instances between different
nodes of the same Ganeti backend, by modifying the same options at the
nodegroup level (see `gnt-group` manpage for mor details).


632
633
634
Managing Virtual Machines
~~~~~~~~~~~~~~~~~~~~~~~~~

635
636
637
638
639
640
As mentioned, Cyclades uses Ganeti for management of VMs. The administrator can
handle Cyclades VMs just like any other Ganeti instance, via `gnt-instance`
commands. All Ganeti instances that belong to Synnefo, are separated from
others, by a prefix in their names. This prefix is defined in
``BACKEND_PREFIX_ID`` setting in
``/etc/synnefo/20-snf-cyclades-app-backend.conf``.
641

642
Apart from handling instances directly in the Ganeti level, a number of `snf-manage`
643
644
commands are available:

645
646
647
648
649
650
* ``snf-manage server-list``: List servers
* ``snf-manage server-show``: Show information about a server in the Cyclades DB
* ``snf-manage server-inspect``: Inspect the state of a server both in DB and Ganeti
* ``snf-manage server-modify``: Modify the state of a server in the Cycldes DB
* ``snf-manage server-create``: Create a new server
* ``snf-manage server-import``: Import an existing Ganeti instance to Cyclades
651
652
653
654
655


Managing Virtual Networks
~~~~~~~~~~~~~~~~~~~~~~~~~

656
657
658
659
Cyclades is able to create and manage Virtual Networks. Networking is
desployment specific and must be customized based on the specific needs of the
system administrator. For better understanding of networking please refer to
the :ref:`Network <networks>` section.
660
661
662

Exactly as Cyclades VMs can be handled like Ganeti instances, Cyclades Networks
can also by handled as Ganeti networks, via `gnt-network commands`. All Ganeti
663
networks that belong to Synnefo are named with the prefix
664
665
`${BACKEND_PREFIX_ID}-net-`.

666
There are also the following `snf-manage` commands for managing networks:
667

668
669
670
671
672
673
* ``snf-manage network-list``: List networks
* ``snf-manage network-show``: Show information about a network in the Cyclades DB
* ``snf-manage network-inspect``: Inspect the state of the network in DB and Ganeti backends
* ``snf-manage network-modify``: Modify the state of a network in the Cycldes DB
* ``snf-manage network-create``: Create a new network
* ``snf-manage network-remove``: Remove an existing network
674

675
Managing Network Resources
676
``````````````````````````
677

678
679
680
Proper operation of the Cyclades Network Service depends on the unique
assignment of specific resources to each type of virtual network. Specifically,
these resources are:
681

682
683
684
685
686
* IP addresses. Cyclades creates a Pool of IPs for each Network, and assigns a
  unique IP address to each VM, thus connecting it to this Network. You can see
  the IP pool of each network by running `snf-manage network-inspect
  <network_ID>`. IP pools are automatically created and managed by Cyclades,
  depending on the subnet of the Network.
687
* Bridges corresponding to physical VLANs, which are required for networks of
688
689
690
  type `PRIVATE_PHYSICAL_VLAN`.
* One Bridge corresponding to one physical VLAN which is required for networks of
  type `PRIVATE_MAC_PREFIX`.
691

692
693
694
695
696
697
698
699
700
701
702
IPv4 addresses
**************

An allocation pool of IPv4 addresses is automatically created for every network
that has the attribute `dhcp` set to True. The allocation pool contains the
range of IP addresses that are included in the subnet. The gateway and the
broadcast address of the network are excluded from the allocation pool. The
admin can externally reserve IP addresses to exclude them from automatic
allocation with the `--add-reserved-ips` option of `snf-manage network-modify`
command. For example the following command will reserve two IP addresses
from network with ID `42`:
703

704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
.. code-block:: console

 snf-manage network-modify --add-reserved-ips=10.0.0.21,10.0.0.22 42

.. warning:: Externally reserving IP addresses is also available at the Ganeti.
 However, when using Cyclades with multiple Ganeti backends, the handling of
 IP pools must be performed from Cyclades!

Bridges
*******

As already mentioned Cyclades use a pool of Bridges that must correspond
to Physical VLAN at the Ganeti level. A bridge from the pool is assigned to
each network of flavor `PHYSICAL_VLAN`. Creation of this pool is done
using `snf-manage pool-create` command. For example the following command
will create a pool containing the brdiges from `prv1` to `prv21`.
Christos Stavrakakis's avatar
Christos Stavrakakis committed
720

721
722
723
724
725
.. code-block:: console

   # snf-manage pool-create --type=bridge --base=prv --size=20

You can verify the creation of the pool, and check its contents by running:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
726

727
728
729
730
731
.. code-block:: console

   # snf-manage pool-list
   # snf-manage pool-show --type=bridge 1

732
733
734
735
736
737
738
739
740
Finally you can use the `pool-modify` management command in order to externally
reserve the values from pool, extend or shrink the pool if possible.

MAC Prefixes
************

Cyclades also use a pool of MAC prefixes to assign to networks of flavor
`MAC_FILTERED`. Handling of this pool is done exactly as with pool of bridges,
except that the type option must be set to mac-prefix:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
741

742
743
744
745
.. code-block:: console

   # snf-manage pool-create --type=mac-prefix --base=aa:00:0 --size=65536

746
747
748
749
750
751
752
753
754
755
756
The above command will create a pool of MAC prefixes from ``aa:00:1`` to
``b9:ff:f``. The MAC prefix pool is responsible for providing only unicast and
locally administered MAC addresses, so many of these prefixes will be
externally reserved, to exclude from allocation.

Pool reconciliation
*******************

The management command `snf-manage reconcile-pools` can be used that all the
above mentioned pools are consistent and that all values that come from the
pool are not used more than once.
757

758

759
760
761
762
763
Cyclades advanced operations
----------------------------

Reconciliation mechanism
~~~~~~~~~~~~~~~~~~~~~~~~
764

765
766
767
768
769
770
On certain occasions, such as a Ganeti or RabbitMQ failure, the state of
Cyclades database may differ from the real state of VMs and networks in the
Ganeti backends. The reconciliation process is designed to synchronize
the state of the Cyclades DB with Ganeti. There are two management commands
for reconciling VMs and Networks

771
772
773
Reconciling Virtual Machines
````````````````````````````

774
Reconciliation of VMs detects the following conditions:
775

776
777
 * Stale DB servers without corresponding Ganeti instances
 * Orphan Ganeti instances, without corresponding DB entries
778
 * Out-of-sync state for DB entries wrt to Ganeti instances
779

780
To detect all inconsistencies you can just run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
781

782
.. code-block:: console
783
784

  $ snf-manage reconcile-servers
785

786
Adding the `--fix-all` option, will do the actual synchronization:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
787

788
.. code-block:: console
789

790
  $ snf-manage reconcile-servers --fix-all
791

792
Please see ``snf-manage reconcile-servers --help`` for all the details.
793

794
Reconciling Networks
Christos Stavrakakis's avatar
Christos Stavrakakis committed
795
````````````````````
796

797
Reconciliation of Networks detects the following conditions:
798

799
800
801
802
803
804
  * Stale DB networks without corresponding Ganeti networks
  * Orphan Ganeti networks, without corresponding DB entries
  * Private networks that are not created to all Ganeti backends
  * Unsynchronized IP pools

To detect all inconsistencies you can just run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
805

806
.. code-block:: console
807

808
809
810
  $ snf-manage reconcile-networks

Adding the `--fix-all` option, will do the actual synchronization:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
811

812
.. code-block:: console
813

814
815
816
  $ snf-manage reconcile-networks --fix-all

Please see ``snf-manage reconcile-networks --help`` for all the details.
817
818


819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
Cyclades internals
------------------

Asynchronous communication with Ganeti backends
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Synnefo uses Google Ganeti backends for VM cluster management. In order for
Cyclades to be able to handle thousands of user requests, Cyclades and Ganeti
communicate asynchronously. Briefly, requests are submitted to Ganeti through
Ganeti's RAPI/HTTP interface, and then asynchronous notifications about the
progress of Ganeti jobs are being created and pushed upwards to Cyclades. The
architecture and communication with a Ganeti backend is shown in the graph
below:

.. image:: images/cyclades-ganeti-communication.png
   :width: 50%
   :target: _images/cyclades-ganeti-communication.png

The Cyclades API server is responsible for handling user requests. Read-only
requests are directly served by looking up the Cyclades DB. If the request
needs an action in the Ganeti backend, Cyclades submit jobs to the Ganeti
master using the `Ganeti RAPI interface
<http://docs.ganeti.org/ganeti/2.2/html/rapi.html>`_.

While Ganeti executes the job, `snf-ganeti-eventd`, `snf-ganeti-hook` and
`snf-progress-monitor` are monitoring the progress of the job and send
corresponding messages to the RabbitMQ servers. These components are part
of `snf-cyclades-gtools` and must be installed on all Ganeti nodes. Specially:

* *snf-ganeti-eventd* sends messages about operations affecting the operating
  state of instances and networks. Works by monitoring the Ganeti job queue.
* *snf-ganeti_hook* sends messages about the NICs of instances. It includes a
  number of `Ganeti hooks <http://docs.ganeti.org/ganeti/2.2/html/hooks.html>`_
  for customisation of operations.
* *snf-progress_monitor* sends messages about the progress of the Image deployment
  phase which is done by the Ganeti OS Definition `snf-image`.

Finally, `snf-dispatcher` consumes messages from the RabbitMQ queues, processes
these messages and properly updates the state of the Cyclades DB. Subsequent
requests to the Cyclades API, will retrieve the updated state from the DB.


860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
Synnefo management commands ("snf-manage")
==========================================

Each Synnefo service, Astakos, Pithos and Cyclades are controlled by the
administrator using the "snf-manage" admin tool. This tool is an extension of
the Django command-line management utility. It is run on the host that runs
each service and provides different types of commands depending the services
running on the host. If you are running more than one service on the same host
"snf-manage" adds all the corresponding commands for each service dynamically,
providing a unified admin environment.

To run "snf-manage" you just type:

.. code-block:: console

   # snf-manage <command> [arguments]

on the corresponding host that runs the service. For example, if you have all
services running on different physical hosts you would do:

.. code-block:: console

   root@astakos-host # snf-manage <astakos-command> [argument]
   root@pithos-host # snf-manage <pithos-command> [argument]
   root@cyclades-host # snf-manage <cyclades-command> [argument]

If you have all services running on the same host you would do:

.. code-block:: console

   root@synnefo-host # snf-manage <{astakos,pithos,cyclades}-command> [argument]

Note that you cannot execute a service's command on a host that is not running
this service. For example, the following will return an error if Astakos and
Cyclades are installed on different physical hosts:

.. code-block:: console

   root@astakos-host # snf-manage <cyclades-command> [argument]
   Unknown command: 'cyclades-command'
   Type 'snf-manage help' for usage.

This is the complete list of "snf-manage" commands for each service.

Astakos snf-manage commands
---------------------------

============================  ===========================
Name                          Description
============================  ===========================
fix-superusers                Transform superusers created by syncdb into AstakosUser instances
911
cleanup-full                  Cleanup sessions and session catalog
912
913
commission-list               List pending commissions
commission-show               Show details for a pending commission
914
915
916
component-add                 Register a component
component-list                List components
component-modify              Modify component attributes
917
component-show                Show component details
918
919
920
921
922
923
project-control               Manage projects and applications
project-list                  List projects
project-show                  Show project details
quota                         List and check the integrity of user quota
reconcile-resources-astakos   Reconcile resource usage of Quotaholder with Astakos DB
resource-list                 List resources
924
resource-modify               Modify a resource's default base quota and boolean flags
925
service-export-astakos        Export Astakos services and resources in JSON format
926
service-import                Register services
927
service-list                  List services
928
service-show                  Show service details
929
930
931
term-add                      Add approval terms
user-activation-send          Send user activation
user-add                      Add user
932
933
934
935
936
937
938
authpolicy-add                Create a new authentication provider policy profile
authpolicy-list               List existing authentication provider policy profiles
authpolicy-remove             Remove an authentication provider policy
authpolicy-set                Assign an existing authentication provider policy profile to a user or group
authpolicy-show               Show authentication provider profile details
group-add                     Create a group with the given name
group-list                    List available groups
939
940
941
942
943
944
945
946
947
948
949
user-list                     List users
user-modify                   Modify user
user-show                     Show user details
============================  ===========================

Pithos snf-manage commands
--------------------------

============================  ===========================
Name                          Description
============================  ===========================
950
reconcile-commissions-pithos  Display unresolved commissions and trigger their recovery
951
service-export-pithos         Export Pithos services and resources in JSON format
952
reconcile-resources-pithos    Detect unsynchronized usage between Astakos and Pithos DB resources and synchronize them if specified so.
953
954
955
956
957
============================  ===========================

Cyclades snf-manage commands
----------------------------

958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
============================== ===========================
Name                           Description
============================== ===========================
backend-add                    Add a new Ganeti backend
backend-list                   List backends
backend-modify                 Modify a backend
backend-update-status          Update backend statistics for instance allocation
backend-remove                 Remove a Ganeti backend
server-create                  Create a new server
server-show                    Show server details
server-list                    List servers
server-modify                  Modify a server
server-import                  Import an existing Ganeti VM into synnefo
server-inspect                 Inspect a server in DB and Ganeti
network-create                 Create a new network
network-list                   List networks
network-modify                 Modify a network
network-inspect                Inspect network state in DB and Ganeti
network-remove                 Delete a network
flavor-create                  Create a new flavor
flavor-list                    List flavors
flavor-modify                  Modify a flavor
image-list                     List images
image-show                     Show image details
pool-create                    Create a bridge or mac-prefix pool
pool-show                      Show pool details
pool-list                      List pools
pool-modify                    Modify a pool
pool-remove                    Delete a pool
987
988
989
990
991
992
993
994
995
port-create                    Create a port connecting a server to a network
port-inspect                   Inspect the state of a port in DB and Ganeti
port-list                      List ports
port-remove                    Delete a port
floating-ip-create             Create a new floating IP
floating-ip-attach             Attach a floating IP to a server
floating-ip-dettach            Dettach a flotaing IP from a server
floating-ip-list               List floating IPs
floating-ip-remove             Delete a floating IP
996
997
queue-inspect                  Inspect the messages of a RabbitMQ queue
queue-retry                    Resend messages from Dead Letter queues to original exchanges
998
service-export-cyclades        Export Cyclades services and resources in JSON format
999
1000
1001
1002
subnet-create                  Create a subnet
subnet-inspect                 Inspect a subnet in DB
subnet-list                    List subnets
subnet-modify                  Modify a subnet
1003
1004
1005
1006
1007
1008
reconcile-servers              Reconcile servers of Synnefo DB with state of Ganeti backend
reconcile-networks             Reconcile networks of Synnefo DB with state of Ganeti backend
reconcile-pools                Check consistency of pool resources
reconcile-commissions-cyclades Detect and resolve pending commissions to Quotaholder
reconcile-resources-cyclades   Reconcile resource usage of Astakos with Cyclades DB.
============================== ===========================
1009

1010

1011
1012
1013
1014
1015
1016
1017
1018
Astakos helper scripts
======================

Astakos includes two scripts to facilitate the installation procedure.
Running:

.. code-block:: console

1019
   snf-component-register [<component_name>]
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030

automates the registration of the standard Synnefo components (astakos,
cyclades, and pithos) in astakos database. It internally uses the script:

.. code-block:: console

   snf-service-export <component_name> <base_url>

which simulates the export of service and resource definitions of the
standard Synnefo components.

1031

1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
Pithos managing accounts
========================

Pithos provides a utility tool for managing accounts.
To run you just type:

.. code-block:: console

   # pithos-manage-accounts <command> [arguments]

This is the list of the available commands:

============================  ===========================
Name                          Description
============================  ===========================
delete                        Remove an account from the Pithos DB
export-quota                  Export account quota in a file
list                          List existing/dublicate accounts
merge                         Move an account contents in another account
set-container-quota           Set container quota for all or a specific account
============================  ===========================

1054

1055
The "kamaki" API client
1056
=======================
1057
1058
1059

To upload, register or modify an image you will need the **kamaki** tool.
Before proceeding make sure that it is configured properly. Verify that
1060
*image.url*, *file.url*, *user.url* and *token* are set as needed:
1061
1062
1063
1064
1065

.. code-block:: console

   $ kamaki config list

1066
To change a setting use ``kamaki config set``:
1067
1068
1069

.. code-block:: console

1070
   $ kamaki config set image.url https://cyclades.example.com/image
1071
1072
   $ kamaki config set file.url https://pithos.example.com/v1
   $ kamaki config set user.url https://accounts.example.com
1073
1074
   $ kamaki config set token ...

1075
1076
To test that everything works, try authenticating the current account with
kamaki:
1077
1078
1079
1080
1081

.. code-block:: console

  $ kamaki user authenticate

1082
This will output user information.
1083

1084
1085
1086
Upload Image
------------

1087
1088
By convention, images are stored in a container called ``images``. Check if the
container exists, by listing all containers in your account:
1089
1090
1091

.. code-block:: console

1092
   $ kamaki file list
1093

1094
If the container ``images`` does not exist, create it:
1095
1096
1097

.. code-block:: console

1098
  $ kamaki file create images
1099

1100
You are now ready to upload an image to container ``images``. You can upload it
1101
with a Pithos client, or use kamaki directly:
1102
1103
1104

.. code-block:: console

1105
   $ kamaki file upload ubuntu.iso images
1106

1107
You can use any Pithos client to verify that the image was uploaded correctly,
1108
or you can list the contents of the container with kamaki:
1109

1110
1111
1112
.. code-block:: console

  $ kamaki file list images
1113
1114

The full Pithos URL for the previous example will be
1115
1116
``pithos://u53r-un1qu3-1d/images/ubuntu.iso`` where ``u53r-un1qu3-1d`` is the
unique user id (uuid).
1117
1118
1119
1120

Register Image
--------------

1121
To register an image you will need to use the full Pithos URL. To register as
1122
1123
1124
1125
a public image the one from the previous example use:

.. code-block:: console

1126
   $ kamaki image register Ubuntu pithos://u53r-un1qu3-1d/images/ubuntu.iso --public
1127
1128

The ``--public`` flag is important, if missing the registered image will not
1129
be listed by ``kamaki image list``.
1130

1131
Use ``kamaki image register`` with no arguments to see a list of available
1132
1133
1134
1135
options. A more complete example would be the following:

.. code-block:: console

1136
   $ kamaki image register Ubuntu pithos://u53r-un1qu3-1d/images/ubuntu.iso \
1137
1138
1139
1140
1141
1142
            --public --disk-format diskdump --property kernel=3.1.2

To verify that the image was registered successfully use:

.. code-block:: console

1143
   $ kamaki image list --name-like=ubuntu
1144

1145
1146
1147
1148

Miscellaneous
=============

1149
.. _branding:
Olga Brani's avatar
Olga Brani committed
1150

1151
Branding
1152
--------
Olga Brani's avatar
Olga Brani committed
1153

1154
1155
1156
1157
Since Synnefo v0.14, you are able to adapt the Astakos, Pithos and Cyclades Web
UI to your company’s visual identity. This is possible using the snf-branding
component, which is automatically installed on the nodes running the API
servers for Astakos, Pithos and Cyclades. 
Olga Brani's avatar
Olga Brani committed
1158
1159
1160
1161

Configuration
~~~~~~~~~~~~~

1162
1163
1164
1165
This can be done by modifing the settings provided by the snf-branding component
to match your service identity. The settings for the snf-branding application
can be found inside the configuration file ``/etc/synnefo/15-snf-branding.conf``
on the nodes that have Astakos, Pithos and Cyclades installed.
Olga Brani's avatar
Olga Brani committed
1166

1167
1168
1169
By default, the global service name is "Synnefo" and the company name is
"GRNET". These names and their respective logos and URLs are used throughout
the Astakos, Pithos and Cyclades UI.
Olga Brani's avatar
Olga Brani committed
1170

1171
**Names and URLs:**
Olga Brani's avatar
Olga Brani committed
1172

1173
1174
The first group of branding customization refers to the service's and company's
information.
Olga Brani's avatar
Olga Brani committed
1175

1176
1177
You can overwrite the company and the service name and URL respectively by
uncommenting and setting the following:
Olga Brani's avatar
Olga Brani committed
1178

1179
1180
1181
1182
1183
.. code-block:: python
  
  # setting used in Astakos Dashboard/Projects pages
  BRANDING_SERVICE_NAME = 'My cloud'
  BRANDING_SERVICE_URL = 'http://www.mycloud.synnefo.org/'
Olga Brani's avatar
Olga Brani committed
1184

1185
1186
1187
1188
1189
  # settings used in Astakos, Pithos, Cyclades footer only if 
  # BRANDING_SHOW_COPYRIGHT is set to True
  BRANDING_SHOW_COPYRIGHT = True
  BRANDING_COMPANY_NAME = 'Company LTD'
  BRANDING_COMPANY_URL = 'https://www.company-ltd.synnefo.org/'
Olga Brani's avatar
Olga Brani committed
1190
1191


Olga Brani's avatar
Olga Brani committed
1192
**Copyright and footer options:**
Olga Brani's avatar
Olga Brani committed
1193

1194
1195
By default, no Copyright message is shown in the UI footer. If you want to make
it visible in the footer of Astakos, Pithos and Cyclades UI, you can uncomment
1196
and set to ``True`` the ``BRANDING_SHOW_COPYRIGHT`` setting:
Olga Brani's avatar
Olga Brani committed
1197

1198
.. code-block:: python
Olga Brani's avatar
Olga Brani committed
1199
1200
1201

  #BRANDING_SHOW_COPYRIGHT = False

1202
Copyright message defaults to 'Copyright (c) 2011-<current_year>
1203
1204
<BRANDING_COMPANY_NAME>.' but you can overwrite it to a completely custom one by
setting the following option:
Olga Brani's avatar
Olga Brani committed
1205

1206
1207
1208
.. code-block:: python

  BRANDING_COPYRIGHT_MESSAGE = 'Copyright (c) 2011-2013 GRNET'
Olga Brani's avatar
Olga Brani committed
1209

Olga Brani's avatar
Olga Brani committed
1210
1211
1212
1213
1214
1215
1216
1217
1218
If you want to include a custom message in the footer, you can uncomment and 
set the ``BRANDING_FOOTER_EXTRA_MESSAGE`` setting. You can use html markup. 
Your custom message will appear  above Copyright message at the Compute 
templates and the Dashboard UI.

.. code-block:: python

  #BRANDING_FOOTER_EXTRA_MESSAGE = ''

Olga Brani's avatar
Olga Brani committed
1219
1220
1221

**Images:**

1222
1223
The Astakos, Pithos and Cyclades Web UI has some logos and images.
 
Olga Brani's avatar
Olga Brani committed
1224
1225
1226
1227
1228
1229
1230
1231
The branding-related images are presented in  the following table:

===============  ============================  =========
Image            Name/extension  convention    Usage
===============  ============================  =========
Favicon          favicon.ico                   Favicon for all services
Dashboard logo   dashboard_logo.png            Visible in all Astakos UI pages
Compute logo     compute_logo.png              Visible in all Cyclades UI pages
1232
1233
Console logo     console_logo.png              Visible in the Cyclades Console Window
Storage logo     storage_logo.png              Visible in all Pithos UI pages
Olga Brani's avatar
Olga Brani committed
1234
1235
===============  ============================  =========

1236
There are two methods  available for replacing all, or individual, 
1237
branding-related images:
Olga Brani's avatar
Olga Brani committed
1238

1239
1240
1. Create a new directory inside ``/usr/share/synnefo/static/`` (e.g.
   ``mybranding``) and place there some or all of your images.
Olga Brani's avatar
Olga Brani committed
1241

1242
   If you want to replace all of your images, keep the name/extension
1243
1244
   conventions as indicated in the above table and change the
   ``BRANDING_IMAGE_MEDIA_URL`` setting accordingly:
Olga Brani's avatar
Olga Brani committed
1245

1246
1247
1248
1249
   .. code-block:: python
        
      # using relative path
      BRANDING_IMAGE_MEDIA_URL= '/static/mybranding/images/' 
Olga Brani's avatar
Olga Brani committed
1250

1251
1252
      # or if you already host them in a separate domain (e.g. cdn)
      BRANDING_IMAGE_MEDIA_URL= 'https://cdn.synnefo.org/branding/images/'
Olga Brani's avatar
Olga Brani committed
1253
1254


1255
1256
1257
1258
   If you wish to replace individual images, **do not uncomment**
   ``BRANDING_IMAGE_MEDIA_URL``, but instead provide a relative path, pointing to
   the file inside your directory for each ``BRANDING_<image>_URL`` that you wish
   to replace.
Olga Brani's avatar
Olga Brani committed
1259

1260
2. Upload some or all of your images to a server and replace each 
1261
   ``BRANDING_<image>_URL`` with the absolute url of the image (i.e.
1262
   ``BRANDING_DASHBOARD_URL = 'https://www.synnefo.com/images/my_dashboard.jpg'``).
Olga Brani's avatar
Olga Brani committed
1263

1264
   Note that the alternative text  for each image tag inside html documents is 
1265
   alt=“BRANDING_SERVICE_NAME {Dashboard, Compute. Console, Storage}” respectively.
Olga Brani's avatar
Olga Brani committed
1266

1267
.. note:: Retina optimized images:
Olga Brani's avatar
Olga Brani committed
1268

1269
1270
   Synnefo UI is optimized for Retina displays. As far as images are concerned,  
   `retina.js <http://retinajs.com/>`_ is used.
Olga Brani's avatar
Olga Brani committed
1271

1272
   Retina.js checks each image on a page to see if there is a high-resolution 
1273
1274
   version of that image on your server. If a high-resolution variant exists, 
   the script will swap in that image in-place.
Olga Brani's avatar
Olga Brani committed
1275

1276
1277
1278
1279
1280
   The script assumes you use  `Apple's prescribed high-resolution modifier (@2x)
   <http://developer.apple.com/library/ios/#documentation/2DDrawing/Conceptual/
   DrawingPrintingiOS/SupportingHiResScreensInViews/SupportingHiResScreensInViews
   .html#//apple_ref/doc/uid/TP40010156-CH15-SW1>`_ to denote high-resolution 
   image variants on your server.
Olga Brani's avatar
Olga Brani committed
1281

1282
1283
1284
1285
1286
   For each of the images that you wish the script to  replace, you must have a 
   high-resolution variant in the same folder  named correctly and it will be 
   detected automatically. For example if your image is in <my_directory> and is 
   named "my_image.jpg" the script will look in the same directory for an image 
   named "my_image@2x.jpg".
Olga Brani's avatar
Olga Brani committed
1287

1288
1289
   In case that you don’t want to use a high-resolution image, the 
   normal-resolution image will be visible.
Olga Brani's avatar
Olga Brani committed
1290

1291
1292
More branding
~~~~~~~~~~~~~
Olga Brani's avatar
Olga Brani committed
1293

1294
1295
Although, it is not 100% branding-related, further verbal customization is
feasible. 
Olga Brani's avatar
Olga Brani committed
1296
1297
1298

**EMAILS**

1299
1300
1301
The output of all email `*`.txt files will be already customized to contain your
company and service names but you can further alter their content if you feel it
best fits your needs as simple as creasynnefo template.    
1302

1303
1304
In order to overwrite one or more email-templates you need to place your 
modified <email-file>.txt files respecting the following structure:
1305
  
1306
1307
  **/etc/synnefo/templates/**
      **im/**
1308
1309
1310
1311
1312
          | activation_email.txt
          | email.txt
          | invitation.txt
          | switch_accounts_email.txt
          | welcome_email.txt
1313
          **projects/**
1314
1315
1316
1317
1318
1319
1320
1321
              | project_approval_notification.txt
              | project_denial_notification.txt    
              | project_membership_change_notification.txt
              | project_membership_enroll_notification.txt
              | project_membership_leave_request_notification.txt
              | project_membership_request_notification.txt
              | project_suspension_notification.txt
              | project_termination_notification.txt
1322
      **registration/**
1323
1324
1325
1326
1327
          | email_change_email.txt
          | password_email.txt

Feel free to omit any of the above files you do not wish to overwrite.

Olga Brani's avatar
Olga Brani committed
1328
1329
1330
Below is a list of all emails sent by Synnefo to users along with a short 
description and a link to their content:

1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
* ``snf-astakos-app/astakos/im/templates/im/email.txt``
  Base email template. Contains a contact email and a “thank you” message.
  (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/activation_email.txt`` Email sent to
  user that prompts  him/her to click on a link provided to activate the account.
  Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/activation_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/invitation.txt`` Email sent to an
  invited user. He/she has to click on a link provided to activate the account.
  Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/invitation.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/switch_accounts_email.txt`` Email
  sent to user upon his/her request to associate this email address with a
  shibboleth account. He/she has to click on a link provided to activate the
  association. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/switch_accounts_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/welcome_email.txt`` Email sent to
  inform the user that his/ her account has been activated. Extends “email.txt”
  (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/welcome_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/registration/email_change_email.txt``
  Email sent to user when he/she has requested new email address assignment. The
  user has to click on a link provided to validate this action. Extends
  “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/registration/email_change_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/registration/password_email.txt`` Email
  sent for resetting password purpose. The user has to click on a link provided
  to validate this action. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/registration/password_email.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_approval_notification.txt``
  Informs  the project owner that his/her project has been approved. Extends
  “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_approval_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_denial_notification.txt``
  Informs the project owner that his/her  project application has been denied
  explaining the reasons. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_denial_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_change_notification.txt``
  An email is sent to a user containing information about his project membership
  (whether he has been accepted, rejected or removed). Extends “email.txt” (`Link
  <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_change_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_enroll_notification.txt``
  Informs a user that he/she  has been enrolled to a project. Extends
  “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_enroll_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_leave_request_notification.txt``
  An email is sent to the project owner to make him aware of a  user having
  requested to leave his project. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_leave_request_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_request_notification.txt``
  An email is sent to the project owner to make him/her aware of a user having
  requested to join  his project. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_request_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_suspension_notification.txt``
  An email is sent to the project owner to make him/her aware of his/her project
  having been suspended. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_suspension_notification.txt>`_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_termination_notification.txt``
  An email is sent to the project owner to make him/her aware of his/her project
  having been terminated. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_termination_notification.txt>`_)
Olga Brani's avatar
Olga Brani committed
1379
1380
1381
1382

.. warning:: Django templates language:

  If you choose to  overwrite these email templates, be mindful of the necessary 
1383
1384
1385
  information contained in django template variables that must not be omitted, 
  such as the activation link for activating one’s account and many more. 
  These variables are contained into {{}} inside the templates.
Olga Brani's avatar
Olga Brani committed
1386
1387


1388
.. RabbitMQ
1389

1390
1391
1392
1393
1394
1395
RabbitMQ Broker
---------------

Queue nodes run the RabbitMQ sofware, which provides AMQP functionality. To
guarantee high-availability, more than one Queue nodes should be deployed, each
of them belonging to the same `RabbitMQ cluster
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
<http://www.rabbitmq.com/clustering.html>`_. Synnefo uses the RabbitMQ
active/active `High Available Queues <http://www.rabbitmq.com/ha.html>`_ which
are mirrored between two nodes within a RabbitMQ cluster.

The RabbitMQ nodes that form the cluster, are declared to Synnefo through the
`AMQP_HOSTS` setting. Each time a Synnefo component needs to connect to
RabbitMQ, one of these nodes is chosen in a random way. The client that Synnefo
uses to connect to RabbitMQ, handles connection failures transparently and
tries to reconnect to a different node. As long as one of these nodes are up
and running, functionality of Synnefo should not be downgraded by the RabbitMQ
node failures.
1407
1408

All the queues that are being used are declared as durable, meaning that
1409
1410
messages are persistently stored to RabbitMQ, until they get successfully
processed by a client.
1411
1412
1413

Currently, RabbitMQ is used by the following components:

1414
* `snf-ganeti-eventd`, `snf-ganeti-hook` and `snf-progress-monitor`:
1415
1416
  These components send messages concerning the status and progress of
  jobs in the Ganeti backend.
1417
1418
* `snf-dispatcher`: This daemon, consumes the messages that are sent from
  the above components, and updates the Cyclades DB accordingly.
1419

Christos Stavrakakis's avatar
Christos Stavrakakis committed
1420

1421
Installation
1422
1423
~~~~~~~~~~~~

1424
1425
1426
1427
1428
1429
Please check the RabbitMQ documentation which covers extensively the
`installation of RabbitMQ server <http://www.rabbitmq.com/download.html>`_ and
the setup of a `RabbitMQ cluster <http://www.rabbitmq.com/clustering.html>`_.
Also, check out the `web management plugin
<http://www.rabbitmq.com/management.html>`_ that can be useful for managing and
monitoring RabbitMQ.
1430
1431

For a basic installation of RabbitMQ on two nodes (node1 and node2) you can do
1432
the following:
1433

1434
On both nodes, install rabbitmq-server and create a Synnefo user:
1435
1436
1437
1438
1439
1440
1441

.. code-block:: console

  $ apt-get install rabbitmq-server
  $ rabbitmqctl add_user synnefo "example_pass"
  $ rabbitmqctl set_permissions synnefo  ".*" ".*" ".*"

1442
Also guarantee that both nodes share the same cookie, by running:
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470

.. code-block:: console

  $ scp node1:/var/lib/rabbitmq/.erlang.cookie node2:/var/lib/rabbitmq/.erlang.cookie

and restart the nodes:

.. code-block:: console

  $ /etc/init.d/rabbitmq-server restart


To setup the RabbitMQ cluster run:

.. code-block:: console

  root@node2: rabbitmqctl stop_app
  root@node2: rabbitmqctl reset
  root@node2: rabbitmqctl cluster rabbit@node1 rabbit@node2
  root@node2: rabbitmqctl start_app

You can verify that the cluster is set up correctly by running:

.. code-block:: console

  root@node2: rabbitmqctl cluster_status


1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
Logging
-------

Logging in Synnefo is using Python's logging module. The module is configured
using dictionary configuration, whose format is described here:

http://docs.python.org/release/2.7.1/library/logging.html#logging-config-dictschema

Note that this is a feature of Python 2.7 that we have backported for use in
Python 2.6.

1482
The logging configuration dictionary is defined in
1483
``/etc/synnefo/10-snf-webproject-logging.conf``
1484

1485
1486
1487
1488
The administrator can have finer logging control by modifying the
``LOGGING_SETUP`` dictionary, and defining subloggers with different handlers
and log levels.  e.g. To enable debug messages only for the API set the level
of 'synnefo.api' to ``DEBUG``
1489