quick-install-admin-guide.rst 78.4 KB
Newer Older
1
2
.. _quick-install-admin-guide:

3
4
Administrator's Installation Guide
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5

6
This is the Administrator's installation guide.
7

8
It describes how to install the whole Synnefo stack on two (2) physical nodes,
9
with minimum configuration. It installs synnefo from Debian packages, and
10
assumes the nodes run Debian Wheezy. After successful installation, you will
11
12
have the following services running:

13
    * Identity Management (Astakos)
14
    * Object Storage Service (Pithos)
15
    * Compute Service (Cyclades)
16
17
    * Image Service (part of Cyclades)
    * Network Service (part of Cyclades)
18
19
20

and a single unified Web UI to manage them all.

21
22
If you just want to install the Object Storage Service (Pithos), follow the
guide and just stop after the "Testing of Pithos" section.
23
24
25
26
27


Installation of Synnefo / Introduction
======================================

28
29
30
We will install the services with the above list's order. The last three
services will be installed in a single step (at the end), because at the moment
they are contained in the same software component (Cyclades). Furthermore, we
31
will install all services in the first physical node, except Pithos which will
32
33
be installed in the second, due to a conflict between the snf-pithos-app and
snf-cyclades-app component (scheduled to be fixed in the next version).
34
35
36

For the rest of the documentation we will refer to the first physical node as
"node1" and the second as "node2". We will also assume that their domain names
37
38
are "node1.example.com" and "node2.example.com" and their public IPs are "203.0.113.1" and
"203.0.113.2" respectively. It is important that the two machines are under the same domain name.
39
In case you choose to follow a private installation you will need to
40
41
set up a private dns server, using dnsmasq for example. See node1 below for 
more information on how to do so.
42
43
44
45
46

General Prerequisites
=====================

These are the general synnefo prerequisites, that you need on node1 and node2
47
and are related to all the services (Astakos, Pithos, Cyclades).
48
49
50
51

To be able to download all synnefo components you need to add the following
lines in your ``/etc/apt/sources.list`` file:

52
53
| ``deb http://apt.dev.grnet.gr wheezy/``
| ``deb-src http://apt.dev.grnet.gr wheezy/``
54

55
56
57
58
and import the repo's GPG key:

| ``curl https://dev.grnet.gr/files/apt-grnetdev.pub | apt-key add -``

59
60
61
Update your list of packages and continue with the installation:

.. code-block:: console
62

63
   # apt-get update
64

65
You also need a shared directory visible by both nodes. Pithos will save all
66
data inside this directory. By 'all data', we mean files, images, and Pithos
67
68
69
specific mapping data. If you plan to upload more than one basic image, this
directory should have at least 50GB of free space. During this guide, we will
assume that node1 acts as an NFS server and serves the directory ``/srv/pithos``
70
71
to node2 (be sure to set no_root_squash flag). Node2 has this directory
mounted under ``/srv/pithos``, too.
72
73
74
75
76
77
78

Before starting the synnefo installation, you will need basic third party
software to be installed and configured on the physical nodes. We will describe
each node's general prerequisites separately. Any additional configuration,
specific to a synnefo service for each node, will be described at the service's
section.

79
80
81
Finally, it is required for Cyclades and Ganeti nodes to have synchronized
system clocks (e.g. by running ntpd).

82
83
84
Node1
-----

85

86
87
88
General Synnefo dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

89
90
91
92
93
94
95
		* apache (http server)
		* public certificate
		* gunicorn (WSGI http server)
		* postgresql (database)
		* rabbitmq (message queue)
		* ntp (NTP daemon)
		* gevent
96
		* dnsmasq (DNS server)
97

98
You can install apache2, postgresql, ntp and rabbitmq by running:
99
100
101

.. code-block:: console

102
   # apt-get install apache2 postgresql ntp rabbitmq-server
103

104
To install gunicorn and gevent, run:
105
106
107

.. code-block:: console

108
   # apt-get install gunicorn python-gevent
109

110
111
On node1, we will create our databases, so you will also need the
python-psycopg2 package:
112
113
114
115
116
117
118
119
120
121
122
123
124
125

.. code-block:: console

   # apt-get install python-psycopg2

Database setup
~~~~~~~~~~~~~~

On node1, we create a database called ``snf_apps``, that will host all django
apps related tables. We also create the user ``synnefo`` and grant him all
privileges on the database. We do this by running:

.. code-block:: console

126
127
128
129
130
    root@node1:~ # su - postgres
    postgres@node1:~ $ psql
    postgres=# CREATE DATABASE snf_apps WITH ENCODING 'UTF8' LC_COLLATE='C' LC_CTYPE='C' TEMPLATE=template0;
    postgres=# CREATE USER synnefo WITH PASSWORD 'example_passw0rd';
    postgres=# GRANT ALL PRIVILEGES ON DATABASE snf_apps TO synnefo;
131

132
We also create the database ``snf_pithos`` needed by the Pithos backend and
133
134
135
136
137
138
grant the ``synnefo`` user all privileges on the database. This database could
be created on node2 instead, but we do it on node1 for simplicity. We will
create all needed databases on node1 and then node2 will connect to them.

.. code-block:: console

139
140
    postgres=# CREATE DATABASE snf_pithos WITH ENCODING 'UTF8' LC_COLLATE='C' LC_CTYPE='C' TEMPLATE=template0;
    postgres=# GRANT ALL PRIVILEGES ON DATABASE snf_pithos TO synnefo;
141
142

Configure the database to listen to all network interfaces. You can do this by
143
editting the file ``/etc/postgresql/9.1/main/postgresql.conf`` and change
144
145
146
147
``listen_addresses`` to ``'*'`` :

.. code-block:: console

148
    listen_addresses = '*'
149

150
Furthermore, edit ``/etc/postgresql/9.1/main/pg_hba.conf`` to allow node1 and
151
152
153
154
155
node2 to connect to the database. Add the following lines under ``#IPv4 local
connections:`` :

.. code-block:: console

156
157
    host		all	all	203.0.113.1/32	md5
    host		all	all	203.0.113.2/32	md5
158

159
Make sure to substitute "203.0.113.1" and "203.0.113.2" with node1's and node2's
160
161
162
163
164
165
166
actual IPs. Now, restart the server to apply the changes:

.. code-block:: console

   # /etc/init.d/postgresql restart


167
168
169
Certificate Creation
~~~~~~~~~~~~~~~~~~~~~

170
171
172
Node1 will host Cyclades. Cyclades should communicate with the other Synnefo 
Services and users over a secure channel. In order for the connection to be 
trusted, the keys provided to Apache below should be signed with a certificate.
173
This certificate should be added to all nodes. In case you don't have signed keys you can create a self-signed certificate
174
and sign your keys with this. To do so on node1 run:
175
176
177

.. code-block:: console

178
		# apt-get install openvpn
179
180
181
182
183
184
185
186
187
188
189
190
191
192
		# mkdir /etc/openvpn/easy-rsa
		# cp -ai /usr/share/doc/openvpn/examples/easy-rsa/2.0/ /etc/openvpn/easy-rsa
		# cd /etc/openvpn/easy-rsa/2.0
		# vim vars

In vars you can set your own parameters such as KEY_COUNTRY

.. code-block:: console

	# . ./vars
	# ./clean-all

Now you can create the certificate

193
194
.. code-block:: console

195
196
		# ./build-ca

197
198
The previous will create a ``ca.crt`` file in the directory ``/etc/openvpn/easy-rsa/2.0/keys``.
Copy this file under ``/usr/local/share/ca-certificates/`` directory and run :
199
200
201
202
203
204
205
206
207
208

.. code-block:: console

		# update-ca-certificates

to update the records. You will have to do the following on node2 as well.

Now you can create the keys and sign them with the certificate

.. code-block:: console
209

210
211
		# ./build-key-server node1.example.com

212
213
This will create a ``01.pem`` and a ``node1.example.com.key`` files in the 
``/etc/openvpn/easy-rsa/2.0/keys`` directory. Copy these in ``/etc/ssl/certs/``
214
and ``/etc/ssl/private/`` respectively and use them in the apache2
215
configuration file below instead of the defaults.
216

217
218
219
Apache2 setup
~~~~~~~~~~~~~

220
221
Create the file ``/etc/apache2/sites-available/synnefo`` containing the
following:
222
223
224

.. code-block:: console

225
226
    <VirtualHost *:80>
        ServerName node1.example.com
227

228
229
230
231
232
        RewriteEngine On
        RewriteCond %{THE_REQUEST} ^.*(\\r|\\n|%0A|%0D).* [NC]
        RewriteRule ^(.*)$ - [F,L]
        RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
    </VirtualHost>
233

234

235
236
Create the file ``/etc/apache2/sites-available/synnefo-ssl`` containing the
following:
237
238
239

.. code-block:: console

240
241
242
    <IfModule mod_ssl.c>
    <VirtualHost _default_:443>
        ServerName node1.example.com
243

244
        Alias /static "/usr/share/synnefo/static"
245

246
247
        #  SetEnv no-gzip
        #  SetEnv dont-vary
248

249
       AllowEncodedSlashes On
250

251
       RequestHeader set X-Forwarded-Protocol "https"
252

253
254
255
256
    <Proxy * >
        Order allow,deny
        Allow from all
    </Proxy>
257

258
259
260
        SetEnv                proxy-sendchunked
        SSLProxyEngine        off
        ProxyErrorOverride    off
261

262
263
264
        ProxyPass        /static !
        ProxyPass        / http://localhost:8080/ retry=0
        ProxyPassReverse / http://localhost:8080/
265

266
267
268
        RewriteEngine On
        RewriteCond %{THE_REQUEST} ^.*(\\r|\\n|%0A|%0D).* [NC]
        RewriteRule ^(.*)$ - [F,L]
269

270
271
272
273
274
        SSLEngine on
        SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
        SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
    </VirtualHost>
    </IfModule>
275
276
277
278
279
280
281
282
283
284
285
286
287

Now enable sites and modules by running:

.. code-block:: console

   # a2enmod ssl
   # a2enmod rewrite
   # a2dissite default
   # a2ensite synnefo
   # a2ensite synnefo-ssl
   # a2enmod headers
   # a2enmod proxy_http

288
289
290
291
292
293
.. note:: This isn't really needed, but it's a good security practice to disable
    directory listing in apache::

        # a2dismod autoindex


294
.. warning:: Do NOT start/restart the server yet. If the server is running::
295

296
       # /etc/init.d/apache2 stop
297

298

299
300
.. _rabbitmq-setup:

301
302
303
304
305
306
307
308
309
Message Queue setup
~~~~~~~~~~~~~~~~~~~

The message queue will run on node1, so we need to create the appropriate
rabbitmq user. The user is named ``synnefo`` and gets full privileges on all
exchanges:

.. code-block:: console

Ilias Tsitsimpis's avatar
Ilias Tsitsimpis committed
310
   # rabbitmqctl add_user synnefo "example_rabbitmq_passw0rd"
311
312
313
314
315
   # rabbitmqctl set_permissions synnefo ".*" ".*" ".*"

We do not need to initialize the exchanges. This will be done automatically,
during the Cyclades setup.

316
317
Pithos data directory setup
~~~~~~~~~~~~~~~~~~~~~~~~~~~
318

319
320
As mentioned in the General Prerequisites section, there should be a directory 
called ``/srv/pithos`` visible by both nodes. We create and setup the ``data``
321
322
323
324
directory inside it:

.. code-block:: console

325
   # mkdir /srv/pithos
326
327
328
329
330
   # cd /srv/pithos
   # mkdir data
   # chown www-data:www-data data
   # chmod g+ws data

331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
This directory must be shared via `NFS <https://en.wikipedia.org/wiki/Network_File_System>`_.
In order to do this, run:

.. code-block:: console

   # apt-get install rpcbind nfs-kernel-server

Now edit ``/etc/exports`` and add the following line:

.. code-block:: console
   
   /srv/pithos/ 203.0.113.2(rw,no_root_squash,sync,subtree_check)

Once done, run:

.. code-block:: console

   # /etc/init.d/nfs-kernel-server restart


351
352
353
DNS server setup
~~~~~~~~~~~~~~~~

354
355
If your machines are not under the same domain name you have to set up a dns server.
In order to set up a dns server using dnsmasq do the following:
356
357

.. code-block:: console
358

359
   # apt-get install dnsmasq
360

361
Then edit your ``/etc/hosts/`` file as follows:
362
363
364

.. code-block:: console

365
366
		203.0.113.1     node1.example.com
		203.0.113.2     node2.example.com
367

368
dnsmasq will serve any IPs/domains found in ``/etc/resolv.conf``.
369

370
371
372
373
There is a `"bug" in libevent 2.0.5 <http://sourceforge.net/p/levent/bugs/193/>`_
, where if you have multiple nameservers in your ``/etc/resolv.conf``, libevent
will round-robin against them. To avoid this, you must use a single nameserver
for all your needs. Edit your ``/etc/resolv.conf`` to include your dns server: 
374
375
376

.. code-block:: console

377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
   nameserver 203.0.113.1

Because of the aforementioned bug, you can't specify more than one DNS servers
in your ``/etc/resolv.conf``. In order for dnsmasq to serve domains not in 
``/etc/hosts``, edit ``/etc/dnsmasq.conf`` and change the line starting with 
``#resolv-file=`` to:

.. code-block:: console

   resolv-file=/etc/external-dns

Now create the file ``/etc/external-dns`` and specify any extra DNS servers you
want dnsmasq to query for domains, e.g., 8.8.8.8:

.. code-block:: console

   nameserver 8.8.8.8

In the ``/etc/dnsmasq.conf`` file, you can also specify the ``listen-address`` 
and the ``interface`` you would like dnsmasq to listen to.

Finally, restart dnsmasq:

.. code-block:: console

   # /etc/init.d/dnsmasq restart
403

404
405
406
407
408
409
410
411
412
You are now ready with all general prerequisites concerning node1. Let's go to
node2.

Node2
-----

General Synnefo dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

413
414
415
416
417
    * apache (http server)
    * gunicorn (WSGI http server)
    * postgresql (database)
    * ntp (NTP daemon)
    * gevent
418
    * certificates
419
    * dnsmasq (DNS server)
420
421
422
423
424

You can install the above by running:

.. code-block:: console

425
   # apt-get install apache2 postgresql ntp
426

427
To install gunicorn and gevent, run:
428
429
430

.. code-block:: console

431
   # apt-get install gunicorn python-gevent
432

433
434
Node2 will connect to the databases on node1, so you will also need the
python-psycopg2 package:
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450

.. code-block:: console

   # apt-get install python-psycopg2

Database setup
~~~~~~~~~~~~~~

All databases have been created and setup on node1, so we do not need to take
any action here. From node2, we will just connect to them. When you get familiar
with the software you may choose to run different databases on different nodes,
for performance/scalability/redundancy reasons, but those kind of setups are out
of the purpose of this guide.

Apache2 setup
~~~~~~~~~~~~~
451

452
453
Create the file ``/etc/apache2/sites-available/synnefo`` containing the
following:
454

455
456
.. code-block:: console

457
458
    <VirtualHost *:80>
        ServerName node2.example.com
459

460
461
462
463
464
        RewriteEngine On
        RewriteCond %{THE_REQUEST} ^.*(\\r|\\n|%0A|%0D).* [NC]
        RewriteRule ^(.*)$ - [F,L]
        RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
    </VirtualHost>
465
466
467
468
469
470

Create the file ``synnefo-ssl`` under ``/etc/apache2/sites-available/``
containing the following:

.. code-block:: console

471
472
473
    <IfModule mod_ssl.c>
    <VirtualHost _default_:443>
        ServerName node2.example.com
474

475
        Alias /static "/usr/share/synnefo/static"
476

477
478
479
        SetEnv no-gzip
        SetEnv dont-vary
        AllowEncodedSlashes On
480

481
        RequestHeader set X-Forwarded-Protocol "https"
482

483
484
485
486
        <Proxy * >
            Order allow,deny
            Allow from all
        </Proxy>
487

488
489
490
        SetEnv                proxy-sendchunked
        SSLProxyEngine        off
        ProxyErrorOverride    off
491

492
493
494
        ProxyPass        /static !
        ProxyPass        / http://localhost:8080/ retry=0
        ProxyPassReverse / http://localhost:8080/
495

496
497
498
499
500
        SSLEngine on
        SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
        SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
    </VirtualHost>
    </IfModule>
501
502
503
504
505
506
507
508
509
510
511
512
513

As in node1, enable sites and modules by running:

.. code-block:: console

   # a2enmod ssl
   # a2enmod rewrite
   # a2dissite default
   # a2ensite synnefo
   # a2ensite synnefo-ssl
   # a2enmod headers
   # a2enmod proxy_http

514
515
516
517
518
.. note:: This isn't really needed, but it's a good security practice to disable
    directory listing in apache::

        # a2dismod autoindex

519
.. warning:: Do NOT start/restart the server yet. If the server is running::
520

521
       # /etc/init.d/apache2 stop
522

523
524
525
526
527

Acquire certificate
~~~~~~~~~~~~~~~~~~~

Copy the certificate you created before on node1 (`ca.crt`) under the directory
528
``/usr/local/share/ca-certificate`` and run:
529
530
531

.. code-block:: console

532
   # update-ca-certificates
533
534
535
536
537
538
539
540
541
542

to update the records.


DNS Setup
~~~~~~~~~

Add the following line in ``/etc/resolv.conf`` file

.. code-block:: console
543

544
   nameserver 203.0.113.1
545

546
547
548
549
to inform the node about the new DNS server.

As mentioned before, this should be the only ``nameserver`` entry in 
``/etc/resolv.conf``.
550

551
552
553
554
555
556
557
We are now ready with all general prerequisites for node2. Now that we have
finished with all general prerequisites for both nodes, we can start installing
the services. First, let's install Astakos on node1.

Installation of Astakos on node1
================================

558
559
560
To install Astakos, grab the package from our repository (make sure  you made
the additions needed in your ``/etc/apt/sources.list`` file and updated, as 
described previously), by running:
561
562
563

.. code-block:: console

564
   # apt-get install snf-astakos-app snf-pithos-backend
565

566
567
.. _conf-astakos:

568
569
570
Configuration of Astakos
========================

571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
Gunicorn setup
--------------

Copy the file ``/etc/gunicorn.d/synnefo.example`` to
``/etc/gunicorn.d/synnefo``, to make it a valid gunicorn configuration file:

.. code-block:: console

    # mv /etc/gunicorn.d/synnefo.example /etc/gunicorn.d/synnefo


.. warning:: Do NOT start the server yet, because it won't find the
    ``synnefo.settings`` module. Also, in case you are using ``/etc/hosts``
    instead of a DNS to get the hostnames, change ``--worker-class=gevent`` to
    ``--worker-class=sync``. We will start the server after successful
    installation of Astakos. If the server is running::

       # /etc/init.d/gunicorn stop

590
591
592
Conf Files
----------

593
After Astakos is successfully installed, you will find the directory
594
595
596
597
598
599
``/etc/synnefo`` and some configuration files inside it. The files contain
commented configuration options, which are the default options. While installing
new snf-* components, new configuration files will appear inside the directory.
In this guide (and for all services), we will edit only the minimum necessary
configuration options, to reflect our setup. Everything else will remain as is.

600
After getting familiar with Synnefo, you will be able to customize the software
601
602
603
as you wish and fits your needs. Many options are available, to empower the
administrator with extensively customizable setups.

604
For the snf-webproject component (installed as an Astakos dependency), we
605
606
607
608
609
610
611
need the following:

Edit ``/etc/synnefo/10-snf-webproject-database.conf``. You will need to
uncomment and edit the ``DATABASES`` block to reflect our database:

.. code-block:: console

612
613
614
    DATABASES = {
     'default': {
         # 'postgresql_psycopg2', 'postgresql','mysql', 'sqlite3' or 'oracle'
615
         'ENGINE': 'django.db.backends.postgresql_psycopg2',
616
617
         # ATTENTION: This *must* be the absolute path if using sqlite3.
         # See: http://docs.djangoproject.com/en/dev/ref/settings/#name
618
619
620
621
         'NAME': 'snf_apps',
         'USER': 'synnefo',                      # Not used with sqlite3.
         'PASSWORD': 'example_passw0rd',         # Not used with sqlite3.
         # Set to empty string for localhost. Not used with sqlite3.
622
         'HOST': '203.0.113.1',
623
624
625
         # Set to empty string for default. Not used with sqlite3.
         'PORT': '5432',
     }
626
627
628
    }

Edit ``/etc/synnefo/10-snf-webproject-deploy.conf``. Uncomment and edit
629
``SECRET_KEY``. This is a Django specific setting which is used to provide a
630
seed in secret-key hashing algorithms. Set this to a random string of your
Giorgos Korfiatis's avatar
Giorgos Korfiatis committed
631
choice and keep it private:
632
633
634

.. code-block:: console

635
    SECRET_KEY = 'sy6)mw6a7x%n)-example_secret_key#zzk4jo6f2=uqu!1o%)'
636

637
For Astakos specific configuration, edit the following options in
638
``/etc/synnefo/20-snf-astakos-app-settings.conf`` :
639
640
641

.. code-block:: console

642
    ASTAKOS_COOKIE_DOMAIN = '.example.com'
643

Giorgos Korfiatis's avatar
Giorgos Korfiatis committed
644
    ASTAKOS_BASE_URL = 'https://node1.example.com/astakos'
645

646
The ``ASTAKOS_COOKIE_DOMAIN`` should be the base url of our domain (for all
647
services). ``ASTAKOS_BASE_URL`` is the Astakos top-level URL. Appending an
648
649
extra path (``/astakos`` here) is recommended in order to distinguish
components, if more than one are installed on the same machine.
650

651
652
.. note:: For the purpose of this guide, we don't enable recaptcha authentication.
    If you would like to enable it, you have to edit the following options:
653
654
655
656
657
658
659
660
661
662

    .. code-block:: console

        ASTAKOS_RECAPTCHA_PUBLIC_KEY = 'example_recaptcha_public_key!@#$%^&*('
        ASTAKOS_RECAPTCHA_PRIVATE_KEY = 'example_recaptcha_private_key!@#$%^&*('
        ASTAKOS_RECAPTCHA_USE_SSL = True
        ASTAKOS_RECAPTCHA_ENABLED = True

    For the ``ASTAKOS_RECAPTCHA_PUBLIC_KEY`` and ``ASTAKOS_RECAPTCHA_PRIVATE_KEY``
    go to https://www.google.com/recaptcha/admin/create and create your own pair.
663

664
665
666
667
Then edit ``/etc/synnefo/20-snf-astakos-app-cloudbar.conf`` :

.. code-block:: console

668
    CLOUDBAR_LOCATION = 'https://node1.example.com/static/im/cloudbar/'
669

Giorgos Korfiatis's avatar
Giorgos Korfiatis committed
670
    CLOUDBAR_SERVICES_URL = 'https://node1.example.com/astakos/ui/get_services'
671

Giorgos Korfiatis's avatar
Giorgos Korfiatis committed
672
    CLOUDBAR_MENU_URL = 'https://node1.example.com/astakos/ui/get_menu'
673

674
675
676
Those settings have to do with the black cloudbar endpoints and will be
described in more detail later on in this guide. For now, just edit the domain
to point at node1 which is where we have installed Astakos.
677

678
679
If you are an advanced user and want to use the Shibboleth Authentication
method, read the relative :ref:`section <shibboleth-auth>`.
Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
680

681
.. _email-configuration:
682
683
684
685

Email delivery configuration
----------------------------

686
687
688
689
690
Many of the ``Astakos`` operations require the server to notify service users 
and administrators via email. e.g. right after the signup process, the service 
sents an email to the registered email address containing an verification url. 
After the user verifies the email address, Astakos once again needs to 
notify administrators with a notice that a new account has just been verified.
691

692
More specifically Astakos sends emails in the following cases
693
694

- An email containing a verification link after each signup process.
695
696
697
- An email to the people listed in ``ADMINS`` setting after each email
  verification if ``ASTAKOS_MODERATION`` setting is ``True``. The email
  notifies administrators that an additional action is required in order to
698
  activate the user.
699
- A welcome email to the user email and an admin notification to ``ADMINS``
700
  right after each account activation.
701
- Feedback messages submited from Astakos contact view and Astakos feedback
702
  API endpoint are sent to contacts listed in ``HELPDESK`` setting.
703
- Project application request notifications to people included in ``HELPDESK``
704
  and ``MANAGERS`` settings.
705
- Notifications after each project members action (join request, membership
706
707
  accepted/declinde etc.) to project members or project owners.

708
709
710
Astakos uses the Django internal email delivering mechanism to send email
notifications. A simple configuration, using an external smtp server to
deliver messages, is shown below. Alter the following example to meet your
711
smtp server characteristics. Notice that the smtp server is needed for a proper
712
713
714
installation.

Edit ``/etc/synnefo/00-snf-common-admins.conf``:
715
716

.. code-block:: python
717

718
    EMAIL_HOST = "mysmtp.server.example.com"
719
720
721
722
    EMAIL_HOST_USER = "<smtpuser>"
    EMAIL_HOST_PASSWORD = "<smtppassword>"

    # this gets appended in all email subjects
723
    EMAIL_SUBJECT_PREFIX = "[example.com] "
724

725
    # Address to use for outgoing emails
726
    DEFAULT_FROM_EMAIL = "server@example.com"
727

728
    # Email where users can contact for support. This is used in html/email
729
    # templates.
730
    CONTACT_EMAIL = "server@example.com"
731
732

    # The email address that error messages come from
733
    SERVER_EMAIL = "server-errors@example.com"
734
735

Notice that since email settings might be required by applications other than
736
737
Astakos, they are defined in a different configuration file than the one
previously used to set Astakos specific settings.
738

739
Refer to
740
`Django documentation <https://docs.djangoproject.com/en/1.4/topics/email/>`_
741
742
for additional information on available email settings.

743
As refered in the previous section, based on the operation that triggers
744
an email notification, the recipients list differs. Specifically, for
745
746
emails whose recipients include contacts from your service team
(administrators, managers, helpdesk etc) synnefo provides the following
747
settings located in ``00-snf-common-admins.conf``:
748
749
750

.. code-block:: python

751
752
753
754
    ADMINS = (('Admin name', 'admin@example.com'),
              ('Admin2 name', 'admin2@example.com))
    MANAGERS = (('Manager name', 'manager@example.com'),)
    HELPDESK = (('Helpdesk user name', 'helpdesk@example.com'),)
755

756
757
758
759
760
Alternatively, it may be convenient to send e-mails to a file, instead of an actual smtp server, using the file backend. Do so by creating a configuration file ``/etc/synnefo/99-local.conf`` including the folowing:

.. code-block:: python

    EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
761
    EMAIL_FILE_PATH = '/tmp/app-messages'
762
763


764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
Enable Pooling
--------------

This section can be bypassed, but we strongly recommend you apply the following,
since they result in a significant performance boost.

Synnefo includes a pooling DBAPI driver for PostgreSQL, as a thin wrapper
around Psycopg2. This allows independent Django requests to reuse pooled DB
connections, with significant performance gains.

To use, first monkey-patch psycopg2. For Django, run this before the
``DATABASES`` setting in ``/etc/synnefo/10-snf-webproject-database.conf``:

.. code-block:: console

779
780
    from synnefo.lib.db.pooled_psycopg2 import monkey_patch_psycopg2
    monkey_patch_psycopg2()
781

782
783
Since we are running with greenlets, we should modify psycopg2 behavior, so it
works properly in a greenlet context:
784
785
786

.. code-block:: console

787
788
    from synnefo.lib.db.psyco_gevent import make_psycopg_green
    make_psycopg_green()
789
790
791
792

Use the Psycopg2 driver as usual. For Django, this means using
``django.db.backends.postgresql_psycopg2`` without any modifications. To enable
connection pooling, pass a nonzero ``synnefo_poolsize`` option to the DBAPI
793
driver, through ``DATABASES.OPTIONS`` in Django.
794
795
796
797
798
799

All the above will result in an ``/etc/synnefo/10-snf-webproject-database.conf``
file that looks like this:

.. code-block:: console

800
801
802
    # Monkey-patch psycopg2
    from synnefo.lib.db.pooled_psycopg2 import monkey_patch_psycopg2
    monkey_patch_psycopg2()
803

804
805
806
    # If running with greenlets
    from synnefo.lib.db.psyco_gevent import make_psycopg_green
    make_psycopg_green()
807

808
809
810
    DATABASES = {
     'default': {
         # 'postgresql_psycopg2', 'postgresql','mysql', 'sqlite3' or 'oracle'
811
         'ENGINE': 'django.db.backends.postgresql_psycopg2',
812
         'OPTIONS': {'synnefo_poolsize': 8},
813
814
815

         # ATTENTION: This *must* be the absolute path if using sqlite3.
         # See: http://docs.djangoproject.com/en/dev/ref/settings/#name
816
817
818
819
         'NAME': 'snf_apps',
         'USER': 'synnefo',                      # Not used with sqlite3.
         'PASSWORD': 'example_passw0rd',         # Not used with sqlite3.
         # Set to empty string for localhost. Not used with sqlite3.
820
         'HOST': '203.0.113.1',
821
822
823
         # Set to empty string for default. Not used with sqlite3.
         'PORT': '5432',
     }
824
    }
Christos Stavrakakis's avatar
Christos Stavrakakis committed
825

826
827
828
Database Initialization
-----------------------

829
After configuration is done, we initialize the database by running:
830

831
832
.. code-block:: console

833
    # snf-manage syncdb
834
835
836

At this example we don't need to create a django superuser, so we select
``[no]`` to the question. After a successful sync, we run the migration needed
837
for Astakos:
838
839
840

.. code-block:: console

841
    # snf-manage migrate im
842
    # snf-manage migrate quotaholder_app
843

844
Then, we load the pre-defined user groups
Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
845
846
847

.. code-block:: console

848
    # snf-manage loaddata groups
Sofia Papagiannaki's avatar
Sofia Papagiannaki committed
849

850
851
852
853
854
.. _services-reg:

Services Registration
---------------------

855
When the database is ready, we need to register the services. The following
856
857
858
command will ask you to register the standard Synnefo components (Astakos,
Cyclades and Pithos) along with the services they provide. Note that you
have to register at least Astakos in order to have a usable authentication
859
860
861
862
863
system. For each component, you will be asked to provide two URLs: its base
URL and its UI URL.

The former is the location where the component resides; it should equal
the ``<component_name>_BASE_URL`` as specified in the respective component
864
settings. For example, the base URL for Astakos would be
865
866
867
868
869
870
871
872
873
``https://node1.example.com/astakos``.

The latter is the URL that appears in the Cloudbar and leads to the
component UI. If you want to follow the default setup, set
the UI URL to ``<base_url>/ui/`` where ``base_url`` the component's base
URL as explained before. (You can later change the UI URL with
``snf-manage component-modify <component_name> --url new_ui_url``.)

The command will also register automatically the resource definitions
874
offered by the services.
875
876
877

.. code-block:: console

878
    # snf-component-register
879
880
881
882

.. note::

   This command is equivalent to running the following series of commands;
883
   it registers the three components in Astakos and then in each host it
884
   exports the respective service definitions, copies the exported json file
885
   to the Astakos host, where it finally imports it:
886
887
888

    .. code-block:: console

889
890
891
       astakos-host$ snf-manage component-add astakos --base-url astakos_base_url --ui-url astakos_ui_url
       astakos-host$ snf-manage component-add cyclades --base-url cyclades_base_url --ui-url cyclades_ui_url
       astakos-host$ snf-manage component-add pithos --base-url pithos_base_url --ui-url pithos_ui_url
892
893
894
895
896
897
898
899
       astakos-host$ snf-manage service-export-astakos > astakos.json
       astakos-host$ snf-manage service-import --json astakos.json
       cyclades-host$ snf-manage service-export-cyclades > cyclades.json
       # copy the file to astakos-host
       astakos-host$ snf-manage service-import --json cyclades.json
       pithos-host$ snf-manage service-export-pithos > pithos.json
       # copy the file to astakos-host
       astakos-host$ snf-manage service-import --json pithos.json
900

901
Notice that in this installation astakos and cyclades are in node1 and pithos is in node2.
902

903
904
905
906
Setting Default Base Quota for Resources
----------------------------------------

We now have to specify the limit on resources that each user can employ
907
908
909
(exempting resources offered by projects). When specifying storage or
memory size limits consider to add an appropriate size suffix to the
numeric value, i.e. 10240 MB, 10 GB etc.
910
911
912

.. code-block:: console

913
    # snf-manage resource-modify --default-quota-interactive
914

915
916
917
918
919
920
921
922
923
924
925
926
927
.. _pithos_view_registration:

Register pithos view as an OAuth 2.0 client
-------------------------------------------

Starting from synnefo version 0.15, the pithos view, in order to get access to
the data of a protect pithos resource, has to be granted authorization for the
specific resource by astakos.

During the authorization grant procedure, it has to authenticate itself with
astakos since the later has to prevent serving requests by unknown/unauthorized
clients.

928
929
930
931
932
933
934
935
936
Each oauth 2.0 client is identified by a client identifier (client_id).
Moreover, the confidential clients are authenticated via a password
(client_secret).
Then, each client has to declare at least a redirect URI so that astakos will
be able to validate the redirect URI provided during the authorization code
request.
If a client is trusted (like a pithos view) astakos grants access on behalf
of the resource owner, otherwise the resource owner has to be asked.

937
938
939
940
To register the pithos view as an OAuth 2.0 client in astakos, we have to run
the following command::

    snf-manage oauth2-client-add pithos-view --secret=<secret> --is-trusted --url https://node2.example.com/pithos/ui/view
941
942
943
944
945
946
947
948

Servers Initialization
----------------------

Finally, we initialize the servers on node1:

.. code-block:: console

949
950
    root@node1:~ # /etc/init.d/gunicorn restart
    root@node1:~ # /etc/init.d/apache2 restart
951
952

We have now finished the Astakos setup. Let's test it now.
953
954
955
956
957
958
959


Testing of Astakos
==================

Open your favorite browser and go to:

Giorgos Korfiatis's avatar
Giorgos Korfiatis committed
960
``http://node1.example.com/astakos``
961

Giorgos Korfiatis's avatar
Giorgos Korfiatis committed
962
If this redirects you to ``https://node1.example.com/astakos/ui/`` and you can see
963
964
965
966
967
the "welcome" door of Astakos, then you have successfully setup Astakos.

Let's create our first user. At the homepage click the "CREATE ACCOUNT" button
and fill all your data at the sign up form. Then click "SUBMIT". You should now
see a green box on the top, which informs you that you made a successful request
968
969
and the request has been sent to the administrators. So far so good, let's
assume that you created the user with username ``user@example.com``.
970
971
972
973
974

Now we need to activate that user. Return to a command prompt at node1 and run:

.. code-block:: console

975
    root@node1:~ # snf-manage user-list
976
977

This command should show you a list with only one user; the one we just created.
Giorgos Korfiatis's avatar
Giorgos Korfiatis committed
978
979
This user should have an id with a value of ``1`` and flag "active" and
"verified" set to False. Now run:
980
981
982

.. code-block:: console

Giorgos Korfiatis's avatar
Giorgos Korfiatis committed
983
    root@node1:~ # snf-manage user-modify 1 --verify --accept
984

Giorgos Korfiatis's avatar
Giorgos Korfiatis committed
985
This verifies the user email and activates the user.
986
987
988
When running in production, the activation is done automatically with different
types of moderation, that Astakos supports. You can see the moderation methods
(by invitation, whitelists, matching regexp, etc.) at the Astakos specific
989
990
991
documentation. In production, you can also manually activate a user, by sending
him/her an activation email. See how to do this at the :ref:`User
activation <user_activation>` section.
992

Giorgos Korfiatis's avatar
Giorgos Korfiatis committed
993
Now let's go back to the homepage. Open ``http://node1.example.com/astkos/ui/`` with
994
your browser again. Try to sign in using your new credentials. If the Astakos
995
996
997
menu appears and you can see your profile, then you have successfully setup
Astakos.

998
Let's continue to install Pithos now.
999
1000


1001
1002
Installation of Pithos on node2
===============================
1003

1004
To install Pithos, grab the packages from our repository (make sure  you made
1005
1006
1007
1008
1009
the additions needed in your ``/etc/apt/sources.list`` file, as described
previously), by running:

.. code-block:: console

1010
   # apt-get install snf-pithos-app snf-pithos-backend
1011

1012
Now, install the pithos web interface:
1013

1014
1015
1016
1017
.. code-block:: console

   # apt-get install snf-pithos-webclient

1018
1019
This package provides the standalone Pithos web client. The web client is the
web UI for Pithos and will be accessible by clicking "Pithos" on the Astakos
1020
interface's cloudbar, at the top of the Astakos homepage.
1021

1022

1023
1024
.. _conf-pithos:

1025
1026
Configuration of Pithos
=======================
1027

1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
Gunicorn setup
--------------

Copy the file ``/etc/gunicorn.d/synnefo.example`` to
``/etc/gunicorn.d/synnefo``, to make it a valid gunicorn configuration file
(as happened for node1):

.. code-block:: console

    # cp /etc/gunicorn.d/synnefo.example /etc/gunicorn.d/synnefo


.. warning:: Do NOT start the server yet, because it won't find the
    ``synnefo.settings`` module. Also, in case you are using ``/etc/hosts``
    instead of a DNS to get the hostnames, change ``--worker-class=gevent`` to
    ``--worker-class=sync``. We will start the server after successful
    installation of Astakos. If the server is running::

       # /etc/init.d/gunicorn stop

1048
1049
1050
Conf Files
----------

1051
After Pithos is successfully installed, you will find the directory
1052
``/etc/synnefo`` and some configuration files inside it, as you did in node1
1053
after installation of Astakos. Here, you will not have to change anything that
1054
has to do with snf-common or snf-webproject. Everything is set at node1. You
1055
only need to change settings that have to do with Pithos. Specifically:
1056
1057

Edit ``/etc/synnefo/20-snf-pithos-app-settings.conf``. There you need to set
Ilias Tsitsimpis's avatar
Ilias Tsitsimpis committed
1058
this options:
1059
1060
1061

.. code-block