admin-guide.rst 39.3 KB
Newer Older
1
2
3
4
5
6
7
8
9
.. _admin-guide:

Synnefo Administrator's Guide
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is the complete Synnefo Administrator's Guide.



10
11
12
13
14
15
16
17
18
General Synnefo Architecture
============================

The following graph shows the whole Synnefo architecture and how it interacts
with multiple Ganeti clusters. We hope that after reading the Administrator's
Guide you will be able to understand every component and all the interactions
between them. It is a good idea to first go through the Quick Administrator's
Guide before proceeding.

19
.. image:: images/synnefo-arch2.png
20
   :width: 100%
21
   :target: _images/synnefo-arch2.png
22
23


24

25
26
Identity Service (Astakos)
==========================
27
28


29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
Overview
--------

Authentication methods
~~~~~~~~~~~~~~~~~~~~~~

Local Authentication
````````````````````

LDAP Authentication
```````````````````

.. _shibboleth-auth:

Shibboleth Authentication
`````````````````````````

Astakos can delegate user authentication to a Shibboleth federation.

To setup shibboleth, install package::

  apt-get install libapache2-mod-shib2

Change appropriately the configuration files in ``/etc/shibboleth``.

Add in ``/etc/apache2/sites-available/synnefo-ssl``::

  ShibConfig /etc/shibboleth/shibboleth2.xml
  Alias      /shibboleth-sp /usr/share/shibboleth

  <Location /im/login/shibboleth>
    AuthType shibboleth
    ShibRequireSession On
    ShibUseHeaders On
    require valid-user
  </Location>

and before the line containing::

  ProxyPass        / http://localhost:8080/ retry=0

add::

  ProxyPass /Shibboleth.sso !
73

74
75
76
Then, enable the shibboleth module::

  a2enmod shib2
77

78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
After passing through the apache module, the following tokens should be
available at the destination::

  eppn # eduPersonPrincipalName
  Shib-InetOrgPerson-givenName
  Shib-Person-surname
  Shib-Person-commonName
  Shib-InetOrgPerson-displayName
  Shib-EP-Affiliation
  Shib-Session-ID

Finally, add 'shibboleth' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Architecture
------------

Prereqs
-------

Installation
------------

Configuration
-------------

Working with Astakos
--------------------

User activation methods
~~~~~~~~~~~~~~~~~~~~~~~
109
110
111
112
113
114

When a new user signs up, he/she is not marked as active. You can see his/her
state by running (on the machine that runs the Astakos app):

.. code-block:: console

115
   $ snf-manage user-list
116
117

There are two different ways to activate a new user. Both need access to a
118
running :ref:`mail server <mail-server>`.
119
120

Manual activation
121
`````````````````
122
123
124
125
126
127
128
129

You can manually activate a new user that has already signed up, by sending
him/her an activation email. The email will contain an approriate activation
link, which will complete the activation process if followed. You can send the
email by running:

.. code-block:: console

130
   $ snf-manage user-activation-send <user ID or email>
131

132
Be sure to have already setup your mail server and defined it in your Synnefo
133
134
135
settings, before running the command.

Automatic activation
136
137
138
139
````````````````````

FIXME: Describe Regex activation method

140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
Setting quota limits
~~~~~~~~~~~~~~~~~~~~

Set default quotas
``````````````````

In 20-snf-astakos-app-settings.conf, 
uncomment the default setting ``ASTAKOS_SERVICES``
and customize the ``'uplimit'`` values.
These are the default base quotas for all users.

To apply your configuration run::

    # snf-manage astakos-init --load-service-resources
    # snf-manage astakos-quota --sync

Set base quotas for individual users
````````````````````````````````````

For individual users that need different quotas than the default
you can set it for each resource like this::

    # use this to display quotas / uuid
    # snf-manage user-show 'uuid or email'

    # snf-manage user-set-initial-quota --set-capacity 'user-uuid' 'cyclades.vm' 10

    # this applies the configuration
    # snf-manage astakos-quota --sync --user 'user-uuid'


Enable the Projects feature
~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you want to enable the projects feature so that users may apply
on their own for resources by creating and joining projects,
in ``20-snf-astakos-app-settings.conf`` set::

    # this will allow at most one pending project application per user
    ASTAKOS_PENDING_APPLICATION_LIMIT = 1
    # this will make the 'projects' page visible in the dashboard
    ASTAKOS_PROJECTS_VISIBLE = True

183
184
185
186
187
You can specify a user-specific limit on pending project applications
with::

    # snf-manage user-update <user id> --max-pending-projects=2

188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
When users apply for projects they are not automatically granted
the resources. They must first be approved by the administrator.

To list pending project applications in astakos::

    # snf-manage project-list --pending

Note the last column, the application id. To approve it::

    # <app id> from the last column of project-list
    # snf-manage project-control --approve <app id>

To deny an application::

    # snf-manage project-control --deny <app id>

204
205
206
207
208
209
Users designated as *project admins* can approve, deny, or modify
an application through the web interface. In
``20-snf-astakos-app-settings.conf`` set::

    # UUIDs of users that can approve or deny project applications from the web.
    ASTAKOS_PROJECT_ADMINS = [<uuid>, ...]
210
211


212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
Astakos advanced operations
---------------------------

Adding "Terms of Use"
~~~~~~~~~~~~~~~~~~~~~

Astakos supports versioned terms-of-use. First of all you need to create an
html file that will contain your terms. For example, create the file
``/usr/share/synnefo/sample-terms.html``, which contains the following:

.. code-block:: console

   <h1>~okeanos terms</h1>

   These are the example terms for ~okeanos

Then, add those terms-of-use with the snf-manage command:

.. code-block:: console

   $ snf-manage term-add /usr/share/synnefo/sample-terms.html

Your terms have been successfully added and you will see the corresponding link
appearing in the Astakos web pages' footer.

237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
Enabling reCAPTCHA
~~~~~~~~~~~~~~~~~~

Astakos supports the `reCAPTCHA <http://www.google.com/recaptcha>`_ feature.
If enabled, it protects the Astakos forms from bots. To enable the feature, go
to https://www.google.com/recaptcha/admin/create and create your own reCAPTCHA
key pair. Then edit ``/etc/synnefo/20-snf-astakos-app-settings.conf`` and set
the corresponding variables to reflect your newly created key pair. Finally, set
the ``ASTAKOS_RECAPTCHA_ENABLED`` variable to ``True``:

.. code-block:: console

   ASTAKOS_RECAPTCHA_PUBLIC_KEY = 'example_recaptcha_public_key!@#$%^&*('
   ASTAKOS_RECAPTCHA_PRIVATE_KEY = 'example_recaptcha_private_key!@#$%^&*('

   ASTAKOS_RECAPTCHA_ENABLED = True

Restart the service on the Astakos node(s) and you are ready:

.. code-block:: console

   # /etc/init.d/gunicorn restart

Checkout your new Sign up page. If you see the reCAPTCHA box, you have setup
everything correctly.

263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305


File Storage Service (Pithos)
=============================

Overview
--------

Architecture
------------

Prereqs
-------

Installation
------------

Configuration
-------------

Working with Pithos
-------------------

Pithos advanced operations
--------------------------



Compute/Network/Image Service (Cyclades)
========================================

Compute Overview
----------------

Network Overview
----------------

Image Overview
--------------

Architecture
------------

306
Asynchronous communication with Ganeti backends
Christos Stavrakakis's avatar
Christos Stavrakakis committed
307
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
308
309
310
311
312
313
314
Synnefo uses Google Ganeti backends for VM cluster management. In order for
Cyclades to be able to handle thousands of user requests, Cyclades and Ganeti
communicate asynchronously. Briefly, requests are submitted to Ganeti through
Ganeti's RAPI/HTTP interface, and then asynchronous notifications about the
progress of Ganeti jobs are being created and pushed upwards to Cyclades. The
architecture and communication with a Ganeti backend is shown in the graph
below:
315
316
317
318

.. image:: images/cyclades-ganeti-communication.png
   :width: 50%
   :target: _images/cyclades-ganeti-communication.png
319

320
The Cyclades API server is responsible for handling user requests. Read-only
321
322
requests are directly served by looking up the Cyclades DB. If the request
needs an action in the Ganeti backend, Cyclades submit jobs to the Ganeti
323
324
master using the `Ganeti RAPI interface
<http://docs.ganeti.org/ganeti/2.2/html/rapi.html>`_.
325

326
327
While Ganeti executes the job, `snf-ganeti-eventd`, `snf-ganeti-hook` and
`snf-progress-monitor` are monitoring the progress of the job and send
328
corresponding messages to the RabbitMQ servers. These components are part
329
of `snf-cyclades-gtools` and must be installed on all Ganeti nodes. Specially:
330
331

* *snf-ganeti-eventd* sends messages about operations affecting the operating
332
333
  state of instances and networks. Works by monitoring the Ganeti job queue.
* *snf-ganeti_hook* sends messages about the NICs of instances. It includes a
Christos Stavrakakis's avatar
Christos Stavrakakis committed
334
335
  number of `Ganeti hooks <http://docs.ganeti.org/ganeti/2.2/html/hooks.html>`_
  for customisation of operations.
336
337
* *snf-progress_monitor* sends messages about the progress of the Image deployment
  phase which is done by the Ganeti OS Definition `snf-image`.
338

339
340
341
Finally, `snf-dispatcher` consumes messages from the RabbitMQ queues, processes
these messages and properly updates the state of the Cyclades DB. Subsequent
requests to the Cyclades API, will retrieve the updated state from the DB.
342
343


344
345
346
Prereqs
-------

347
348
Work in progress. Please refer to :ref:`quick administrator quide <quick-install-admin-guide>`.

349
350
351
Installation
------------

352
353
Work in progress. Please refer to :ref:`quick administrator quide <quick-install-admin-guide>`.

354
355
356
Configuration
-------------

357
358
Work in progress. Please refer to :ref:`quick administrator quide <quick-install-admin-guide>`.

359
360
361
Working with Cyclades
---------------------

362
Managing Ganeti Backends
Christos Stavrakakis's avatar
Christos Stavrakakis committed
363
~~~~~~~~~~~~~~~~~~~~~~~~
364

365
366
367
Since v0.11, Synnefo is able to manage multiple Ganeti clusters (backends)
making it capable to scale linearly to tens of thousands of VMs. Backends
can be dynamically added or removed via `snf-manage` commands.
368

369
370
371
372
373
Each newly created VM is allocated to a Ganeti backend by the Cyclades backend
allocator. The VM is "pinned" to this backend, and can not change through its
lifetime. The backend allocator decides in which backend to spawn the VM based
on the available resources of each backend, trying to balance the load between
them.
374
375

Handling of Networks, as far as backends are concerned, is based on whether the
376
377
378
379
network is public or not. Public networks are created through the `snf-manage
network-create` command, and are only created on one backend. Private networks
are created on all backends, in order to ensure that VMs residing on different
backends can be connected to the same private network.
380

381
382
383
Listing existing backends
`````````````````````````
To list all the Ganeti backends known to Synnefo, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
384

385
386
387
388
389
.. code-block:: console

   $ snf-manage backend-list

Adding a new Ganeti backend
Christos Stavrakakis's avatar
Christos Stavrakakis committed
390
```````````````````````````
391
392
Backends are dynamically added under the control of Synnefo with `snf-manage
backend-add` command. In this section it is assumed that a Ganeti cluster,
393
394
named ``cluster.example.com`` is already up and running and configured to be
able to host Synnefo VMs.
395

396
To add this Ganeti cluster, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
397

398
399
400
401
.. code-block:: console

   $ snf-manage backend-add --clustername=cluster.example.com --user="synnefo_user" --pass="synnefo_pass"

402
403
404
405
406
where ``clustername`` is the Cluster hostname of the Ganeti cluster, and
``user`` and ``pass`` are the credentials for the `Ganeti RAPI user
<http://docs.ganeti.org/ganeti/2.2/html/rapi.html#users-and-passwords>`_.  All
backend attributes can be also changed dynamically using the `snf-manage
backend-modify` command.
407

408
409
410
``snf-manage backend-add`` will also create all existing private networks to
the new backend. You can verify that the backend is added, by running
`snf-manage backend-list`.
411

412
413
414
415
416
417
418
419
420
421
Note that no VMs will be spawned to this backend, since by default it is in a
``drained`` state after addition and also it has no public network assigned to
it.

So, first you need to create its public network, make sure everything works as
expected and finally make it active by un-setting the ``drained`` flag. You can
do this by running:

.. code-block:: console

422
   $ snf-manage backend-modify --drained=False <backend_id>
423
424

Removing an existing Ganeti backend
Christos Stavrakakis's avatar
Christos Stavrakakis committed
425
```````````````````````````````````
426
In order to remove an existing backend from Synnefo, we run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
427

428
429
.. code-block:: console

430
   # snf-manage backend-remove <backend_id>
431

432
433
434
This command will fail if there are active VMs on the backend. Also, the
backend is not cleaned before removal, so all the Synnefo private networks
will be left on the Ganeti nodes. You need to remove them manually.
435

436
437
438
439
440
441
Allocation of VMs in Ganeti backends
````````````````````````````````````
As already mentioned, the Cyclades backend allocator is responsible for
allocating new VMs to backends. This allocator does not choose the exact Ganeti
node that will host the VM but just the Ganeti backend. The exact node is
chosen by the Ganeti cluster's allocator (hail).
442
443
444
445

The decision about which backend will host a VM is based on the available
resources. The allocator computes a score for each backend, that shows its load
factor, and the one with the minimum score is chosen. The admin can exclude
446
backends from the allocation phase by marking them as ``drained`` by running:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
447

448
449
.. code-block:: console

450
   $ snf-manage backend-modify --drained=True <backend_id>
451
452

The backend resources are periodically updated, at a period defined by
453
the ``BACKEND_REFRESH_MIN`` setting, or by running `snf-manage backend-update-status`
454
command. It is advised to have a cron job running this command at a smaller
455
interval than ``BACKEND_REFRESH_MIN`` in order to remove the load of refreshing
456
457
the backends stats from the VM creation phase.

458
459
460
Finally, the admin can decide to have a user's VMs being allocated to a
specific backend, with the ``BACKEND_PER_USER`` setting. This is a mapping
between users and backends. If the user is found in ``BACKEND_PER_USER``, then
461
462
Synnefo allocates all his/hers VMs to the specific backend in the variable,
even if is marked as drained (useful for testing).
463

464
465
466
467

Managing Virtual Machines
~~~~~~~~~~~~~~~~~~~~~~~~~

468
469
470
471
472
473
As mentioned, Cyclades uses Ganeti for management of VMs. The administrator can
handle Cyclades VMs just like any other Ganeti instance, via `gnt-instance`
commands. All Ganeti instances that belong to Synnefo, are separated from
others, by a prefix in their names. This prefix is defined in
``BACKEND_PREFIX_ID`` setting in
``/etc/synnefo/20-snf-cyclades-app-backend.conf``.
474

475
Apart from handling instances directly in the Ganeti level, a number of `snf-manage`
476
477
commands are available:

478
479
480
481
482
483
* ``snf-manage server-list``: List servers
* ``snf-manage server-show``: Show information about a server in the Cyclades DB
* ``snf-manage server-inspect``: Inspect the state of a server both in DB and Ganeti
* ``snf-manage server-modify``: Modify the state of a server in the Cycldes DB
* ``snf-manage server-create``: Create a new server
* ``snf-manage server-import``: Import an existing Ganeti instance to Cyclades
484
485
486
487
488


Managing Virtual Networks
~~~~~~~~~~~~~~~~~~~~~~~~~

489
490
491
492
Cyclades is able to create and manage Virtual Networks. Networking is
desployment specific and must be customized based on the specific needs of the
system administrator. For better understanding of networking please refer to
the :ref:`Network <networks>` section.
493
494
495

Exactly as Cyclades VMs can be handled like Ganeti instances, Cyclades Networks
can also by handled as Ganeti networks, via `gnt-network commands`. All Ganeti
496
networks that belong to Synnefo are named with the prefix
497
498
`${BACKEND_PREFIX_ID}-net-`.

499
There are also the following `snf-manage` commands for managing networks:
500

501
502
503
504
505
506
* ``snf-manage network-list``: List networks
* ``snf-manage network-show``: Show information about a network in the Cyclades DB
* ``snf-manage network-inspect``: Inspect the state of the network in DB and Ganeti backends
* ``snf-manage network-modify``: Modify the state of a network in the Cycldes DB
* ``snf-manage network-create``: Create a new network
* ``snf-manage network-remove``: Remove an existing network
507

508
Managing Network Resources
509
``````````````````````````
510

511
512
513
Proper operation of the Cyclades Network Service depends on the unique
assignment of specific resources to each type of virtual network. Specifically,
these resources are:
514

515
516
517
518
519
* IP addresses. Cyclades creates a Pool of IPs for each Network, and assigns a
  unique IP address to each VM, thus connecting it to this Network. You can see
  the IP pool of each network by running `snf-manage network-inspect
  <network_ID>`. IP pools are automatically created and managed by Cyclades,
  depending on the subnet of the Network.
520
* Bridges corresponding to physical VLANs, which are required for networks of
521
522
523
  type `PRIVATE_PHYSICAL_VLAN`.
* One Bridge corresponding to one physical VLAN which is required for networks of
  type `PRIVATE_MAC_PREFIX`.
524

525
526
Cyclades allocates those resources from pools that are created by the
administrator with the `snf-manage pool-create` management command.
527

528
529
530
Pool Creation
`````````````
Pools are created using the `snf-manage pool-create` command:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
531

532
533
534
535
536
537
538
.. code-block:: console

   # snf-manage pool-create --type=bridge --base=prv --size=20

will create a pool of bridges, containing bridges prv1, prv2,..prv21.

You can verify the creation of the pool, and check its contents by running:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
539

540
541
542
543
544
.. code-block:: console

   # snf-manage pool-list
   # snf-manage pool-show --type=bridge 1

545
With the same commands you can handle a pool of MAC prefixes. For example:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
546

547
548
549
550
.. code-block:: console

   # snf-manage pool-create --type=mac-prefix --base=aa:00:0 --size=65536

551
552
553
554
will create a pool of MAC prefixes from ``aa:00:1`` to ``b9:ff:f``. The MAC
prefix pool is responsible for providing only unicast and locally administered
MAC addresses, so many of these prefixes will be externally reserved, to
exclude from allocation.
555

556
557
558
559
560
Cyclades advanced operations
----------------------------

Reconciliation mechanism
~~~~~~~~~~~~~~~~~~~~~~~~
561

562
563
564
565
566
567
On certain occasions, such as a Ganeti or RabbitMQ failure, the state of
Cyclades database may differ from the real state of VMs and networks in the
Ganeti backends. The reconciliation process is designed to synchronize
the state of the Cyclades DB with Ganeti. There are two management commands
for reconciling VMs and Networks

568
569
570
Reconciling Virtual Machines
````````````````````````````

571
Reconciliation of VMs detects the following conditions:
572

573
574
 * Stale DB servers without corresponding Ganeti instances
 * Orphan Ganeti instances, without corresponding DB entries
575
 * Out-of-sync state for DB entries wrt to Ganeti instances
576

577
To detect all inconsistencies you can just run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
578

579
.. code-block:: console
580
581

  $ snf-manage reconcile-servers
582

583
Adding the `--fix-all` option, will do the actual synchronization:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
584

585
.. code-block:: console
586
587

  $ snf-manage reconcile --fix-all
588
589
590
591

Please see ``snf-manage reconcile --help`` for all the details.


592
Reconciling Networks
Christos Stavrakakis's avatar
Christos Stavrakakis committed
593
````````````````````
594

595
Reconciliation of Networks detects the following conditions:
596

597
598
599
600
601
602
  * Stale DB networks without corresponding Ganeti networks
  * Orphan Ganeti networks, without corresponding DB entries
  * Private networks that are not created to all Ganeti backends
  * Unsynchronized IP pools

To detect all inconsistencies you can just run:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
603

604
.. code-block:: console
605

606
607
608
  $ snf-manage reconcile-networks

Adding the `--fix-all` option, will do the actual synchronization:
Christos Stavrakakis's avatar
Christos Stavrakakis committed
609

610
.. code-block:: console
611

612
613
614
  $ snf-manage reconcile-networks --fix-all

Please see ``snf-manage reconcile-networks --help`` for all the details.
615
616


617

618
619
620
621
622
Block Storage Service (Archipelago)
===================================

Overview
--------
623
624
625
626
627
628
Archipelago offers Copy-On-Write snapshotable volumes. Pithos images can be used
to provision a volume with Copy-On-Write semantics (i.e. a clone). Snapshots
offer a unique deduplicated image of a volume, that reflects the volume state
during snapshot creation and are indistinguishable from a Pithos image.

Archipelago is used by Cyclades and Ganeti for fast provisioning of VMs based on
629
630
CoW volumes. Moreover, it enables live migration of thinly-provisioned VMs with
no physically shared storage.
631

632
633
634
Archipelago Architecture
------------------------

635
636
637
.. image:: images/archipelago-architecture.png
   :width: 50%
   :target: _images/archipelago-architecture.png
638

639
640
641
642
643
644
645
646
647
.. _syn+archip+rados:

Overview of Synnefo + Archipelago + RADOS
-----------------------------------------

.. image:: images/synnefo-arch3.png
   :width: 100%
   :target: _images/synnefo-arch3.png

648
649
Prereqs
-------
650

651
652
653
654
655
656
657
658
659
660
661
662
663
The administrator must initialize the storage backend where archipelago volume
blocks will reside.

In case of a files backend, the administrator must create two directories. One
for the archipelago data blocks and one for the archipelago map blocks. These
should probably be over shared storage to enable sharing archipelago volumes
between multiple nodes. He or she, must also be able to supply a directory where
the pithos data and map blocks reside.

In case of a RADOS backend, the administrator must create two rados pools, one
for data blocks, and one for the map blocks. These pools, must be the same pools
used in pithos, in order to enable volume creation based on pithos images.

664
665
Installation
------------
666

667
668
669
Archipelago consists of

* ``libxseg0``: libxseg used to communicate over shared memory segments
670
* ``python-xseg``: python bindings for libxseg
671
672
* ``archipelago-kernel-dkms``: contains archipelago kernel modules to provide
  block devices to be used as vm disks
673
674
* ``python-archipelago``: archipelago python module. Includes archipelago and
  vlmc functionality.
675
676
677
678
679
680
681
682
683
684
685
686
687
688
* ``archipelago``: user space tools and peers for the archipelago management and
  volume composition
* ``archipelago-ganeti``: ganeti ext storage scripts, that enable ganeti to
  provision VMs over archipelago

Performing

.. code-block:: console

  $ apt-get install archipelago-ganeti 

should fetch all the required packages and get you up 'n going with archipelago

Bare in mind, that custom librados is required, which is provided in the apt
689
repo of GRNet.
690
691


692
693
For now, librados is a dependency of archipelago, even if you do not intend to
use archipelago over RADOS.
694
695
696

Configuration
-------------
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
Archipelago should work out of the box with a RADOS backend, but basic
configuration can be done in ``/etc/default/archipelago`` .

If you wish to change the storage backend to files, set

.. code-block:: console

   STORAGE="files"

and provide the appropriate settings for files storage backend in the conf file.

These are:

* ``FILED_IMAGES``: directory for archipelago data blocks.
* ``FILED_MAPS``: directory for archipelago map blocks.
* ``PITHOS``: directory of pithos data blocks.
* ``PITHOSMAPS``: directory of pithos map blocks.

The settings for RADOS storage backend are:

* ``RADOS_POOL_MAPS``: The pool where archipelago and pithos map blocks reside.
* ``RADOS_POOL_BLOCKS``: The pool where archipelago and pithos data blocks
719
  reside.
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743

Examples can be found in the conf file.

Be aware that archipelago infrastructure doesn't provide default values for this
settings. If they are not set in the conf file, archipelago will not be able to
function.

Archipelago also provides ``VERBOSITY`` config options to control the output
generated by the userspace peers.

The available options are:

* ``VERBOSITY_BLOCKERB``
* ``VERBOSITY_BLOCKERM``
* ``VERBOSITY_MAPPER``
* ``VERBOSITY_VLMC``

and the available values are:

* 0 : Error only logging.
* 1 : Warning logging.
* 2 : Info logging.
* 3 : Debug logging. WARNING: This options produces tons of output, but the
  logrotate daemon should take care of it.
744
745
746
747

Working with Archipelago
------------------------

748
``archipelago`` provides basic functionality for archipelago.
749
750
751
752
753
754
755
756

Usage:

.. code-block:: console

  $ archipelago [-u] command


757
758
Currently it supports the following commands:

759
760
761
762
763
764
* ``start [peer]``
  Starts archipelago or the specified peer.
* ``stop [peer]``
  Stops archipelago or the specified peer.
* ``restart [peer]``
  Restarts archipelago or the specified peer.
765
* ``status``
766
767
768
769
  Show the status of archipelago.

Available peers: ``blockerm``, ``blockerb``, ``mapperd``, ``vlmcd``.

770
771
772
773
774
775

``start``, ``stop``, ``restart`` can be combined with the ``-u / --user`` option
to affect only the userspace peers supporting archipelago.



776
777
Archipelago advanced operations
-------------------------------
778
779
780
The ``vlmc`` tool provides a way to interact with archipelago volumes

* ``vlmc map <volumename>``: maps the volume to a xsegbd device.
781

782
* ``vlmc unmap </dev/xsegbd[1-..]>``: unmaps the specified device from the
783
  system.
784

785
786
* ``vlmc create <volumename> --snap <snapname> --size <size>``: creates a new
  volume named <volumename> from snapshot name <snapname> with size <size>.
787
788
  The ``--snap`` and ``--size`` are optional, but at least one of them is
  mandatory. e.g:
789

790
791
792
  ``vlmc create <volumename> --snap <snapname>`` creates a volume named
  volumename from snapshot snapname. The size of the volume is the same as
  the size of the snapshot.
793

794
795
  ``vlmc create <volumename> --size <size>`` creates an empty volume of size
  <size> named <volumename>.
796

797
798
* ``vlmc remove <volumename>``: removes the volume and all the related
  archipelago blocks from storage.
799

800
801
* ``vlmc list``: provides a list of archipelago volumes. Currently only works
  with RADOS storage backend.
802

803
804
805
* ``vlmc info <volumename>``: shows volume information. Currently returns only
  volume size.

806
807
808
809
810
811
812
813
814
815
816
817
818
819
* ``vlmc open <volumename>``: opens an archipelago volume. That is, taking all
  the necessary locks and also make the rest of the infrastructure aware of the
  operation.

  This operation succeeds if the volume is alread opened.

* ``vlmc close <volumename>``: closes an archipelago volume. That is, performing
  all the necessary functions in the insfrastrure to successfully release the
  volume. Also releases all the acquired locks.

  ``vlmc close`` should be performed after a ``vlmc open`` operation.

* ``vlmc lock <volumename>``: locks a volume. This step allow the administrator
  to lock an archipelago volume, independently from the rest of the
820
  infrastrure.
821
822
823
824
825
826
827
828

* ``vlmc unlock [-f] <volumename>``: unlocks a volume. This allow the
  administrator to unlock a volume, independently from the rest of the
  infrastructure.
  The unlock option can be performed only by the blocker that acquired the lock
  in the first place. To unlock a volume from another blocker, ``-f`` option
  must be used to break the lock.

829

830
The "kamaki" API client
831
=======================
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906

To upload, register or modify an image you will need the **kamaki** tool.
Before proceeding make sure that it is configured properly. Verify that
*image_url*, *storage_url*, and *token* are set as needed:

.. code-block:: console

   $ kamaki config list

To chage a setting use ``kamaki config set``:

.. code-block:: console

   $ kamaki config set image_url https://cyclades.example.com/plankton
   $ kamaki config set storage_url https://pithos.example.com/v1
   $ kamaki config set token ...

Upload Image
------------

As a shortcut, you can configure a default account and container that will be
used by the ``kamaki store`` commands:

.. code-block:: console

   $ kamaki config set storage_account images@example.com
   $ kamaki config set storage_container images

If the container does not exist, you will have to create it before uploading
any images:

.. code-block:: console

   $ kamaki store create images

You are now ready to upload an image. You can upload it with a Pithos+ client,
or use kamaki directly:

.. code-block:: console

   $ kamaki store upload ubuntu.iso

You can use any Pithos+ client to verify that the image was uploaded correctly.
The full Pithos URL for the previous example will be
``pithos://images@example.com/images/ubuntu.iso``.


Register Image
--------------

To register an image you will need to use the full Pithos+ URL. To register as
a public image the one from the previous example use:

.. code-block:: console

   $ kamaki glance register Ubuntu pithos://images@example.com/images/ubuntu.iso --public

The ``--public`` flag is important, if missing the registered image will not
be listed by ``kamaki glance list``.

Use ``kamaki glance register`` with no arguments to see a list of available
options. A more complete example would be the following:

.. code-block:: console

   $ kamaki glance register Ubuntu pithos://images@example.com/images/ubuntu.iso \
            --public --disk-format diskdump --property kernel=3.1.2

To verify that the image was registered successfully use:

.. code-block:: console

   $ kamaki glance list -l


907
908
909
910

Miscellaneous
=============

911
.. RabbitMQ
912

913
914
915
916
917
918
RabbitMQ Broker
---------------

Queue nodes run the RabbitMQ sofware, which provides AMQP functionality. To
guarantee high-availability, more than one Queue nodes should be deployed, each
of them belonging to the same `RabbitMQ cluster
919
920
921
922
923
924
925
926
927
928
929
<http://www.rabbitmq.com/clustering.html>`_. Synnefo uses the RabbitMQ
active/active `High Available Queues <http://www.rabbitmq.com/ha.html>`_ which
are mirrored between two nodes within a RabbitMQ cluster.

The RabbitMQ nodes that form the cluster, are declared to Synnefo through the
`AMQP_HOSTS` setting. Each time a Synnefo component needs to connect to
RabbitMQ, one of these nodes is chosen in a random way. The client that Synnefo
uses to connect to RabbitMQ, handles connection failures transparently and
tries to reconnect to a different node. As long as one of these nodes are up
and running, functionality of Synnefo should not be downgraded by the RabbitMQ
node failures.
930
931

All the queues that are being used are declared as durable, meaning that
932
933
messages are persistently stored to RabbitMQ, until they get successfully
processed by a client.
934
935
936

Currently, RabbitMQ is used by the following components:

937
* `snf-ganeti-eventd`, `snf-ganeti-hook` and `snf-progress-monitor`:
938
939
  These components send messages concerning the status and progress of
  jobs in the Ganeti backend.
940
941
* `snf-dispatcher`: This daemon, consumes the messages that are sent from
  the above components, and updates the Cyclades DB accordingly.
942

Christos Stavrakakis's avatar
Christos Stavrakakis committed
943

944
Installation
945
946
~~~~~~~~~~~~

947
948
949
950
951
952
Please check the RabbitMQ documentation which covers extensively the
`installation of RabbitMQ server <http://www.rabbitmq.com/download.html>`_ and
the setup of a `RabbitMQ cluster <http://www.rabbitmq.com/clustering.html>`_.
Also, check out the `web management plugin
<http://www.rabbitmq.com/management.html>`_ that can be useful for managing and
monitoring RabbitMQ.
953
954

For a basic installation of RabbitMQ on two nodes (node1 and node2) you can do
955
the following:
956

957
On both nodes, install rabbitmq-server and create a Synnefo user:
958
959
960
961
962
963
964

.. code-block:: console

  $ apt-get install rabbitmq-server
  $ rabbitmqctl add_user synnefo "example_pass"
  $ rabbitmqctl set_permissions synnefo  ".*" ".*" ".*"

965
Also guarantee that both nodes share the same cookie, by running:
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996

.. code-block:: console

  $ scp node1:/var/lib/rabbitmq/.erlang.cookie node2:/var/lib/rabbitmq/.erlang.cookie

and restart the nodes:

.. code-block:: console

  $ /etc/init.d/rabbitmq-server restart


To setup the RabbitMQ cluster run:

.. code-block:: console

  root@node2: rabbitmqctl stop_app
  root@node2: rabbitmqctl reset
  root@node2: rabbitmqctl cluster rabbit@node1 rabbit@node2
  root@node2: rabbitmqctl start_app

You can verify that the cluster is set up correctly by running:

.. code-block:: console

  root@node2: rabbitmqctl cluster_status





997
998
999
1000
1001
1002
1003
1004
Admin tool: snf-manage
----------------------

``snf-manage`` is a tool used to perform various administrative tasks. It needs
to be able to access the django database, so the following should be able to
import the Django settings.

Additionally, administrative tasks can be performed via the admin web interface
1005
located in /admin. Only users of type ADMIN can access the admin pages. To
1006
change the type of a user to ADMIN, snf-manage can be used:
1007
1008
1009

.. code-block:: console

1010
   $ snf-manage user-modify 42 --type ADMIN
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022

Logging
-------

Logging in Synnefo is using Python's logging module. The module is configured
using dictionary configuration, whose format is described here:

http://docs.python.org/release/2.7.1/library/logging.html#logging-config-dictschema

Note that this is a feature of Python 2.7 that we have backported for use in
Python 2.6.

1023
The logging configuration dictionary is defined in
1024
``/etc/synnefo/10-snf-webproject-logging.conf``
1025

1026
1027
1028
1029
The administrator can have finer logging control by modifying the
``LOGGING_SETUP`` dictionary, and defining subloggers with different handlers
and log levels.  e.g. To enable debug messages only for the API set the level
of 'synnefo.api' to ``DEBUG``
1030

1031
1032
By default, the Django webapp and snf-manage logs to syslog, while
`snf-dispatcher` logs to `/var/log/synnefo/dispatcher.log`.
1033

1034

1035
1036
.. _scale-up:

1037
1038
1039
Scaling up to multiple nodes
============================

1040
1041
1042
1043
1044
1045
Here we will describe how should a large scale Synnefo deployment look like. Make
sure you are familiar with Synnefo and Ganeti before proceeding with this section.
This means you should at least have already set up successfully a working Synnefo
deployment as described in the :ref:`Admin's Quick Installation Guide
<quick-install-admin-guide>` and also read the Administrator's Guide until this
section.
1046

1047
1048
Graph of a scale-out Synnefo deployment
---------------------------------------
1049

1050
Each box in the following graph corresponds to a distinct physical node:
1051

1052
1053
1054
1055
1056
1057
1058
1059
1060
.. image:: images/synnefo-arch2-roles.png
   :width: 100%
   :target: _images/synnefo-arch2-roles.png

The above graph is actually the same with the one at the beginning of this
:ref:`guide <admin-guide>`, with the only difference that here we show the
Synnefo roles of each physical node. These roles are described in the
following section.

1061
1062
.. _physical-node-roles:

1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
Physical Node roles
-------------------

As appears in the previous graph, a scale-out Synnefo deployment consists of
multiple physical nodes that have the following roles:

* **WEBSERVER**: A web server running in front of gunicorn (e.g.: Apache, nginx)
* **ASTAKOS**: The Astakos application (gunicorn)
* **ASTAKOS_DB**: The Astakos database (postgresql)
* **PITHOS**: The Pithos application (gunicorn)
* **PITHOS_DB**: The Pithos database (postgresql)
* **CYCLADES**: The Cyclades application (gunicorn)
* **CYCLADES_DB**: The Cyclades database (postgresql)
* **MQ**: The message queue (RabbitMQ)
* **GANETI_MASTER**: The Ganeti master of a Ganeti cluster
* **GANETI_NODE** : A VM-capable Ganeti node of a Ganeti cluster
1079

1080
You will probably also have:
1081

1082
* **CMS**: The CMS used as a frotend portal for the Synnefo services
1083
* **NS**: A nameserver serving all other Synnefo nodes and resolving Synnefo FQDNs
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
* **CLIENT**: A machine that runs the Synnefo clients (e.g.: kamaki, Web UI),
              most of the times, the end user's local machine

From this point we will also refer to the following groups of roles:

* **SYNNEFO**: [ **ASTAKOS**, **ASTAKOS_DB**, **PITHOS**, **PITHOS_DB**, **CYCLADES**, **CYCLADES_DB**, **MQ**, **CMS**]
* **G_BACKEND**: [**GANETI_MASTER**, **GANETI_NODE**]

Of course, when deploying Synnefo you can combine multiple of the above roles on a
single physical node, but if you are trying to scale out, the above separation
gives you significant advantages.

So, in the next section we will take a look on what components you will have to
install on each physical node depending on its Synnefo role. We assume the graph's
architecture.

Components for each role
------------------------
1102

1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
When deploying Synnefo in large scale, you need to install different Synnefo
or/and third party components on different physical nodes according to their
Synnefo role, as stated in the previous section.

Specifically:

Role **WEBSERVER**
    * Synnefo components: `None`
    * 3rd party components: Apache
Role **ASTAKOS**
    * Synnefo components: `snf-webproject`, `snf-astakos-app`
    * 3rd party components: Django, Gunicorn
Role **ASTAKOS_DB**
    * Synnefo components: `None`
    * 3rd party components: PostgreSQL
Role **PITHOS**
    * Synnefo components: `snf-webproject`, `snf-pithos-app`, `snf-pithos-webclient`
    * 3rd party components: Django, Gunicorn
Role **PITHOS_DB**
    * Synnefo components: `None`
    * 3rd party components: PostgreSQL
Role **CYCLADES**
    * Synnefo components: `snf-webproject`, `snf-cyclades-app`, `snf-vncauthproxy`
    * 3rd party components: Django Gunicorn
Role **CYCLADES_DB**
    * Synnefo components: `None`
    * 3rd party components: PostgreSQL
Role **MQ**
    * Synnefo components: `None`
    * 3rd party components: RabbitMQ
Role **GANETI_MASTER**
    * Synnefo components: `snf-cyclades-gtools`
    * 3rd party components: Ganeti
Role **GANETI_NODE**
    * Synnefo components: `snf-cyclades-gtools`, `snf-network`, `snf-image`, `nfdhcpd`
    * 3rd party components: Ganeti
Role **CMS**
    * Synnefo components: `snf-webproject`, `snf-cloudcms`
    * 3rd party components: Django, Gunicorn
Role **NS**
    * Synnefo components: `None`
    * 3rd party components: BIND
Role **CLIENT**
    * Synnefo components: `kamaki`, `snf-image-creator`
    * 3rd party components: `None`

Example scale out installation
------------------------------

In this section we describe an example of a medium scale installation which
combines multiple roles on 10 different physical nodes. We also provide a
:ref:`guide <i-synnefo>` to help with such an install.

We assume that we have the following 10 physical nodes with the corresponding
roles:

Node1:
    **WEBSERVER**, **ASTAKOS**
      Guide sections:
        * :ref:`apt <i-apt>`
        * :ref:`gunicorn <i-gunicorn>`
        * :ref:`apache <i-apache>`
        * :ref:`snf-webproject <i-webproject>`
        * :ref:`snf-astakos-app <i-astakos>`
Node2:
    **WEBSERVER**, **PITHOS**
      Guide sections:
        * :ref:`apt <i-apt>`
        * :ref:`gunicorn <i-gunicorn>`
        * :ref:`apache <i-apache>`
        * :ref:`snf-webproject <i-webproject>`
        * :ref:`snf-pithos-app <i-pithos>`
        * :ref:`snf-pithos-webclient <i-pithos>`
Node3:
    **WEBSERVER**, **CYCLADES**
      Guide sections:
        * :ref:`apt <i-apt>`
        * :ref:`gunicorn <i-gunicorn>`
        * :ref:`apache <i-apache>`
        * :ref:`snf-webproject <i-webproject>`
        * :ref:`snf-cyclades-app <i-cyclades>`
        * :ref:`snf-vncauthproxy <i-cyclades>`
Node4:
    **WEBSERVER**, **CMS**
      Guide sections:
        * :ref:`apt <i-apt>`
        * :ref:`gunicorn <i-gunicorn>`
        * :ref:`apache <i-apache>`
        * :ref:`snf-webproject <i-webproject>`
        * :ref:`snf-cloudcms <i-cms>`
Node5:
    **ASTAKOS_DB**, **PITHOS_DB**, **CYCLADES_DB**
      Guide sections:
        * :ref:`apt <i-apt>`
        * :ref:`postgresql <i-db>`
Node6:
    **MQ**
      Guide sections:
        * :ref:`apt <i-apt>`
        * :ref:`rabbitmq <i-mq>`
Node7:
    **GANETI_MASTER**, **GANETI_NODE**
      Guide sections:
        * :ref:`apt <i-apt>`
        * :ref:`general <i-backends>`
        * :ref:`ganeti <i-ganeti>`
        * :ref:`snf-cyclades-gtools <i-gtools>`
        * :ref:`snf-network <i-network>`
        * :ref:`snf-image <i-image>`
        * :ref:`nfdhcpd <i-network>`
Node8:
    **GANETI_NODE**
      Guide sections:
        * :ref:`apt <i-apt>`
        * :ref:`general <i-backends>`
        * :ref:`ganeti <i-ganeti>`
        * :ref:`snf-cyclades-gtools <i-gtools>`
        * :ref:`snf-network <i-network>`
        * :ref:`snf-image <i-image>`
        * :ref:`nfdhcpd <i-network>`
Node9:
    **GANETI_NODE**
      Guide sections:
        `Same as Node8`
Node10:
    **GANETI_NODE**
      Guide sections:
        `Same as Node8`

All sections: :ref:`Scale out Guide <i-synnefo>`
1233

1234

1235
1236
Upgrade Notes
=============
1237

1238
1239
1240
.. toctree::
   :maxdepth: 1

1241
   v0.12 -> v0.13 <upgrade/upgrade-0.13>
1242

1243
1244
1245
1246
1247
1248
1249

Changelog, News
===============

* v0.13 :ref:`Changelog <Changelog-0.13>`, :ref:`NEWS <NEWS-0.13>`


1250
Older Cyclades Upgrade Notes
1251
============================
1252
1253
1254
1255

.. toctree::
   :maxdepth: 2

1256
   Upgrade <older/cyclades-upgrade>