Commit 28b54d26 authored by Constantinos Venetsanopoulos's avatar Constantinos Venetsanopoulos
Browse files

Major documentation fixes/updates for 0.13

 * Various semantic fixes on different guides
 * Various shpinx related fixes
 * New pictures for the overall Synnefo architecture
 * Update the scaling-up section of the admin guide
parent 5f06527f
......@@ -16,9 +16,9 @@ Guide you will be able to understand every component and all the interactions
between them. It is a good idea to first go through the Quick Administrator's
Guide before proceeding.
.. image:: images/synnefo-architecture1.png
.. image:: images/synnefo-arch2.png
:width: 100%
:target: _images/synnefo-architecture1.png
:target: _images/synnefo-arch2.png
......@@ -557,18 +557,28 @@ Archipelago is used by Cyclades and Ganeti for fast provisioning of VMs based on
CoW volumes. Moreover, it enables live migration of thinly-provisioned VMs with
no physically shared storage.
Architecture
------------
Archipelago Architecture
------------------------
.. image:: images/archipelago-architecture.png
:width: 50%
:target: _images/archipelago-architecture.png
.. _syn+archip+rados:
Overview of Synnefo + Archipelago + RADOS
-----------------------------------------
.. image:: images/synnefo-arch3.png
:width: 100%
:target: _images/synnefo-arch3.png
Prereqs
-------
The administrator must initialize the storage backend where archipelago volume
blocks will reside.
In case of a files backend, the administrator must create two directories. One
for the archipelago data blocks and one for the archipelago map blocks. These
should probably be over shared storage to enable sharing archipelago volumes
......@@ -579,9 +589,9 @@ In case of a RADOS backend, the administrator must create two rados pools, one
for data blocks, and one for the map blocks. These pools, must be the same pools
used in pithos, in order to enable volume creation based on pithos images.
Installation
------------
Archipelago consists of
* ``libxseg0``: libxseg used to communicate over shared memory segments
......@@ -702,15 +712,15 @@ The ``vlmc`` tool provides a way to interact with archipelago volumes
* ``vlmc create <volumename> --snap <snapname> --size <size>``: creates a new
volume named <volumename> from snapshot name <snapname> with size <size>.
The ``--snap`` and ``--size`` are optional, but at least one of them is
mandatory. e.g:
The ``--snap`` and ``--size`` are optional, but at least one of them is
mandatory. e.g:
``vlmc create <volumename> --snap <snapname>`` creates a volume named
volumename from snapshot snapname. The size of the volume is the same as
the size of the snapshot.
``vlmc create <volumename> --snap <snapname>`` creates a volume named
volumename from snapshot snapname. The size of the volume is the same as
the size of the snapshot.
``vlmc create <volumename> --size <size>`` creates an empty volume of size
<size> named <volumename>.
``vlmc create <volumename> --size <size>`` creates an empty volume of size
<size> named <volumename>.
* ``vlmc remove <volumename>``: removes the volume and all the related
archipelago blocks from storage.
......@@ -859,7 +869,8 @@ Currently, RabbitMQ is used by the following components:
the above components, and updates the Cyclades DB accordingly.
Installation
````````````
~~~~~~~~~~~~
Please check the RabbitMQ documentation which covers extensively the
`installation of RabbitMQ server <http://www.rabbitmq.com/download.html>`_ and
the setup of a `RabbitMQ cluster <http://www.rabbitmq.com/clustering.html>`_.
......@@ -951,35 +962,197 @@ By default, the Django webapp and snf-manage logs to syslog, while
Scaling up to multiple nodes
============================
Here we will describe how to deploy all services, interconnected with each
other, on multiple physical nodes.
Here we will describe how should a large scale Synnefo deployment look like. Make
sure you are familiar with Synnefo and Ganeti before proceeding with this section.
This means you should at least have already set up successfully a working Synnefo
deployment as described in the :ref:`Admin's Quick Installation Guide
<quick-install-admin-guide>` and also read the Administrator's Guide until this
section.
synnefo components
------------------
Graph of a scale-out Synnefo deployment
---------------------------------------
You need to install the appropriate synnefo software components on each node,
depending on its type, see :ref:`Architecture <cyclades-architecture>`.
Each box in the following graph corresponds to a distinct physical node:
Please see the page of each synnefo software component for specific
installation instructions, where applicable.
.. image:: images/synnefo-arch2-roles.png
:width: 100%
:target: _images/synnefo-arch2-roles.png
The above graph is actually the same with the one at the beginning of this
:ref:`guide <admin-guide>`, with the only difference that here we show the
Synnefo roles of each physical node. These roles are described in the
following section.
Physical Node roles
-------------------
As appears in the previous graph, a scale-out Synnefo deployment consists of
multiple physical nodes that have the following roles:
* **WEBSERVER**: A web server running in front of gunicorn (e.g.: Apache, nginx)
* **ASTAKOS**: The Astakos application (gunicorn)
* **ASTAKOS_DB**: The Astakos database (postgresql)
* **PITHOS**: The Pithos application (gunicorn)
* **PITHOS_DB**: The Pithos database (postgresql)
* **CYCLADES**: The Cyclades application (gunicorn)
* **CYCLADES_DB**: The Cyclades database (postgresql)
* **MQ**: The message queue (RabbitMQ)
* **GANETI_MASTER**: The Ganeti master of a Ganeti cluster
* **GANETI_NODE** : A VM-capable Ganeti node of a Ganeti cluster
Install the following synnefo components:
You will probably also have:
Nodes of type :ref:`APISERVER <APISERVER_NODE>`
Components
:ref:`snf-common <snf-common>`,
:ref:`snf-webproject <snf-webproject>`,
:ref:`snf-cyclades-app <snf-cyclades-app>`
Nodes of type :ref:`GANETI-MASTER <GANETI_MASTER>` and :ref:`GANETI-NODE <GANETI_NODE>`
Components
:ref:`snf-common <snf-common>`,
:ref:`snf-cyclades-gtools <snf-cyclades-gtools>`
Nodes of type :ref:`LOGIC <LOGIC_NODE>`
Components
:ref:`snf-common <snf-common>`,
:ref:`snf-webproject <snf-webproject>`,
:ref:`snf-cyclades-app <snf-cyclades-app>`.
* **CMS**: The CMS used as a frotend portal for the Synnefo services
* **NS**: A nameserver serving all other nodes
* **CLIENT**: A machine that runs the Synnefo clients (e.g.: kamaki, Web UI),
most of the times, the end user's local machine
From this point we will also refer to the following groups of roles:
* **SYNNEFO**: [ **ASTAKOS**, **ASTAKOS_DB**, **PITHOS**, **PITHOS_DB**, **CYCLADES**, **CYCLADES_DB**, **MQ**, **CMS**]
* **G_BACKEND**: [**GANETI_MASTER**, **GANETI_NODE**]
Of course, when deploying Synnefo you can combine multiple of the above roles on a
single physical node, but if you are trying to scale out, the above separation
gives you significant advantages.
So, in the next section we will take a look on what components you will have to
install on each physical node depending on its Synnefo role. We assume the graph's
architecture.
Components for each role
------------------------
When deploying Synnefo in large scale, you need to install different Synnefo
or/and third party components on different physical nodes according to their
Synnefo role, as stated in the previous section.
Specifically:
Role **WEBSERVER**
* Synnefo components: `None`
* 3rd party components: Apache
Role **ASTAKOS**
* Synnefo components: `snf-webproject`, `snf-astakos-app`
* 3rd party components: Django, Gunicorn
Role **ASTAKOS_DB**
* Synnefo components: `None`
* 3rd party components: PostgreSQL
Role **PITHOS**
* Synnefo components: `snf-webproject`, `snf-pithos-app`, `snf-pithos-webclient`
* 3rd party components: Django, Gunicorn
Role **PITHOS_DB**
* Synnefo components: `None`
* 3rd party components: PostgreSQL
Role **CYCLADES**
* Synnefo components: `snf-webproject`, `snf-cyclades-app`, `snf-vncauthproxy`
* 3rd party components: Django Gunicorn
Role **CYCLADES_DB**
* Synnefo components: `None`
* 3rd party components: PostgreSQL
Role **MQ**
* Synnefo components: `None`
* 3rd party components: RabbitMQ
Role **GANETI_MASTER**
* Synnefo components: `snf-cyclades-gtools`
* 3rd party components: Ganeti
Role **GANETI_NODE**
* Synnefo components: `snf-cyclades-gtools`, `snf-network`, `snf-image`, `nfdhcpd`
* 3rd party components: Ganeti
Role **CMS**
* Synnefo components: `snf-webproject`, `snf-cloudcms`
* 3rd party components: Django, Gunicorn
Role **NS**
* Synnefo components: `None`
* 3rd party components: BIND
Role **CLIENT**
* Synnefo components: `kamaki`, `snf-image-creator`
* 3rd party components: `None`
Example scale out installation
------------------------------
In this section we describe an example of a medium scale installation which
combines multiple roles on 10 different physical nodes. We also provide a
:ref:`guide <i-synnefo>` to help with such an install.
We assume that we have the following 10 physical nodes with the corresponding
roles:
Node1:
**WEBSERVER**, **ASTAKOS**
Guide sections:
* :ref:`apt <i-apt>`
* :ref:`gunicorn <i-gunicorn>`
* :ref:`apache <i-apache>`
* :ref:`snf-webproject <i-webproject>`
* :ref:`snf-astakos-app <i-astakos>`
Node2:
**WEBSERVER**, **PITHOS**
Guide sections:
* :ref:`apt <i-apt>`
* :ref:`gunicorn <i-gunicorn>`
* :ref:`apache <i-apache>`
* :ref:`snf-webproject <i-webproject>`
* :ref:`snf-pithos-app <i-pithos>`
* :ref:`snf-pithos-webclient <i-pithos>`
Node3:
**WEBSERVER**, **CYCLADES**
Guide sections:
* :ref:`apt <i-apt>`
* :ref:`gunicorn <i-gunicorn>`
* :ref:`apache <i-apache>`
* :ref:`snf-webproject <i-webproject>`
* :ref:`snf-cyclades-app <i-cyclades>`
* :ref:`snf-vncauthproxy <i-cyclades>`
Node4:
**WEBSERVER**, **CMS**
Guide sections:
* :ref:`apt <i-apt>`
* :ref:`gunicorn <i-gunicorn>`
* :ref:`apache <i-apache>`
* :ref:`snf-webproject <i-webproject>`
* :ref:`snf-cloudcms <i-cms>`
Node5:
**ASTAKOS_DB**, **PITHOS_DB**, **CYCLADES_DB**
Guide sections:
* :ref:`apt <i-apt>`
* :ref:`postgresql <i-db>`
Node6:
**MQ**
Guide sections:
* :ref:`apt <i-apt>`
* :ref:`rabbitmq <i-mq>`
Node7:
**GANETI_MASTER**, **GANETI_NODE**
Guide sections:
* :ref:`apt <i-apt>`
* :ref:`general <i-backends>`
* :ref:`ganeti <i-ganeti>`
* :ref:`snf-cyclades-gtools <i-gtools>`
* :ref:`snf-network <i-network>`
* :ref:`snf-image <i-image>`
* :ref:`nfdhcpd <i-network>`
Node8:
**GANETI_NODE**
Guide sections:
* :ref:`apt <i-apt>`
* :ref:`general <i-backends>`
* :ref:`ganeti <i-ganeti>`
* :ref:`snf-cyclades-gtools <i-gtools>`
* :ref:`snf-network <i-network>`
* :ref:`snf-image <i-image>`
* :ref:`nfdhcpd <i-network>`
Node9:
**GANETI_NODE**
Guide sections:
`Same as Node8`
Node10:
**GANETI_NODE**
Guide sections:
`Same as Node8`
All sections: :ref:`Scale out Guide <i-synnefo>`
Upgrade Notes
......
......@@ -563,7 +563,7 @@ overLimit (413)
This operation changes the name of the network in the Compute system.
**Example Update Network Name Request: JSON**::
**Example Update Network Name Request: JSON**:
.. code-block:: javascript
......
......@@ -3,13 +3,8 @@
Synnefo Developer's Guide
^^^^^^^^^^^^^^^^^^^^^^^^^
This is the complete Synnefo Developer's Guide
Tying it all up with kamaki
===========================
kamaki
------
This is the complete Synnefo Developer's Guide. Here we document all Synnefo APIs
to allow external developers write independent tools that interact with Synnefo.
IM API (Astakos)
================
......@@ -49,10 +44,10 @@ This is the Plankton Image API:
Image API <plankton-api-guide>
Storage API (Pithos+)
Storage API (Pithos)
=====================
This is the Pithos+ Object Storage API:
This is the Pithos Object Storage API:
.. toctree::
:maxdepth: 2
......@@ -67,8 +62,8 @@ take into account before writing his own client for the above APIs. Before,
starting your client implementation, make sure you have thoroughly read the
corresponding Synnefo API.
Pithos+ clients
---------------
Pithos clients
--------------
User Experience
~~~~~~~~~~~~~~~
......
......@@ -17,79 +17,71 @@ Welcome to Synnefo's documentation
:maxdepth: 1
Identity Management (codename: astakos) <astakos>
Object Storage Service (codename: pithos+) <pithos>
Object Storage Service (codename: pithos) <pithos>
Compute Service (codename: cyclades) <cyclades>
Network Service (part of Cyclades) <networks>
Image Registry (codename: plankton) <plankton>
Billing Service (codename: aquarium) <http://docs.dev.grnet.gr/aquarium/latest/index.html>
Image Service (codename: plankton) <plankton>
Volume Storage Service (codename: archipelago) <archipelago>
.. image:: images/synnefo-overview.png
:target: _images/synnefo-overview.png
There are also components for:
There are also the following tools:
.. toctree::
:maxdepth: 1
Secure image deployment (snf-image tool) <snf-image>
Command-line cloud management (kamaki tool) <http://docs.dev.grnet.gr/kamaki/latest/index.html>
Image bundling/uploading/registering (snf-image-creator tool) <http://docs.dev.grnet.gr/snf-image-creator/latest/index.html>
kamaki: Command-line client <http://docs.dev.grnet.gr/kamaki/latest/index.html>
snf-image-creator: Image bundling/uploading/registering tool <http://docs.dev.grnet.gr/snf-image-creator/latest/index.html>
snf-image: Secure image deployment tool <snf-image>
Synnefo is designed to be as simple, scalable and production ready as possible.
Furthermore, although it can be deployed in small configurations, its prime
target is large installations. If you are planning for the latter, you should
first be completely aware of what you want to provide, the architecture of your
cluster/s and Synnefo's overall architecture before you start deploying.
target is large installations.
All Synnefo components use an intuitive settings mechanism, that gives you the
ability to either deploy the above services independently and standalone, or
interconnected with each other, in large configurations.
All Synnefo components use an intuitive settings mechanism, that adds and removes
settings dynamically as components are getting added or removed from a physical
node. All settings are stored in a single location.
.. _general-arch:
Synnefo General Architecture
============================
The following graph shows the whole Synnefo architecture and how it interacts
with multiple Ganeti clusters. Right click on the image and select "Open image
in new tab" to be able to zoom in.
with multiple Ganeti clusters.
.. image:: images/synnefo-architecture1.png
.. image:: images/synnefo-arch2.png
:width: 100%
:target: _images/synnefo-architecture1.png
:target: _images/synnefo-arch2.png
Synnefo also supports RADOS as an alternative storage backend for
Files/Images/VM disks. :ref:`Here <syn+archip+rados>` is a graph that shows
Synnefo running with two different storage backends.
Synnefo Guides
==============
There are 5 guides for Synnefo.
There are 4 guides for Synnefo.
The installation overview walks through the synnefo components and defines roles
per node so that each role can be installed separately. Please note that
different roles can still coexist in the same node and there is also a posibility
to install everything on one node. For the sake of scalability this is not recommended.
The quick installation guide describes how to install the whole synnefo stack
The quick installation guide describes how to install the whole Synnefo stack
in just two physical nodes, for testing purposes. This guide is useful to those
interested in deploying synnefo in large scale, as a starting point that will
help them get familiar with the synnefo components and overall architecture, as
interested in deploying Synnefo in large scale, as a starting point that will
help them get familiar with the Synnefo components and overall architecture, as
well as the interconnection between different services. Such an installation,
also provides a quick preview of the basic synnefo features, although we would
like to think that synnefo unveils its real power while scaling.
also provides a quick preview of the basic Synnefo features, although we would
like to think that Synnefo's real power unveils while scaling.
The Administrator's Guide targets system administrators, who want to dive into
more details and common tasks regarding Synnefo. The Developer's Guide targets
developers, who want to build on top of Synnefo and so describes all the
different types of interfaces Synnefo provides to the external world. The
Integrator's Guide targets developers, who want to actually
extend/modify/change Synnefo itself, so describes Synnefo's indepth
architecture and the internals of Synnefo components.
more details and common tasks regarding Synnefo. For the experienced Synnefo
administrator, there is also a section that describes how to deploy Synnefo in
large scale with a corresponding guide.
.. toctree::
:maxdepth: 1
Installation Guide/Overview <i-synnefo>
The Developer's Guide targets developers, who want to build on top of Synnefo
and so describes all the different types of interfaces Synnefo provides to the
external world. The Integrator's Guide targets developers, who want to actually
extend/modify/change Synnefo itself, so describes Synnefo's indepth
architecture and the internals of Synnefo components (currently out-of-date!).
.. toctree::
:maxdepth: 1
......@@ -107,14 +99,6 @@ architecture and the internals of Synnefo components.
List of all Synnefo components
==============================
Here are all Synnefo components. Combined in different ways, they provide all
Synnefo services. All components are released as:
.. toctree::
debian packages <http://docs.dev.grnet.gr/debs/>
python packages <http://docs.dev.grnet.gr/pypi/>
They are also available from our apt repository: ``apt.okeanos.grnet.gr``
* `snf-common <http://docs.dev.grnet.gr/snf-common/latest/index.html>`_
......@@ -122,7 +106,6 @@ They are also available from our apt repository: ``apt.okeanos.grnet.gr``
* `snf-astakos-app <http://docs.dev.grnet.gr/astakos/latest/index.html>`_
* `snf-pithos-backend <http://docs.dev.grnet.gr/pithos/latest/backends.html>`_
* `snf-pithos-app <http://docs.dev.grnet.gr/pithos/latest/index.html>`_
* `snf-pithos-tools <http://docs.dev.grnet.gr/pithos/latest/index.html>`_
* `snf-pithos-webclient <http://docs.dev.grnet.gr/pithos-webclient/latest/index.html>`_
* `snf-cyclades-app <http://docs.dev.grnet.gr/snf-cyclades-app/latest/index.html>`_
* `snf-cyclades-gtools <http://docs.dev.grnet.gr/snf-cyclades-gtools/latest/index.html>`_
......@@ -142,6 +125,10 @@ You can contact the Synnefo team at the following mailing lists:
* Users list: synnefo@googlegroups.com
* Developers list: synnefo-devel@googlegroups.com
The official site is:
`http://www.synnefo.org <http://www.synnefo.org>`_
Indices and tables
==================
......
.. _cyclades-api-guide:
.. _plankton-api-guide:
Plankton API Guide
^^^^^^^^^^^^^^^^^^
......
......@@ -1258,11 +1258,10 @@ In the above command:
* ``img_passwd``: the arbitrary root password of your new instance
* ``img_format``: set to ``diskdump`` to reflect the type of the uploaded Image
* ``img_id``: If you want to deploy an Image stored on Pithos+ (our case), this
should have the format
``pithos://<username>/<container>/<filename>``:
* ``username``: ``user@example.com`` (defined during Astakos sign up)
* ``container``: ``pithos`` (default, if the Web UI was used)
* ``filename``: the name of file (visible also from the Web UI)
should have the format ``pithos://<username>/<container>/<filename>``:
* ``username``: ``user@example.com`` (defined during Astakos sign up)
* ``container``: ``pithos`` (default, if the Web UI was used)
* ``filename``: the name of file (visible also from the Web UI)
* ``img_properties``: taken from the metadata file. Used only the two mandatory
properties ``OSFAMILY`` and ``ROOT_PARTITION``. `Learn more
<https://code.grnet.gr/projects/snf-image/wiki/Image_Format#Image-Properties>`_
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment