Commit 079f88d3 authored by Constantinos Venetsanopoulos's avatar Constantinos Venetsanopoulos
Browse files

Integrate Pithos+ and Astakos documentation

Move and rearrange documentation concerning Pithos+ and Astakos.
Most content from the Pithos+ and Astakos repositories have been
integrated inside Synnefo's generic documentation.

Images have not yet been transfered. This is a WIP.
parent c76a7518
Astakos API
===========
This is Astakos API guide.
Overview
--------
Astakos serves as the point of authentication for GRNET (http://www.grnet.gr)
services. It is a platform-wide service, allowing users to register, login, and
keep track of permissions.
Users in astakos can be authenticated via several identity providers:
* Local
* Twitter
* Shibboleth
It provides also a command line tool for managing user accounts.
It is build over django and extends its authentication mechanism.
This document's goals is to describe the APIs to the outer world.
Make sure you have read the :ref:`astakos` general architecture first.
The present document is meant to be read alongside the Django documentation
(https://www.djangoproject.com/). Thus, it is suggested that the reader is
familiar with associated technologies.
Document Revisions
^^^^^^^^^^^^^^^^^^
========================= ================================
Revision Description
========================= ================================
0.1 (Feb 10, 2012) Initial release.
========================= ================================
API Operations
--------------
.. _authenticate-api-label:
Authenticate
^^^^^^^^^^^^
Authenticate API requests require a token. An application that wishes to connect to Astakos, but does not have a token, should redirect the user to ``/login``. (see :ref:`authentication-label`)
==================== ========= ==================
Uri Method Description
==================== ========= ==================
``/im/authenticate`` GET Authenticate user using token
==================== ========= ==================
|
==================== ===========================
Request Header Name Value
==================== ===========================
X-Auth-Token Authentication token
==================== ===========================
Extended information on the user serialized in the json format will be returned:
=========================== ============================
Name Description
=========================== ============================
username User uniq identifier
uniq User email (uniq identifier used by Astakos)
auth_token Authentication token
auth_token_expires Token expiration date
auth_token_created Token creation date
has_credits Whether user has credits
has_signed_terms Whether user has aggred on terms
=========================== ============================
Example reply:
::
{"username": "4ad9f34d6e7a4992b34502d40f40cb",
"uniq": "papagian@example.com"
"auth_token": "0000",
"auth_token_expires": "Tue, 11-Sep-2012 09:17:14 ",
"auth_token_created": "Sun, 11-Sep-2011 09:17:14 ",
"has_credits": false,
"has_signed_terms": true}
|
=========================== =====================
Return Code Description
=========================== =====================
204 (No Content) The request succeeded
400 (Bad Request) The request is invalid
401 (Unauthorized) Missing token or inactive user or penging approval terms
500 (Internal Server Error) The request cannot be completed because of an internal error
=========================== =====================
Get Services
^^^^^^^^^^^^
Returns a json formatted list containing information about the supported cloud services.
==================== ========= ==================
Uri Method Description
==================== ========= ==================
``/im/get_services`` GET Get cloud services
==================== ========= ==================
Example reply:
::
[{"url": "/", "icon": "home-icon.png", "name": "grnet cloud", "id": "cloud"},
{"url": "/okeanos.html", "name": "~okeanos", "id": "okeanos"},
{"url": "/ui/", "name": "pithos+", "id": "pithos"}]
Get Menu
^^^^^^^^
Returns a json formatted list containing the cloud bar links.
==================== ========= ==================
Uri Method Description
==================== ========= ==================
``/im/get_menu`` GET Get cloud bar menu
==================== ========= ==================
|
====================== =========================
Request Parameter Name Value
====================== =========================
location Location to pass in the next parameter
====================== =========================
Example reply if request user is not authenticated:
::
[{"url": "/im/login?next=", "name": "login..."}]
Example reply if request user is authenticated:
[{"url": "/im/profile", "name": "spapagian@grnet.gr"},
{"url": "/im/profile", "name": "view your profile..."},
{"url": "/im/password", "name": "change your password..."},
{"url": "/im/feedback", "name": "feedback..."},
{"url": "/im/logout", "name": "logout..."}]
......@@ -5,5 +5,166 @@ Identity Management Service (astakos)
Astakos is the synnefo Identity Management Service.
Introduction
============
Astakos serves as the point of authentication for GRNET (http://www.grnet.gr)
services. It is a platform-wide service, allowing users to register, login, and
keep track of permissions.
Users in astakos can be authenticated via several identity providers:
* Local
* Twitter
* Shibboleth
It provides also a command line tool for managing user accounts.
It is build over django and extends its authentication mechanism.
This document's goals are:
* present the overall architectural design.
* provide basic use cases.
The present document is meant to be read alongside the Django documentation
(https://www.djangoproject.com/). Thus, it is suggested that the reader is
familiar with associated technologies.
Astakos Architecture
====================
Overview
--------
Astakos service co-ordinates the access to resources (and the subsequent
permission model) and acts as the single point of registry and entry to the
GRNET cloud offering, comprising of Cyclades and Pithos+ subsystems.
It also propagates the user state to the Aquarium pricing subsystem.
.. image:: images/~okeanos.jpg
Registration Use Cases
----------------------
The following subsections describe two basic registration use cases. All the
registration cases are covered in :ref:`registration-flow-label`
Invited user
~~~~~~~~~~~~
A registered ~okeanos user, invites student Alice to subscribe to ~okeanos
services. Alice receives an email and through a link is navigated to Astakos's
signup page. The system prompts her to select one of the available
authentication mechanisms (Shibboleth, Twitter or local authentication) in
order to register to the system. Alice already has a Shibboleth account so
chooses that and then she is redirected to her institution's login page. Upon
successful login, her account is created.
Since she is invited his account is automaticaly activated and she is
redirected to Astakos's login page. As this is the first time Alice has
accessed the system she is redirected to her profile page where she can edit or
provide more information.
Not invited user
~~~~~~~~~~~~~~~~
Tony while browsing in the internet finds out about ~okeanos services. He
visits the signup page and since his has already a twitter account selects the
twitter authentication mechanism and he is redirected to twitter login page
where he is promted to provide his credentials. Upon successful login, twitter
redirects him back to the Astakos and the account is created.
Since his not an invited user his account has to be activated from an
administrator first, in order to be able to login. Upon the account's
activation he receives an email and through a link he is redirected to the
login page.
Authentication Use Cases
------------------------
Cloud service user
~~~~~~~~~~~~~~~~~~
Alice requests a specific resource from a cloud service ex. Pithos. In the
request supplies the `X-Auth-Token`` to identify whether she is eligible to
perform the specific task. The service contacts Astakos through its
``/im/authenticate`` api call (see :ref:`authenticate-api-label`) providing the
specific ``X-Auth-Token``. Astakos checkes whether the token belongs to an
active user and it has not expired and returns a dictionary containing user
related information. Finally the service uses the ``uniq`` field included in
the dictionary as the account string to identify the user accessible resources.
.. _registration-flow-label:
Registration Flow
-----------------
.. image:: images/signup.jpg
:scale: 100%
Login Flow
----------
.. image:: images/login.jpg
:scale: 100%
.. _authentication-label:
Astakos Users and Authentication
--------------------------------
Astakos incorporates django user authentication system and extends its User model.
Since username field of django User model has a limitation of 30 characters,
AstakosUser is **uniquely** identified by the ``email`` instead. Therefore,
``astakos.im.authentication_backends.EmailBackend`` is served to authenticate a
user using email if the first argument is actually an email, otherwise tries
the username.
A new AstakosUser instance is assigned with a uui as username and also with a
``auth_token`` used by the cloud services to authenticate the user.
``astakos.im.authentication_backends.TokenBackend`` is also specified in order
to authenticate the user using the email and the token fields.
Logged on users can perform a number of actions:
* access and edit their profile via: ``/im/profile``.
* change their password via: ``/im/password``
* invite somebody else via: ``/im/invite``
* send feedback for grnet services via: ``/im/send_feedback``
* logout (and delete cookie) via: ``/im/logout``
User entries can also be modified/added via the ``snf-manage activateuser`` command.
A superuser account can be created the first time you run the ``manage.py
syncdb`` django command and then loading the extra user data from the
``admin_user`` fixture. At a later date, the ``manage.py createsuperuser``
command line utility can be used (as long as the extra user data for Astakos is
added with a fixture or by hand).
Internal Astakos requests are handled using cookie-based django user sessions.
External systems in the same domain can delgate ``/login`` URI. The server,
depending on its configuration will redirect to the appropriate login page.
When done with logging in, the service's login URI should redirect to the URI
provided with next, adding user and token parameters, which contain the email
and token fields respectively.
The login URI accepts the following parameters:
====================== =========================
Request Parameter Name Value
====================== =========================
next The URI to redirect to when the process is finished
renew Force token renewal (no value parameter)
force Force logout current user (no value parameter)
====================== =========================
External systems outside the domain scope can acquire the user information by a
cookie set identified by ASTAKOS_COOKIE_NAME setting.
Finally, backend systems having acquired a token can use the
:ref:`authenticate-api-label` api call from a private network or through HTTPS.
......@@ -11,6 +11,16 @@ Tying it all up with kamaki
kamaki
------
IM API (Astakos)
================
This is the Identity Management API:
.. toctree::
:maxdepth: 2
IM API <astakos-api-guide>
Compute API (Cyclades)
======================
......@@ -34,8 +44,327 @@ Images API body
Storage API (Pithos+)
=====================
Files API body
This is the Pithos+ File Storage API:
.. toctree::
:maxdepth: 2
File Storage API <pithos-api-guide>
Implementing new clients
========================
In this section we discuss implementation guidelines, that a developer should
take into account before writing his own client for the above APIs. Before,
starting your client implementation, make sure you have thoroughly read the
corresponding Synnefo API.
Pithos+ clients
---------------
User Experience
~~~~~~~~~~~~~~~
Hopefully this API will allow for a multitude of client implementations, each
supporting a different device or operating system. All clients will be able to
manipulate containers and objects - even software only designed for OOS API
compatibility. But a Pithos interface should not be only about showing
containers and folders. There are some extra user interface elements and
functionalities that should be common to all implementations.
Upon entrance to the service, a user is presented with the following elements -
which can be represented as folders or with other related icons:
* The ``home`` element, which is used as the default entry point to the user's
"files". Objects under ``home`` are represented in the usual hierarchical
organization of folders and files.
* The ``trash`` element, which contains files that have been marked for
deletion, but can still be recovered.
* The ``shared`` element, which contains all objects shared by the user to
other users of the system.
* The ``others`` element, which contains all objects that other users share
with the user.
* The ``groups`` element, which contains the names of groups the user has
defined. Each group consists of a user list. Group creation, deletion, and
manipulation is carried out by actions originating here.
* The ``history`` element, which allows browsing past instances of ``home``
and - optionally - ``trash``.
Objects in Pithos+ can be:
* Moved to trash and then deleted.
* Shared with specific permissions.
* Made public (shared with non-Pithos+ users).
* Restored from previous versions.
Some of these functions are performed by the client software and some by the
Pithos+ server.
In the first version of Pithos, objects could also be assigned custom tags.
This is no longer supported. Existing deployments can migrate tags into a
specific metadata value, i.e. ``X-Object-Meta-Tags``.
Implementation Guidelines
~~~~~~~~~~~~~~~~~~~~~~~~~
Pithos+ clients should use the ``pithos`` and ``trash`` containers for active
and inactive objects respectively. If any of these containers is not found, the
client software should create it, without interrupting the user's workflow. The
``home`` element corresponds to ``pithos`` and the ``trash`` element to
``trash``. Use ``PUT`` with the ``X-Move-From`` header, or ``MOVE`` to transfer
objects from one container to the other. Use ``DELETE`` to remove from
``pithos`` without trashing, or to remove from ``trash``. When moving objects,
detect naming conflicts with the ``If-Match`` or ``If-None-Match`` headers.
Such conflicts should be resolved by the user.
Object names should use the ``/`` delimiter to impose a hierarchy of folders
and files.
The ``shared`` element should be implemented as a read-only view of the
``pithos`` container, using the ``shared`` parameter when listing objects. The
``others`` element, should start with a top-level ``GET`` to retrieve the list
of accounts accessible to the user. It is suggested that the client software
hides the next step of navigation - the container - if it only includes
``pithos`` and forwards the user directly to the objects.
Public objects are not included in ``shared`` and ``others`` listings. It is
suggested that they are marked in a visually distinctive way in ``pithos``
listings (for example using an icon overlay).
A special application menu, or a section in application preferences, should be
devoted to managing groups (the ``groups`` element). All group-related actions
are implemented at the account level.
Browsing past versions of objects should be available both at the object and
the container level. At the object level, a list of past versions can be
included in the screen showing details or more information on the object
(metadata, permissions, etc.). At the container level, it is suggested that
clients use a ``history`` element, which presents to the user a read-only,
time-variable view of ``pithos`` contents. This can be accomplished via the
``until`` parameter in listings. Optionally, ``history`` may include ``trash``.
Uploading and downloading data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By using hashmaps to upload and download objects the corresponding operations
can complete much faster.
In the case of an upload, only the missing blocks will be submitted to the
server:
* Calculate the hash value for each block of the object to be uploaded. Use
the hash algorithm and block size of the destination container.
* Send a hashmap ``PUT`` request for the object.
* Server responds with status ``201`` (Created):
* Blocks are already on the server. The object has been created. Done.
* Server responds with status ``409`` (Conflict):
* Server's response body contains the hashes of the blocks that do not
exist on the server.
* For each hash value in the server's response (or all hashes together):
* Send a ``POST`` request to the destination container with the
corresponding data.
* Repeat hashmap ``PUT``. Fail if the server's response is not ``201``.
Consulting hashmaps when downloading allows for resuming partially transferred
objects. The client should retrieve the hashmap from the server and compare it
with the hashmap computed from the respective local file. Any missing parts can
be downloaded with ``GET`` requests with the additional ``Range`` header.
Syncing
~~~~~~~
Consider the following algorithm for synchronizing a local folder with the
server. The "state" is the complete object listing, with the corresponding
attributes.
::
L: local state (stored state from last sync with the server)
C: current state (state computed right before sync)
S: server state
if C == L:
# No local changes
if S == L:
# No remote changes, nothing to do
else:
# Update local state to match that of the server
L = S
else:
# We have local changes
if S == L:
# No remote changes, update the server
S = C
L = S
else:
# Both we and server have changes
if C == S:
# We were lucky, we did the same change
L = S
else:
# We have conflicting changes
resolve conflict
Notes:
* States represent file hashes (it is suggested to use Merkle). Deleted or
non-existing files are assumed to have a magic hash (e.g. empty string).
* Updating a state (either local or remote) implies downloading, uploading or
deleting the appropriate file.
Recommended Practices and Examples
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Assuming an authentication token is obtained, the following high-level
operations are available - shown with ``curl``:
* Get account information ::
curl -X HEAD -D - \
-H "X-Auth-Token: 0000" \
https://pithos.dev.grnet.gr/v1/user
* List available containers ::
curl -X GET -D - \
-H "X-Auth-Token: 0000" \
https://pithos.dev.grnet.gr/v1/user
* Get container information ::
curl -X HEAD -D - \
-H "X-Auth-Token: 0000" \
https://pithos.dev.grnet.gr/v1/user/pithos
* Add a new container ::
curl -X PUT -D - \
-H "X-Auth-Token: 0000" \
https://pithos.dev.grnet.gr/v1/user/test
* Delete a container ::
curl -X DELETE -D - \
-H "X-Auth-Token: 0000" \
https://pithos.dev.grnet.gr/v1/user/test
* List objects in a container ::
curl -X GET -D - \
-H "X-Auth-Token: 0000" \
https://pithos.dev.grnet.gr/v1/user/pithos
* List objects in a container (extended reply) ::
curl -X GET -D - \
-H "X-Auth-Token: 0000" \
https://pithos.dev.grnet.gr/v1/user/pithos?format=json
It is recommended that extended replies are cached and subsequent requests
utilize the ``If-Modified-Since`` header.
* List metadata keys used by objects in a container
Will be in the ``X-Container-Object-Meta`` reply header, included in
container information or object list (``HEAD`` or ``GET``). (**TBD**)
* List objects in a container having a specific meta defined ::
curl -X GET -D - \
-H "X-Auth-Token: 0000" \
https://pithos.dev.grnet.gr/v1/user/pithos?meta=favorites
* Retrieve an object ::
curl -X GET -D - \
-H "X-Auth-Token: 0000" \
https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
* Retrieve an object (specific ranges of data) ::
curl -X GET -D - \
-H "X-Auth-Token: 0000" \
-H "Range: bytes=0-9" \
https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
This will return the first 10 bytes. To get the first 10, bytes 30-39 and the
last 100 use ``Range: bytes=0-9,30-39,-100``.
* Add a new object (folder type) (**TBD**) ::
curl -X PUT -D - \
-H "X-Auth-Token: 0000" \
-H "Content-Type: application/directory" \
https://pithos.dev.grnet.gr/v1/user/pithos/folder
* Add a new object ::
curl -X PUT -D - \
-H "X-Auth-Token: 0000" \
-H "Content-Type: text/plain" \
-T EXAMPLE.txt
https://pithos.dev.grnet.gr/v1/user/pithos/folder/EXAMPLE.txt
* Update an object ::
curl -X POST -D - \
-H "X-Auth-Token: 0000" \
-H "Content-Length: 10" \
-H "Content-Type: application/octet-stream" \
-H "Content-Range: bytes 10-19/*" \
-d "0123456789" \
https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
This will update bytes 10-19 with the data specified.
* Update an object (append) ::
curl -X POST -D - \
-H "X-Auth-Token: 0000" \
-H "Content-Length: 10" \
-H "Content-Type: application/octet-stream" \
-H "Content-Range: bytes */*" \
-d "0123456789" \
https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
* Update an object (truncate) ::
curl -X POST -D - \
-H "X-Auth-Token: 0000" \
-H "X-Source-Object: /folder/EXAMPLE.txt" \
-H "Content-Range: bytes 0-0/*" \
-H "X-Object-Bytes: 0" \
https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
This will truncate the object to 0 bytes.
* Add object metadata ::
curl -X POST -D - \
-H "X-Auth-Token: 0000" \
-H "X-Object-Meta-First: first_meta_value" \
-H "X-Object-Meta-Second: second_meta_value" \
https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
* Delete object metadata ::
curl -X POST -D - \
-H "X-Auth-Token: 0000" \
-H "X-Object-Meta-First: first_meta_value" \
https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
Metadata can only be "set". To delete ``X-Object-Meta-Second``, reset all