Commit 47195121 authored by Stavros Sachtouris's avatar Stavros Sachtouris
Browse files

Merge branch 'develop' into feature-image

parents f769a16a cae76f25
CHANGELOG for version 0.7
New features:
1. Unify and improve data size units presentation
2. Ask for user permission at store-delete
3. Intuitive semantics for store-move/copy/download/upload (modified syntax and clients)
4. Use UUID instead of email in pithos client calls, make account setting obsolete
5. Allow character : in container operations in store interface
6. Recursively download remote directories
Improvements:
1. Clean up CLI error handling code
2. Unify and improve data size units presentation
3. Ask for user permission at store-delete
4. Intuitive semantics for store-move/copy/download/upload (modified syntax and clients)
5. Dynamically limit max number of threads
6. Various CLI and dependency fixes
\ No newline at end of file
2. Dynamically limit max number of threads
3. Cache user info in astakos client
4. Organize client unittests in a package and update to comply with new synnefo specs
Bug fixes:
1. kamaki image * commands should not send non-flag URL params without values
2. Trace errors through all parts of kamaki code
3. Quoted text in kamaki shell behaves same way as in one command mode
4. Pithos container names with spaces fail
5. URL-encode all url paths before requests
#!/usr/bin/env sh
set -e
BUILD_DIR=$1
BUILD_NUMBER=$2
PACKAGES_DIR=$1/$2
PACKAGES_DIR=$1
shift
shift
TEMP_DIR=$(mktemp -d /tmp/devflow_autopkg_XXXXXXX)
......
#!/usr/bin/env sh
set -e
BUILD_DIR=$1
BUILD_NUMBER=$2
DOCS_DIR=$1/$2
DOCS_DIR=$1
cd docs
make html
......
......@@ -24,15 +24,14 @@ In the following, the token has been set in a previous step (see `setup section
* Authenticate user *
[astakos]:authenticate
auth_token : s0m3t0k3nth@t1sr3m0v3d==
auth_token_created: 2012-11-13T14:12:40.917034
auth_token_expires: 2012-12-13T14:12:40.917035
groups :
default
has_credits : False
has_signed_terms : True
uniq : myaccount@grnet.gr
username : 4215th3b357num9323v32
email :
myaccount@grnet.gr
myotheraccount@grnet.gr
name : My Real Name
username : usually@an.email.org
uuid : ab1cde23-45fg-6h7i-8j9k-10l1m11no2pq
flavor (Compute/Cyclades)
-------------------------
......@@ -60,26 +59,27 @@ Showcase: show details for flavor with id 43
name : C4R2048D10
ram : 2048
image (Compute/Cyclades + Plankton)
-----------------------------------
image (Plankton commands + Compute Image subcommands)
-----------------------------------------------------
.. code-block:: text
addmember : Add a member to an image
addproperty: Add an image property
delete : Delete image
delmember : Remove a member from an image
delproperty: Delete an image property
info : Get image details
list : List images
list : List images accessible by user
members : Get image members
meta : Get image metadata
properties : Get image properties
public : List public images
register : (Re)Register an image
setmembers : Set the members of an image
setproperty: Update an image property
shared : List shared images
compute : Compute Image API commands
list : List images
delete : Delete image
info : Get image details
properties : Get image properties
delproperty: Delete an image property
setproperty: Update an image property
Showcase: Pick an image and list the properties
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
......@@ -92,20 +92,33 @@ Showcase: Pick an image and list the properties
* list all available images *
[image]:list
1395fdfb-51b4-419f-bb02-f7d632860611 Ubuntu Desktop LTS
1580deb4-edb3-4496-a27f-7a246c4c0528 Ubuntu Desktop
18a82962-43eb-4b32-8e28-8f8880af89d7 Kubuntu LTS
6aa6eafd-dccb-422d-a904-67fe2bdde87e Debian Desktop
6b5681e4-7502-46ae-b1e9-9fd837932095 maelstrom
78262ee7-949e-4d70-af3a-85360c3de57a Windows Server 2012
86bc2414-0fb3-4898-a637-240292243302 Fedora
926ab1c5-2d85-49d4-aebe-0fce712789b9 Windows Server 2008
b2dffe52-64a4-48c3-8a4c-8214cc3165cf Debian Base
baf2321c-57a0-4a69-825d-49f49cea163a CentOS
c1d27b46-d875-4f5c-b7f1-f39b5af62905 Kubuntu
* Get properties of image with id b2dffe52-64a4-48c3-8a4c-8214cc3165cf *
[image]:properties b2dffe52-64a4-48c3-8a4c-8214cc3165cf
1. Windows Server 2008
container_format: bare
disk_format : diskdump
id : 926ab1c5-2d85-49d4-aebe-0fce712789b9
size : 11917066240
status : available
2. Windows Server 2012
container_format: bare
disk_format : diskdump
id : 78262ee7-949e-4d70-af3a-85360c3de57a
size : 11697913856
status : available
3. ubuntu
container_format: bare
disk_format : diskdump
id : 5ed5a29b-292c-4fe0-b32c-2e2b65628635
size : 2578100224
status : available
4. Debian_Wheezy_Base
container_format: bare
disk_format : diskdump
id : 1f8454f0-8e3e-4b6c-ab8e-5236b728dffe
size : 795107328
status : available
* Get properties of image with id 1f8454f0-8e3e-4b6c-ab8e-5236b728dffe *
[image]:compute properties 1f8454f0-8e3e-4b6c-ab8e-5236b728dffe
description : Debian 6.0.6 (Squeeze) Base System
gui : No GUI
kernel : 2.6.32
......@@ -165,7 +178,7 @@ Showcase: Create a server
-s, --silent Do not output anything
* List all available images *
[server]:/image list
[server]:/image compute list
1395fdfb-51b4-419f-bb02-f7d632860611 Ubuntu Desktop LTS
1580deb4-edb3-4496-a27f-7a246c4c0528 Ubuntu Desktop
18a82962-43eb-4b32-8e28-8f8880af89d7 Kubuntu LTS
......@@ -300,11 +313,11 @@ store (Storage/Pithos+)
copy : Copy an object
create : Create a container
delete : Delete a container [or an object]
delgroup : Delete a user group on an account
delmeta : Delete an existing metadatum of account [, container [or object]]
delgroup : Delete a user group
delmeta : Delete an existing metadatum for an account [, container [or object]]
delpermissions: Delete all sharing permissions
download : Download a file
group : Get user groups details for account
group : Get user groups details
hashmap : Get the hashmap of an object
info : Get information for account [, container [or object]]
list : List containers, object trees or objects in a directory
......@@ -317,7 +330,7 @@ store (Storage/Pithos+)
publish : Publish an object
purge : Purge a container
quota : Get quota for account [or container]
setgroup : Create/update a new user group on account
setgroup : Create/update a new user group
setmeta : Set a new metadatum for account [, container [or object]]
setpermissions: Set sharing permissions
setquota : Set new quota (in KB) for account [or container]
......
......@@ -20,11 +20,11 @@ from sys import path, stderr
import os
try:
from objpool import http
http
from objpool.http import PooledHTTPConnection
PooledHTTPConnection
except ImportError:
stderr.write("`objpool` package is required to build kamaki docs.\n")
#exit()
raise
path.insert(0, os.path.join(os.path.abspath(os.path.dirname(__file__)), '..'))
......@@ -58,9 +58,9 @@ copyright = u'2012, GRNET'
# built documents.
#
# The short X.Y version.
version = '0.6'
version = '0.7'
# The full version, including alpha/beta/rc tags.
release = '0.6.3'
release = '0.7'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
......@@ -102,7 +102,30 @@ pygments_style = 'sphinx'
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'nature'
#html_theme = 'nature'
html_theme = 'default'
html_theme_options = {
'collapsiblesidebar': 'true',
'footerbgcolor': '#55b577',
'footertextcolor': '#000000',
'sidebarbgcolor': '#ffffff',
'sidebarbtncolor': '#f2f2f2',
'sidebartextcolor': '#000000',
'sidebarlinkcolor': '#328e4a',
'relbarbgcolor': '#55b577',
'relbartextcolor': '#ffffff',
'relbarlinkcolor': '#ffffff',
'bgcolor': '#ffffff',
'textcolor': '#000000',
'headbgcolor': '#ffffff',
'headtextcolor': '#000000',
'headlinkcolor': '#c60f0f',
'linkcolor': '#328e4a',
'visitedlinkcolor': '#63409b',
'codebgcolor': '#eeffcc',
'codetextcolor': '#333333'
}
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
......
......@@ -16,7 +16,7 @@ External applications may instantiate one or more kamaki clients.
.. code-block:: python
:emphasize-lines: 1
Example 1.1: Instantiate a Cyclades client
Example 1.1: Instantiate Cyclades and Pithos client
from kamaki.clients.cyclades import CycladesClient
......@@ -27,6 +27,14 @@ External applications may instantiate one or more kamaki clients.
.. note:: *cyclades* and *pithos* clients inherit all methods of *compute* and *storage* clients respectively. Separate compute or storage objects should be used only when implementing applications for strict OS Compute or OS Storage services.
.. note:: *account* variable is usually acquired by the following astakos call
.. code-block:: python
from kamaki.clients.astakos import AstakosClient
astakos = AstakosClient(astakos_base_url, token)
account = astakos.term('uuid')
Use client methods
------------------
......
......@@ -121,19 +121,13 @@ utils
The clients API
---------------
Imports
^^^^^^^
.. toctree::
connection
Modules list
^^^^^^^^^^^^
compute_rest_api
compute ReST API
^^^^^^^^^^^^^^^^
.. automodule:: kamaki.clients.compute_rest_api
.. automodule:: kamaki.clients.compute.rest_api
:members:
:show-inheritance:
:undoc-members:
......@@ -148,6 +142,14 @@ compute
:undoc-members:
cyclades ReST API
^^^^^^^^^^^^^^^^^
.. automodule:: kamaki.clients.cyclades.rest_api
:members:
:show-inheritance:
:undoc-members:
cyclades
^^^^^^^^
......@@ -165,24 +167,22 @@ storage
:show-inheritance:
:undoc-members:
pithos_rest_api
^^^^^^^^^^^^^^^
pithos
^^^^^^
.. automodule:: kamaki.clients.pithos
.. automodule:: kamaki.clients.pithos.rest_api
:members:
:show-inheritance:
:undoc-members:
pithos_rest_api
^^^^^^^^^^^^^^^
pithos
^^^^^^
.. automodule:: kamaki.clients.pithos_rest_api
.. automodule:: kamaki.clients.pithos
:members:
:show-inheritance:
:undoc-members:
image
^^^^^
......
Connection
==========
An http connection package with connection pooling.
Since version 0.6 it is safe to use threaded connections.
The Connection package uses httplib, standard python threads and a connection pooling mechanism.
.. note:: Since version 0.6.2 the underlying pooling mechanism is packed in a new GRNET package called *objpool*.
.. automodule:: kamaki.clients.connection
:members:
:show-inheritance:
:undoc-members:
Extending kamaki.clients
========================
By default, kamaki clients are REST clients (they manage HTTP requests and responses to communicate with services). This is achieved by importing the connection module, which is an httplib wrapper.
Connection
----------
The connection module features an error handling and logging system, a lazy response mechanism, a pooling mechanism as well as concurrency control for thread-demanding client functions (e.g. store upload).
By default, kamaki clients are REST clients (they manage HTTP requests and responses to communicate with services).
How to build a client
---------------------
......@@ -94,4 +89,166 @@ Kamaki clients may handle multiple requests at once, using threads. In that case
event = SilentEvent(self._single_threaded_method, **args)
event.start()
thread_list.append(event)
thread_list = self._watch_thread_limit(thread_list)
\ No newline at end of file
thread_list = self._watch_thread_limit(thread_list)
Going agile
-----------
The kamaki.clients package contains a set of fine-grained unit-tests for all its packages.
.. note:: unit tests require the optional python-mock package, version 1.X or better
Using the tests
^^^^^^^^^^^^^^^
To run the tests, the kamaki source code has to be downloaded.
.. code-block:: console
$ git clone https://code.grnet.gr/git/kamaki
$ cd kamaki/kamaki/clients
In each package under kamaki.clients, there is a test module (test.py) where the tests are implemented. To run all tests, run the test.py file from kamaki.clients
.. code-block:: console
$ python test.py
To test a specific class, add the class name as an argument. E.g. for the Client class:
.. code-block:: console
$ python test.py Client
To test a specific method in a class, apply an extra argument, e.g. for the request method in the Client class:
.. code-block:: console
$ python test.py Client request
Each package contains a test module (test.py) which is also runnable from the command line. E.g. in the pithos package there is a test module which contains, among others, the **download** sub-test:
.. code-block:: console
$ cd pithos
# Run all kamaki.clients.pithos tests
$ python test.py
# Run all kamaki.clients.pithos.PithoClient tests
$ python test.py Pithos
# Test kamaki.clients.pithos.PithosClient.download
$ python test.py Pithos download
To fully test a specific package, run test.py from the package location. E.g. to test everything in kamaki.clients.pithos package:
.. code-block:: console
$ cd pithos
$ python test.py
Mechanism
^^^^^^^^^
Each folder / package contains a test.py file, that represents the test module of this package. All test modules contain a set of classes that extent the TestCase class. They also contain a main method to run the tests.
By convention, testing classes are named as <Tested Class> where <Test Class> is the name of the tested class or module. Methods not grouped in classes are tested by classes named after their respective module.
For example, the kamaki.clients.pithos.PithosClient class is tested by the kamaki.clients.pithos.test.PithosClient class, while the methods in kamaki.clients.utils module are tested by the kamaki.clients.utils.test.Utils testing class.
Adding unit tests
^^^^^^^^^^^^^^^^^
After modifying or extending kamaki.clients method, classes, modules or packages, it is a good practice to also modify or extend the corresponding unit tests. What's more, it is recommended to modify or implement the testing of new behavior before implementing the behavior itself. The aim for kamaki.clients package is an 1 to 1 mapping between methods and their tests.
Modifying an existing method
""""""""""""""""""""""""""""
In case of an existing method modification, the programmer has to modify the corresponding test as well. By convention, the test method is located in the test module under the same package, in a TestCase subclass that is named with a name similar to the package or class that contains the tested method.
Example 1: to modify the kamaki.clients.utils.filter_in method, the programmer has to also adjust the kamaki.clients.utils.test.Utils.test_filter_in method.
Example 2: to modify the kamaki.clients.pithos.PithosRestClient.object_get, the programmer has to also adjust the kamaki.clients.pithos.test.PithosRestClient.test_object_get method.
Adding a new method
"""""""""""""""""""
Programmers who want to implement a new method in an existing class, are encouraged to implement the corresponding unit test first. In order to do that, they should find the testing class that is mapped to the class or module they need to extend.
Example 1: To add a **list_special** method to kamaki.clients.astakos.AstakosClient, extend the kamaki.clients.astakos.test.AstakosClient class, as shown bellow:
.. code-block:: python
# file: ${kamaki}/kamaki/clients/astakos/__init__.py
class AstakosClient(TestCase):
...
def test_list_special(self):
"""Test the list_special method"""
...
Example 2: To add a **get_random_int** method in kamaki.clients.utils module, extend the kamaki.clients.utils.test.Utils test class, as shown bellow:
.. code-block:: python
# file: ${kamaki}/kamaki/clients/utils/__init__.py
class Utils(TestCase):
...
def test_get_random_int(self):
"""Test the get_random_int method"""
...
Implementing a new class or module
""""""""""""""""""""""""""""""""""
Each class or module needs a seperate test sub-module. By convention, each class or module under the kamaki.clients should be located in a separate directory.
Example 1: To add a NewService class that implements the kamaki.clients.Client interface:
* create a new_service package and implement the unit tests in the kamaki.clients.new_service.test module:
.. code-block:: console
$ mkdir new_service && touch new_service/test.py
* create the file that will hold the package code and implement the module there:
.. code-block:: console
$ touch new_service/__init__.py
* Create the test class and methods in kamaki.clients.new_service.test
.. code-block:: python
# file: ${kamaki}/kamaki/clients/new_service/test.py
from unittest import TestCase
class NewService(TestCase):
def test_method1(self):
...
* Create the NewService and its actual functionality in kamaki.clients.new_service
.. code-block:: python
# file: ${kamaki}/kamaki/clients/new_service/__init__.py
from kamaki.clients import Client
class NewService(Client):
def method1(self, ...):
...
* Expose the new tests to top test module, by importing the test class to kamaki.clients.test
..code-block:: python
# file: ${kamaki}/kamaki/clients/test.py
from kamaki.clients.new_service.test import NewService
.. note:: If the new class or module is part of an existing sub-package, it is acceptable to append its testing class in the existing test.py file of the sub-package it belongs to. For example, the kamaki.clients.pithos.PithosClient and kamaki.clients.pithos.rest_api.PithosRestClient classes are tested by two different classes (PithosClient and PithosRestClient respectively) in the same module (kamaki.clients.pithos.test).
......@@ -10,7 +10,7 @@ Kamaki project documentation
./kamaki is a simple, yet intuitive, multipurpose, interactive command-line tool and client API for managing clouds.
As a develpment API is an initial implementation of the OpenStack Compute API, v1.1, with custom extensions specific to the `Synnefo IaaS <http://synnefo.org/>`_ cloud management software.
As a develpment API is an initial implementation of an OpenStack inspired API designed for the `Synnefo IaaS <http://www.synnefo.org/>`_ cloud management software.
./kamaki is open source and released under a 2-clause BSD License.
......
......@@ -6,7 +6,9 @@ This guide describes the standard installation process for kamaki, with the aspi
* Kamaki repository: `http://code.grnet.gr/git/kamaki <http://code.grnet.gr/git/kamaki>`_
* Synnefo Linux packages: `http://apt.dev.grnet.gr <http://apt.dev.grnet.gr>`_, `http://apt2.dev.grnet.gr <http://apt2.dev.grnet.gr>`_
* Kamaki at pypi: `http://pypi.python.org/pypi/kamaki <https://pypi.python.org/pypi/kamaki>`_
* Synnefo Linux packages: `http://apt2.dev.grnet.gr <http://apt2.dev.grnet.gr>`_
Linux and Unix-like enviroments
-------------------------------
......@@ -18,7 +20,6 @@ The following steps describe a command-line approach, but any graphic package ma
* As root, append the following to */etc/apt/sources.list* ::
deb http://apt.dev.grnet.gr/ squeeze main
deb http://apt2.dev.grnet.gr stable/
* Make sure the GPG public key for the GRNET dev team is added:
......@@ -65,18 +66,28 @@ The following steps describe a command-line approach, but any graphic package ma
$ sudo apt-get install kamaki
Install ansicolors and/or progress (Optional but recommended)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Install ansicolors (optional but recommended)
"""""""""""""""""""""""""""""""""""""""""""""
.. code-block:: console
$ sudo apt-get install python-ansicolors
$ sudo apt-get install python-progress
.. _installing-from-source-ref:
Install mock (for developers only)
""""""""""""""""""""""""""""""""""
.. code-block:: console
$ sudo apt-get install python-mock
.. warning:: kamaki.clients unit-tests need python-mock 1.X or better. e.g.::
Installing from source (git repos.)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
$ sudo apt-get install python-mock=1.0.1
.. _installing-from-pypi-ref:
Installing from pypi
^^^^^^^^^^^^^^^^^^^^
Requirements
""""""""""""
......@@ -102,83 +113,68 @@ With virtualenv users can setup kamaki and synnefo services in a sandbox environ
A more detailed example of using virtual env can be found at the `snf-image-creator setup guide <http://docs.dev.grnet.gr/snf-image-creator/latest/install.html#python-virtual-environment>`_
Install objpool (was: snf-common)
"""""""""""""""""""""""""""""""""
Since 0.6.2, kamaki is based on python-objpool. The objpool package is easy to install from source (even on windows platforms):
.. code-block:: console
$ git clone http://code.grnet.gr/git/objpool
$ cd objpool
$ ./setup build install
$ cd -
Install kamaki
""""""""""""""
Kamaki can be downloaded from `this location <https://code.grnet.gr/projects/kamaki/files>`_, where users can pick the version they prefer and unzip it locally:
.. code-block:: console
$ tar xvfz kamaki-0.6.2.tar.gz
or it can be downloaded directly from the git repository:
$ pip install kamaki
.. code-block:: console
$ git clone http://code.grnet.gr/git/kamaki
Install ansicolors
""""""""""""""""""
and then installed by the setup script:
The **ansicolors** package is not required for running kamaki, but it is
recommended as a user experience improvement. In specific, ansicolors
adds colors to kamaki responses.
.. code-block:: console
$ cd kamaki
$ ./setup build install
Install progress and/or ansicolors (optional)
"""""""""""""""""""""""""""""""""""""""""""""
$ pip install ansicolors