Commit 55c36638 authored by Stavros Sachtouris's avatar Stavros Sachtouris

Devise a CI mechanism based on docker containers

Rename "paste_deploy/" to "ci/" and add a Dockerfile to create an
image which runs the snf-occi service.
Write instructions in ci/README.md on how to build and run the
container.
Update the docs and fix emerging bugs and typos.
parent 12b2c6cc
# Copyright (C) 2016 GRNET S.A.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# VERSION: 0.1
#
# docker run -ti --name occi-ci --net host -p 127.0.0.1:8080:8080 \
# -e AUTH_URL='https://accounts.okeanos.grnet.gr/identity/v2.0' \
# -e KEYSTONE_URL='https://okeanos-astavoms.hellasgrid.gr' -d \
# snf-occi-ci:latest
# docker logs -ft occi-ci
FROM debian:jessie
MAINTAINER Stavros Sachtouris <saxtouri@grnet.gr>
RUN apt update && apt upgrade -y \
&& apt install -y python-pip python-dev git curl
RUN pip install Paste PasteScript pytz ooi
RUN git clone https://github.com/grnet/snf-occi.git
COPY config.py /snf-occi/soi/config.py
RUN cd /snf-occi && python setup.py clean install
EXPOSE 8080
CMD ["python", "/snf-occi/ci/run-server.py"]
There are two options to setup a testing snf-occi deployment: with docker or
from scratch.
Deploy with docker (recommended)
---
Use docker to build the snf-occi image and deploy the snf-occi in a container.
::
$ echo "Build the docker snf-occi-ci image"
$ docker build -t snf-occi-ci https://github.com/grnet/snf-occi.git#master:ci
...
$ echo "Run the occi-ci container"
$ docker run -ti --name occi-ci --net host -p 127.0.0.1:8080:8080 \
-e AUTH_URL='https://accounts.okeanos.grnet.gr/identity/v2.0' \
-e KEYSTONE_URL='https://okeanos-astavoms.hellasgrid.gr' -d \
snf-occi-ci
...
$ echo "(optional) check the service logs"
$ docker logs -f occi-ci
serving on http://127.0.0.1:8080
...
.. note: To use a different host port, try "-p 127.0.0.1:<PORT>:8080" e.g.,
"-p 127.0.0.1:9000:8080"
Deploy from scratch
---
Here is how to run a local paste server. This is useful only for experimenting
and development and should not be used in production. We suggest to run this
test in a sandboxed environment e.g., virtualenv
::
$ virtualenv mytest
$ source mytest/bin/activate
$ pip install Paste PasteDeploy
$ cp soi/config.py.template soi/config.py
$ vim soi/config.py
... (set the appropriate values here) ...
$ python setup.py install
$ python ci/run-server.py .
serving on http://127.0.0.1:8080
...
# Copyright (C) 2016 GRNET S.A.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>
# Copy this file as config.py and fill in the appropriate values
import os
AUTH_URL = os.environ.get('AUTH_URL')
CA_CERTS = os.environ.get('CA_CERTS', '/etc/ssl/certs/ca-certificates.crt')
KEYSTONE_URL = os.environ.get('KEYSTONE_URL')
HOST = os.environ.get('HOST', '127.0.0.1')
PORT = os.environ.get('PORT', '8080')
PASTE_INI = '/snf-occi/ci/soi.ini'
......@@ -15,7 +15,7 @@
from paste import deploy
import logging
from soi.config immport PASTE_INI
from soi.config import PASTE_INI
LOG = logging.getLogger(__name__)
......
......@@ -16,7 +16,16 @@
from paste import deploy
import logging
from paste import httpserver
from soi.config import PASTE_INI, HOST, PORT
try:
from soi.config import PASTE_INI, HOST, PORT
except (ImportError, NameError) as err:
from sys import stderr, exit
stderr.write('{err}\n'.format(err=err))
stderr.write(
'Make sure soi/config.py exists and contains appropriate values.\n')
stderr.write('Refer to the documentation for more information\n')
exit(1)
LOG = logging.getLogger(__name__)
......
......@@ -13,9 +13,6 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# snf_voms authentication PasteDeploy configuration file
[composite:main]
use = egg:Paste#urlmap
/:snf_occiapp
......
......@@ -23,9 +23,9 @@ copyright = u'2012-2016, GRNET'
# built documents.
#
# The short X.Y version.
version = '0.2'
version = '0.3'
# The full version, including alpha/beta/rc tags.
release = '0.2'
release = '0.3'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
......
......@@ -6,26 +6,32 @@
About snf-occi
==============
**snf-occi** snf-occi implements the OCCI specification on top of Synnefo’s API in order to achieve greater interoperability in common tasks referring cyclades management. This module is a translation bridge between OCCI and the Openstack API and is designed to be as independent as possible from the rest IaaS, providing an OCCI compatibility layer to other services using Openstack API.
**snf-occi** snf-occi is an OCCI API proxy for the Synnefo IaaS cloud. The
purpose of snf-occi is to provide and OCCI REST API for Synnefo's Cyclades API,
which provides compute, networking and volume services.
**snf-occi** is based in modules provided by kamaki library-tool when dealing with REST API calls to Openstack.
The implementation of snf-occi is based on Openstack OCCI Interface (OOI)
project, since version 0.3. OOI is a WSGI proxy for OpenStack. Synnefo API is
(almost) compatible with OpenStack, so snf-occi patches OOI wherever needer to
ensure the desired API compatibility. Clients running on OOI should be able to
run on snf-occi as well.
**snf-occi** is depends on kamaki library for communicating with Synnefo, and
OOI for receiving and understanding OCCI calls.
.. toctree::
:maxdepth: 2
About Open Cloud Computing Interface (OCCI)
-------------------------------------------
The current OCCI specification consists of the following three documents:
* `OCCI Core <http://ogf.org/documents/GFD.183.pdf>`_
* `OCCI Infrastructure <http://ogf.org/documents/GFD.184.pdf>`_
* `OCCI HTTP rendering <http://ogf.org/documents/GFD.185.pdf>`_
The master document for the OCCI specification is at `OCCI Specification <http://occi-wg.org/about/specification/>`_
OCCI unifies various cloud compute services in a common API and a product of
the EGI initiative. For more information on OCCI: www.ogf.org
OCCI and Cyclades
-----------------
The OCCI implementation for Cyclades is going to be based in the OCCI 1.1 Infrastructure specification, in which common Cloud IaaS components are described. The correspondence between OCCI and Cyclades is as follows:
The OCCI implementation for Cyclades currently implements OCCI 1.1
Infrastructure specification. The correspondence between OCCI and Cyclades is
as follows:
+-------------------------+-------------------------+
|OCCI |Cyclades |
......@@ -36,167 +42,153 @@ The OCCI implementation for Cyclades is going to be based in the OCCI 1.1 Infras
+-------------------------+-------------------------+
|Resource Template |Synnefo flavors |
+-------------------------+-------------------------+
|Networking |NA |
|Networking |Synnefo Networking |
+-------------------------+-------------------------+
|Storage |NA |
|Storage |Synnefo Volumes |
+-------------------------+-------------------------+
**Note:** Metadata info in Synnefo's servers cannot be represented (clearly) using OCCI's components.
.. Note: Metadata info in Synnefo's servers cannot be represented (clearly) using OCCI's components.
OCCI requirements
-----------------
Due to OCCI's structure there cannot be straightforward mapping to Cyclades/OpenStack API. The missing elements are networking and storage capabilities using current Cyclades API.
Current progress
=================
We currently have an adequate implementation of the **compute** API and we are
planning to extend it for **networking** and **volumes** as soon as possible.
We will also provide the corresponding implementations for OCCI 1.2, as soon as
they are implemented in OOI.
OCCI operations
****************
Installation
-------------
Below you can see the required procedures/operations for OCCI compatibility.
* Handling the query interface
* Query interface must be found under path /-/
* Retrieve all registered Kinds, Actions and Mixins
* Add a mixin definition
* Remove a mixin definition
The following instructions have been tested on Debian Jessy with pythoh-pip
installed. It is suggested to install Paste and PasteScript for deployment.
* Operation on paths in the name-space
* Retrieving the state of the name-space hierarchy
* Retrieving all Resource instances below a path
* Deletion of all Resource instances below a path
::
* Operations on Mixins and Kinds
* Retrieving all Resource instances belonging to Mixin or Kind
* Triggering actions to all instances of a Mixin or a Kind
* Associate resource instances with a Mixin or a Kind
* Full update of a Mixin collection
* Dissociate resource instances from a Mixin
$ git clone https://github.com/grnet/snf-occi
$ cd snf-occi
$ cp soi/config.py.template snfOCCI/config.py
$ vim soi/confg.py
...
$ python setup.py install
* Operations on Resource instances
* Creating a resource instance
* Retrieving a resource instance
* Partial update of a resource instance
* Full update of a resource instance
* Delete a resource instance
* Triggering an action on a resource instance
**NOTE**: edit the **config.py** before running the service. In the following
example, we replicate the settings of the hellasgrid service, but you can set
your own cloud and astavoms settings in your own deployment
* Handling Link instances
* Inline creation of a Link instance
* Retrieving Resource instances with defined Links
* Creating of Link Resource instance
::
# Copy this file as config.py and fill in the appropriate values
AUTH_URL = 'https://accounts.okeanos.grnet.gr/identity/v2.0'
CA_CERTS = '/etc/ssl/certs/ca-certificates.crt'
KEYSTONE_URL = 'https://okeanos-astavoms.hellasgrid.gr'
OCCI client/server library
==========================
HOST = '127.0.0.1'
PORT = '8080'
PASTE_INI = '/path/to/snf-occi/ci/soi.ini''
pyssf is a collection of OCCI python modules. It aims to provide a high-level interface for the integration of OCCI to other new or existing applications.
snf-occi is a simple WSGI python application with basic paste support. A full
scale deployment is out of the scope of this document, but deployments with
apache and gunicorn have been tested and work well.
Features:
---------
Running with docker
===================
* It includes a REST API service with the OCCI specifications already implemented
* It only requires a custom backend and registry to interact with Cyclades
To test snf-occi, you can build and use a docker image, as described in the
"ci/README.md" file, or just follow these steps:
Current progress
=================
By now we have considered implementing only the **Compute** backend of the OCCI to Cyclades/Openstack API bridge and we are planning to extend it for **networking** and **storage** capabilities. It is possible to implement the remaining capabilities directly for OCCI 1.2, though.
::
Installation
-------------
Install **snf-occi** API translation server by cloning our latest source code:
$ docker build -t snf-occi-ci https://github.com/grnet/snf-occi#master:ci
$ docker run -ti --name occi-ci --net host -p 127.0.0.1:8080:8080 \
-e AUTH_URL='https://accounts.okeanos.grnet.gr/identity/v2.0' \
-e KEYSTONE_URL='https://okeanos-astavoms.hellasgrid.gr' -d \
snf-occi-ci
::
git clone https://github.com/grnet/snf-occi
cd snf-occi
cp snfOCCI/config.py.template snfOCCI/config.py
python setup.py install
Testing
=======
**NOTE**: edit the **config.py** before running the service. In the following
example, we replicate the settings of the hellasgrid service, but you can set
your own cloud and astavoms settings in your own deployment
Functional tests
----------------
The current snf-occi has good test coverage. Two equivalent sets of functional
tests (curl and rOCCI) in the form of bash scripts can be found in
"soi/tests/functional/". To run the script, make sure you have setup following
environment variables:
::
# Copy this file as config.py and fill in the appropriate values
COMPUTE = {
'arch': 'x86',
}
KAMAKI = {
'compute_url': 'https://cyclades.okeanos.grnet.gr/compute/v2.0/',
'astakos_url': 'https://accounts.okeanos.grnet.gr/identity/v2.0',
'network_url': 'https://cyclades.okeanos.grnet.gr/network/v2.0'
}
KEYSTONE_URL = 'https://okeanos-astavoms.hellasgrid.gr'
HOSTNAME = 'https://okeanos-occi2.hellasgrid.gr:9000'
PASTEDEPLOY = '/home/user/src/snf-occi/snfOCCI/paste_deploy/snf-occi-paste.ini'
$ export OCCI_ENDPOINT="http:127.0.0.1:8080"
$ export TOKEN="Your-Synnefo-User-Token"
$ export USER_PROXY="/path/to/your/VOMS/proxy"
$ export OS_TPL="13"
$ export RESOURCE_TPL="6f1f7205-cf4c-4b8c-ae77-7c419747bcbd"
The USER_PROXY is only needed if you run the rOCCI-based "run_function_tests.sh",
while TOKEN is only needed if you run the "run_curl_tests.sh" script.
snf-occi is a simple WSGI python application with basic paste support. A full scale deployment is out of the scope of this document, but it is expected to use standard tools like apache and gunicorn to setup the service.
You can setup a docker client for testing. Follow the
"soi/tests/functional/README.md" instructions and test the application with the
prepared queries or the examples of the current document. The instructions will
guide you to build a docker image based on the
`egifedcloud/fedcloud-userinterface` with all the functional tests loaded in
the container.
Examples:
---------
For the examples below we assume server is running on localhost (port 8888) and authentication token is $AUTH. For the HTTP requests we are using **curl**.
Examples
--------
For the examples below we assume server is running on localhost (port 8080) and authentication token is $TOKEN. For the HTTP requests we are using **curl**. All
these tests are also programmed in "soi/tests/functional/run_curl_tests.sh"
* Retrieve all registered Kinds, Actions and Mixins:
::
curl -v -X GET localhost:8888/-/ -H 'Auth-Token: $AUTH'
curl -X GET localhost:8080/-/ -H 'X-Auth-Token: $TOKEN'
* Create a new VM described by the flavor 'C2R2048D20' and using the image 'Debian'
* List all VMs:
::
curl -X GET localhost:8080/compute -H 'X-Auth-Token: $TOKEN'
* Create a new VM described by the flavor 13 and using the image
6f1f7205-cf4c-4b8c-ae77-7c419747bcbd:
::
curl -v -X POST localhost:8888/compute/
-H 'Category: compute; scheme=http://schemas.ogf.org/occi/infrastructure#; class="kind";'
-H 'X-OCCI-Attribute: occi.core.title = newVM' -H 'Category: C2R2048D20; scheme=http://schemas.ogf.org/occi/infrastructure#; '
-H 'Category: Debian; scheme=http://schemas.ogf.org/occi/infrastructure#;' -H 'Auth-Token: $AUTH'
-H 'Content-type: text/occi'
curl -X POST localhost:8080/compute/ \
-H 'Category: compute; scheme=http://schemas.ogf.org/occi/infrastructure#; class="kind";' \
-H 'X-OCCI-Attribute: occi.core.title = newVM' -H 'Category: 13; scheme=http://schemas.ogf.org/occi/infrastructure#;' \
-H 'Category: 6f1f7205-cf4c-4b8c-ae77-7c419747bcbd; scheme=http://schemas.ogf.org/occi/infrastructure#;' \
-H 'X-Auth-Token: $TOKEN' -H 'Content-type: text/occi'
* Retrieve all the details of th VM with identifier $ID
::
curl -v -X GET localhost:8888/compute/$ID -H 'Auth-Token: $AUTH'
curl -X GET localhost:8080/compute/$ID -H 'X-Auth-Token: $TOKEN'
* Delete the VM with identifier $ID
* Start/Stop/Restart a VM with $ID:
::
curl -v -X DELETE localhost:8888/compute/$ID -H 'Auth-Token: $AUTH'
Testing
-------
Here is how to run a local paste server. This is useful only for experimenting
and development and should not be used in production. We suggest to run this
test in a sandboxed environment e.g., virtualenv
curl -X POST -H 'X-Auth-Token: $TOKEN' localhost:8080/compute/$ID?action=start \
-H 'Category: start ; scheme=\"http://schemas.ogf.org/occi/infrastructure/compute/action#\"; class=\"action\"'
::
virtualenv mytest
source mytest/bin/activate
pip install Paste PasteDeploy
cp snfOCCI/paste_deploy/test-server.py .
python test-server.py
server is running on 127.0.0.1:8080
curl -X POST -H 'X-Auth-Token: $TOKEN' localhost:8080/compute/$ID?action=stop \
-H 'Category: stop ; scheme=\"http://schemas.ogf.org/occi/infrastructure/compute/action#\"; class=\"action\"'
Follow the test/README.md instructions to setup a client e.g., with docker, and
test the application with the prepared queries or the examples of the current
document.
curl -X POST -H 'X-Auth-Token: $TOKEN' localhost:8080/compute/$ID?action=restart \
-H 'Category: restart ; scheme=\"http://schemas.ogf.org/occi/infrastructure/compute/action#\"; class=\"action\"'
A smart way to test the application is by using the `egifedcloud/fedcloud-userinterface`. Make sure you have valid and authorized proxy certificates in your ${HOME}/.globus directory, and then start a cointainer shell loaded with all necessary client tools. E.g., to perform a "list servers" operation:
* Delete the VM with identifier $ID
::
$ docker run -v /home/user/.globus:/root/.globus -it egifedcloud/fedcloud-userinterface /bin/bash
# fetch-crl -p 20
# voms-proxy-init --voms fedcloud.egi.eu -rfc
Your proxy is stored at /tmp/x509up_u0
# occi --endpoint https://snf-occi.example.com --action list --resource compute -n x509 -x /tmp/x509up_u0 -X
curl -X DELETE localhost:8080/compute/$ID -H 'X-Auth-Token: $TOKEN'
Indices and tables
......
......@@ -26,5 +26,5 @@ setup(
[paste.app_factory]
snf_occi_app=soi:main
''',
install_requires=['kamaki', 'ooi', ]
install_requires=['kamaki', 'ooi==0.3.2', ]
)
......@@ -21,4 +21,4 @@ KEYSTONE_URL = 'https://okeanos-astavoms.example.com'
HOST = 'this-host.example.com or IP'
PORT = '8080'
PASTE_INI = '/home/user/snf-occi/paste_deploy/soi.ini'
PASTE_INI = '/home/user/snf-occi/ci/soi.ini'
# Copyright (C) 2016 GRNET S.A.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Dockerfile for snf-occi-manual-ci
# contains functional testing scripts
#
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment