Commit cb51ce93 authored by Christos Stavrakakis's avatar Christos Stavrakakis
Browse files

Merge branch 'release-0.13' into develop

Conflicts:
	version
parents 9a43e008 de1fe683
......@@ -101,6 +101,7 @@ General Synnefo dependencies
* postgresql (database)
* rabbitmq (message queue)
* ntp (NTP daemon)
* gevent
You can install apache2, progresql and ntp by running:
......@@ -115,6 +116,12 @@ the official debian backports:
# apt-get -t squeeze-backports install gunicorn
Also, make sure to install gevent >= 0.13.6. Again from the debian backports:
.. code-block:: console
# apt-get -t squeeze-backports install python-gevent
On node1, we will create our databases, so you will also need the
python-psycopg2 package:
......@@ -209,6 +216,7 @@ Create the file ``synnefo`` under ``/etc/gunicorn.d/`` containing the following:
'group': 'www-data',
'args': (
'--bind=127.0.0.1:8080',
'--worker-class=gevent',
'--workers=8',
'--log-level=debug',
),
......@@ -338,6 +346,7 @@ General Synnefo dependencies
* gunicorn (WSGI http server)
* postgresql (database)
* ntp (NTP daemon)
* gevent
You can install the above by running:
......@@ -352,6 +361,12 @@ the official debian backports:
# apt-get -t squeeze-backports install gunicorn
Also, make sure to install gevent >= 0.13.6. Again from the debian backports:
.. code-block:: console
# apt-get -t squeeze-backports install python-gevent
Node2 will connect to the databases on node1, so you will also need the
python-psycopg2 package:
......@@ -386,6 +401,7 @@ Create the file ``synnefo`` under ``/etc/gunicorn.d/`` containing the following
'group': 'www-data',
'args': (
'--bind=127.0.0.1:8080',
'--worker-class=gevent',
'--workers=4',
'--log-level=debug',
'--timeout=43200'
......@@ -629,8 +645,8 @@ To use, first monkey-patch psycopg2. For Django, run this before the
from synnefo.lib.db.pooled_psycopg2 import monkey_patch_psycopg2
monkey_patch_psycopg2()
If running with greenlets, we should modify psycopg2 behavior, so it works
properly in a greenlet context:
Since we are running with greenlets, we should modify psycopg2 behavior, so it
works properly in a greenlet context:
.. code-block:: console
......@@ -899,9 +915,9 @@ Pithos is pooling-ready without the need of further configuration, because it
doesn't use a Django DB. It pools HTTP connections to Astakos and pithos
backend objects for access to the Pithos DB.
However, as in Astakos, if running with Greenlets, it is also recommended to
modify psycopg2 behavior so it works properly in a greenlet context. This means
adding the following lines at the top of your
However, as in Astakos, since we are running with Greenlets, it is also
recommended to modify psycopg2 behavior so it works properly in a greenlet
context. This means adding the following lines at the top of your
``/etc/synnefo/10-snf-webproject-database.conf`` file:
.. code-block:: console
......@@ -909,29 +925,6 @@ adding the following lines at the top of your
from synnefo.lib.db.psyco_gevent import make_psycopg_green
make_psycopg_green()
Furthermore, add the ``--worker-class=gevent`` argument on your
``/etc/gunicorn.d/synnefo`` configuration file. The file should look something like
this:
.. code-block:: console
CONFIG = {
'mode': 'django',
'environment': {
'DJANGO_SETTINGS_MODULE': 'synnefo.settings',
},
'working_dir': '/etc/synnefo',
'user': 'www-data',
'group': 'www-data',
'args': (
'--bind=127.0.0.1:8080',
'--workers=4',
'--worker-class=gevent',
'--log-level=debug',
'--timeout=43200'
),
}
Servers Initialization
----------------------
......@@ -1013,75 +1006,21 @@ For the purpose of this guide, we will assume that the :ref:`GANETI-MASTER
:ref:`GANETI-NODE <GANETI_NODES>` and is Master-capable and VM-capable too.
We highly recommend that you read the official Ganeti documentation, if you are
not familiar with Ganeti. If you are extremely impatient, you can result with
the above assumed setup by running on both nodes:
.. code-block:: console
# apt-get install -t squeeze-backports ganeti2 ganeti-htools
# modprobe drbd minor_count=255 usermode_helper=/bin/true
Unfortunatelly, stock Ganeti doesn't support IP pool management yet (we are
working hard to merge it upstream for Ganeti 2.7). Synnefo depends on the IP
pool functionality of Ganeti, so you have to use GRNET's patches for now. To
do so you have to build your own package from source. Please clone our local
repo:
.. code-block:: console
# git clone https://code.grnet.gr/git/ganeti-local
# cd ganeti-local
# git checkout stable-2.6-ippool-hotplug-esi
# git checkout debian-2.6
Then please check if you can complile ganeti:
not familiar with Ganeti.
.. code-block:: console
# cd ganeti-local
# ./automake.sh
# ./configure
# make
To do so you must have a correct build environment. Please refer to INSTALL
file in the source tree. Most of the packages needed are refered here:
.. code-block:: console
# apt-get install graphviz automake lvm2 ssh bridge-utils iproute iputils-arping \
ndisc6 python python-pyopenssl openssl \
python-pyparsing python-simplejson \
python-pyinotify python-pycurl socat \
python-elementtree kvm qemu-kvm \
ghc6 libghc6-json-dev libghc6-network-dev \
libghc6-parallel-dev libghc6-curl-dev \
libghc-quickcheck2-dev hscolour hlint
python-support python-paramiko \
python-fdsend python-ipaddr python-bitarray libjs-jquery fping pandoc
Now lets try to build the package:
Unfortunatelly, the current stable version of the stock Ganeti (v2.6.2) doesn't
support IP pool management. This feature will be available in Ganeti >= 2.7.
Synnefo depends on the IP pool functionality of Ganeti, so you have to use
GRNET provided packages until stable 2.7 is out. To do so:
.. code-block:: console
# apt-get install git-buildpackage
# mkdir ../build-area
# git-buildpackage --git-upstream-branch=stable-2.6-ippool-hotplug-esi \
--git-debian-branch=debian-2.6 \
--git-export=INDEX \
--git-ignore-new
To be able to sign the packages a key must be found in the system to comply to the
name and email of the last debian/changelog entry in debian branch. Please note
that signing is optional.
This will create two deb packages in build-area. You should then run in both
nodes:
# apt-get install snf-ganeti ganeti-htools
# modprobe drbd minor_count=255 usermode_helper=/bin/true
.. code-block:: console
You should have:
# dpkg -i ../build-area/ganeti-htools.*deb
# dpkg -i ../build-area/snf-ganeti.*deb
# apt-get install -f
Ganeti >= 2.6.2+ippool11+hotplug5+extstorage3+rdbfix1+kvmfix2-1
We assume that Ganeti will use the KVM hypervisor. After installing Ganeti on
both nodes, choose a domain name that resolves to a valid floating IP (let's
......@@ -1653,10 +1592,6 @@ corresponding package by running on node1:
# apt-get install snf-cyclades-app
.. warning:: Make sure you have installed ``python-gevent`` version >= 0.13.6.
This version is available at squeeze-backports and can be installed by
running: ``apt-get install -t squeeze-backports python-gevent``
If all packages install successfully, then Cyclades and Plankton are installed
and we proceed with their configuration.
......@@ -1867,7 +1802,6 @@ Ganeti MASTER:
$ gnt-network list
$ gnt-network info <network_name>
Create pools for Private Networks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
......
......@@ -43,10 +43,11 @@ from astakos.im.settings import (
QUOTAHOLDER_URL, QUOTAHOLDER_TOKEN, LOGGING_LEVEL)
if QUOTAHOLDER_URL:
from kamaki.clients.quotaholder import QuotaholderClient
from kamaki.clients.quotaholder import QH_PRACTICALLY_INFINITE
from synnefo.lib.quotaholder import (
QuotaholderClient, QH_PRACTICALLY_INFINITE)
from synnefo.util.number import strbigdec
from astakos.im.settings import QUOTAHOLDER_POOLSIZE
ENTITY_KEY = '1'
......@@ -66,7 +67,8 @@ def get_client():
return _client
if not QUOTAHOLDER_URL:
return
_client = QuotaholderClient(QUOTAHOLDER_URL, token=QUOTAHOLDER_TOKEN)
_client = QuotaholderClient(QUOTAHOLDER_URL, token=QUOTAHOLDER_TOKEN,
poolsize=QUOTAHOLDER_POOLSIZE)
return _client
......@@ -507,7 +509,8 @@ def timeline_charge(entity, resource, after, before, details, charge_type):
m = 'charge type %s not supported' % charge_type
raise ValueError(m)
quotaholder = QuotaholderClient(QUOTAHOLDER_URL, token=QUOTAHOLDER_TOKEN)
quotaholder = QuotaholderClient(QUOTAHOLDER_URL, token=QUOTAHOLDER_TOKEN,
poolsize=QUOTAHOLDER_POOLSIZE)
timeline = quotaholder.get_timeline(
context={},
after=after,
......
......@@ -159,6 +159,7 @@ PROJECT_MEMBERSHIP_LEAVE_REQUEST_SUBJECT = getattr(
# Set the quota holder component URI
QUOTAHOLDER_URL = getattr(settings, 'ASTAKOS_QUOTAHOLDER_URL', '')
QUOTAHOLDER_TOKEN = getattr(settings, 'ASTAKOS_QUOTAHOLDER_TOKEN', '')
QUOTAHOLDER_POOLSIZE = getattr(settings, 'ASTAKOS_QUOTAHOLDER_POOLSIZE', 50)
# Set the cloud service properties
SERVICES = getattr(settings, 'ASTAKOS_SERVICES', {
......
......@@ -91,7 +91,8 @@ def login(
shibboleth_headers = {}
for token in dir(Tokens):
if token == token.upper():
shibboleth_headers[token] = tokens.get(token, 'NOT_SET')
shibboleth_headers[token] = request.META.get(token, 'NOT_SET')
# log shibboleth headers
# TODO: info -> debug
logger.info("shibboleth request: %r" % shibboleth_headers)
......
......@@ -111,6 +111,12 @@
# Set the quotaholder component URI and token
#ASTAKOS_QUOTAHOLDER_URL = ''
#ASTAKOS_QUOTAHOLDER_TOKEN = ''
#
# Tune the size of the quotaholder http client connection pool.
# This limits the number of concurrent requests to quotaholder.
# If quotaholder is installed in the same system as astakos,
# This must be at most half the synnefo_poolsize for the django database
#ASTAKOS_QUOTAHOLDER_POOLSIZE = 50
# Setup quotaholder URL and token when snf-quotaholder-app is installed
# in the same server as astakos (recommended)
......
from .api import *
from .http import QuotaholderClient
QH_PRACTICALLY_INFINITE = 10 ** 32
......@@ -32,10 +32,11 @@
# or implied, of GRNET S.A.
from .quotaholder import QuotaholderAPI, QH_PRACTICALLY_INFINITE
from .exception import ( InvalidKeyError, NoEntityError,
NoQuantityError, NoCapacityError,
ExportLimitError, ImportLimitError,
CorruptedError, InvalidDataError,
DuplicateError)
from .exception import (CallError,
InvalidKeyError, NoEntityError,
NoQuantityError, NoCapacityError,
ExportLimitError, ImportLimitError,
CorruptedError, InvalidDataError,
DuplicateError)
API_Spec = QuotaholderAPI
# Copyright 2012 GRNET S.A. All rights reserved.
#
# Redistribution and use in source and binary forms, with or
# without modification, are permitted provided that the following
# conditions are met:
#
# 1. Redistributions of source code must retain the above
# copyright notice, this list of conditions and the following
# disclaimer.
#
# 2. Redistributions in binary form must reproduce the above
# copyright notice, this list of conditions and the following
# disclaimer in the documentation and/or other materials
# provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
#
# The views and conclusions contained in the software and
# documentation are those of the authors and should not be
# interpreted as representing official policies, either expressed
# or implied, of GRNET S.A.
from synnefo.lib.commissioning import Callpoint, CallError
from synnefo.lib.pool.http import get_http_connection
from .api import QuotaholderAPI
from json import loads as json_loads, dumps as json_dumps
import logging
from urlparse import urlparse
logger = logging.getLogger(__name__)
class QuotaholderClient(Callpoint):
api_spec = QuotaholderAPI()
def __init__(self, base_url, token='', poolsize=1000):
super(QuotaholderClient, self).__init__()
self._url = base_url
parsed = urlparse(base_url)
self._netloc = parsed.netloc
self._scheme = parsed.scheme
basepath = parsed.path
if not basepath.endswith('/'):
basepath += '/'
self._basepath = basepath
self._token = token
self._poolsize = poolsize
def do_make_call(self, api_call, data):
gettable = ['list', 'get', 'read']
method = ('GET' if any(api_call.startswith(x) for x in gettable)
else 'POST')
path = self._basepath + api_call
json_data = json_dumps(data)
logger.debug("%s %s\n%s\n<<<\n", method, path, json_data[:128])
headers = {'X-Auth-Token': self._token}
conn = get_http_connection(scheme=self._scheme, netloc=self._netloc,
pool_size=self._poolsize)
try:
conn.request(method, path, body=json_data, headers=headers)
resp = conn.getresponse()
finally:
conn.close()
logger.debug(">>>\nStatus: %s", resp.status)
body = resp.read()
logger.debug("\n%s\n<<<\n", body[:128] if body else None)
status = int(resp.status)
if status == 200:
return json_loads(body)
else:
try:
error = json_loads(body)
except ValueError:
exc = CallError(body, call_error='ValueError')
else:
exc = CallError.from_dict(error)
raise exc
......@@ -12,4 +12,9 @@
#
# CYCLADES_QUOTAHOLDER_URL = "https://accounts.example.synnefo.org/quotaholder/v"
#
# CYCLADES_QUOTAHOLDER_TOKEN = ""
# # CYCLADES_QUOTAHOLDER_TOKEN = ""
# # Tune the size of the http pool for the quotaholder client.
# # It limits the maximum number of quota-querying requests
# # that Cyclades can serve. Extra requests will be blocked
# # until another has completed.
# CYCLADES_QUOTAHOLDER_POOLSIZE = 200
......@@ -13,3 +13,9 @@ CYCLADES_USE_QUOTAHOLDER = False
CYCLADES_QUOTAHOLDER_URL = "http://127.0.0.1:8008/api/quotaholder/v"
CYCLADES_QUOTAHOLDER_TOKEN = ""
# Tune the size of the http pool for the quotaholder client.
# It limits the maximum number of quota-querying requests
# that Cyclades can serve. Extra requests will be blocked
# until another has completed.
CYCLADES_QUOTAHOLDER_POOLSIZE = 200
......@@ -285,7 +285,7 @@ class VirtualMachine(models.Model):
name = models.CharField('Virtual Machine Name', max_length=255)
userid = models.CharField('User ID of the owner', max_length=100,
db_index=True)
db_index=True, null=False)
backend = models.ForeignKey(Backend, null=True,
related_name="virtual_machines",)
backend_hash = models.CharField(max_length=128, null=True, editable=False)
......
......@@ -265,11 +265,15 @@ def update_network_state(serials, network):
release_resource(res_type="bridge", value=network.link)
# Issue commission
serial = quotas.issue_network_commission(network.userid, delete=True)
serials.append(serial)
network.serial = serial
serial.accepted = True
serial.save()
if network.userid:
serial = quotas.issue_network_commission(network.userid,
delete=True)
serials.append(serial)
network.serial = serial
serial.accepted = True
serial.save()
elif not network.public:
log.warning("Network %s does not have an owner!", network.id)
network.save()
......
......@@ -65,7 +65,8 @@ import logging
from synnefo.settings import (CYCLADES_USE_QUOTAHOLDER,
CYCLADES_QUOTAHOLDER_URL,
CYCLADES_QUOTAHOLDER_TOKEN)
CYCLADES_QUOTAHOLDER_TOKEN,
CYCLADES_QUOTAHOLDER_POOLSIZE)
logger = logging.getLogger(__name__)
......@@ -117,12 +118,14 @@ class NotAllowedError(BackendException):
from pithos.backends.util import PithosBackendPool
POOL_SIZE = 8
_pithos_backend_pool = \
PithosBackendPool(POOL_SIZE,
quotaholder_enabled=CYCLADES_USE_QUOTAHOLDER,
quotaholder_url=CYCLADES_QUOTAHOLDER_URL,
quotaholder_token=CYCLADES_QUOTAHOLDER_TOKEN,
db_connection=settings.BACKEND_DB_CONNECTION,
block_path=settings.BACKEND_BLOCK_PATH)
PithosBackendPool(
POOL_SIZE,
quotaholder_enabled=CYCLADES_USE_QUOTAHOLDER,
quotaholder_url=CYCLADES_QUOTAHOLDER_URL,
quotaholder_token=CYCLADES_QUOTAHOLDER_TOKEN,
quotaholder_client_poolsize=CYCLADES_QUOTAHOLDER_POOLSIZE,
db_connection=settings.BACKEND_DB_CONNECTION,
block_path=settings.BACKEND_BLOCK_PATH)
def get_pithos_backend():
......
......@@ -37,15 +37,15 @@ from synnefo.settings import CYCLADES_USE_QUOTAHOLDER
if CYCLADES_USE_QUOTAHOLDER:
from synnefo.settings import (CYCLADES_QUOTAHOLDER_URL,
CYCLADES_QUOTAHOLDER_TOKEN)
from kamaki.clients.quotaholder import QuotaholderClient
CYCLADES_QUOTAHOLDER_TOKEN,
CYCLADES_QUOTAHOLDER_POOLSIZE)
from synnefo.lib.quotaholder import QuotaholderClient
else:
from synnefo.settings import (VMS_USER_QUOTA, MAX_VMS_PER_USER,
NETWORKS_USER_QUOTA, MAX_NETWORKS_PER_USER)
from kamaki.clients.quotaholder.api import (NoCapacityError, NoQuantityError,
NoEntityError)
from kamaki.clients.commissioning import CallError
from synnefo.lib.quotaholder.api import (NoCapacityError, NoQuantityError,
NoEntityError, CallError)
import logging
log = logging.getLogger(__name__)
......@@ -110,7 +110,8 @@ def get_quota_holder():
"""Context manager for using a QuotaHolder."""
if CYCLADES_USE_QUOTAHOLDER:
quotaholder = QuotaholderClient(CYCLADES_QUOTAHOLDER_URL,
token=CYCLADES_QUOTAHOLDER_TOKEN)
token=CYCLADES_QUOTAHOLDER_TOKEN,
poolsize=CYCLADES_QUOTAHOLDER_POOLSIZE)
else:
quotaholder = DummyQuotaholderClient()
......
......@@ -827,10 +827,14 @@
}
if (this.__added_flavors.ram.indexOf(values.mem) == -1) {
var mem = $(('<li class="option mem value-{0}">' +
var mem_value = parseInt(_.escape(values.mem))*1024*1024;
var displayvalue = synnefo.util.readablizeBytes(mem_value,
0).split(" ");
var mem = $(('<li class="option mem value-{2}">' +
'<span class="value">{0}</span>' +
'<span class="metric">MB</span></li>').format(
_.escape(values.mem))).data('value', values.mem);
'<span class="metric">{1}</span></li>').format(
displayvalue[0], displayvalue[1], values.mem)).data(
'value', values.mem);
this.mems.append(mem);
this.__added_flavors.ram.push(values.mem);
}
......
......@@ -236,10 +236,11 @@
return string.substring(0, len) + append;
}
synnefo.util.readablizeBytes = function(bytes) {
synnefo.util.readablizeBytes = function(bytes, fix) {
if (fix === undefined) { fix = 2; }
var s = ['bytes', 'kb', 'MB', 'GB', 'TB', 'PB'];
var e = Math.floor(Math.log(bytes)/Math.log(1024));
return (bytes/Math.pow(1024, Math.floor(e))).toFixed(2)+" "+s[e];
return (bytes/Math.pow(1024, Math.floor(e))).toFixed(fix)+" "+s[e];
}
......
......@@ -48,3 +48,10 @@
#PITHOS_USE_QUOTAHOLDER = False
#PITHOS_QUOTAHOLDER_URL = ''
#PITHOS_QUOTAHOLDER_TOKEN = ''
#
# Tune the size of the http pool for the quotaholder client.
# It limits the maximum number of quota changing requests
# that pithos can serve. Extra requests will be blocked
# until another has completed.
#
#PITHOS_QUOTAHOLDER_POOLSIZE = 200
......@@ -64,3 +64,4 @@ USER_LOGIN_URL = getattr(settings, 'PITHOS_USER_LOGIN_URL',
USE_QUOTAHOLDER = getattr(settings, 'PITHOS_USE_QUOTAHOLDER', False)
QUOTAHOLDER_URL = getattr(settings, 'PITHOS_QUOTAHOLDER_URL', '')
QUOTAHOLDER_TOKEN = getattr(settings, 'PITHOS_QUOTAHOLDER_TOKEN', '')
QUOTAHOLDER_POOLSIZE = getattr(settings, 'PITHOS_QUOTAHOLDER_POOLSIZE', 200)
......@@ -62,6 +62,7 @@ from pithos.api.settings import (BACKEND_DB_MODULE, BACKEND_DB_CONNECTION,
BACKEND_QUEUE_MODULE, BACKEND_QUEUE_HOSTS,
BACKEND_QUEUE_EXCHANGE, USE_QUOTAHOLDER,
QUOTAHOLDER_URL, QUOTAHOLDER_TOKEN,
QUOTAHOLDER_POOLSIZE,
BACKEND_QUOTA, BACKEND_VERSIONING,
BACKEND_FREE_VERSIONING,
AUTHENTICATION_URL, AUTHENTICATION_USERS,
......@@ -967,20 +968,22 @@ else:
}
_pithos_backend_pool = PithosBackendPool(size=POOL_SIZE,
db_module=BACKEND_DB_MODULE,
db_connection=BACKEND_DB_CONNECTION,
block_module=BACKEND_BLOCK_MODULE,
block_path=BACKEND_BLOCK_PATH,
block_umask=BACKEND_BLOCK_UMASK,
queue_module=BACKEND_QUEUE_MODULE,
queue_hosts=BACKEND_QUEUE_HOSTS,
queue_exchange=BACKEND_QUEUE_EXCHANGE,
quotaholder_enabled=USE_QUOTAHOLDER,
quotaholder_url=QUOTAHOLDER_URL,
quotaholder_token=QUOTAHOLDER_TOKEN,
free_versioning=BACKEND_FREE_VERSIONING,
block_params=BLOCK_PARAMS)
_pithos_backend_pool = PithosBackendPool(
size=POOL_SIZE,
db_module=BACKEND_DB_MODULE,
db_connection=BACKEND_DB_CONNECTION,
block_module=BACKEND_BLOCK_MODULE,
block_path=BACKEND_BLOCK_PATH,
block_umask=BACKEND_BLOCK_UMASK,
queue_module=BACKEND_QUEUE_MODULE,
queue_hosts=BACKEND_QUEUE_HOSTS,
queue_exchange=BACKEND_QUEUE_EXCHANGE,
quotaholder_enabled=USE_QUOTAHOLDER,
quotaholder_url=QUOTAHOLDER_URL,
quotaholder_token=QUOTAHOLDER_TOKEN,
quotaholder_client_poolsize=QUOTAHOLDER_POOLSIZE,
free_versioning=BACKEND_FREE_VERSIONING,
block_params=BLOCK_PARAMS)