Commit 6e7c8a04 authored by Christos Stavrakakis's avatar Christos Stavrakakis

Merge branch 'release-0.16' into debian-release-0.16

parents a48a7c10 d81e06b4
......@@ -8,10 +8,46 @@ repository and have aligned versions.
.. _Changelog-0.16:
v0.16rc2
========
Released: Tue Sep 30 10:39:13 EEST 2014
Admin
-----
* Define the Admin node URL and path with the ``ADMIN_BASE_URL`` setting
* The ``ADMIN_BASE_URL`` setting should be adjusted in every node that Admin
is installed.
UI
--
* Remove UI_SNAPSHOTS_ENABLED setting. UI now uses CYCLADES_SNAPSHOTS_ENABLED
setting to resolve whether or not snapshot functionality should be enabled.
Cyclades
--------
* Make snf-dispatcher update status of snapshots when Ganeti job fails.
* Implement reconciliation for stale snapshots.
Pithos
------
* Restore PostgreSQL 8.4 compatibility.
Tools
-----
* Extend snf-burnin to include testing of snapshots.
v0.16rc1
========
Released: Mon Sep 8 17:26:35 EEST 201
Released: Mon Sep 8 17:26:35 EEST 2014
Synnefo-wide
------------
......@@ -92,29 +128,30 @@ Astakos
* Admin users API
* Extend astakos identity API to support user management functionality.
* Admin API related settings
* ``ADMIN_API_ENABLED``
Whether or not the admin api endpoints will be enabled.
* Extend astakos identity API to support user management functionality.
* Admin API related settings
* ``ADMIN_API_ENABLED``
Whether or not the admin api endpoints will be enabled.
* ``ADMIN_API_PERMITTED_GROUPS``
A list of group names for which users which belong to any of
them be granted full access to the admin API endpoints.
* ``ADMIN_API_PERMITTED_GROUPS``
A list of group names for which users which belong to any of
them be granted full access to the admin API endpoints.
* Updated projects integration in UI
* Display both max per member/total in group quota in project views.
* Enhanced usage view to display individual resource usage details for
each of the projects the user is associated with.
* Display project usage statistics in project details view.
* Display system projects in admin project list view.
* Handle display of infinite resource quota values.
* Display both max per member/total in group quota in project views.
* Enhanced usage view to display individual resource usage details for
each of the projects the user is associated with.
* Display project usage statistics in project details view.
* Display system projects in admin project list view.
* Handle display of infinite resource quota values.
Cyclades
--------
* Introduce Volumes and Snapshots:
* Implement 'cyclades_volumes' service, containing the /volumes, /types,
and /snapshots API endpoints under '/volumes/v2.0'.
* Implement `snf-manage volume-{create, list, show, modify, inspect, remove}'
......@@ -124,32 +161,37 @@ Cyclades
* Implement 'volume-type-{create, list, modify, show, remove}` management
commands for handling of volume types.
* Add new settings:
* GANETI_MAX_DISKS_PER_INSTANCE: Maximum number of disks that each Ganeti
instance can have.
* CYCLADES_VOLUME_MAX_SIZE: The maximum allowed size (in GB) for Cyclades
volumes.
* CYCLADES_SNAPSHOTS_ENABLED: Enable/Disable the snapshots feature
altogether at the API level.
* Integrate Cyclades resources with Astakos projects.
* Extend API calls that create resources with the 'project' attribute
in order to assign resources to the specified project.
* Implement API calls for reassigning resources to a new project.
* Export the project that a resource belongs in the 'tenant_id' API
attribute.
attribute.
* Add support for snf-vncauthproxy 1.6 (configurable VNC console types).
* Add support for more types of VM console:
* Modify the current 'console' action to support VNC WebSockets (requires
snf-vncauthproxy=1.6).
* Add support for the three OpenStack Compute console actions
(VNC, RDP, Spice). RDP and Spice are currently not implemented.
(VNC, RDP, Spice). RDP and Spice are currently not implemented.
* Update Cyclades to work with Pithos with integrated Archipelago v2
mapfiles.
* Include functionality for checking the status of the Cyclades update path,
which includes Ganeti, AMQP, snf-ganeti-eventd and snf-dispatcher. This
functionality is implemented as part of the snf-dispatcher and can be used
by passing the '--status-check' option.
which includes Ganeti, AMQP, snf-ganeti-eventd and snf-dispatcher. This
functionality is implemented as part of the snf-dispatcher and can be used
by passing the '--status-check' option.
* Add setting `CYCLADES_VM_MAX_METADATA` to limit the maximum number of
metadata per vm.
* Add setting `CYCLADES_VOLUME_MAX_METADATA` to limit the maximum number of
......@@ -157,8 +199,8 @@ Cyclades
* Use common -u/--user option to specify the UUID or email of the user, for
all management commands.
* Store basic information about images that have been used to create servers,
in order to preserve this information even if images are deleted from
Pithos.
in order to preserve this information even if images are deleted from
Pithos.
* Make 'snf-ganeti-eventd' tolerate failures when processing Ganeti jobs. The
daemon will not crash but continue to run in order to process jobs that can
be processed.
......@@ -177,43 +219,46 @@ Cyclades UI
* In sync with the updated astakos projects API
* Include a project select widget within all resource create views, to let user
decide the project that the created resources will be assigned to.
* Display resource assigned project in resource list views.
* Let user reassign resource project.
* Include a project select widget within all resource create views, to let user
decide the project that the created resources will be assigned to.
* Display resource assigned project in resource list views.
* Let user reassign resource project.
* Volumes API integration.
* Introduce the Disks list/create views.
* Display machine attached disks in icon/single machine views.
* Use ``CYCLADES_VOLUME_MAX_SIZE`` setting to determine the
maximum allowed disk size.
* Introduce the Disks list/create views.
* Display machine attached disks in icon/single machine views.
* Use ``CYCLADES_VOLUME_MAX_SIZE`` setting to determine the
maximum allowed disk size.
* Integrate volume snapshots
* Include available snapshots in disk/vm create wizards.
* Disk snaphot create actions.
* The ui snapshoting functionality (snapshot actions and listing)
can be enabled/disabled using the introduced ``UI_SNAPSHOTS_ENABLED``
setting.
* Include available snapshots in disk/vm create wizards.
* Disk snaphot create actions.
* The ui snapshoting functionality (snapshot actions and listing)
can be enabled/disabled using the introduced ``UI_SNAPSHOTS_ENABLED``
setting.
* Replace TigerVNC Java client with an HTML5 Websocket-based client (noVNC)
* Other UI fixes
* Disabled suspended vm actions
* Custom error message for 413 api error response codes
* Enable resize actions from both info/actions subviews for active
machines
* Update images collection each time machine create wizard opens
* Fixed a couple of title truncate in several views
* Handle display of infinite resource quota values.
* Improve network create wizard. Support for custom gateway.
* Common font styling for all resource titles
* Disabled suspended vm actions
* Custom error message for 413 api error response codes
* Enable resize actions from both info/actions subviews for active
machines
* Update images collection each time machine create wizard opens
* Fixed a couple of title truncate in several views
* Handle display of infinite resource quota values.
* Improve network create wizard. Support for custom gateway.
* Common font styling for all resource titles
Pithos
------
* Backend modifications to enable/support snapshot creation.
* Backend and API modifications to reflect the modifications in the resource
allocation mechanism via the accumulative projects with pool projects.
* Change default disposition type: If no disposition-type is
specifically requested or an invalid value is passed, the disposition type
is set to 'inline'.
......@@ -223,11 +268,11 @@ Admin
* Introduce Administrator Dashboard, which provides the following:
* Graphic access to the details of various Synnefo entities (users, VMs,
Projects).
* Intuitive filtering.
* Multiple actions and notifications.
* Charts and statistics generation.
* Graphic access to the details of various Synnefo entities (users, VMs,
Projects).
* Intuitive filtering.
* Multiple actions and notifications.
* Charts and statistics generation.
.. _Changelog-0.15.2:
......
......@@ -7,10 +7,10 @@ Since v0.13 all Synnefo components have been merged into a single repository.
.. _NEWS-0.16:
v0.16rc1
v0.16rc2
========
Released: Mon Sep 8 17:26:35 EEST 201
Released: Tue Sep 30 10:39:13 EEST 2014
This is the first release candidate for Synnefo 0.16.
......
......@@ -30,10 +30,12 @@ except ImportError:
import json
from astakosclient.utils import \
retry_dec, scheme_to_class, parse_request, check_input, join_urls
retry_dec, scheme_to_class, parse_request, check_input, join_urls, \
render_overlimit_exception
from astakosclient.errors import \
AstakosClientException, Unauthorized, BadRequest, NotFound, Forbidden, \
NoUserName, NoUUID, BadValue, QuotaLimit, InvalidResponse, NoEndpoints
NoUserName, NoUUID, BadValue, QuotaLimit, InvalidResponse, NoEndpoints, \
ConnectionError
# --------------------------------------------------------------------
......@@ -244,7 +246,7 @@ class AstakosClient(object):
hashed_token.update(self.token)
self.logger.debug(
"Make a %s request to %s, using token with hash %s, "
"with headers %s and body %s",
"with headers %r and body %r",
method, request_path, hashed_token.hexdigest(), headers,
body if log_body else "(not logged)")
......@@ -286,11 +288,11 @@ class AstakosClient(object):
self.log_response = dict(
status=status, message=message, data=data)
except Exception as err:
self.logger.error("Failed to send request: %s" % repr(err))
raise AstakosClientException(str(err))
self.logger.error("Failed to send request: %r", err)
raise ConnectionError(err)
# Return
self.logger.debug("Request returned with status %s" % status)
self.logger.debug("Request returned with status %s", status)
if status == 400:
raise BadRequest(message, data)
elif status == 401:
......@@ -300,7 +302,8 @@ class AstakosClient(object):
elif status == 404:
raise NotFound(message, data)
elif status < 200 or status >= 300:
raise AstakosClientException(message, data, status)
raise AstakosClientException(
message=message, status=status, response=data)
try:
if data:
......@@ -308,9 +311,9 @@ class AstakosClient(object):
else:
return None
except Exception as err:
msg = "Cannot parse response \"%s\" with simplejson: %s"
msg = "Cannot parse response \"%r\" with simplejson: %s"
self.logger.error(msg % (data, str(err)))
raise InvalidResponse(str(err), data)
raise InvalidResponse(message=str(err), response=data)
# ----------------------------------
# do a POST to ``API_USERCATALOGS`` (or ``API_SERVICE_USERCATALOGS``)
......@@ -324,10 +327,10 @@ class AstakosClient(object):
if "uuid_catalog" in data:
return data.get("uuid_catalog")
else:
msg = "_uuid_catalog request returned %s. No uuid_catalog found" \
msg = "_uuid_catalog request returned %r. No uuid_catalog found" \
% data
self.logger.error(msg)
raise AstakosClientException(msg)
raise AstakosClientException(message=msg, response=data)
def get_usernames(self, uuids):
"""Return a uuid_catalog dictionary for the given uuids
......@@ -375,10 +378,10 @@ class AstakosClient(object):
if "displayname_catalog" in data:
return data.get("displayname_catalog")
else:
msg = "_displayname_catalog request returned %s. " \
msg = "_displayname_catalog request returned %r. " \
"No displayname_catalog found" % data
self.logger.error(msg)
raise AstakosClientException(msg)
raise AstakosClientException(message=msg, response=data)
def get_uuids(self, display_names):
"""Return a displayname_catalog for the given names
......@@ -600,17 +603,27 @@ class AstakosClient(object):
method="POST")
except AstakosClientException as err:
if err.status == 413:
raise QuotaLimit(err.message, err.details)
try:
msg, details = render_overlimit_exception(
err.response, self.logger)
except Exception as perr:
self.logger.error("issue_commission request returned '413'"
" but response '%r' could not be parsed:"
" %s", err.response, str(perr))
msg, details = err.message, ""
raise QuotaLimit(message=msg,
details=details,
response=err.response)
else:
raise
if "serial" in response:
return response['serial']
else:
msg = "issue_commission_core request returned %s. " + \
msg = "issue_commission_core request returned %r. " + \
"No serial found" % response
self.logger.error(msg)
raise AstakosClientException(msg)
raise AstakosClientException(message=msg, response=response)
def _mk_user_provision(self, holder, source, resource, quantity):
holder = "user:" + holder
......
......@@ -20,15 +20,25 @@ Astakos Client Exceptions
class AstakosClientException(Exception):
"""Base AstakosClientException Class"""
def __init__(self, message='', details='', status=500):
def __init__(self, message='', details='', status=500,
response=None, errobject=None):
self.message = message
self.details = details
self.errobject = errobject
self.response = response
if not hasattr(self, 'status'):
self.status = status
super(AstakosClientException,
self).__init__(self.message, self.details, self.status)
class ConnectionError(AstakosClientException):
"""Connection failed"""
def __init__(self, errobject):
super(ConnectionError, self).__init__(
message=str(errobject), errobject=errobject)
class BadValue(AstakosClientException):
"""Re-define ValueError Exception under AstakosClientException"""
def __init__(self, details):
......@@ -38,8 +48,6 @@ class BadValue(AstakosClientException):
class InvalidResponse(AstakosClientException):
"""Return simplejson parse Exception as AstakosClient one"""
def __init__(self, message, details):
super(InvalidResponse, self).__init__(message, details)
class BadRequest(AstakosClientException):
......
......@@ -20,10 +20,14 @@ Astakos Client utility module
from httplib import HTTPConnection, HTTPSConnection
from contextlib import closing
import simplejson
from objpool.http import PooledHTTPConnection
from astakosclient.errors import AstakosClientException, BadValue
try:
import simplejson as json
except ImportError:
import json
def retry_dec(func):
"""Class Method Decorator"""
......@@ -81,7 +85,7 @@ def scheme_to_class(scheme, use_pool, pool_size):
def parse_request(request, logger):
"""Parse request with simplejson to convert it to string"""
try:
return simplejson.dumps(request)
return json.dumps(request)
except Exception as err:
msg = "Cannot parse request \"%s\" with simplejson: %s" \
% (request, str(err))
......@@ -102,3 +106,35 @@ def check_input(function_name, logger, **kwargs):
def join_urls(url_a, url_b):
"""Join_urls from synnefo.lib"""
return url_a.rstrip("/") + "/" + url_b.lstrip("/")
def render_overlimit_exception(response, logger):
"""Render a human readable message for QuotaLimit Exception"""
resource_name = {
"cyclades.disk": "Disk",
"cyclades.vm": "Virtual Machine",
"cyclades.cpu": "CPU",
"cyclades.ram": "RAM",
"cyclades.floating_ip": "Floating IP address",
"cyclades.network.private": "Private Network",
"pithos.diskspace": "Storage space",
"astakos.pending_app": "Pending Applications"
}
response = json.loads(response)
data = response['overLimit']['data']
usage = data["usage"]
limit = data["limit"]
available = limit - usage
provision = data['provision']
requested = provision['quantity']
resource = provision['resource']
try:
resource = resource_name[resource]
except KeyError:
logger.error("Unknown resource name '%s'", resource)
msg = "Resource Limit Exceeded for your account."
details = "Limit for resource '%s' exceeded for your account."\
" Available: %s, Requested: %s"\
% (resource, available, requested)
return msg, details
[debian]
python-nfqueue = 0.4+physindev-1~wheezy
python-scapy = 2.2.0+rfc6355-1
libcurl4-openssl-dev = 7.26.0-1+wheezy9
curl = 7.26.0-1+wheezy9
snf-ganeti = unstable
ganeti2 = unstable
python-django-eztables = 0.3.3-1~snf~0.2
qemu-kvm = wheezy-backports
[ubuntu]
......@@ -338,6 +338,16 @@ You can modify the default system quota limit for all future users with::
# snf-manage resource-modify <resource_name> --system-default <value>
You can also control the default quota a new project offers to its members
if a limit is not specified in the project application (`project default`).
In particular, if a resource is not meant to be visible to the end user,
then it's best to set its project default to infinite.
.. code-block:: console
# snf-manage resource-modify cyclades.total_ram --project-default inf
Grant extra quota through projects
``````````````````````````````````
......@@ -1290,7 +1300,7 @@ externally reserved, to exclude from allocation.
Quotas
~~~~~~
The andling of quotas for Cyclades resources is powered by Astakos quota
The handling of quotas for Cyclades resources is powered by Astakos quota
mechanism. During registration of Cyclades service to Astakos, the Cyclades
resources are also imported to Astakos for accounting and presentation.
......@@ -1662,44 +1672,52 @@ from the Astakos/Cyclades stats.
Access and permissions
----------------------
The Admin dashboard can be accessed by default from the `/admin/` URL. Since
there is no login form, the user must login on Astakos first and then visit the
above URL. Access will be granted only to users that belong to a predefined
list of Astakos groups. By default, there are three group categories that
are mapped 1-to-1 to Astakos groups:
The Admin dashboard can be accessed by default from the ``ADMIN_BASE_URL`` URL.
Since there is no login form, the user must login on Astakos first and then
visit the above URL. Access will be granted only to users that belong to a
predefined list of Astakos groups. By default, there are three group categories
that are mapped 1-to-1 to Astakos groups:
* ADMIN_READONLY_GROUP: 'admin-readonly'
* ADMIN_HELPDESK_GROUP: 'helpdesk'
* ADMIN_GROUP: 'admin'
The group categories can be changed using the `ADMIN_PERMITTED_GROUPS` setting.
In order to change the Astakos group that a category corresponds to, the
administrator can specify the group that he/she wants in the
`ADMIN_READONLY_GROUP`, `ADMIN_HELPDESK_GROUP` or `ADMIN_GROUP` settings.
The group categories can be changed using the ``ADMIN_PERMITTED_GROUPS``
setting. In order to change the Astakos group that a category corresponds to,
the administrator can specify the group that he/she wants in the
``ADMIN_READONLY_GROUP``, ``ADMIN_HELPDESK_GROUP`` or ``ADMIN_GROUP`` settings.
Note that while any user that belongs to the `ADMIN_PERMITTED_GROUPS` has the
Note that while any user that belongs to the ``ADMIN_PERMITTED_GROUPS`` has the
same access to the administrator dashboard, the actions that are allowed for a
group may differ. That's because Admin implements a Role-Based Access Control
(RBAC) policy, which can be changed from the `ADMIN_RBAC` setting. By default,
users in the `ADMIN_READONLY_GROUP` cannot perform any actions. On the other
hand, users in the `ADMIN_GROUP` can perform all actions. In the middle of the
spectrum is the `ADMIN_HELPDESK_GROUP`, which by default performs a small
subset of reversible actions.
(RBAC) policy, which can be changed from the ``ADMIN_RBAC`` setting. By
default, users in the ``ADMIN_READONLY_GROUP`` cannot perform any actions. On
the other hand, users in the ``ADMIN_GROUP`` can perform all actions. In the
middle of the spectrum is the ``ADMIN_HELPDESK_GROUP``, which by default
performs a small subset of reversible actions.
Seting up Admin
---------------
Admin is bundled by default with a list of sane settings. The most important
one, `ADMIN_ENABLED`, is set to `True` and defines whether Admin will be used
or not. Therefore, the administrator only needs to create the necessary Astakos
groups and add trusted users in them. The following example will create an
admin group and will add a user in it:
one, ``ADMIN_ENABLED``, is set to ``True`` and defines whether Admin will be used
or not.
The administrator simply has to create the necessary Astakos groups and
add trusted users in them. The following example will create an admin group and
will add a user in it:
.. code-block:: console
snf-manage group-add admin
snf-manage user-modify --add-group=admin <user_id>
Finally, the administrator must edit the ``20-snf-admin-app-general.conf``
settings file, uncomment the ``ADMIN_BASE_URL`` setting and assign the
appropriate URL to it. In most cases, this URL will be the top-level URL of the
Admin node, with the optional addition of an extra path (e.g. ``/admin``) in
order to distinguish it from different components.
That's all that is required for a single-node setup. For a multi-node setup,
please consult the following section:
......@@ -1712,22 +1730,18 @@ settings of their nodes. As a result, when installing Admin in a node, the
Astakos and Cyclades packages will also be installed.
In order to disable the Astakos/Cyclades API in the Admin node, the
administrator can add the following line in `99-local.conf`:
.. code-block:: console
ROOT_URLCONF="synnefo_admin.admin.urls"
As a result, Admin will be accessible by this URL: `<Admin_Node_URL>/`. If the
administrator wishes to use the default URL (`<Admin_Node_URL>/admin/`), then
the following line must be added instead.
administrator can add the following line in ``99-locals.conf`` (you can create
it if doesn't exist):