Merge branch 'release-0.14'

Conflicts:
	version

.gitignore

 *.db
+*.egg
+*.tar.gz
 *.pyc
 *~
 *.*.swp
@@ -20,6 +22,7 @@ settings.d/*-local.conf
 *.egg-info
 dist
 _build
+.coverage
 
 # pithos temp files
 pithos/data
@@ -34,3 +37,6 @@ snf-quotaholder-app/quotaholder_django/version.py
 *.iml
 *.graffle
 snf-stats-app/synnefo_stats/version.py
+astakosclient/astakosclient/version.py
+snf-django-lib/snf_django/version.py
+snf-branding/synnefo_branding/version.py

Changelog

@@ -6,6 +6,208 @@ Unified Changelog file for Synnefo versions >= 0.13
 Since v0.13 most of the Synnefo components have been merged into a single
 repository and have aligned versions.
 
+
+.. _Changelog-0.13next:
+
+v0.13next
+=========
+
+Released: UNRELEASED
+
+Synnefo-wide
+------------
+
+* Create 'snf_django' Python package to hold common code for all Synnefo
+  components.
+
+* Create a JSON-exportable definition document for each Synnefo Components
+  (Astakos, Cyclades, Pithos, etc.) that consolidates APIs (services),
+  resources, and other standardized properties (e.g. default URL prefixes).
+
+* Standardize URLs for Synnefo Components, impose structure and naming
+  conventions to related settings. Make each component deployable under
+  a user-configurable <COMPONENT>_BASE_URL. Each API (compute, image, etc.)
+  is deployable under a developer-configurable prefix beneath BASE_URL.
+
+* Deprecate CLOUDBAR_ACTIVE_SERVICE setting from all apps. 
+
+* Common synnefo 404/500 templates (located in snf-webproject)
+
+Astakos
+-------
+
+* Redesign of the accounting system (quotaholder) and integration into
+  Astakos.
+
+  * Simplified the quotaholder model; removed tables Entity and Policy; now
+    table Holding contains limit and usage for every holding.
+  * Extended table Holding, so that we can keep track of quota for every
+    valid combination of holder (e.g. user), resource, and source (e.g. the
+    default system or some specific project).
+  * Refactored code for issuing and resolving commissions for robustness;
+    added a 'force' option to bypass the upper limit check when issuing a
+    commission.
+  * Simplified syncing to the quotaholder; removed fields from models
+    Project and ProjectMembership, previously needed for syncing; removed
+    state PROJECT_DEACTIVATED from ProjectMembership.
+  * Removed settings ASTAKOS_QUOTAHOLDER_URL, ASTAKOS_QUOTAHOLDER_TOKEN,
+    and ASTAKOS_QUOTAHOLDER_POOLSIZE.
+
+* API-related changes:
+
+  * Implemented API calls for quota, resources, and commissions.
+  * Moved all API calls under '/account/v1.0'.
+  * Implemented the keystone API call POST /tokens under '/identity/v2.0'.
+
+* Service and resource specification and handling:
+
+  * Specified a format for defining services along with the API endpoints
+    and the resources they expose. Migrated internal resource name by
+    prefixing it with service name (e.g. 'vm' becomes 'cyclades.vm');
+    renamed registered service 'pithos+' to 'pithos'.
+  * Specified a procedure to register a Synnefo component, its services and
+    their resources in astakos and set the resources' default base quota
+    limit. Removed resource definitions from settings.
+  * Moved service and resource presentation data out of the respective db
+    models into a separate file of UI constants.
+
+* Converted the limit on pending applications from a setting to a quotable
+  resource. Converted the related user setting to a user-specific base quota
+  limit. Deprecated model UserSetting; removed setting
+  ASTAKOS_PENDING_APPLICATION_LIMIT.
+
+* Changes in locking strategy:
+
+  * Lock only project's chain for all project operations; lock user before
+    syncing to quotaholder.
+  * When locking multiple rows (e.g. users or holdings) include an ORDER BY
+    clause in the query to impose ordering on locking.
+
+* Changes in views:
+
+  * Replaced custom transaction context with a simple decorator for managing
+    transactions and a context 'ExceptionHandler', which logs and suppresses
+    exceptions
+
+* Added fine grain user auth provider's policies.
+
+  * Administrator can override default auth provider policies to a specific 
+    user or group of users.
+  * Optionally a user can be assigned to a list of groups, based on the
+    authentication method he choosed to signup.
+
+* Removed explicit handling of SMTP errors on each email delivery. Exceptions 
+  are now propagated to base django exception handler.
+
+* Email used in html/email tempaltes which prompt user to contact for service 
+  support prompts is now defined in ``CONTACT_EMAIL`` setting introduced in 
+  snf-common settings.
+
+* Improvements in user activation flow
+
+  * User moderation now takes place after the user has verified his email
+    address.
+  * User model enriched with additional user state fields
+  * Split activation email from moderation process. Administrator is required 
+    to moderate user explicitly using the `user-modify --accept` or
+    `user-modify --reject` commands.
+  * Improved logging throught out user activation procedures.
+
+* Remove deprecated AstakosUser model fields: `provider`,
+  `third_party_identifier`
+
+* Allow override of authentication provider messages using the following 
+  format in setting names: ``ASTAKOS_<PROVIDER_MODULE>_<MSGID>_MSG``
+
+* Cloudbar automatically tries to identify the active service based on window 
+  location.
+
+* Removing authentication provider view is now CSRF protected.
+
+* New `API access` view, containing useful information to users on how to 
+  access available Synnefo services API's.
+
+* Remove of ASTAKOS_*_EMAIL_SUBJECT settings. All email subjects are now 
+  defined in astakos.im.messages module. Overriding default values can be 
+  achieved using custom gettext files or using astakos messages settings::
+	
+	#change of greeting email subject
+  	ASTAKOS_GREETING_EMAIL_SUBJECT_MESSAGE = 'Welcome to my cloud'
+
+* Remove ``ASTAKOS_ACTIVATION_REDIRECT_URL`` and ``ASTAKOS_LOGIN_SUCCESS_URL`` 
+  from astakos .conf file. Settings are dynamically computed based on 
+  ``ASTAKOS_BASE_URL``.
+
+* Management commands:
+
+  * Introduced new commands:
+     * authpolicy-{add, list, remove, set, show}
+     * group-{add, list}
+     * component-{add, list, modify, remove}
+     * reconcile-resources-astakos
+     * resource-{export-astakos, import, modify}
+     * service-{export-astakos, import, show}
+  * Renamed commands:
+     * astakos-quota to quota
+     * user-update to user-modify
+     * full-cleanup to cleanup-full
+  * Removed commands:
+     * astakos-init
+     * invitation-{details, list}
+     * project-sync
+     * resource-{add, remove}
+     * service-{add, remove, token-renew, update}
+     * user-invite
+     * user-set-initial-quota (integrated its functionality in user-modify and quota)
+  * Added quota and project-related information in user-show command; added
+    membership information in project-show.
+
+Cyclades
+--------
+
+* Make 'type' attribute required for network create API request.
+* Networks not created to all Ganeti backends upon creation, they are instead
+  created to a backend only when a VM connects to the network.
+* Add 'CYCLADES_ASTAKOSCLIENT_POOLSIZE' setting which tunes the size of the
+  http connection pool to astakos.
+* Remove 'CYCLADES_USER_CATALOG_URL' and 'CYCLADES_USER_FEEDBACK_URL' settings
+* Remove CYCLADES_USE_QUOTAHOLDER, CYCLADES_QUOTAHOLDER_TOKEN,
+  CYCLADES_QUOTAHOLDER_URL, CYCLADES_QUOTAHOLDER_POOLSIZE settings
+* Rename 'cyclades-usage-verify' management command to
+  'reconcile-resources-cyclades'. Also, remove 'cyclades-usage-reset' command,
+  which is equivalent to 'reconcile-resources-cyclades --fix'.
+* Rename 'cyclades-reconcile-commissions' management command to
+  'reconcile-commissions-cyclades'.
+* Remove obsolete 'MAX_VMS_PER_USER', 'MAX_NETWORKS_PER_USER',
+  "VMS_USER_QUOTA" and "NETWORKS_USER_QUOTA" settings, since their usage
+  is covered by Quotaholder.
+* Remove obsolete setting 'API_ROOT_URL', since it is being covered by
+  the use of CYCLADES_BASE_URL* Remove obsolete setting 'API_ROOT_URL', since
+  it is being covered by 'CYCLADES_BASE_URL'.
+* Remove obsolete settings GANETI_DISK_TEMPLATES and
+  DEFAULT_GANETI_DISK_TEMPLATE
+
+Cyclades helpdesk
+-----------------
+
+* Additional start/stop vm action
+* Display extend backend info in vm's view
+* Fixed IP lookup
+
+Pithos
+------
+
+* Remove PITHOS_AUTHENTICATION_USERS setting, which was used to override
+  astakos users.
+* Remove 'PITHOS_USER_CATALOG_URL', 'PITHOS_USER_FEEDBACK_URL' and
+  'PITHOS_USER_LOGIN_URL' settings.
+* Remove PITHOS_USE_QUOTAHOLDER, PITHOS_QUOTAHOLDER_URL,
+  PITHOS_QUOTAHOLDER_TOKEN and PITHOS_ASTAKOSCLIENT_POOLSIZE
+
+Tools
+-----
+
+
 .. _Changelog-0.13:
 
 v0.13
@@ -69,6 +271,15 @@ Astakos
   * Project management UI
   * New Overview page
 
+* refactored/improved /login endpoint used by desktop/mobile clients.
+  * endpoint url is now exposed by `weblogin` service
+  * clients should use unauthenticated identity/tokens api to resolve the 
+    endpoint url
+  * view only allows redirects to `pithos://` scheme urls
+  * removed uuid from redirect parameters. Client should use authenticated 
+    request to identity/tokens to retrieve user uuid.
+
+
 Cyclades
 --------
 
@@ -87,6 +298,18 @@ Cyclades
 * Helpdesk actions are now logged using the synnefo's common login
   infrastructure
 
+UI
+^^
+* Removed feedback endpoint. Feedback requests delegate to astakos feedback
+  service. ``FEEDBACK_CONTACTS``, ``FEEDBACK_EMAIL_FROM`` settings removed, 
+  and no longer used.
+* ``UI_LOGIN_URL``, ``UI_GLANCE_URL``, ``COMPUTE_URL`` settings no longer 
+  required to be set and are dynamically computed based on ``ASTAKOS_BASE_URL``
+  and ``CYCLADES_BASE_URL`` settings.
+* File group is no longer included in ssh keys personality metadata sent in 
+  create vm calls.
+  
+
 Pithos
 ------
 
@@ -95,6 +318,9 @@ Pithos
   * new settings:
     PITHOS_RADOS_STORAGE, PITHOS_RADOS_POOL_BLOCKS, PITHOS_RADOS_POOL_MAPS
 
+* X-Object-Public now contains full url (domain + proper component prefix + 
+  file path)
+
 * Rewritten support for public URLs, with admin-selectable length
 
   * new settings:

NEWS

@@ -5,6 +5,30 @@ Unified NEWS file for Synnefo versions >= 0.13
 
 Since v0.13 all Synnefo components have been merged into a single repository.
 
+
+.. _NEWS-0.13next:
+
+v0.13next
+=========
+
+Released: UNRELEASED
+
+Synnefo-wide
+------------
+
+Astakos
+-------
+
+Cyclades
+--------
+
+Pithos
+------
+
+Tools
+-----
+
+
 .. _NEWS-0.13:
 
 v0.13

snf-cyclades-app/README → astakosclient/COPYRIGHT

-README
-=======
-
-This is the top-level documentation file for the
-Synnefo cloud management software.
-
-Consult:
-  * docs/src/develop.rst: for information on how to setup a development environment
-  * docs/src/install.rst: for information on how to install the application
-  * docs/src/deploy.rst:  for information on how to deploy the application
-  * docs/src/ci.rst:      for information on how to setup a Jenkins-based
-                          continuous integration system
-  * docs/src/i18n.rst:    for information on application internationalization
-
-
-Synnefo may be distributed under the terms of the following license:
-
-Copyright 2011 GRNET S.A. All rights reserved.
+Copyright (C) 2012, 2013 GRNET S.A. All rights reserved.
 
 Redistribution and use in source and binary forms, with or
 without modification, are permitted provided that the following

astakosclient/MANIFEST.in

+include distribute_setup.py

snf-quotaholder-app/quotaholder_django/quotaholder_app/views.py → astakosclient/astakosclient/errors.py

-# Copyright 2012, 2013 GRNET S.A. All rights reserved.
+# Copyright (C) 2012, 2013 GRNET S.A. All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or
 # without modification, are permitted provided that the following
@@ -32,57 +32,58 @@
 # or implied, of GRNET S.A.
 
 
-from django.http import HttpResponse
-from django.db import transaction
-from django.conf import settings
-from synnefo.lib.commissioning import CallError
+class AstakosClientException(Exception):
+    def __init__(self, message='', details='', status=500):
+        self.message = message
+        self.details = details
+        if not hasattr(self, 'status'):
+            self.status = status
+        super(AstakosClientException,
+              self).__init__(self.message, self.details, self.status)
 
-from .callpoint import API_Callpoint
 
-import json
-from traceback import format_exc
+class BadValue(AstakosClientException):
+    def __init__(self, details):
+        """Re-define ValueError Exception under AstakosClientException"""
+        message = "ValueError"
+        super(BadValue, self).__init__(message, details)
 
-import logging
-logger = logging.getLogger(__name__)
 
-try:
-    from django.views.decorators.csrf import csrf_exempt
-except ImportError:
-    def csrf_exempt(func):
-        return func
+class InvalidResponse(AstakosClientException):
+    def __init__(self, message, details):
+        """Return simplejson parse Exception as AstakosClient one"""
+        super(InvalidResponse, self).__init__(message, details)
 
 
-def _get_body(request):
-    body = request.raw_post_data
-    if body is None:
-        body = request.GET.get('body', None)
-    return body
+class BadRequest(AstakosClientException):
+    status = 400
 
-callpoints = {('quotaholder', 'v'): API_Callpoint()}
 
+class Unauthorized(AstakosClientException):
+    status = 401
 
-@transaction.commit_manually
-@csrf_exempt
-def view(request, appname='quotaholder', version=None, callname=None):
-    if (appname, version) not in callpoints:
-        return HttpResponse(status=404)
 
-    if request.META.get('HTTP_X_AUTH_TOKEN') != settings.QUOTAHOLDER_TOKEN:
-        return HttpResponse(status=403, content='invalid token')
+class Forbidden(AstakosClientException):
+    status = 403
 
-    callpoint = callpoints[(appname, version)]
-    body = _get_body(request)
-    try:
-        body = callpoint.make_call_from_json(callname, body)
-        status = 200
-    except Exception as e:
-        logger.exception(e)
-        status = 450
-        if not isinstance(e, CallError):
-            e.args += (''.join(format_exc()),)
-            e = CallError.from_exception(e)
-            status = 500
 
-        body = json.dumps(e.to_dict())
+class NotFound(AstakosClientException):
+    status = 404
 
-    return HttpResponse(status=status, content=body)
+
+class QuotaLimit(AstakosClientException):
+    status = 413
+
+
+class NoUserName(AstakosClientException):
+    def __init__(self, uuid):
+        """No display name for the given uuid"""
+        message = "No display name for the given uuid: %s" % uuid
+        super(NoUserName, self).__init__(message)
+
+
+class NoUUID(AstakosClientException):
+    def __init__(self, display_name):
+        """No uuid for the given display name"""
+        message = "No uuid for the given display name: %s" % display_name
+        super(NoUUID, self).__init__(message)

astakosclient/astakosclient/keypath.py

+# Copyright 2012, 2013 GRNET S.A. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or
+# without modification, are permitted provided that the following
+# conditions are met:
+#
+#   1. Redistributions of source code must retain the above
+#      copyright notice, this list of conditions and the following
+#      disclaimer.
+#
+#   2. Redistributions in binary form must reproduce the above
+#      copyright notice, this list of conditions and the following
+#      disclaimer in the documentation and/or other materials
+#      provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
+# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+#
+# The views and conclusions contained in the software and
+# documentation are those of the authors and should not be
+# interpreted as representing official policies, either expressed
+# or implied, of GRNET S.A.
+
+
+def dict_merge(a, b):
+    """
+    http://www.xormedia.com/recursively-merge-dictionaries-in-python/
+    """
+    if not isinstance(b, dict):
+        return b
+    result = copy.deepcopy(a)
+    for k, v in b.iteritems():
+        if k in result and isinstance(result[k], dict):
+                result[k] = dict_merge(result[k], v)
+        else:
+            result[k] = copy.deepcopy(v)
+    return result
+
+
+def lookup_path(container, path, sep='.', createpath=False):
+    """
+    return (['a','b'],
+            [container['a'], container['a']['b']],
+            'c')  where path=sep.join(['a','b','c'])
+
+    """
+    names = path.split(sep)
+    dirnames = names[:-1]
+    basename = names[-1]
+
+    node = container
+    name_path = []
+    node_path = [node]
+    for name in dirnames:
+        name_path.append(name)
+        if name not in node:
+            if not createpath:
+                m = "'{0}': path not found".format(sep.join(name_path))
+                raise KeyError(m)
+            node[name] = {}
+        try:
+            node = node[name]
+        except TypeError as e:
+            m = "'{0}': cannot traverse path beyond this node: {1}"
+            m = m.format(sep.join(name_path), str(e))
+            raise ValueError(m)
+        node_path.append(node)
+
+    return name_path, node_path, basename
+
+
+def walk_paths(container):
+    for name, node in container.iteritems():
+        if not hasattr(node, 'items'):
+            yield [name], [node]
+        else:
+            for names, nodes in walk_paths(node):
+                yield [name] + names, [node] + nodes
+
+
+def list_paths(container, sep='.'):
+    """
+    >>> sorted(list_paths({'a': {'b': {'c': 'd'}}}))
+    [('a.b.c', 'd')]
+    >>> sorted(list_paths({'a': {'b': {'c': 'd'}, 'e': 3}}))
+    [('a.b.c', 'd'), ('a.e', 3)]
+    >>> sorted(list_paths({'a': {'b': {'c': 'd'}, 'e': {'f': 3}}}))
+    [('a.b.c', 'd'), ('a.e.f', 3)]
+    >>> list_paths({})
+    []
+
+    """
+    return [(sep.join(name_path), node_path[-1])
+            for name_path, node_path in walk_paths(container)]
+
+
+def del_path(container, path, sep='.', collect=True):
+    """
+    del container['a']['b']['c'] where path=sep.join(['a','b','c'])
+
+    >>> d = {'a': {'b': {'c': 'd'}}}; del_path(d, 'a.b.c'); d
+    {}
+    >>> d = {'a': {'b': {'c': 'd'}}}; del_path(d, 'a.b.c', collect=False); d
+    {'a': {'b': {}}}
+    >>> d = {'a': {'b': {'c': 'd'}}}; del_path(d, 'a.b.c.d')
+    Traceback (most recent call last):
+    ValueError: 'a.b.c': cannot traverse path beyond this node:\
+ 'str' object does not support item deletion
+    """
+
+    name_path, node_path, basename = \
+            lookup_path(container, path, sep=sep, createpath=False)
+
+    lastnode = node_path.pop()
+    lastname = basename
+    try:
+        if basename in lastnode:
+            del lastnode[basename]
+    except (TypeError, KeyError) as e:
+        m = "'{0}': cannot traverse path beyond this node: {1}"
+        m = m.format(sep.join(name_path), str(e))
+        raise ValueError(m)
+
+    if collect:
+        while node_path and not lastnode:
+            basename = name_path.pop()
+            lastnode = node_path.pop()
+            del lastnode[basename]
+
+
+def get_path(container, path, sep='.'):
+    """
+    return container['a']['b']['c'] where path=sep.join(['a','b','c'])
+
+    >>> get_path({'a': {'b': {'c': 'd'}}}, 'a.b.c.d')
+    Traceback (most recent call last):
+    ValueError: 'a.b.c.d': cannot traverse path beyond this node:\
+ string indices must be integers, not str
+    >>> get_path({'a': {'b': {'c': 1}}}, 'a.b.c.d')
+    Traceback (most recent call last):
+    ValueError: 'a.b.c.d': cannot traverse path beyond this node:\
+ 'int' object is unsubscriptable
+    >>> get_path({'a': {'b': {'c': 1}}}, 'a.b.c')
+    1
+    >>> get_path({'a': {'b': {'c': 1}}}, 'a.b')
+    {'c': 1}
+
+    """
+    name_path, node_path, basename = \
+            lookup_path(container, path, sep=sep, createpath=False)
+    name_path.append(basename)
+    node = node_path[-1]
+
+    try:
+        return node[basename]
+    except TypeError as e:
+        m = "'{0}': cannot traverse path beyond this node: {1}"
+        m = m.format(sep.join(name_path), str(e))
+        raise ValueError(m)
+    except KeyError as e:
+        m = "'{0}': path not found: {1}"
+        m = m.format(sep.join(name_path), str(e))
+        raise KeyError(m)
+
+
+def set_path(container, path, value, sep='.',
+             createpath=False, overwrite=True):
+    """
+    container['a']['b']['c'] = value where path=sep.join(['a','b','c'])
+
+    >>> set_path({'a': {'b': {'c': 'd'}}}, 'a.b.c.d', 1)
+    Traceback (most recent call last):
+    ValueError: 'a.b.c.d': cannot traverse path beyond this node:\
+ 'str' object does not support item assignment
+    >>> set_path({'a': {'b': {'c': 'd'}}}, 'a.b.x.d', 1)
+    Traceback (most recent call last):
+    KeyError: "'a.b.x': path not found"
+    >>> set_path({'a': {'b': {'c': 'd'}}}, 'a.b.x.d', 1, createpath=True)
+    
+    >>> set_path({'a': {'b': {'c': 'd'}}}, 'a.b.c', 1)
+     
+    >>> set_path({'a': {'b': {'c': 'd'}}}, 'a.b.c', 1, overwrite=False)
+    Traceback (most recent call last):
+    ValueError: will not overwrite path 'a.b.c'
+
+    """