Commit 92fd6010 authored by Giorgos Korfiatis's avatar Giorgos Korfiatis

Merge branch 'feature-pool-projects' into develop

Refs #5007 #5024
parents c79cdaa8 5164eb98
......@@ -6,6 +6,70 @@ Unified Changelog file for Synnefo versions >= 0.13
Since v0.13 most of the Synnefo components have been merged into a single
repository and have aligned versions.
v0.15next
=============
Released: UNRELEASED
Synnefo-wide
------------
* Replace accumulative projects with pool projects:
* Projects are now viewed as a source of finite resources. A member can
reserve a part of these resources up to a specified limit.
* Base quota are now offered through a special purpose user-specific base
project, identified with the same UUID as the user.
* Each actual resource (Cyclades VM, network, floating IP and Pithos
container) is now also associated with a project besides the owner.
* In resource creation, project defaults to the user-specific base
project, if not specified otherwise. It is also possible to change the
project assignment of an existing resource.
* All existing resources have been assigned to the respective
user-specific base projects.
Astakos
-------
* Decouple projects from applications:
* Support project creation (by the system) and modification (by a
privileged user) without the need to submit/approve an application.
* View applications as modifications. When a project is uninitialized
(e.g. an application for a new project is pending), no further
modification is allowed.
* Applications are removed from the API. A project's last application is
only accessible as part of the project details.
* Decouple project state from application state; they can be combined by
an API client, if needed.
* Changes concerning quota and pool projects:
* A project must provide limits for all registered resources. On project
activation, resources missing are automatically completed using a
skeleton.
* Field `uplimit' of registed resources is exposed as `base_default' and
provide the skeleton for user-specific base projects. A new field
`project_default' is introduce to act as a skeleton for conventional
projects.
* The quotaholder now also records project quota besides user quota. The
two types of holders are distinguished with a prefix: `user:' and
`project:'.
* The quota API is extended to make project quota available.
* Projects can be set `private', making it accessible only to its owner and
members.
v0.14next
=========
......
......@@ -213,6 +213,10 @@ class AstakosClient(object):
def api_service_quotas(self):
return join_urls(self.account_prefix, "service_quotas")
@property
def api_service_project_quotas(self):
return join_urls(self.account_prefix, "service_project_quotas")
@property
def api_commissions(self):
return join_urls(self.account_prefix, "commissions")
......@@ -229,10 +233,6 @@ class AstakosClient(object):
def api_projects(self):
return join_urls(self.account_prefix, "projects")
@property
def api_applications(self):
return join_urls(self.api_projects, "apps")
@property
def api_memberships(self):
return join_urls(self.api_projects, "memberships")
......@@ -558,9 +558,27 @@ class AstakosClient(object):
query += "?user=" + user
return self._call_astakos(query)
# ----------------------------------
# do a GET to ``API_SERVICE_PROJECT_QUOTAS``
def service_get_project_quotas(self, project=None):
"""Get all project quotas for resources associated with the service
Keyword arguments:
project -- optionally, the uuid of a specific project
In case of success return a dict of dicts with current quotas
for all projects, or of a specified project, if project argument is set.
Otherwise raise an AstakosClientException
"""
query = self.api_service_project_quotas
if project is not None:
query += "?project=" + project
return self._call_astakos(query)
# ----------------------------------
# do a POST to ``API_COMMISSIONS``
def issue_commission(self, request):
def _issue_commission(self, request):
"""Issue a commission
Keyword arguments:
......@@ -591,6 +609,57 @@ class AstakosClient(object):
self.logger.error(msg)
raise AstakosClientException(msg)
def _mk_user_provision(self, holder, source, resource, quantity):
holder = "user:" + holder
source = "project:" + source
return {"holder": holder, "source": source,
"resource": resource, "quantity": quantity}
def _mk_project_provision(self, holder, resource, quantity):
holder = "project:" + holder
return {"holder": holder, "source": None,
"resource": resource, "quantity": quantity}
def mk_provisions(self, holder, source, resource, quantity):
return [self._mk_user_provision(holder, source, resource, quantity),
self._mk_project_provision(source, resource, quantity)]
def issue_commission_generic(self, user_provisions, project_provisions,
name="", force=False, auto_accept=False):
"""Issue commission (for multiple holder/source pairs)
keyword arguments:
user_provisions -- dict mapping user holdings
(user, project, resource) to integer quantities
project_provisions -- dict mapping project holdings
(project, resource) to integer quantities
name -- description of the commission (string)
force -- force this commission (boolean)
auto_accept -- auto accept this commission (boolean)
In case of success return commission's id (int).
Otherwise raise an AstakosClientException.
"""
request = {}
request["force"] = force
request["auto_accept"] = auto_accept
request["name"] = name
try:
request["provisions"] = []
for (holder, source, resource), quantity in \
user_provisions.iteritems():
p = self._mk_user_provision(holder, source, resource, quantity)
request["provisions"].append(p)
for (holder, resource), quantity in project_provisions.iteritems():
p = self._mk_project_provision(holder, resource, quantity)
request["provisions"].append(p)
except Exception as err:
self.logger.error(str(err))
raise BadValue(str(err))
return self._issue_commission(request)
def issue_one_commission(self, holder, source, provisions,
name="", force=False, auto_accept=False):
"""Issue one commission (with specific holder and source)
......@@ -605,7 +674,6 @@ class AstakosClient(object):
In case of success return commission's id (int).
Otherwise raise an AstakosClientException.
(See also issue_commission)
"""
check_input("issue_one_commission", self.logger,
......@@ -619,14 +687,37 @@ class AstakosClient(object):
try:
request["provisions"] = []
for resource, quantity in provisions.iteritems():
prov = {"holder": holder, "source": source,
"resource": resource, "quantity": quantity}
request["provisions"].append(prov)
ps = self.mk_provisions(holder, source, resource, quantity)
request["provisions"].extend(ps)
except Exception as err:
self.logger.error(str(err))
raise BadValue(str(err))
return self._issue_commission(request)
def issue_resource_reassignment(self, holder, from_source,
to_source, provisions, name="",
force=False, auto_accept=False):
"""Change resource assignment to another project
"""
request = {}
request["force"] = force
request["auto_accept"] = auto_accept
request["name"] = name
try:
request["provisions"] = []
for resource, quantity in provisions.iteritems():
ps = self.mk_provisions(
holder, from_source, resource, -quantity)
ps += self.mk_provisions(holder, to_source, resource, quantity)
request["provisions"].extend(ps)
except Exception as err:
self.logger.error(str(err))
raise BadValue(str(err))
return self.issue_commission(request)
return self._issue_commission(request)
# ----------------------------------
# do a GET to ``API_COMMISSIONS``
......@@ -713,13 +804,15 @@ class AstakosClient(object):
# ----------------------------
# do a GET to ``API_PROJECTS``
def get_projects(self, name=None, state=None, owner=None):
def get_projects(self, name=None, state=None, owner=None, mode=None):
"""Retrieve all accessible projects
Arguments:
name -- filter by name (optional)
state -- filter by state (optional)
owner -- filter by owner (optional)
mode -- if value is 'member', return only active projects in which
the request user is an active member
In case of success, return a list of project descriptions.
"""
......@@ -730,11 +823,13 @@ class AstakosClient(object):
filters["state"] = state
if owner is not None:
filters["owner"] = owner
if mode is not None:
filters["mode"] = mode
path = self.api_projects
if filters:
path += "?" + urllib.urlencode(filters)
req_headers = {'content-type': 'application/json'}
req_body = (parse_request({"filter": filters}, self.logger)
if filters else None)
return self._call_astakos(self.api_projects,
headers=req_headers, body=req_body)
return self._call_astakos(path, headers=req_headers)
# -----------------------------------------
# do a GET to ``API_PROJECTS``/<project_id>
......@@ -766,7 +861,7 @@ class AstakosClient(object):
method="POST")
# ------------------------------------------
# do a POST to ``API_PROJECTS``/<project_id>
# do a PUT to ``API_PROJECTS``/<project_id>
def modify_project(self, project_id, specs):
"""Submit application to modify an existing project
......@@ -780,7 +875,7 @@ class AstakosClient(object):
req_headers = {'content-type': 'application/json'}
req_body = parse_request(specs, self.logger)
return self._call_astakos(path, headers=req_headers,
body=req_body, method="POST")
body=req_body, method="PUT")
# -------------------------------------------------
# do a POST to ``API_PROJECTS``/<project_id>/action
......@@ -798,56 +893,30 @@ class AstakosClient(object):
path = join_urls(self.api_projects, str(project_id))
path = join_urls(path, "action")
req_headers = {'content-type': 'application/json'}
req_body = parse_request({action: reason}, self.logger)
req_body = parse_request({action: {"reason": reason}}, self.logger)
return self._call_astakos(path, headers=req_headers,
body=req_body, method="POST")
# --------------------------------
# do a GET to ``API_APPLICATIONS``
def get_applications(self, project=None):
"""Retrieve all accessible applications
Arguments:
project -- filter by project (optional)
In case of success, return a list of application descriptions.
"""
req_headers = {'content-type': 'application/json'}
body = {"project": project} if project is not None else None
req_body = parse_request(body, self.logger) if body else None
return self._call_astakos(self.api_applications,
headers=req_headers, body=req_body)
# -----------------------------------------
# do a GET to ``API_APPLICATIONS``/<app_id>
def get_application(self, app_id):
"""Retrieve application description, if accessible
Arguments:
app_id -- application identifier
In case of success, return application description.
"""
path = join_urls(self.api_applications, str(app_id))
return self._call_astakos(path)
# -------------------------------------------------
# do a POST to ``API_APPLICATIONS``/<app_id>/action
def application_action(self, app_id, action, reason=""):
"""Perform action on an application
# do a POST to ``API_PROJECTS``/<project_id>/action
def application_action(self, project_id, app_id, action, reason=""):
"""Perform action on a project application
Arguments:
app_id -- application identifier
action -- action to perform, one of "approve", "deny",
"dismiss", "cancel"
reason -- reason of performing the action
project_id -- project identifier
app_id -- application identifier
action -- action to perform, one of "approve", "deny",
"dismiss", "cancel"
reason -- reason of performing the action
In case of success, return nothing.
"""
path = join_urls(self.api_applications, str(app_id))
path = join_urls(self.api_projects, str(project_id))
path = join_urls(path, "action")
req_headers = {'content-type': 'application/json'}
req_body = parse_request({action: reason}, self.logger)
req_body = parse_request({action: {
"reasons": reason,
"app_id": app_id}}, self.logger)
return self._call_astakos(path, headers=req_headers,
body=req_body, method="POST")
......@@ -862,10 +931,13 @@ class AstakosClient(object):
In case of success, return a list of membership descriptions.
"""
req_headers = {'content-type': 'application/json'}
body = {"project": project} if project is not None else None
req_body = parse_request(body, self.logger) if body else None
return self._call_astakos(self.api_memberships,
headers=req_headers, body=req_body)
filters = {}
if project is not None:
filters["project"] = project
path = self.api_memberships
if filters:
path += '?' + urllib.urlencode(filters)
return self._call_astakos(path, headers=req_headers)
# -----------------------------------------
# do a GET to ``API_MEMBERSHIPS``/<memb_id>
......
......@@ -781,7 +781,7 @@ class TestCommissions(unittest.TestCase):
global auth_url
try:
client = AstakosClient(token['id'], auth_url)
response = client.issue_commission(commission_request)
response = client._issue_commission(commission_request)
except Exception as err:
self.fail("Shouldn't raise Exception %s" % err)
self.assertEqual(response, commission_successful_response['serial'])
......@@ -795,7 +795,7 @@ class TestCommissions(unittest.TestCase):
new_request['provisions'][1]['quantity'] = 520000000
try:
client = AstakosClient(token['id'], auth_url)
client.issue_commission(new_request)
client._issue_commission(new_request)
except QuotaLimit:
pass
except Exception as err:
......
......@@ -139,9 +139,19 @@ retry=0, use_pool=False, pool_size=8, logger=None\ **)**
quotas of a specific user with argument user=UUID. In case of error it
raises an AstakosClientException exception.
**issue_commission(**\ request\ **)**
Issue a commission. In case of success it returns commission's id
(int). Otherwise it raises an AstakosClientException exception.
**service_get_project_quotas(**\ project=None\ **)**
It returns all projects' current quotas for the resources
associated with the service (as dict of dicts).
Optionally, one can query the quotas of a specific project with
argument project=UUID. In case of error it raises an
AstakosClientException exception.
**issue_commission_generic(**\ user_provisions, project_provisions, name="", force=False, auto_accept=False\ **)**
Issue a commission. User provisions are specified as a dict from
(user, project, resource) to int; project provisions as a dict from
(project, resource) to int.
In case of success return commission's id (int).
Otherwise raise an AstakosClientException exception.
**issue_one_commission(**\ holder, source, provisions, name="", force=False, auto_accept=False\ **)**
Issue a commission. We have to specify the holder, the source and the
......@@ -150,6 +160,9 @@ retry=0, use_pool=False, pool_size=8, logger=None\ **)**
commission's id (int). Otherwise it raises an AstakosClientException
exception.
**issue_resource_reassignment(**\ holder, from_source, to_source, provisions, name="", force=False, auto_accept=False\ **)**
Change resource assignment to another project
**get_pending_commissions()**
It returns the pending commissions (list of integers). In case of
error it raises an AstakosClientException exception.
......@@ -176,7 +189,7 @@ retry=0, use_pool=False, pool_size=8, logger=None\ **)**
rejected and which failed to resolved. Otherwise raise an
AstakosClientException exception.
**get_projects(**\ name=None, state=None, owner=None\ **)**
**get_projects(**\ name=None, state=None, owner=None, mode=None\ **)**
Retrieve all accessible projects
**get_project(**\ project_id\ **)**
......@@ -191,14 +204,8 @@ retry=0, use_pool=False, pool_size=8, logger=None\ **)**
**project_action(**\ project_id, action, reason=""\ **)**
Perform action on a project
**get_applications(**\ project=None\ **)**
Retrieve all accessible applications
**get_application(**\ app_id\ **)**
Retrieve application description, if accessible
**application_action(**\ app_id, action, reason=""\ **)**
Perform action on an application
**application_action(**\ project_id, app_id, action, reason=""\ **)**
Perform action on a project application
**get_memberships(**\ project=None\ **)**
Retrieve all accessible memberships
......
......@@ -18,6 +18,7 @@ DEPLOY_SYNNEFO_CMD = "deploy"
TEST_SYNNEFO_CMD = "test"
RUN_BURNIN_CMD = "burnin"
CREATE_X2GO_FILE = "x2goplugin"
SHELL_CONNECT = "shell"
ALL_CMDS = "all"
COMMANDS_IN_ALL_MODE = [
......@@ -32,6 +33,7 @@ COMMANDS_IN_ALL_MODE = [
AVAILABLE_COMMANDS = [
CREATE_X2GO_FILE,
DELETE_SERVER_CMD,
SHELL_CONNECT,
] + COMMANDS_IN_ALL_MODE
USAGE = """usage: %%prog [options] command[,command...]
......@@ -45,6 +47,7 @@ command:
* %s: Run snf-burnin in the deployed Synnefo
* %s: Create x2go plugin file
* %s: Delete the slave server
* %s: Connect to the server using ssh
* %s: Run all the available commands
""" % tuple([CREATE_SERVER_CMD,
......@@ -55,10 +58,11 @@ command:
RUN_BURNIN_CMD,
CREATE_X2GO_FILE,
DELETE_SERVER_CMD,
SHELL_CONNECT,
ALL_CMDS])
def main(): # Too many branches. pylint: disable-msg=R0912
def main(): # pylint: disable=too-many-statements, too-many-branches
"""Parse command line options and run the specified actions"""
parser = OptionParser(usage=USAGE)
parser.add_option("-c", "--conf", dest="config_file", default=None,
......@@ -79,7 +83,7 @@ def main(): # Too many branches. pylint: disable-msg=R0912
help="Upload/Install the public ssh keys contained"
" in this file to the server")
parser.add_option("--name", dest="server_name", default=None,
help=""),
help="")
parser.add_option("-n", "--build-id", dest="build_id", default=None,
type="int",
help="Specify a number to use to identify this build."
......@@ -172,6 +176,8 @@ def main(): # Too many branches. pylint: disable-msg=R0912
synnefo_ci.x2go_plugin(options.x2go_output)
if getattr(options, DELETE_SERVER_CMD, False):
synnefo_ci.destroy_server()
if getattr(options, SHELL_CONNECT, False):
synnefo_ci.shell_connect()
if __name__ == "__main__":
......
......@@ -261,6 +261,12 @@ class SynnefoCI(object):
fip['floating_ip_address'])
self.network_client.delete_floatingip(fip['id'])
# pylint: disable= no-self-use
@_check_fabric
def shell_connect(self):
"""Open shell to remote server"""
fabric.open_shell("export TERM=xterm")
def _create_floating_ip(self):
"""Create a new floating ip"""
networks = self.network_client.list_networks(detail=True)
......
......@@ -316,61 +316,50 @@ Upon success, the system renews the token (if it has expired), logins the user
and sets the cookie, before redirecting the user to the ``next`` parameter
value.
Setting quota limits
~~~~~~~~~~~~~~~~~~~~
Set default quota
`````````````````
To inspect current default base quota limits, run::
# snf-manage resource-list
You can modify the default base quota limit for all future users with::
# snf-manage resource-modify <resource_name> --default-quota <value>
Set base quota for individual users
```````````````````````````````````
For individual users that need different quota than the default
you can set it for each resource like this::
# use this to display quota / uuid
# snf-manage user-show 'uuid or email' --quota
# snf-manage user-modify <user-uuid> --base-quota 'cyclades.vm' 10
You can set base quota for all existing users, with possible exceptions, using::
Projects and quota
~~~~~~~~~~~~~~~~~~
# snf-manage user-modify --all --base-quota cyclades.vm 10 --exclude uuid1,uuid2
Synnefo supports granting resources and controling their quota through the
mechanism of *projects*. A project is considered as a pool of finite
resources. Every actual resources allocated by a user (e.g. a Cyclades VM or
a Pithos container) is also assigned to a project where the user is a
member to. For each resource a project specifies the maximum amount that can
be assigned to it and the maximum amount that a single member can assign to it.
All quota for which values different from the default have been set,
can be listed with::
Default quota
`````````````
# snf-manage quota-list --with-custom=True
Upon user creation, a special purpose user-specific project is automatically
created in order to hold the base quota provided by the system. These *base*
projects are identified with the same UUID as the user.
To inspect the quota that future users will receive by default through their
base projects, check column ``base_default`` in::
Enable the Projects feature
~~~~~~~~~~~~~~~~~~~~~~~~~~~
# snf-manage resource-list
If you want to enable the projects feature so that users may apply
on their own for resources by creating and joining projects,
in ``20-snf-astakos-app-settings.conf`` set::
You can modify the default base quota limit for all future users with::
# this will make the 'projects' page visible in the dashboard
ASTAKOS_PROJECTS_VISIBLE = True
# snf-manage resource-modify <resource_name> --base-default <value>
You can change the maximum allowed number of pending project applications
per user with::
Grant extra quota through projects
``````````````````````````````````
# snf-manage resource-modify astakos.pending_app --default-quota <number>
A user can apply for a new project through the web interface or the API.
Once it is approved by the administrators, the applicant can join the
project and let other users in too.