Commit f3bc27bb authored by Christos Stavrakakis's avatar Christos Stavrakakis
Browse files

Merge branch 'feature-resource-defaults' into develop

parents cc7f14b7 53efa0aa
......@@ -49,6 +49,21 @@ Astakos
Re-registration with `snf-component-register' affects both the base and
the ui URL.
* Changes in resource and quota handling:
* New resources are registered with unlimited default base quota,
represented by 2**63-1.
* Each newly accepted user copies the default value for all resources
as their own base quota. A base quota is considered 'custom' if its
value differs from the default.
* Changing resource's default quota affects the base quota *only* of
future users.
* Resource definition got flags 'api_visible' and 'ui_visible',
replacing flag 'allow_in_projects'. They control whether a user can
access these resources. The system internally always accounts for
all resources, and a user can get off quota even for a resource that
is not visible.
* Remove API call GET /account/v1.0/authenticate in favor of
POST /identity/v2.0/tokens.
......@@ -61,8 +76,17 @@ Astakos
* Management commands:
* Introduced new commands:
* component-show
* quota-list (replacing quota, supports various filters)
* quota-verify (replacing quota)
* Changed commands:
* component-add got options --base-url and --ui-url
* resource-modify --limit became --default-quota
* user-modify can operate on multiple users with --all and --exclude
* user-modify --set-base-quota became --base-quota
* Removed commands:
* quota
* resource-import (subsumed by service-import)
* resource-export-astakos (subsumed by service-export-astakos)
......@@ -110,6 +134,9 @@ Cyclades
to the Astakos groups that are defined in the 'ADMIN_STATS_PERMITTED_GROUPS'
setting. Statistics are also availble from 'snf-manage stats-cyclades'
management command.
* Support enforcing quota through command 'enforce-resources-cyclades'.
* Remove command 'resource-export-cyclades' subsumed by
......@@ -119,6 +146,8 @@ objects by domain attribute. This is used by Plankton for listing VM images.
* Enforce container-level atomicity in (most) Pithos API calls.
* Remove command 'resource-export-pithos' subsumed by 'service-export-pithos'.
.. _Changelog-0.14.10:
......@@ -300,16 +300,13 @@ Setting quota limits
Set default quota
To inspect current default base quota limits, run::
In 20-snf-astakos-app-settings.conf,
uncomment the default setting ``ASTAKOS_SERVICES``
and customize the ``'uplimit'`` values.
These are the default base quota for all users.
# snf-manage resource-list
To apply your configuration run::
You can modify the default base quota limit for all future users with::
# snf-manage astakos-init --load-service-resources
# snf-manage quota --sync
# snf-manage resource-modify <resource_name> --default-quota <value>
Set base quota for individual users
......@@ -320,7 +317,16 @@ you can set it for each resource like this::
# use this to display quota / uuid
# snf-manage user-show 'uuid or email' --quota
# snf-manage user-modify 'user-uuid' --set-base-quota 'cyclades.vm' 10
# snf-manage user-modify <user-uuid> --base-quota 'cyclades.vm' 10
You can set base quota for all existing users, with possible exceptions, using::
# snf-manage user-modify --all --base-quota cyclades.vm 10 --exclude uuid1,uuid2
All quota for which values different from the default have been set,
can be listed with::
# snf-manage quota-list --with-custom=True
Enable the Projects feature
......@@ -336,11 +342,11 @@ in ``20-snf-astakos-app-settings.conf`` set::
You can change the maximum allowed number of pending project applications
per user with::
# snf-manage resource-modify astakos.pending_app --limit <number>
# snf-manage resource-modify astakos.pending_app --default-quota <number>
You can also set a user-specific limit with::
# snf-manage user-modify 'user-uuid' --set-base-quota 'astakos.pending_app' 5
# snf-manage user-modify <user-uuid> --base-quota 'astakos.pending_app' 5
When users apply for projects they are not automatically granted
the resources. They must first be approved by the administrator.
......@@ -918,12 +924,12 @@ component-show Show component details
project-control Manage projects and applications
project-list List projects
project-show Show project details
quota List and check the integrity of user quota
quota-list List user quota
quota-verify Check the integrity of user quota
reconcile-resources-astakos Reconcile resource usage of Quotaholder with Astakos DB
resource-export-astakos Export astakos resources in json format
resource-import Register resources
resource-list List resources
resource-modify Modify a resource's default base quota and boolean flags
service-export-astakos Export Astakos services and resources in JSON format
service-import Register services
service-list List services
service-show Show service details
......@@ -949,7 +955,7 @@ Pithos snf-manage commands
Name Description
============================ ===========================
reconcile-commissions-pithos Display unresolved commissions and trigger their recovery
resource-export-pithos Export pithos resources in json format
service-export-pithos Export Pithos services and resources in JSON format
reconcile-resources-pithos Detect unsynchronized usage between Astakos and Pithos DB resources and synchronize them if specified so.
============================ ===========================
......@@ -964,6 +970,7 @@ backend-list List backends
backend-modify Modify a backend
backend-update-status Update backend statistics for instance allocation
backend-remove Remove a Ganeti backend
enforce-resources-cyclades Check and fix quota violations for Cyclades resources
server-create Create a new server
server-show Show server details
server-list List servers
......@@ -996,8 +1003,7 @@ floating-ip-list List floating IPs
floating-ip-remove Delete a floating IP
queue-inspect Inspect the messages of a RabbitMQ queue
queue-retry Resend messages from Dead Letter queues to original exchanges
resource-export-cyclades Export Cyclades resources in JSON format.
service-export-cyclades Export Cyclades services in JSON format.
service-export-cyclades Export Cyclades services and resources in JSON format
subnet-create Create a subnet
subnet-inspect Inspect a subnet in DB
subnet-list List subnets
......@@ -69,11 +69,11 @@ For clarity, option ``--limit`` will be renamed ``--default-quota``.
We can currently change a user's base quota with::
snf-manage user-modify <id> --set-base-quota <resource_name> <value>
snf-manage user-modify <id> --set-base-quota <resource_name> <value>
This command will be extended with option ``--all`` to allow changing base
quota for multiple users; option ``--exclude`` will allow introducing
exceptions. ``--set-base-quota`` will be renamed ``--base-quota``.
Inspecting base quota
......@@ -402,7 +402,7 @@ applying/approving applications in order to modify some project settings,
such as the quota limits.
Currently, the administrator can change the user base quota with:
``snf-manage user-modify <id> --set-base-quota <resource> <capacity>``.
``snf-manage user-modify <id> --base-quota <resource> <capacity>``.
This will be removed in favor of the ``project-modify`` command, so that all
quota are handled in a uniform way. Similar to ``user-modify --all``,
``project-modify`` will get options ``--all-base`` and ``--all-non-base`` to
......@@ -914,7 +914,7 @@ We now have to specify the limit on resources that each user can employ
.. code-block:: console
# snf-manage resource-modify --limit-interactive
# snf-manage resource-modify --default-quota-interactive
Servers Initialization
......@@ -7,7 +7,7 @@ The upgrade to v0.15 consists in the following steps:
2. Upgrade packages, migrate the databases and configure settings.
3. Re-register components and services in astakos.
3. Register services and resources.
4. Bring up all services.
......@@ -119,11 +119,14 @@ setting will have the value of
For Pithos service we have to change the ``20-snf-pithos-app-settings.conf``
file in the same way as above.
3. Re-register components and services in astakos
3. Register services and resources
Component registration has changed; you will thus need to repeat the
process. On the astakos node, run::
3.1 Re-register service and resource definitions
You will need to register again all Synnefo components, updating the
service and resource definitions. On the astakos node, run::
astakos-host$ snf-component-register
......@@ -131,6 +134,66 @@ This will detect that the Synnefo components are already registered and ask
to re-register. Answer positively. You need to enter the base URL and the UI
URL for each component, just like during the initial registration.
.. note::
You can run ``snf-manage component-list -o name,ui_url`` to inspect the
current registered UI URL. In the default installation, the base URL can
be found by stripping ``/ui`` from the UI URL.
The meaning of resources ``cyclades.cpu`` and ``cyclades.ram`` has changed:
they now denote the number of CPUs and, respectively, RAM of *active* VMs
rather than all VMs. To represent total CPUs and total RAM, as previously,
new resources ``cyclades.total_cpu`` and ``cyclades.total_ram`` are
introduced. We now also control the usage of floating IPs through resource
3.2 Tweek resource settings
New resources (``cyclades.total_cpu``, ``cyclades.total_ram``, and
``cyclades.floating_ip``) are registered with infinite default base quota.
You will probably need to restrict them, especially
``cyclades.floating_ip``. In order to change the default for all *future*
users, for instance restricting floating IPs to 2, run::
astakos-host$ snf-manage resource-modify cyclades.floating_ip --default-quota 2
Note that this command does not affect *existing* users any more. They can
still have infinite floating IPs. You can update base quota of existing
users in bulk, possibly excluding some users, with::
astakos-host$ snf-manage user-modify --all --base-quota cyclades.floating_ip 2 --exclude uuid1,uuid2
.. note::
You can inspect base quota with ``snf-manage quota-list`` before applying
any changes, for example::
# Get users with cyclades.vm base quota that differ from the default value
astakos-host$ snf-manage quota-list --with-custom=True --filter-by "resource=cyclades.vm"
# Get users with cyclades.vm base quota greater than 3
astakos-host$ snf-manage quota-list --filter-by "resource=cyclades.vm,base_quota>3"
It is now possible to control whether a resource is visible for the users
through the API or the UI. Note that the system always checks resource
quota, regardless of their visibility. By default, ``cyclades.total_cpu``,
``cyclades.total_ram`` and ``astakos.pending_app`` are not visible. You can
change this behavior with::
astakos-host$ snf-manage resource-modify <resource> --api-visible=True (or --ui-visible=True)
3.3 Update the Quotaholder
To update quota for all new or modified Cyclades resources, bring up Astakos::
astakos-host$ service gunicorn start
and run on the Cyclades node::
cyclades-host$ snf-manage reconcile-resources-cyclades --fix --force
4. Bring all services up
......@@ -95,6 +95,8 @@ def _application_details(application, all_grants):
grants = all_grants.get(, [])
resources = {}
for grant in grants:
if not grant.resource.api_visible:
resources[] = {
"member_capacity": grant.member_capacity,
"project_capacity": grant.project_capacity,
......@@ -39,7 +39,7 @@ from django.db import transaction
from snf_django.lib import api
from snf_django.lib.api.faults import BadRequest, ItemNotFound
from import get_resources
from import register
from import get_user_quotas, service_get_quotas
import astakos.quotaholder_app.exception as qh_exception
......@@ -52,7 +52,9 @@ from .util import (json_response, is_integer, are_integer,
@api.api_method(http_method='GET', token_required=True, user_required=False)
def quotas(request):
result = get_user_quotas(request.user)
visible_resources = register.get_api_visible_resources()
resource_names = [ for r in visible_resources]
result = get_user_quotas(request.user, resources=resource_names)
return json_response(result)
......@@ -71,7 +73,8 @@ def service_quotas(request):
@api.api_method(http_method='GET', token_required=False, user_required=False)
def resources(request):
result = get_resources()
resources = register.get_api_visible_resources()
result = register.resources_to_dict(resources)
return json_response(result)
......@@ -48,7 +48,8 @@ astakos_services = {
'name': "astakos.pending_app",
'service_type': "account",
'service_origin': "astakos_account",
'allow_in_projects': False},
'ui_visible': False,
'api_visible': False},
......@@ -40,7 +40,7 @@ from import functions
from import settings
from import forms
from import qh_sync_user
from import qh_sync_new_user
import as astakos_messages
......@@ -257,7 +257,7 @@ class ActivationBackend(object):
default=lambda obj:
if user.is_rejected:
logger.warning("User has previously been "
......@@ -886,7 +886,7 @@ class ProjectApplicationForm(forms.ModelForm):
# keep only resource limits for selected resource groups
if'is_selected_%s' %, "0") == "1":
if not resource.allow_in_projects:
if not resource.ui_visible:
raise forms.ValidationError("Invalid resource %s" %
d = model_to_dict(resource)
......@@ -738,7 +738,8 @@ def validate_resource_policies(policies):
raise ProjectBadRequest("Malformed resource policies")
resource_names = policies.keys()
resources = Resource.objects.filter(name__in=resource_names)
resources = Resource.objects.filter(name__in=resource_names,
resource_d = {}
for resource in resources:
resource_d[] = resource
......@@ -40,7 +40,7 @@ from import CommandError
from synnefo.util import units
from import AstakosUser
from import get_resources
from import register
import sys
......@@ -68,6 +68,28 @@ def get_user(email_or_id, **kwargs):
return None
def get_accepted_user(user_ident):
if is_uuid(user_ident):
user = AstakosUser.objects.get(uuid=user_ident)
except AstakosUser.DoesNotExist:
raise CommandError('There is no user with uuid: %s' %
elif is_email(user_ident):
user = AstakosUser.objects.get(username=user_ident)
except AstakosUser.DoesNotExist:
raise CommandError('There is no user with email: %s' %
raise CommandError('Please specify user by uuid or email')
if not user.is_accepted():
raise CommandError('%s is not an accepted user.' % user.uuid)
return user
def get_astakosuser_content_type():
return ContentType.objects.get(app_label='im',
......@@ -163,7 +185,8 @@ class ResourceDict(object):
def get(cls):
if cls._object is None:
cls._object = get_resources()
rs = register.get_resources()
cls._object = register.resources_to_dict(rs)
return cls._object
......@@ -174,27 +197,51 @@ def show_resource_value(number, resource, style):
return, unit, style)
def collect_holder_quotas(holder_quotas, h_initial, style=None):
print_data = []
for source, source_quotas in holder_quotas.iteritems():
s_initial = h_initial[source]
except KeyError:
for resource, values in source_quotas.iteritems():
initial = s_initial[resource]
except KeyError:
initial = show_resource_value(initial, resource, style)
limit = show_resource_value(values['limit'], resource, style)
usage = show_resource_value(values['usage'], resource, style)
fields = (source, resource, initial, limit, usage)
return print_data
def show_user_quotas(holder_quotas, h_initial, style=None):
labels = ('source', 'resource', 'base_quota', 'total_quota', 'usage')
print_data = collect_holder_quotas(holder_quotas, h_initial, style=style)
return print_data, labels
def show_quotas(qh_quotas, astakos_initial, info=None, style=None):
labels = ('source', 'resource', 'base quota', 'total quota', 'usage')
labels = ('user', 'source', 'resource', 'base_quota', 'total_quota',
if info is not None:
labels = ('uuid', 'email') + labels
labels = ('displayname',) + labels
print_data = []
for holder, holder_quotas in qh_quotas.iteritems():
h_initial = astakos_initial.get(holder)
if h_initial is None:
if info is not None:
email = info.get(holder, "")
for source, source_quotas in holder_quotas.iteritems():
s_initial = h_initial.get(source) if h_initial else None
for resource, values in source_quotas.iteritems():
initial = s_initial.get(resource) if s_initial else None
initial = show_resource_value(initial, resource, style)
limit = show_resource_value(values['limit'], resource, style)
usage = show_resource_value(values['usage'], resource, style)
fields = (source, resource, initial, limit, usage)
if info is not None:
fields = (holder, email) + fields
h_data = collect_holder_quotas(holder_quotas, h_initial, style=style)
if info is not None:
h_data = [(email, holder) + fields for fields in h_data]
h_data = [(holder,) + fields for fields in h_data]
print_data += h_data
return print_data, labels
......@@ -31,15 +31,43 @@
# interpreted as representing official policies, either expressed
# or implied, of GRNET S.A.
from django.utils import simplejson as json
from import BaseCommand
from synnefo.util import units
from import CommandError
from django.db.models import Q
from pithos.api import resources
OP_MAP = [
("!=", lambda x: ~x, ""),
(">=", lambda x: x, "__gte"),
("=>", lambda x: x, "__gte"),
(">", lambda x: x, "__gt"),
("<=", lambda x: x, "__lte"),
("=<", lambda x: x, "__lte"),
("<", lambda x: x, "__lt"),
("=", lambda x: x, ""),
class Command(BaseCommand):
help = "Export pithos resources in json format"
def handle(self, *args, **options):
output = json.dumps(resources.resources, indent=4)
self.stdout.write(output + '\n')
def parse_filter(exp):
for s, prepend, op in OP_MAP:
key, sep, value = exp.partition(s)
if s == sep:
return key, prepend, op, value
raise CommandError("Could not parse filter.")
def make_query(flt, handlers):
key, prepend, opstr, value = parse_filter(flt)
(dbkey, parse) = handlers[key]
return prepend(Q(**{dbkey+opstr: parse(value)}))
except KeyError:
return None
def parse_with_unit(value):
return units.parse(value)
except units.ParseError:
raise CommandError("Failed to parse value, should be an integer, "
"possibly followed by a unit, or 'inf'.")
# Copyright 2013 GRNET S.A. All rights reserved.
# Copyright 2012, 2013 GRNET S.A. All rights reserved.
# Redistribution and use in source and binary forms, with or
# without modification, are permitted provided that the following
......@@ -31,15 +31,102 @@
# interpreted as representing official policies, either expressed
# or implied, of GRNET S.A.
from django.utils import simplejson as json
from import BaseCommand
from optparse import make_option
from django.db import transaction
from import resources
from import AstakosUser
from import list_user_quotas
from import SynnefoCommand, CommandError
from import utils
from import _common as common
from import _filtering as filtering
from django.db.models import Q, F
import logging
logger = logging.getLogger(__name__)
class Command(BaseCommand):
help = "Export astakos resources in json format"
class Command(SynnefoCommand):
help = "List user quota"
option_list = SynnefoCommand.option_list + (
help=("Specify display unit for resource values "
"(one of %s); defaults to mb") %
help="Show quota that is over limit"),
help=("Filter quota different from the default or "
"equal to it")),
help="Filter by field; "
"e.g. \"user=uuid,usage>=10M,base_quota<inf\""),
help="Show user display name"),
"total_quota": ("limit", filtering.parse_with_unit),
"usage": ("usage_max", filtering.parse_with_unit),
"user": ("holder", lambda x: x),
"resource": ("resource", lambda x: x),
"source": ("source", lambda x: x),
"base_quota": ("capacity", filtering.parse_with_unit),
def handle(self, *args, **options):
output = json.dumps(resources, indent=4)
self.stdout.write(output + '\n')
output_format = options["output_format"]
displayname = bool(options["displayname"])
unit_style = options["unit_style"]
filteropt = options["filter_by"]
if filteropt is not None:
filters = filteropt.split(",")