Merge branch 'feature-resource-defaults' into develop

Changelog

@@ -49,6 +49,21 @@ Astakos
   Re-registration with `snf-component-register' affects both the base and
   the ui URL.
 
+* Changes in resource and quota handling:
+
+  * New resources are registered with unlimited default base quota,
+    represented by 2**63-1.
+  * Each newly accepted user copies the default value for all resources
+    as their own base quota. A base quota is considered 'custom' if its
+    value differs from the default.
+  * Changing resource's default quota affects the base quota *only* of
+    future users.
+  * Resource definition got flags 'api_visible' and 'ui_visible',
+    replacing flag 'allow_in_projects'. They control whether a user can
+    access these resources. The system internally always accounts for
+    all resources, and a user can get off quota even for a resource that
+    is not visible.
+
 * Remove API call GET /account/v1.0/authenticate in favor of
   POST /identity/v2.0/tokens.
 
@@ -61,8 +76,17 @@ Astakos
 * Management commands:
   * Introduced new commands:
      * component-show
+     * quota-list (replacing quota, supports various filters)
+     * quota-verify (replacing quota)
   * Changed commands:
      * component-add got options --base-url and --ui-url
+     * resource-modify --limit became --default-quota
+     * user-modify can operate on multiple users with --all and --exclude
+     * user-modify --set-base-quota became --base-quota
+  * Removed commands:
+     * quota
+     * resource-import (subsumed by service-import)
+     * resource-export-astakos (subsumed by service-export-astakos)
 
 Cyclades
 --------
@@ -110,6 +134,9 @@ Cyclades
   to the Astakos groups that are defined in the 'ADMIN_STATS_PERMITTED_GROUPS'
   setting.  Statistics are also availble from 'snf-manage stats-cyclades'
   management command.
+* Support enforcing quota through command 'enforce-resources-cyclades'.
+* Remove command 'resource-export-cyclades' subsumed by
+  'service-export-cyclades'.
 
 Pithos
 ------
@@ -119,6 +146,8 @@ objects by domain attribute. This is used by Plankton for listing VM images.
 
 * Enforce container-level atomicity in (most) Pithos API calls.
 
+* Remove command 'resource-export-pithos' subsumed by 'service-export-pithos'.
+
 .. _Changelog-0.14.10:
 
 v0.14.10

docs/admin-guide.rst

@@ -300,16 +300,13 @@ Setting quota limits
 
 Set default quota
 `````````````````
+To inspect current default base quota limits, run::
 
-In 20-snf-astakos-app-settings.conf, 
-uncomment the default setting ``ASTAKOS_SERVICES``
-and customize the ``'uplimit'`` values.
-These are the default base quota for all users.
+   # snf-manage resource-list
 
-To apply your configuration run::
+You can modify the default base quota limit for all future users with::
 
-    # snf-manage astakos-init --load-service-resources
-    # snf-manage quota --sync
+   # snf-manage resource-modify <resource_name> --default-quota <value>
 
 Set base quota for individual users
 ```````````````````````````````````
@@ -320,7 +317,16 @@ you can set it for each resource like this::
     # use this to display quota / uuid
     # snf-manage user-show 'uuid or email' --quota
 
-    # snf-manage user-modify 'user-uuid' --set-base-quota 'cyclades.vm' 10
+    # snf-manage user-modify <user-uuid> --base-quota 'cyclades.vm' 10
+
+You can set base quota for all existing users, with possible exceptions, using::
+
+    # snf-manage user-modify --all --base-quota cyclades.vm 10 --exclude uuid1,uuid2
+
+All quota for which values different from the default have been set,
+can be listed with::
+
+    # snf-manage quota-list --with-custom=True
 
 
 Enable the Projects feature
@@ -336,11 +342,11 @@ in ``20-snf-astakos-app-settings.conf`` set::
 You can change the maximum allowed number of pending project applications
 per user with::
 
-    # snf-manage resource-modify astakos.pending_app --limit <number>
+    # snf-manage resource-modify astakos.pending_app --default-quota <number>
 
 You can also set a user-specific limit with::
 
-    # snf-manage user-modify 'user-uuid' --set-base-quota 'astakos.pending_app' 5
+    # snf-manage user-modify <user-uuid> --base-quota 'astakos.pending_app' 5
 
 When users apply for projects they are not automatically granted
 the resources. They must first be approved by the administrator.
@@ -918,12 +924,12 @@ component-show                Show component details
 project-control               Manage projects and applications
 project-list                  List projects
 project-show                  Show project details
-quota                         List and check the integrity of user quota
+quota-list                    List user quota
+quota-verify                  Check the integrity of user quota
 reconcile-resources-astakos   Reconcile resource usage of Quotaholder with Astakos DB
-resource-export-astakos       Export astakos resources in json format
-resource-import               Register resources
 resource-list                 List resources
 resource-modify               Modify a resource's default base quota and boolean flags
+service-export-astakos        Export Astakos services and resources in JSON format
 service-import                Register services
 service-list                  List services
 service-show                  Show service details
@@ -949,7 +955,7 @@ Pithos snf-manage commands
 Name                          Description
 ============================  ===========================
 reconcile-commissions-pithos  Display unresolved commissions and trigger their recovery
-resource-export-pithos        Export pithos resources in json format
+service-export-pithos         Export Pithos services and resources in JSON format
 reconcile-resources-pithos    Detect unsynchronized usage between Astakos and Pithos DB resources and synchronize them if specified so.
 ============================  ===========================
 
@@ -964,6 +970,7 @@ backend-list                   List backends
 backend-modify                 Modify a backend
 backend-update-status          Update backend statistics for instance allocation
 backend-remove                 Remove a Ganeti backend
+enforce-resources-cyclades     Check and fix quota violations for Cyclades resources
 server-create                  Create a new server
 server-show                    Show server details
 server-list                    List servers
@@ -996,8 +1003,7 @@ floating-ip-list               List floating IPs
 floating-ip-remove             Delete a floating IP
 queue-inspect                  Inspect the messages of a RabbitMQ queue
 queue-retry                    Resend messages from Dead Letter queues to original exchanges
-resource-export-cyclades       Export Cyclades resources in JSON format.
-service-export-cyclades        Export Cyclades services in JSON format.
+service-export-cyclades        Export Cyclades services and resources in JSON format
 subnet-create                  Create a subnet
 subnet-inspect                 Inspect a subnet in DB
 subnet-list                    List subnets

docs/design/resource-defaults.rst

@@ -69,11 +69,11 @@ For clarity, option ``--limit`` will be renamed ``--default-quota``.
 
 We can currently change a user's base quota with::
 
-     snf-manage user-modify <id> --set-base-quota <resource_name> <value>
+  snf-manage user-modify <id> --set-base-quota <resource_name> <value>
 
 This command will be extended with option ``--all`` to allow changing base
 quota for multiple users; option ``--exclude`` will allow introducing
-exceptions.
+exceptions. ``--set-base-quota`` will be renamed ``--base-quota``.
 
 Inspecting base quota
 =====================

docs/design/resource-pool-projects.rst

@@ -402,7 +402,7 @@ applying/approving applications in order to modify some project settings,
 such as the quota limits.
 
 Currently, the administrator can change the user base quota with:
-``snf-manage user-modify <id> --set-base-quota <resource> <capacity>``.
+``snf-manage user-modify <id> --base-quota <resource> <capacity>``.
 This will be removed in favor of the ``project-modify`` command, so that all
 quota are handled in a uniform way. Similar to ``user-modify --all``,
 ``project-modify`` will get options ``--all-base`` and ``--all-non-base`` to

docs/quick-install-admin-guide.rst

@@ -914,7 +914,7 @@ We now have to specify the limit on resources that each user can employ
 
 .. code-block:: console
 
-    # snf-manage resource-modify --limit-interactive
+    # snf-manage resource-modify --default-quota-interactive
 
 
 Servers Initialization

docs/upgrade/upgrade-0.15.rst

@@ -7,7 +7,7 @@ The upgrade to v0.15 consists in the following steps:
 
 2. Upgrade packages, migrate the databases and configure settings.
 
-3. Re-register components and services in astakos.
+3. Register services and resources.
 
 4. Bring up all services.
 
@@ -119,11 +119,14 @@ setting will have the value of
 For Pithos service we have to change the ``20-snf-pithos-app-settings.conf``
 file in the same way as above.
 
-3. Re-register components and services in astakos
-=================================================
+3. Register services and resources
+==================================
 
-Component registration has changed; you will thus need to repeat the
-process. On the astakos node, run::
+3.1 Re-register service and resource definitions
+------------------------------------------------
+
+You will need to register again all Synnefo components, updating the
+service and resource definitions. On the astakos node, run::
 
     astakos-host$ snf-component-register
 
@@ -131,6 +134,66 @@ This will detect that the Synnefo components are already registered and ask
 to re-register. Answer positively. You need to enter the base URL and the UI
 URL for each component, just like during the initial registration.
 
+.. note::
+
+   You can run ``snf-manage component-list -o name,ui_url`` to inspect the
+   current registered UI URL. In the default installation, the base URL can
+   be found by stripping ``/ui`` from the UI URL.
+
+The meaning of resources ``cyclades.cpu`` and ``cyclades.ram`` has changed:
+they now denote the number of CPUs and, respectively, RAM of *active* VMs
+rather than all VMs. To represent total CPUs and total RAM, as previously,
+new resources ``cyclades.total_cpu`` and ``cyclades.total_ram`` are
+introduced. We now also control the usage of floating IPs through resource
+``cyclades.floating_ip``.
+
+3.2 Tweek resource settings
+---------------------------
+
+New resources (``cyclades.total_cpu``, ``cyclades.total_ram``, and
+``cyclades.floating_ip``) are registered with infinite default base quota.
+You will probably need to restrict them, especially
+``cyclades.floating_ip``. In order to change the default for all *future*
+users, for instance restricting floating IPs to 2, run::
+
+    astakos-host$ snf-manage resource-modify cyclades.floating_ip --default-quota 2
+
+Note that this command does not affect *existing* users any more. They can
+still have infinite floating IPs. You can update base quota of existing
+users in bulk, possibly excluding some users, with::
+
+    astakos-host$ snf-manage user-modify --all --base-quota cyclades.floating_ip 2 --exclude uuid1,uuid2
+
+.. note::
+
+   You can inspect base quota with ``snf-manage quota-list`` before applying
+   any changes, for example::
+
+     # Get users with cyclades.vm base quota that differ from the default value
+     astakos-host$ snf-manage quota-list --with-custom=True --filter-by "resource=cyclades.vm"
+
+     # Get users with cyclades.vm base quota greater than 3
+     astakos-host$ snf-manage quota-list --filter-by "resource=cyclades.vm,base_quota>3"
+
+It is now possible to control whether a resource is visible for the users
+through the API or the UI. Note that the system always checks resource
+quota, regardless of their visibility. By default, ``cyclades.total_cpu``,
+``cyclades.total_ram`` and ``astakos.pending_app`` are not visible. You can
+change this behavior with::
+
+    astakos-host$ snf-manage resource-modify <resource> --api-visible=True (or --ui-visible=True)
+
+3.3 Update the Quotaholder
+--------------------------
+
+To update quota for all new or modified Cyclades resources, bring up Astakos::
+
+    astakos-host$ service gunicorn start
+
+and run on the Cyclades node::
+
+   cyclades-host$ snf-manage reconcile-resources-cyclades --fix --force
+
 4. Bring all services up
 ========================
 

snf-astakos-app/astakos/api/projects.py

@@ -95,6 +95,8 @@ def _application_details(application, all_grants):
     grants = all_grants.get(application.id, [])
     resources = {}
     for grant in grants:
+        if not grant.resource.api_visible:
+            continue
         resources[grant.resource.name] = {
             "member_capacity": grant.member_capacity,
             "project_capacity": grant.project_capacity,

snf-astakos-app/astakos/api/quotas.py

@@ -39,7 +39,7 @@ from django.db import transaction
 from snf_django.lib import api
 from snf_django.lib.api.faults import BadRequest, ItemNotFound
 
-from astakos.im.register import get_resources
+from astakos.im import register
 from astakos.im.quotas import get_user_quotas, service_get_quotas
 
 import astakos.quotaholder_app.exception as qh_exception
@@ -52,7 +52,9 @@ from .util import (json_response, is_integer, are_integer,
 @api.api_method(http_method='GET', token_required=True, user_required=False)
 @user_from_token
 def quotas(request):
-    result = get_user_quotas(request.user)
+    visible_resources = register.get_api_visible_resources()
+    resource_names = [r.name for r in visible_resources]
+    result = get_user_quotas(request.user, resources=resource_names)
     return json_response(result)
 
 
@@ -71,7 +73,8 @@ def service_quotas(request):
 
 @api.api_method(http_method='GET', token_required=False, user_required=False)
 def resources(request):
-    result = get_resources()
+    resources = register.get_api_visible_resources()
+    result = register.resources_to_dict(resources)
     return json_response(result)
 
 

snf-astakos-app/astakos/api/services.py

@@ -48,7 +48,8 @@ astakos_services = {
                 'name': "astakos.pending_app",
                 'service_type': "account",
                 'service_origin': "astakos_account",
-                'allow_in_projects': False},
+                'ui_visible': False,
+                'api_visible': False},
         },
     },
 

snf-astakos-app/astakos/im/activation_backends.py

@@ -40,7 +40,7 @@ from astakos.im import functions
 from astakos.im import settings
 from astakos.im import forms
 
-from astakos.im.quotas import qh_sync_user
+from astakos.im.quotas import qh_sync_new_user
 
 import astakos.im.messages as astakos_messages
 
@@ -257,7 +257,7 @@ class ActivationBackend(object):
                                          default=lambda obj:
                                          str(obj))
         user.save()
-        qh_sync_user(user)
+        qh_sync_new_user(user)
 
         if user.is_rejected:
             logger.warning("User has previously been "

snf-astakos-app/astakos/im/forms.py

@@ -886,7 +886,7 @@ class ProjectApplicationForm(forms.ModelForm):
                 # keep only resource limits for selected resource groups
                 if self.data.get('is_selected_%s' %
                                  resource.group, "0") == "1":
-                    if not resource.allow_in_projects:
+                    if not resource.ui_visible:
                         raise forms.ValidationError("Invalid resource %s" %
                                                     resource.name)
                     d = model_to_dict(resource)

snf-astakos-app/astakos/im/functions.py

@@ -738,7 +738,8 @@ def validate_resource_policies(policies):
         raise ProjectBadRequest("Malformed resource policies")
 
     resource_names = policies.keys()
-    resources = Resource.objects.filter(name__in=resource_names)
+    resources = Resource.objects.filter(name__in=resource_names,
+                                        api_visible=True)
     resource_d = {}
     for resource in resources:
         resource_d[resource.name] = resource

snf-astakos-app/astakos/im/management/commands/_common.py

@@ -40,7 +40,7 @@ from django.core.management import CommandError
 
 from synnefo.util import units
 from astakos.im.models import AstakosUser
-from astakos.im.register import get_resources
+from astakos.im import register
 import sys
 
 
@@ -68,6 +68,28 @@ def get_user(email_or_id, **kwargs):
         return None
 
 
+def get_accepted_user(user_ident):
+    if is_uuid(user_ident):
+        try:
+            user = AstakosUser.objects.get(uuid=user_ident)
+        except AstakosUser.DoesNotExist:
+            raise CommandError('There is no user with uuid: %s' %
+                               user_ident)
+    elif is_email(user_ident):
+        try:
+            user = AstakosUser.objects.get(username=user_ident)
+        except AstakosUser.DoesNotExist:
+            raise CommandError('There is no user with email: %s' %
+                               user_ident)
+    else:
+        raise CommandError('Please specify user by uuid or email')
+
+    if not user.is_accepted():
+        raise CommandError('%s is not an accepted user.' % user.uuid)
+
+    return user
+
+
 def get_astakosuser_content_type():
     try:
         return ContentType.objects.get(app_label='im',
@@ -163,7 +185,8 @@ class ResourceDict(object):
     @classmethod
     def get(cls):
         if cls._object is None:
-            cls._object = get_resources()
+            rs = register.get_resources()
+            cls._object = register.resources_to_dict(rs)
         return cls._object
 
 
@@ -174,27 +197,51 @@ def show_resource_value(number, resource, style):
     return units.show(number, unit, style)
 
 
+def collect_holder_quotas(holder_quotas, h_initial, style=None):
+    print_data = []
+    for source, source_quotas in holder_quotas.iteritems():
+        try:
+            s_initial = h_initial[source]
+        except KeyError:
+            continue
+        for resource, values in source_quotas.iteritems():
+            try:
+                initial = s_initial[resource]
+            except KeyError:
+                continue
+            initial = show_resource_value(initial, resource, style)
+            limit = show_resource_value(values['limit'], resource, style)
+            usage = show_resource_value(values['usage'], resource, style)
+            fields = (source, resource, initial, limit, usage)
+            print_data.append(fields)
+    return print_data
+
+
+def show_user_quotas(holder_quotas, h_initial, style=None):
+    labels = ('source', 'resource', 'base_quota', 'total_quota', 'usage')
+    print_data = collect_holder_quotas(holder_quotas, h_initial, style=style)
+    return print_data, labels
+
+
 def show_quotas(qh_quotas, astakos_initial, info=None, style=None):
-    labels = ('source', 'resource', 'base quota', 'total quota', 'usage')
+    labels = ('user', 'source', 'resource', 'base_quota', 'total_quota',
+              'usage')
     if info is not None:
-        labels = ('uuid', 'email') + labels
+        labels = ('displayname',) + labels
 
     print_data = []
     for holder, holder_quotas in qh_quotas.iteritems():
         h_initial = astakos_initial.get(holder)
+        if h_initial is None:
+            continue
+
         if info is not None:
             email = info.get(holder, "")
 
-        for source, source_quotas in holder_quotas.iteritems():
-            s_initial = h_initial.get(source) if h_initial else None
-            for resource, values in source_quotas.iteritems():
-                initial = s_initial.get(resource) if s_initial else None
-                initial = show_resource_value(initial, resource, style)
-                limit = show_resource_value(values['limit'], resource, style)
-                usage = show_resource_value(values['usage'], resource, style)
-                fields = (source, resource, initial, limit, usage)
-                if info is not None:
-                    fields = (holder, email) + fields
-
-                print_data.append(fields)
+        h_data = collect_holder_quotas(holder_quotas, h_initial, style=style)
+        if info is not None:
+            h_data = [(email, holder) + fields for fields in h_data]
+        else:
+            h_data = [(holder,) + fields for fields in h_data]
+        print_data += h_data
     return print_data, labels

snf-pithos-app/pithos/api/management/commands/resource-export-pithos.py → snf-astakos-app/astakos/im/management/commands/_filtering.py

@@ -31,15 +31,43 @@
 # interpreted as representing official policies, either expressed
 # or implied, of GRNET S.A.
 
-from django.utils import simplejson as json
-from django.core.management.base import BaseCommand
+from synnefo.util import units
+from django.core.management import CommandError
+from django.db.models import Q
 
-from pithos.api import resources
 
+OP_MAP = [
+    ("!=", lambda x: ~x, ""),
+    (">=", lambda x: x, "__gte"),
+    ("=>", lambda x: x, "__gte"),
+    (">", lambda x: x, "__gt"),
+    ("<=", lambda x: x, "__lte"),
+    ("=<", lambda x: x, "__lte"),
+    ("<", lambda x: x, "__lt"),
+    ("=", lambda x: x, ""),
+    ]
 
-class Command(BaseCommand):
-    help = "Export pithos resources in json format"
 
-    def handle(self, *args, **options):
-        output = json.dumps(resources.resources, indent=4)
-        self.stdout.write(output + '\n')
+def parse_filter(exp):
+    for s, prepend, op in OP_MAP:
+        key, sep, value = exp.partition(s)
+        if s == sep:
+            return key, prepend, op, value
+    raise CommandError("Could not parse filter.")
+
+
+def make_query(flt, handlers):
+    key, prepend, opstr, value = parse_filter(flt)
+    try:
+        (dbkey, parse) = handlers[key]
+        return prepend(Q(**{dbkey+opstr: parse(value)}))
+    except KeyError:
+        return None
+
+
+def parse_with_unit(value):
+    try:
+        return units.parse(value)
+    except units.ParseError:
+        raise CommandError("Failed to parse value, should be an integer, "
+                           "possibly followed by a unit, or 'inf'.")

snf-astakos-app/astakos/im/management/commands/resource-export-astakos.py → snf-astakos-app/astakos/im/management/commands/quota-list.py

-# Copyright 2013 GRNET S.A. All rights reserved.
+# Copyright 2012, 2013 GRNET S.A. All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or
 # without modification, are permitted provided that the following
@@ -31,15 +31,102 @@
 # interpreted as representing official policies, either expressed
 # or implied, of GRNET S.A.
 
-from django.utils import simplejson as json
-from django.core.management.base import BaseCommand
+from optparse import make_option
+from django.db import transaction
 
-from astakos.im.astakos_resources import resources
+from astakos.im.models import AstakosUser
+from astakos.im.quotas import list_user_quotas
+from snf_django.management.commands import SynnefoCommand, CommandError
+from snf_django.management import utils
+from astakos.im.management.commands import _common as common
+from astakos.im.management.commands import _filtering as filtering
+from django.db.models import Q, F
 
+import logging
+logger = logging.getLogger(__name__)
 
-class Command(BaseCommand):
-    help = "Export astakos resources in json format"
 
+class Command(SynnefoCommand):
+    help = "List user quota"
+
+    option_list = SynnefoCommand.option_list + (
+        make_option('--unit-style',
+                    default='mb',
+                    help=("Specify display unit for resource values "
+                          "(one of %s); defaults to mb") %
+                    common.style_options),
+        make_option('--overlimit',
+                    action='store_true',
+                    help="Show quota that is over limit"),
+        make_option('--with-custom',
+                    metavar='True|False',
+                    help=("Filter quota different from the default or "
+                          "equal to it")),
+        make_option('--filter-by',
+                    help="Filter by field; "
+                    "e.g. \"user=uuid,usage>=10M,base_quota<inf\""),
+        make_option('--displayname',
+                    action='store_true',
+                    help="Show user display name"),
+    )
+
+    QHFLT = {
+        "total_quota": ("limit", filtering.parse_with_unit),
+        "usage": ("usage_max", filtering.parse_with_unit),
+        "user": ("holder", lambda x: x),
+        "resource": ("resource", lambda x: x),
+        "source": ("source", lambda x: x),
+        }
+
+    INITFLT = {
+        "base_quota": ("capacity", filtering.parse_with_unit),
+        }
+
+    @transaction.commit_on_success
     def handle(self, *args, **options):
-        output = json.dumps(resources, indent=4)
-        self.stdout.write(output + '\n')
+        output_format = options["output_format"]
+        displayname = bool(options["displayname"])
+        unit_style = options["unit_style"]
+        common.check_style(unit_style)
+
+        filteropt = options["filter_by"]
+        if filteropt is not None:
+            filters = filteropt.split(",")