Commit 5f8e87c0 authored by Giorgos Korfiatis's avatar Giorgos Korfiatis
Browse files

Merge branch 'release-0.15' into develop

Conflicts:
	snf-deploy/snfdeploy/__init__.py
	version
parents 3bc02f5c 11df26b7
......@@ -93,6 +93,9 @@ Astakos
* Automatically fill third party signup form fields when available by the
the third party provider.
* Omit validation issues for non-required metadata values received from the
third party authentication provider.
* Management commands:
* Introduced new commands:
......@@ -204,6 +207,8 @@ Cyclades
* Remove 'DEFAULT_ROUTING_TABLE' setting. If a link for an IP_LESS_ROUTED
network is not specified, the link will be uniquely named
'snf-link-$network_id'.
* Extend flavors with 'allow_create' attribute. Flavors that have this
attribute unset cannot be used by users to create new servers.
Cyclades UI
......@@ -240,6 +245,10 @@ Cyclades UI
- New setting ``UI_SSH_SUPPORT_OSFAMILY_EXCLUDE_LIST``. A list of image OS
families for which ui will disable ssh key injection in machine wizard.
- Setting ``UI_SUPPORT_SSH_OS_LIST`` removed and no longer used.
- Group public networks by name if setting ``UI_GROUP_PUBLIC_NETWORKS`` is set
to True.
- Setting ``UI_GROUPED_PUBLIC_NETWORK_NAME`` has been deprecated and no longer
used.
Cyclades Userdata
......
......@@ -521,6 +521,122 @@ Finally, backend systems having acquired a token can use the
:ref:`authenticate-api-label` API call from a private network or through HTTPS.
File/Object Storage Service (Pithos+)
====================================
Pithos+ is the Synnefo component that implements a storage service and exposes
the associated OpenStack REST APIs with custom extensions.
Pithos+ advanced operations
---------------------------
Enable separate domain for serving user content
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Since Synnefo v0.15, there is a possibility to serve untrusted user content
in an isolated domain.
Enabling this feature consists of the following steps:
#. **Declare new domain in apache server**
In order to enable the apache server to serve several domains it is required
to setup several virtual hosts.
Therefore, for adding the new domain e.g. "user-content.example.com", append
the following in ``/etc/apache2/sites-available/synnefo-ssl``:
.. code-block:: console
<VirtualHost _default_:443>
ServerName user-content.example.com
Alias /static "/usr/share/synnefo/static"
# SetEnv no-gzip
# SetEnv dont-vary
AllowEncodedSlashes On
RequestHeader set X-Forwarded-Protocol "https"
<Proxy * >
Order allow,deny
Allow from all
</Proxy>
SetEnv proxy-sendchunked
SSLProxyEngine off
ProxyErrorOverride off
ProxyPass /static !
ProxyPass / http://localhost:8080/ retry=0
ProxyPassReverse / http://localhost:8080/
RewriteEngine On
RewriteCond %{THE_REQUEST} ^.*(\\r|\\n|%0A|%0D).* [NC]
RewriteRule ^(.*)$ - [F,L]
SSLEngine on
SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
</VirtualHost>
.. note:: Consider also to purchase and install a certificate for the new
domain.
Finally, restart the apache server::
pithos-host$ /etc/init.d/apache2 restart
#. **Register Pithos+ as an OAuth2 client in Astakos**
Starting from synnefo version 0.15, in order to view the content of a
protected resource, Pithos+ (on behalf of the user) has to be granted
authorization for the specific resource by Astakos.
During the authorization grant procedure, Pithos+ has to authenticate
itself with Astakos since the latter has to prevent serving requests by
unknown/unauthorized clients.
Therefore, in the installation guide you were guided to register Pithos+
as an OAuth2 client in Astakos.
.. note:: You can see the registered clients by running::
astakos-host$ snf-manage oauth2-client-list -o identifier,redirect_urls,is_trusted
However, requests originated from the new domain will be rejected since
Astakos is ignorant about the new domain.
Therefore, you need to register a new client pointing to the unsafe domain.
To do so, use the following command::
astakos-host$ snf-manage oauth2-client-add pithos-unsafe-domain --secret=<secret> --is-trusted --url https://user-content.example.com/pithos/ui/view
.. note:: You can also unregister the client pointing to the safe domain,
since it will no longer be useful.
To do so, run the following::
astakos-host$ snf-manage oauth2-client-remove pithos-view
#. **Update Pithos+ configuration**
Respectively, the ``PITHOS_OAUTH2_CLIENT_CREDENTIALS`` setting should be
updated to contain the credentials of the client registered in the previous
step.
Furthermore, you need to restrict all the requests for user content
to be served exclusively by the unsafe domain.
To enable this, set the ``PITHOS_UNSAFE_DOMAIN`` setting to the value
of the new domain e.g. "user-content.example.com"
Finally, restart the gunicorn server::
pithos-host$ /etc/init.d/gunicorn restart
Compute/Network/Image Service (Cyclades)
========================================
......@@ -589,13 +705,24 @@ command will create two flavors of `40G` disk size with `drbd` disk template,
$ snf-manage flavor-create 2,4 4096 40 drbd
To see the available flavors, run `snf-manage flavor-list` command. Finally,
the administrator can delete a flavor by using `flavor-modify` command:
To see the available flavors, run `snf-manage flavor-list` command. The
administrator can delete a flavor by using `flavor-modify` command:
.. code-block:: console
$ snf-manage flavor-modify --deleted=True <flavor_id>
Finally, the administrator can set if new servers can be created from a flavor
or not, by setting the `allow_create` attribute:
.. code-block:: console
$ snf-manage flavor-modify --allow-create=False <flavor_id>
Flavors that are marked with `allow_create=False` cannot be used by users to
create new servers. However, they can still be used to resize existing VMs.
Images
~~~~~~
......@@ -1141,6 +1268,61 @@ The resources that are exported by Cyclades are the following:
* `cyclades.floating_ip`: Number of floating IP addresses
* `cyclades.network.private`: Number of private virtual networks
Enforcing quotas
~~~~~~~~~~~~~~~~
User quota can get overlimit, for example when a user is removed from a
project granting Cyclades resources. However, no action is automatically
taken to restrict users to their new limits. There is a special tool for
quota enforcement:
.. code-block:: console
# snf-manage enforce-resources-cyclades
This command will check and report which users are overlimit on their
Cyclades quota; it will also suggest actions to be taken in order to enforce
quota limits, dependent on the overlimit resource:
* `cyclades.vm`: Delete VMs
* `cyclades.total_cpu`: Delete VMs
* `cyclades.cpu`: Shutdown VMs
* `cyclades.total_ram`: Delete VMs
* `cyclades.ram`: Shutdown VMs
* `cyclades.disk`: Delete VMs
* `cyclades.floating_ip`: Detach and remove IPs
VMs to be deleted/shutdown are chosen first by state in the following order:
ERROR, BUILD, STOPPED, STARTED or RESIZE and then by decreasing ID. When
needing to remove IPs, we first choose IPs that are free, then those
attached to VMs, using the same VM ordering.
By default, the command checks only the following resources: `cyclades.cpu`,
`cyclades.ram`, and `cyclades.floating_ip`; that is, the less dangerous
ones, those that do not result in *deleting* any VM. One can change the
default behavior by specifying the desired resources with option
``--resources``. It is also possible to specify users to be checked or
excluded.
Actual enforcement is done with option ``--fix``. In order to control the
load that quota enforcement may cause on Cyclades, one can limit the number
of operations per backend. For example,
.. code-block:: console
# snf-manage enforce-resources-cyclades --fix --max-operations 10
will apply only the first 10 listed actions per backend. One can repeat the
operation, until nothing is left to be done.
To control load a timeout can also be set for shutting down VMs (using
option ``--shutdown-timeout <sec>``). This may be needed to avoid
expensive operations triggered by shutdown, such as Windows updates.
The command outputs the list of applied actions and reports whether each
action succeeded or not. Failure is reported if for any reason cyclades
failed to process the job and submit it to the backend.
Cyclades advanced operations
----------------------------
......
......@@ -20,6 +20,7 @@ exclude_patterns = ['_build']
pygments_style = 'sphinx'
html_theme = 'default'
html_theme_options = {
'sidebarwidth': '300',
'collapsiblesidebar': 'true',
'footerbgcolor': '#55b577',
'footertextcolor': '#000000',
......
......@@ -244,5 +244,5 @@ views that serve Pithos file contents will be restricted to be served only
under the domain specified in the setting value.
If an invalid host is identified and request HTTP method is one of ``GET``,
``HOST``, the server will redirect using a clone of the request with host
``HEAD``, the server will redirect using a clone of the request with host
replaced to the one the restriction applies to.
......@@ -132,18 +132,18 @@ Example Image response:
::
[{
"status": "available",
"name": "ubuntu",
"disk_format": "diskdump",
"container_format": "bare",
"id": "5583ffe1-5273-4c84-9e32-2fbe476bd7b7",
"status": "available",
"name": "ubuntu",
"disk_format": "diskdump",
"container_format": "bare",
"id": "5583ffe1-5273-4c84-9e32-2fbe476bd7b7",
"size": 2622562304
}, {
"status": "available",
"name": "Ubuntu-10.04",
"disk_format": "diskdump",
"container_format": "bare",
"id": "907ef618-c03a-4473-9914-9348e12890c1",
"status": "available",
"name": "Ubuntu-10.04",
"disk_format": "diskdump",
"container_format": "bare",
"id": "907ef618-c03a-4473-9914-9348e12890c1",
"size": 761368576
}]
......@@ -194,11 +194,11 @@ properties Custom properties ✔ ✔
Example Image response::
[{
"status": "available",
"status": "available",
"location": "pithos://u53r-1d/images/my/path/example_image_build.diskdump"
"name": "ubuntu",
"disk_format": "diskdump",
"container_format": "bare",
"name": "ubuntu",
"disk_format": "diskdump",
"container_format": "bare",
"created_at": "2013-03-29 14:14:34",
"deleted_at": "",
"id": "5583ffe1-5273-4c84-9e32-2fbe476bd7b7",
......@@ -206,22 +206,22 @@ Example Image response::
"is_public": "True",
"checksum": "a387aaaae583bc65daacf12d6be502bd7cfbbb254dcd452f92ca31f4c06a9208",
"properties": {
"partition_table": "msdos",
"kernel": "3.8.3",
"osfamily": "linux",
"users": "root user",
"gui": "GNOME 3.4.2",
"sortorder": "5",
"os": "fedora",
"root_partition": "1",
"partition_table": "msdos",
"kernel": "3.8.3",
"osfamily": "linux",
"users": "root user",
"gui": "GNOME 3.4.2",
"sortorder": "5",
"os": "fedora",
"root_partition": "1",
"description": "Fedora release 17 (Beefy Miracle)"}
}, {
"location": "pithos://0th3r-u53r-1d/images/ubuntu_10_04.diskdump"
"status": "available",
"name": "Ubuntu-10.04",
"disk_format": "diskdump",
"container_format": "bare",
"id": "907ef618-c03a-4473-9914-9348e12890c1",
"status": "available",
"name": "Ubuntu-10.04",
"disk_format": "diskdump",
"container_format": "bare",
"id": "907ef618-c03a-4473-9914-9348e12890c1",
"size": 761368576
"created_at": "2013-03-29 14:14:34",
"deleted_at": ""
......@@ -232,7 +232,7 @@ Add or update an image
According to the Synnefo approach, this request performs two operations:
* registers a new image to Cyclades/Image
* registers a new image to Cyclades/Image
* commits metadata for the new image
* update the metadata of an existing image
......@@ -272,7 +272,7 @@ X-Image-Meta-Is_public Make image public **✘** ✔
x-image-meta-Min-Ram Minimum ram required (MB) **✘** ✔
x-image-meta-Min-Disk Maximum ram required (MB) **✘** ✔
X-Image-Meta-Owner Image owner ✔ ✔
X-Image-Meta-Property-* Property prefix ✔ ✔
X-Image-Meta-Property-* Property prefix ✔ ✔
============================= ========================= ======== ========
**X-Meta-Location** format is described at :ref:`location-ref`
......@@ -285,7 +285,7 @@ X-Image-Meta-Property-* Property prefix ✔ ✔
**X-Image-Meta-Container-Format** values are listed at :ref:`container-format-ref`
**X-Image-Meta-Size** is optional, but should much the actual image file size.
**X-Image-Meta-Size** is optional, but should match the actual image file size.
**X-Image-Meta-Is-Public** values are true or false (case insensitive)
......@@ -300,12 +300,12 @@ X-Image-Meta-Property-* Property prefix ✔ ✔
Return Code Description
=========================== =====================
200 (OK) The request succeeded
400 (Bad Request)
400 (Bad Request)
\ No name header
\ Illegal header value
\ File not found on given location
\ Invalid size or checksum
401 (Unauthorized) Missing or expired user token
404 (Not Found) File not found on given location
500 (Internal Server Error) The request cannot be completed because of an internal error
501 (Not Implemented) Location header is empty or omitted
=========================== =====================
......@@ -356,16 +356,16 @@ The following refers only to the Cyclades/Image implementation.
|
============================= =========================
Request Header Name Value
Request Header Name Value
============================= =========================
X-Auth-Token User authentication token
X-Image-Meta-Name New image name
X-Image-Meta-Disk-Format New disk format
X-Image-Meta-Container-Format New container format
X-Image-Meta-Status New image status
X-Image-Meta-Is-Public (un)publish the image
X-Image-Meta-Owner Set an owner
X-Image-Meta-Property-* Add / modify properties
X-Image-Meta-Name New image name
X-Image-Meta-Disk-Format New disk format
X-Image-Meta-Container-Format New container format
X-Image-Meta-Status New image status
X-Image-Meta-Is-Public (un)publish the image
X-Image-Meta-Owner Set an owner
X-Image-Meta-Property-* Add / modify properties
============================= =========================
**X-Image-Meta-Disk-Format** values are listed at :ref:`disk-format-ref`
......@@ -377,7 +377,7 @@ X-Image-Meta-Property-* Add / modify properties
**X-Image-Meta-Is-Public** values are true or false (case insensitive)
**X-Image-Meta-Property-*** is used as a prefix to update image property
values, or set some extra proeperties. If a registered image already contains
values, or set some extra properties. If a registered image already contains
some custom properties that are not addressed in the update request, these
properties will remain untouched. For example::
......@@ -390,7 +390,7 @@ properties will remain untouched. For example::
Return Code Description
=========================== =====================
200 (OK) The request succeeded
400 (Bad Request)
400 (Bad Request)
\ Illegal header value
\ Invalid size or checksum
401 (Unauthorized) Missing or expired user token
......@@ -404,20 +404,20 @@ Return Code Description
The following is received when the response code is 200:
============================= =====================
Response Header Description
Response Header Description
============================= =====================
X-Image-Meta-Id Unique img id
X-Image-Meta-Name Img name
X-Image-Meta-Disk-Format Disk format
X-Image-Meta-Container-Format Container format
X-Image-Meta-Size Img file size
X-Image-Meta-Id Unique img id
X-Image-Meta-Name Img name
X-Image-Meta-Disk-Format Disk format
X-Image-Meta-Container-Format Container format
X-Image-Meta-Size Img file size
X-Image-Meta-Checksum Img file MD5 checksum
X-Image-Meta-Location Pithos file location
X-Image-Meta-Created-At Date of img creation
X-Image-Meta-Deleted-At Date of img deletion
X-Image-Meta-Status Img status
X-Image-Meta-Created-At Date of img creation
X-Image-Meta-Deleted-At Date of img deletion
X-Image-Meta-Status Img status
X-Image-Meta-Is-Public True if img is public
X-Image-Meta-Owner Img owner or tentant
X-Image-Meta-Owner Img owner or tentant
X-Image-Meta-Property-* Custom img properties
============================= =====================
......@@ -522,7 +522,7 @@ Example Cyclades/Image Headers response::
x-image-meta-checksum: d0f28e4d72927c90eadf30917d94d0156781fe1351ed16402b538316d404
x-image-meta-created-at: 2013-02-26 12:04:31
x-image-meta-updated-at: 2013-02-26 12:05:28
x-image-meta-deleted-at:
x-image-meta-deleted-at:
x-image-meta-status: available
x-image-meta-is-public: True
x-image-meta-owner: 25cced7-bd53-4145-91ee-cf4737e9fb2
......
......@@ -140,7 +140,7 @@ create all needed databases on node1 and then node2 will connect to them.
postgres=# GRANT ALL PRIVILEGES ON DATABASE snf_pithos TO synnefo;
Configure the database to listen to all network interfaces. You can do this by
editting the file ``/etc/postgresql/9.1/main/postgresql.conf`` and change
editing the file ``/etc/postgresql/9.1/main/postgresql.conf`` and change
``listen_addresses`` to ``'*'`` :
.. code-block:: console
......@@ -2021,7 +2021,7 @@ Add a vncauthproxy user:
# /etc/init.d/vncauthproxy restart
Configure the vncauthproxy settings in
``/etc/synnefo/20/snf-cyclades-app-api.conf``:
``/etc/synnefo/20-snf-cyclades-app-api.conf``:
.. code-block:: console
......@@ -2129,7 +2129,7 @@ created NIC on a bridge.
$ snf-manage network-create --subnet=10.0.0.0/24 \
--gateway=10.0.0.1 \
--public --dhcp --flavor=CUSTOM \
--public --dhcp=True --flavor=CUSTOM \
--link=br1 --mode=bridged \
--name=public_network \
--backend-id=1
......
......@@ -46,43 +46,41 @@ def get_public_stats():
def get_astakos_stats():
stats = {"datetime": datetime.datetime.now().strftime("%c")}
resources = Resource.objects.values_list("name", flat=True)
stats = {"datetime": datetime.datetime.now().strftime("%c"),
"providers": [],
"users": {},
"resources": {}}
users = AstakosUser.objects.all()
verified = users.filter(email_verified=True)
active = users.filter(is_active=True)
user_stats = {}
user_stats["total"] = {"total": users.count(),
"verified": verified.count(),
"active": active.count(),
"usage": {}}
for resource in resources:
usage = Holding.objects.filter(resource=resource)\
.aggregate(summ=Sum("usage_max"))
user_stats["total"]["usage"][resource] = int(usage["summ"])
for provider in settings.ASTAKOS_IM_MODULES:
# Add provider
stats["providers"].append(provider)
# Add stats about users
users = AstakosUser.objects.filter(auth_providers__module=provider)
verified = users.filter(email_verified=True)
active = users.filter(is_active=True)
user_stats[provider] = {"total": users.count(),
"verified": verified.count(),
"active": active.count(),
"usage": {}}
stats["users"][provider] = {"total": users.count(),
"verified": verified.count(),
"active": active.count()}
# Add stats about resources
users_uuids = users.values_list("uuid", flat=True)
for resource in resources:
usage = Holding.objects\
.filter(holder__in=users_uuids, resource=resource)\
.aggregate(summ=Sum("usage_max"))
user_stats[provider]["usage"][resource] = int(usage["summ"])
stats["users"] = user_stats
resources_stats = {}
for resource in Resource.objects.all():
info = Holding.objects\
.filter(holder__in=users_uuids,
resource=resource.name)\
.aggregate(usage_sum=Sum("usage_max"),
limit_sum=Sum("limit"))
resources_stats[resource.name] = {"used": info["usage_sum"],
"limit": info["limit_sum"],
"unit": resource.unit,
"description": resource.desc}
stats["resources"][provider] = resources_stats
return stats
......@@ -36,12 +36,19 @@ from django import http
from django.utils import simplejson as json
from snf_django.lib import api
from astakos.im import settings
from synnefo.lib.services import get_path
from astakos.admin import stats
logger = logging.getLogger(__name__)
PERMITTED_GROUPS = settings.ADMIN_STATS_PERMITTED_GROUPS
try:
AUTH_URL = get_path(settings.astakos_services,
"astakos_identity.endpoints")[0]["publicURL"]
except (IndexError, KeyError) as e:
logger.error("Failed to load Astakos Auth URL: %s", e)
AUTH_URL = None
@api.api_method(http_method='GET', user_required=False, token_required=False,
......@@ -54,6 +61,7 @@ def get_public_stats(request):
@api.api_method(http_method='GET', user_required=True, token_required=True,
astakos_auth_url=AUTH_URL,
logger=logger, serializations=['json'])
@api.user_in_groups(permitted_groups=PERMITTED_GROUPS,
logger=logger)
......
......@@ -111,6 +111,6 @@ class Command(BaseCommand):
profile.save()
if update:
print "Profile updated"
self.stderr.write("Profile updated\n")
else:
print "Profile stored"
self.stderr.write("Profile stored\n")
......@@ -44,10 +44,10 @@ class Command(NoArgsCommand):
help = "Cleanup sessions and session catalog"
def handle_noargs(self, **options):
self.stdout.write('Cleanup sessions ...\n')
self.stderr.write('Cleanup sessions ...\n')
call_command('cleanup')
self.stdout.write('Cleanup session catalog ...\n')
self.stderr.write('Cleanup session catalog ...\n')
engine = import_module(settings.SESSION_ENGINE)
store = engine.SessionStore()
tbd = (entry for entry in SessionCatalog.objects.all()
......
......@@ -49,8 +49,8 @@ class Command(NoArgsCommand):
fixed = fix_superusers()
count = len(fixed)
if count != 0:
self.stdout.write("Fixed %s superuser(s).\n" % count)
self.stderr.write("Fixed %s superuser(s).\n" % count)
else:
self.stdout.write("No superuser needed a fix.\n")
self.stderr.write("No superuser needed a fix.\n")
except BaseException, e:
raise CommandError(e)
......@@ -47,4 +47,4 @@ class Command(BaseCommand):
raise CommandError("Invalid number of arguments")
Group.objects.get_or_create(name=args[0].strip())
print "Group created"
self.stderr.write("Group created\n")
......@@ -124,23 +124,23 @@ class Command(BaseCommand):
s = '1 expired project:\n'
else: