Commit 131a5944 authored by Ilias Tsitsimpis's avatar Ilias Tsitsimpis
Browse files

Merge branch 'release-0.15' into develop

Conflicts:
	ci/ci_wheezy.conf
	ci/utils.py
	snf-cyclades-app/synnefo/db/migrations/0077_migrate_networks_to_subnets.py
	version
parents 68995c0a b0c05399
......@@ -87,29 +87,30 @@ Astakos
* Add API call for validating OAuth 2.0 access tokens
* **Shibboleth module** Extract unique identifier from the
* **Shibboleth module** Extract unique identifier from the
``REMOTE_USER`` header.
* Automatically fill third party signup form fields when available by the
* Automatically fill third party signup form fields when available by the
the third party provider.
* Management commands:
* Introduced new commands:
* component-show
* quota-list (replacing quota, supports various filters)
* quota-verify (replacing quota)
* oauth2-client-add (register OAuth 2.0 client)
* oauth2-client-list (list registered oauth 2.0 clients)
* oauth2-client-remove (remove OAuth 2.0 client)
* component-show
* quota-list (replacing quota, supports various filters)
* quota-verify (replacing quota)
* oauth2-client-add (register OAuth 2.0 client)
* oauth2-client-list (list registered oauth 2.0 clients)
* oauth2-client-remove (remove OAuth 2.0 client)
* Changed commands:
* component-add got options --base-url and --ui-url
* resource-modify --limit became --default-quota
* user-modify can operate on multiple users with --all and --exclude
* user-modify --set-base-quota became --base-quota
* component-add got options --base-url and --ui-url
* resource-modify --limit became --default-quota
* user-modify can operate on multiple users with --all and --exclude
* user-modify --set-base-quota became --base-quota
* Removed commands:
* quota
* resource-import (subsumed by service-import)
* resource-export-astakos (subsumed by service-export-astakos)
* quota
* resource-import (subsumed by service-import)
* resource-export-astakos (subsumed by service-export-astakos)
Cyclades
--------
......@@ -117,30 +118,30 @@ Cyclades
* Major changes to Cyclades networks:
* Implement 'cyclades_network' service, containing the /networks, /ports,
/subnets and /floatingips API endpoints under '/network/v2.0'.
The old /networks API of 'cyclades_compute' (under /compute/v2.0) is
removed.
/subnets and /floatingips API endpoints under '/network/v2.0'. The old
/networks API of 'cyclades_compute' (under /compute/v2.0) is
removed.
* Implement `snf-manage subnet-{create, list, modify, inspect}' management
commands for handling of subnets.
* Implement `snf-manage port-{create, list, remove, inspect}' management
commands for handling of ports.
* Add two new settings, 'CYCLADES_FORCED_SERVER_NETWORKS' and
'CYCLADES_DEFAULT_SERVER_NETWORKS' to control the networks that newly
created servers will be connected.
'CYCLADES_DEFAULT_SERVER_NETWORKS' to control the networks that newly
created servers will be connected.
* Implement Floating IP addresses, which are IPv4 addresses that can be
dynamically added and removed to a running server.
* Add new 'cyclades.floating_ip' resource.
* Implement 'snf-manage floating-ip-{create,list,remove,attach,detach}'
management commands to handle floating IPs.
management commands to handle floating IPs.
* Add 'floating_ip_pool' attribute to networks to mark networks that can
be used as floating IP pools.
* Implement 'resize' server action.
* Implement the 'resize' server action, to change the flavor of a server.
Only 'cpu' and 'memory' resizing is supported.
Only 'cpu' and 'memory' resizing is supported.
* Compute quotas for CPU and memory of running vms.
......@@ -152,7 +153,7 @@ Cyclades
* Make cyclades give a unique name to each Ganeti NIC. NICs are refered by
their unique name and not by their index inside the VM that are connected
to.
to.
* Support firewall profile for all NICs of an instance. Change firewall
settings to be filled with the unique name of the NIC. The affected settings
......@@ -212,80 +213,65 @@ Cyclades UI
service.
- New IPs pane from which user can manage floating IPs.
- Redesign public keys overlay as an additional pane view.
- Split networking configuration into an additional step in machine create
- Split networking configuration into an additional step in machine create
wizard.
- Display forced networks and choices of the available floating IPs which will
- Display forced networks and choices of the available floating IPs which will
be assigned to the created machine.
- Support for machine resize action. Explicit handling when machine is started
by displaying an utility shutdown button within the resize overlay.
- Machine IPs toggling subview in icon/single views.
- Replace IPv4/IPv6 with machine's FQDN in icon/single view. When no FQDN can
be resolved display a message. Message can be configured using the introduced
``UI_NO_FQDN_MESSAGE``. Setting ``UI_VM_HOSTNAME_FORMAT`` has been removed
- Replace IPv4/IPv6 with machine's FQDN in icon/single view. When no FQDN can
be resolved display a message. Message can be configured using the introduced
``UI_NO_FQDN_MESSAGE``. Setting ``UI_VM_HOSTNAME_FORMAT`` has been removed
and no longer used.
- Respect ``SNF:task_state`` machine attribute in order to improve machine
status display.
- Append software version as a url parameter in HTML static files in order
- Append software version as a url parameter in HTML static files in order
to force browser cache invalidation between versions.
- Configurable Google fonts base url. Fonts base url can be changed usint the
- Configurable Google fonts base url. Fonts base url can be changed usint the
``SYNNEFO_FONTS_BASE_URL`` setting.
- Regression fix: Display reboot required notification on machine firewall
parameters.
- Handling of ``GANETI_USE_HOTPLUG`` setting. Do not allow live network actions
- Handling of ``GANETI_USE_HOTPLUG`` setting. Do not allow live network actions
when setting is set to ``False``.
- Double escaping fix in machine create wizard images list and machine details
- Double escaping fix in machine create wizard images list and machine details
subview.
- Fix image ordering in machine create wizard.
- New setting ``UI_SSH_SUPPORT_OSFAMILY_EXCLUDE_LIST``. A list of image OS
families for which ui will disable ssh key injection in machine wizard.
- Setting ``UI_SUPPORT_SSH_OS_LIST`` removed and no longer used.
Pithos
------
* Rewrite tests.
* Performance optimizations in object listing.
* Introduce backend method decorator for handling transaction management if no
transaction is initiated from the frontend.
* Fix Internal Server Errors https://code.grnet.gr/issues/4501 &
https://code.grnet.gr/issues/4502.
* Fix REQUEST ENTITY TOO LARGE request failure during move operations
https://code.grnet.gr/issues/4154.
* Fix FORBIDDEN request failure while listing implicitly shared objects
https://code.grnet.gr/issues/4131.
* Fix issue with the computed size of an updated object.
* Reply with the Merkle hash in the ETag header if MD5 is deactivated.
* Reply with FORBIDDEN (403) to public listing requests performed by non path
owners.
* Change response status to NOT FOUND (404) while trying to delete an
already deleted object.
* Change SQLAlchemy version to 0.7
* Change view authorization
The pithos views do not use the cookie information for user authentication.
They request (from Astakos) and use a short-term access token for a
specific resource.
* Remove PITHOS_ASTAKOS_COOKIE_NAME setting, since it is no longer useful
* Add PITHOS_OAUTH2_CLIENT_CREDENTIALS setting to authenticate the views with
astakos during the resource access token generation procedure
* Add PITHOS_UNSAFE_DOMAIN setting to restrict file serving endpoints to a
specific host
* Management commands:
* Introduced new command:
* file-show
* Added new 'file-show' management command
* Remove command 'resource-export-pithos' subsumed by 'service-export-pithos'.
.. _Changelog-0.14.10:
......@@ -474,7 +460,7 @@ Synnefo-wide
a user-configurable <COMPONENT>_BASE_URL. Each API (compute, image, etc.)
is deployable under a developer-configurable prefix beneath BASE_URL.
* Deprecate CLOUDBAR_ACTIVE_SERVICE setting from all apps.
* Deprecate CLOUDBAR_ACTIVE_SERVICE setting from all apps.
* Common synnefo 404/500 templates (located in snf-webproject)
......@@ -536,16 +522,16 @@ Astakos
* Added fine grain user auth provider's policies.
* Administrator can override default auth provider policies to a specific
* Administrator can override default auth provider policies to a specific
user or group of users.
* Optionally a user can be assigned to a list of groups, based on the
authentication method he choosed to signup.
* Removed explicit handling of SMTP errors on each email delivery. Exceptions
* Removed explicit handling of SMTP errors on each email delivery. Exceptions
are now propagated to base django exception handler.
* Email used in html/email tempaltes which prompt user to contact for service
support prompts is now defined in ``CONTACT_EMAIL`` setting introduced in
* Email used in html/email tempaltes which prompt user to contact for service
support prompts is now defined in ``CONTACT_EMAIL`` setting introduced in
snf-common settings.
* Improvements in user activation flow
......@@ -553,7 +539,7 @@ Astakos
* User moderation now takes place after the user has verified his email
address.
* User model enriched with additional user state fields
* Split activation email from moderation process. Administrator is required
* Split activation email from moderation process. Administrator is required
to moderate user explicitly using the `user-modify --accept` or
`user-modify --reject` commands.
* Improved logging throught out user activation procedures.
......@@ -561,26 +547,26 @@ Astakos
* Remove deprecated AstakosUser model fields: `provider`,
`third_party_identifier`
* Allow override of authentication provider messages using the following
* Allow override of authentication provider messages using the following
format in setting names: ``ASTAKOS_<PROVIDER_MODULE>_<MSGID>_MSG``
* Cloudbar automatically tries to identify the active service based on window
* Cloudbar automatically tries to identify the active service based on window
location.
* Removing authentication provider view is now CSRF protected.
* New `API access` view, containing useful information to users on how to
* New `API access` view, containing useful information to users on how to
access available Synnefo services API's.
* Remove of ASTAKOS_*_EMAIL_SUBJECT settings. All email subjects are now
defined in astakos.im.messages module. Overriding default values can be
* Remove of ASTAKOS_*_EMAIL_SUBJECT settings. All email subjects are now
defined in astakos.im.messages module. Overriding default values can be
achieved using custom gettext files or using astakos messages settings::
#change of greeting email subject
ASTAKOS_GREETING_EMAIL_SUBJECT_MESSAGE = 'Welcome to my cloud'
* Remove ``ASTAKOS_ACTIVATION_REDIRECT_URL`` and ``ASTAKOS_LOGIN_SUCCESS_URL``
from astakos .conf file. Settings are dynamically computed based on
* Remove ``ASTAKOS_ACTIVATION_REDIRECT_URL`` and ``ASTAKOS_LOGIN_SUCCESS_URL``
from astakos .conf file. Settings are dynamically computed based on
``ASTAKOS_BASE_URL``.
* Management commands:
......@@ -721,10 +707,10 @@ Astakos
* refactored/improved /login endpoint used by desktop/mobile clients.
* endpoint url is now exposed by `weblogin` service
* clients should use unauthenticated identity/tokens api to resolve the
* clients should use unauthenticated identity/tokens api to resolve the
endpoint url
* view only allows redirects to `pithos://` scheme urls
* removed uuid from redirect parameters. Client should use authenticated
* removed uuid from redirect parameters. Client should use authenticated
request to identity/tokens to retrieve user uuid.
......@@ -749,14 +735,14 @@ Cyclades
UI
^^
* Removed feedback endpoint. Feedback requests delegate to astakos feedback
service. ``FEEDBACK_CONTACTS``, ``FEEDBACK_EMAIL_FROM`` settings removed,
service. ``FEEDBACK_CONTACTS``, ``FEEDBACK_EMAIL_FROM`` settings removed,
and no longer used.
* ``UI_LOGIN_URL``, ``UI_GLANCE_URL``, ``COMPUTE_URL`` settings no longer
* ``UI_LOGIN_URL``, ``UI_GLANCE_URL``, ``COMPUTE_URL`` settings no longer
required to be set and are dynamically computed based on ``ASTAKOS_BASE_URL``
and ``CYCLADES_BASE_URL`` settings.
* File group is no longer included in ssh keys personality metadata sent in
* File group is no longer included in ssh keys personality metadata sent in
create vm calls.
Pithos
------
......@@ -766,7 +752,7 @@ Pithos
* new settings:
PITHOS_RADOS_STORAGE, PITHOS_RADOS_POOL_BLOCKS, PITHOS_RADOS_POOL_MAPS
* X-Object-Public now contains full url (domain + proper component prefix +
* X-Object-Public now contains full url (domain + proper component prefix +
file path)
* Rewritten support for public URLs, with admin-selectable length
......
......@@ -11,6 +11,7 @@ v0.14next
Released: UNRELEASED
* Support for snf-vncauthproxy-1.5 (drop support for older versions)
* Until release of v0.15, for complete list of changes see the Changelog.
.. _NEWS-0.14.10:
......
......@@ -128,7 +128,16 @@ class AstakosClient(object):
self.auth_prefix = parsed_auth_url.path
self.api_tokens = join_urls(self.auth_prefix, "tokens")
def _fill_endpoints(self, endpoints):
def _fill_endpoints(self, endpoints, extra=False):
"""Fill the endpoints for our AstakosClient
This will be done once (lazily) and the endpoints will be there
to be used afterwards.
The `extra' parameter is there for compatibility reasons. We are going
to fill the oauth2 endpoint only if we need it. This way we are keeping
astakosclient compatible with older Astakos version.
"""
astakos_service_catalog = parse_endpoints(
endpoints, ep_name="astakos_account", ep_version_id="v1.0")
self._account_url = \
......@@ -145,21 +154,22 @@ class AstakosClient(object):
self._ui_prefix = parsed_ui_url.path
self.logger.debug("Got ui_prefix \"%s\"" % self._ui_prefix)
oauth2_service_catalog = parse_endpoints(endpoints,
ep_name="astakos_oauth2")
self._oauth2_url = \
oauth2_service_catalog[0]['endpoints'][0]['publicURL']
parsed_oauth2_url = urlparse.urlparse(self._oauth2_url)
self._oauth2_prefix = parsed_oauth2_url.path
if extra:
oauth2_service_catalog = \
parse_endpoints(endpoints, ep_name="astakos_oauth2")
self._oauth2_url = \
oauth2_service_catalog[0]['endpoints'][0]['publicURL']
parsed_oauth2_url = urlparse.urlparse(self._oauth2_url)
self._oauth2_prefix = parsed_oauth2_url.path
def _get_value(self, s):
def _get_value(self, s, extra=False):
assert s in ['_account_url', '_account_prefix',
'_ui_url', '_ui_prefix',
'_oauth2_url', '_oauth2_prefix']
try:
return getattr(self, s)
except AttributeError:
self.get_endpoints()
self.get_endpoints(extra=extra)
return getattr(self, s)
@property
......@@ -180,11 +190,11 @@ class AstakosClient(object):
@property
def oauth2_url(self):
return self._get_value('_oauth2_url')
return self._get_value('_oauth2_url', extra=True)
@property
def oauth2_prefix(self):
return self._get_value('_oauth2_prefix')
return self._get_value('_oauth2_prefix', extra=True)
@property
def api_usercatalogs(self):
......@@ -456,9 +466,10 @@ class AstakosClient(object):
# -----------------------------------------
# do a POST to ``API_TOKENS`` with no token
def get_endpoints(self):
def get_endpoints(self, extra=False):
""" Get services' endpoints
The extra parameter is to be used by _fill_endpoints.
In case of error raise an AstakosClientException.
"""
......@@ -467,7 +478,7 @@ class AstakosClient(object):
r = self._call_astakos(self.api_tokens, headers=req_headers,
body=req_body, method="POST",
log_body=False)
self._fill_endpoints(r)
self._fill_endpoints(r, extra=extra)
return r
# --------------------------------------
......@@ -500,7 +511,7 @@ class AstakosClient(object):
# --------------------------------------
# do a GET to ``API_TOKENS`` with a token
def validate_token(self, token_id, belongsTo=None):
def validate_token(self, token_id, belongs_to=None):
""" Validate a temporary access token (oath2)
Keyword arguments:
......@@ -509,15 +520,15 @@ class AstakosClient(object):
It returns back the token as well as information about the token
holder.
The belongsTo is optional and if it is given it must be inside the
The belongs_to is optional and if it is given it must be inside the
token's scope.
In case of error raise an AstakosClientException.
"""
path = join_urls(self.api_tokens, str(token_id))
if belongsTo is not None:
params = {'belongsTo': belongsTo}
if belongs_to is not None:
params = {'belongsTo': belongs_to}
path = '%s?%s' % (path, urllib.urlencode(params))
return self._call_astakos(path, method="GET", log_body=False)
......@@ -985,7 +996,7 @@ def parse_endpoints(endpoints, ep_name=None, ep_type=None,
else:
return catalog
except KeyError:
raise NoEndpoints()
raise NoEndpoints(ep_name, ep_type, ep_region, ep_version_id)
# --------------------------------------------------------------------
......
......@@ -61,10 +61,6 @@ ssh_keys = ~/.ssh/id_rsa.pub
# Maybe add some burnin options
# (e.g. tests to run/ignore, timeouts etc)
cmd_options = --images "name:.*" --flavors "name:C1R512D2file" --no-ipv6
# Kamaki version to be used (leave empty for default)
# In some cases there is the need for a specific version
# of kamaki to be used.
kamaki_version = 0.11next-1858-1d7368b
[Unit Tests]
......
......@@ -55,16 +55,16 @@ images = name:SynnefoCIWheezy.*, name:^Debian Base$, id:72d9844f-1024-4a07-a3c3-
# File containing the ssh keys to upload/install to server
# If not set, no ssh keys will be installed
ssh_keys = ~/.ssh/id_rsa.pub
# Allocate a public floating IP for the new server
allocate_floating_ip = True
# List of networks IDs (comma seperated) to connect server
private_networks =
[Burnin]
# Maybe add some burnin options
# (e.g. tests to run/ignore, timeouts etc)
cmd_options = --images "name:.*" --flavors "name:C1R512D2file" --no-ipv6
# Kamaki version to be used (leave empty for default)
# In some cases there is the need for a specific version
# of kamaki to be used.
kamaki_version = 0.12rc2
[Unit Tests]
......
......@@ -7,6 +7,7 @@ user_email = user@synnefo.org
user_name = John
user_lastname = Doe
user_passwd = 12345
oa2_secret = 12345
[roles]
......
......@@ -299,9 +299,15 @@ class SynnefoCI(object):
flavor_id = self._find_flavor(flavor)
# Create Server
fip = self._create_floating_ip()
port = self._create_port(fip)
networks = [{'port': port['id']}]
networks = []
if self.config.get("Deployment", "allocate_floating_ip") == "True":
fip = self._create_floating_ip()
port = self._create_port(fip)
networks.append({'port': port['id']})
private_networks = self.config.get('Deployment', 'private_networks')
if private_networks:
private_networks = [p.strip() for p in private_networks.split(",")]
networks.extend([{"uuid": uuid} for uuid in private_networks])
if server_name is None:
server_name = self.config.get("Deployment", "server_name")
server_name = "%s(BID: %s)" % (server_name, self.build_id)
......@@ -317,7 +323,7 @@ class SynnefoCI(object):
self.write_temp_config('server_passwd', server_passwd)
server = self._wait_transition(server_id, "BUILD", "ACTIVE")
self._get_server_ip_and_port(server)
self._get_server_ip_and_port(server, private_networks)
self._copy_ssh_keys(ssh_keys)
# Setup Firewall
......@@ -461,10 +467,23 @@ class SynnefoCI(object):
self.logger.error("No matching image found.. aborting")
sys.exit(1)
def _get_server_ip_and_port(self, server):
def _get_server_ip_and_port(self, server, private_networks):
"""Compute server's IPv4 and ssh port number"""
self.logger.info("Get server connection details..")
server_ip = server['attachments'][0]['ipv4']
if private_networks:
# Choose the networks that belong to private_networks
networks = [n for n in server['attachments']
if n['network_id'] in private_networks]
else:
# Choose the networks that are public
networks = [n for n in server['attachments']
if self.network_client.
get_network_details(n['network_id'])['public']]
# Choose the networks with IPv4
networks = [n for n in networks if n['ipv4']]
# Use the first network as IPv4
server_ip = networks[0]['ipv4']
if (".okeanos.io" in self.cyclades_client.base_url or
".demo.synnefo.org" in self.cyclades_client.base_url):
tmp1 = int(server_ip.split(".")[2])
......@@ -767,13 +786,12 @@ class SynnefoCI(object):
def build_packages(self):
"""Build packages needed by Synnefo software"""
self.logger.info("Install development packages")
kamaki_version = self.config.get('Burnin', 'kamaki_version')
cmd = """
apt-get update
apt-get install zlib1g-dev dpkg-dev debhelper git-buildpackage \
python-dev python-all python-pip ant --yes --force-yes
pip install -U devflow kamaki{0}
""".format(("==" + kamaki_version) if kamaki_version else "")
pip install -U devflow
"""
_run(cmd, False)
# Patch pydist bug
......
......@@ -642,8 +642,8 @@ image to Cyclades:
.. code-block:: console
$ kamaki image register "Debian Base" \
pithos://u53r-un1qu3-1d/images/debian_base-6.0-7-x86_64.diskdump \
$ kamaki image register --name="Debian Base" \
--location=pithos://u53r-un1qu3-1d/images/debian_base-6.0-7-x86_64.diskdump \
--public \
--disk-format=diskdump \
--property OSFAMILY=linux --property ROOT_PARTITION=1 \
......@@ -1660,10 +1660,8 @@ To change a setting use ``kamaki config set``:
.. code-block:: console
$ kamaki config set image.url https://cyclades.example.com/image
$ kamaki config set file.url https://pithos.example.com/v1
$ kamaki config set user.url https://accounts.example.com
$ kamaki config set token ...
$ kamaki config set cloud.default.url https://example.com/identity/v2.0
$ kamaki config set cloud.default.token ...
To test that everything works, try authenticating the current account with
kamaki:
......@@ -1682,27 +1680,27 @@ container exists, by listing all containers in your account:
.. code-block:: console
$ kamaki file list
$ kamaki file list /images
If the container ``images`` does not exist, create it:
.. code-block:: console
$ kamaki file create images
$ kamaki container create images
You are now ready to upload an image to container ``images``. You can upload it
with a Pithos client, or use kamaki directly:
.. code-block:: console
$ kamaki file upload ubuntu.iso images
$ kamaki file upload ubuntu.iso /images
You can use any Pithos client to verify that the image was uploaded correctly,
or you can list the contents of the container with kamaki:
.. code-block:: console
$ kamaki file list images
$ kamaki file list /images
The full Pithos URL for the previous example will be
``pithos://u53r-un1qu3-1d/images/ubuntu.iso`` where ``u53r-un1qu3-1d`` is the
......@@ -1711,12 +1709,12 @@ unique user id (uuid).
Register Image
--------------
To register an image you will need to use the full Pithos URL. To register as
a public image the one from the previous example use:
To register an image you will need to use the full or the relative Pithos URL.
To register as a public image the one from the previous example use:
.. code-block:: console
$ kamaki image register Ubuntu pithos://u53r-un1qu3-1d/images/ubuntu.iso --public
$ kamaki image register --name=Ubuntu --location=/images/ubuntu.iso --public
The ``--public`` flag is important, if missing the registered image will not
be listed by ``kamaki image list``.
......@@ -1726,14 +1724,14 @@ options. A more complete example would be the following:
.. code-block:: console
$ kamaki image register Ubuntu pithos://u53r-un1qu3-1d/images/ubuntu.iso \
$ kamaki image register --name Ubuntu --location /images/ubuntu.iso \
--public --disk-format diskdump --property kernel=3.1.2
To verify that the image was registered successfully use:
.. code-block:: console
$ kamaki image list --name-like=ubuntu
$ kamaki image list --name-like ubuntu
Miscellaneous
......@@ -1891,7 +1889,7 @@ feasible.
The output of all email `*`.txt files will be already customized to contain your
company and service names but you can further alter their content if you feel it
best fits your needs as simple as creasynnefo template.
best fits your needs.
In order to overwrite one or more email-templates you need to place your
modified <email-file>.txt files respecting the following structure:
......@@ -1977,6 +1975,71 @@ description and a link to their content:
such as the activation link for activating one’s account and many more.
These variables are contained into {{}} inside the templates.
**Astakos landing page**