Commit e3e8c90f authored by Stratos Psomadakis's avatar Stratos Psomadakis
Browse files

Merge branch 'hotfix-0.16.2'

parents 3c49605b c696fef1
Dimitris Aragiorgis <dimitris.aragiorgis@gmail.com>
......@@ -6,6 +6,101 @@ Unified Changelog file for Synnefo versions >= 0.13
Since v0.13 most of the Synnefo components have been merged into a single
repository and have aligned versions.
.. _Changelog-0.16.2:
v0.16.2
=======
Released: Fri Jul 24 14:13:24 EEST 2015
Changelog
---------
* Switch to a more verbose Changelog style, which includes more details
regarding the commits of each relase. A summary of the changes is provided
with the NEWS file.
Admin
-----
* Handle unicode chars in VM search field.
* Check usage to decide if resource is useful.
* Display last login per auth provider if more than one exists.
* Display `comments for review' in project applications.
* Display effective limit in user's quotas.
* Fix IP updated time since timestamp.
* Fetch image info using UUID and version.
Astakos
-------
* Always accept terms for users created via admin API.
* Allow admin to update the 'signed terms' date for a user via the 'signTerms'
admin API action.
* Improve admin API user creation process, in case that user moderation is
disabled.
* Provide South migration to fix initialized state of enrolled memberships.
* Handle empty project limit in application submission form.
* Include terminated projects in project list views.
* Display UUID in expired projects list.
* Fix checkbox behaviour in project members list.
* Fix style error in project creation form.
* Add project_creation_date column to list.
* Fix UI project tests.
* Route oa2 models to Astakos database in Synnefo DB router.
* Catch a couple of potential 500 errors.
* Fix auth_provider limit enforcement.
Bunrin
------
* Fix a race in public network detach.
* Handle exceptions and retry during SSH.
* Create private networks without a gateway.
CI
--
* Pin Debian Wheezy image during ci runs.
* Fix CI image flavor regexp.
Cyclades
--------
* Round up memory sizes in flavor create.
* Change volume ownership on server-modify.
Cyclades UI
-----------
* Add UI_IMAGE_LISTING_USERS setting to enable custom defined image sections
in image selection view of the VM creation wizard. The sections are defined
using the LISTING_SECTION image property.
* Change default private network gateway to 'None'.
* Include machine icon images for bitnami.
Deploy
------
* Substitute IPs with FQDNs.
* Fix CA creation.
* Use domain in ServerName.
* Update qa-init.sh for Ganeti.
* Disable v6 on loopback interface.
Documentation
-------------
* Update install guide to use Debian Wheezy.
Pithos
------
* Fix off-by-one error in Pithos API tests.
.. _Changelog-0.16.1:
v0.16.1
......
......@@ -5,6 +5,29 @@ Unified NEWS file for Synnefo versions >= 0.13
Since v0.13 all Synnefo components have been merged into a single repository.
.. _NEWS-0.16.2:
v0.16.2
=======
Released: Fri Jul 24 14:13:49 EEST 2015
The Synnefo 0.16.2 release is mostly a bug fix version. The most notable
changes are:
* Added custom image sections in the Cyclades UI.
* Round up memory sizes in flavor create.
* Change volume ownership, when reassigning VMs.
* Improved handling of projects (both API- and UI-wise).
* Provide South migration to fix initialized state of enrolled project
memberships.
* Add a `signTerms` admin API action.
* Include machine icon images for bitnami.
* Usability fixes for the admin app.
* Updates and fixes for snf-deploy.
* Various bug fixes and improvements across Synnefo. Please see the Synnefo
:ref:`Changelog <Changelog-0.16.1>` for a complete list.
.. _NEWS-0.16.1:
v0.16.1
......
......@@ -47,7 +47,7 @@ for more information on the Synnefo users and developers lists.
Copyright and license
=====================
Copyright (C) 2010-2014 GRNET S.A.
Copyright (C) 2010-2015 GRNET S.A. and individual contributors
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
......
......@@ -2,7 +2,7 @@
# Timeouts in seconds
build_timeout = 240
# Apt repository to use
apt_repo =
apt_repo = # Synnefo and X2Go repositories
deb http://apt.dev.grnet.gr wheezy/
deb http://packages.x2go.org/debian wheezy main
......@@ -57,7 +57,7 @@ flavors = name:C2R4...D20ext_.*, name:C2R4...D20drbd, id:1
# A list of images (comma seperated) to choose from
# The user can specify an image name (reg expression)
# with "name:" or an image id with "id:".
images = name:SynnefoCIWheezy.*, name:^Debian Base$, id:72d9844f-1024-4a07-a3c3-60d650b8f5cd
images = name:SynnefoCIWheezy.*, name:^Debian Base \(OldStable\)$, id:a1a2ff80-3371-473e-87e9-3389103bb80e
# File containing the ssh keys to upload/install to server
# If not set, no ssh keys will be installed
ssh_keys = ~/.ssh/id_rsa.pub
......
#!/usr/bin/env python
#
# Copyright (C) 2010-2014 GRNET S.A.
# Copyright (C) 2010-2015 GRNET S.A.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
......@@ -51,6 +51,9 @@ DEFAULT_SYSTEM_IMAGES_UUID = [
"04cbe33f-29b7-4ef1-94fb-015929e5fc06", # testing (okeanos.io)
]
# Working directory - Contains the code repository and the build directory
work_dir = "/var/tmp"
def _run(cmd, verbose):
"""Run fabric with verbose level"""
......@@ -288,7 +291,7 @@ class SynnefoCI(object):
resource = resource.replace("__", ".")
project_resource = project_quota.get(resource)
if not project_resource:
raise Exception("Requested resource does not exist %s" \
raise Exception("Requested resource does not exist %s"
% resource)
plimit, ppending, pusage, musage, mlimit, mpending = \
......@@ -298,7 +301,7 @@ class SynnefoCI(object):
mavailable = mlimit - mpending - musage
can_fit = (pavailable - required) >= 0 and \
(mavailable - required) >= 0
(mavailable - required) >= 0
if not can_fit:
return None
return uuid
......@@ -310,7 +313,6 @@ class SynnefoCI(object):
raise Exception("No project available for %r" % resources)
return projects[0]
def _wait_transition(self, server_id, current_status, new_status):
"""Wait for server to go from current_status to new_status"""
self.logger.debug("Waiting for server to become %s" % new_status)
......@@ -468,7 +470,7 @@ class SynnefoCI(object):
echo 'precedence ::ffff:0:0/96 100' >> /etc/gai.conf
apt-get update
apt-get install -q=2 curl --yes --force-yes
echo -e "\n\n{0}" >> /etc/apt/sources.list
echo -e "{0}" >> /etc/apt/sources.list.d/synnefo.wheezy.list
# Synnefo repo's key
curl https://dev.grnet.gr/files/apt-grnetdev.pub | apt-key add -
""".format(self.config.get('Global', 'apt_repo'))
......@@ -493,7 +495,8 @@ class SynnefoCI(object):
echo 'Encoding=UTF-8' >> /usr/share/applications/xterm.desktop
echo 'Icon=xterm-color_48x48' >> /usr/share/applications/xterm.desktop
echo 'Categories=System;TerminalEmulator;' >> \
/usr/share/applications/xterm.desktop"""
/usr/share/applications/xterm.desktop
"""
if self.config.get("Global", "setup_x2go") == "True":
self.logger.debug("Install x2goserver and firefox")
_run(cmd, False)
......@@ -577,8 +580,8 @@ class SynnefoCI(object):
"Trying to find an image with name \"%s\"" % img_value)
accepted_uuids = DEFAULT_SYSTEM_IMAGES_UUID + [user_uuid]
list_imgs = \
[i for i in list_images if i['user_id'] in accepted_uuids
and
[i for i in list_images if i['user_id'] in
accepted_uuids and
re.search(img_value, i['name'], flags=re.I) is not None]
elif img_type == "id":
# Filter images by id
......@@ -854,7 +857,9 @@ class SynnefoCI(object):
# Use local_repo
self.logger.debug("Push local repo to server")
# Firstly create the remote repo
_run("git init synnefo", False)
_run("git init %s/synnefo" % work_dir, False)
# Create a symlink to the userdir
_run("ln -s %s/synnefo ~/synnefo" % work_dir, False)
# Then push our local repo over ssh
# We have to pass some arguments to ssh command
# namely to disable host checking.
......@@ -864,31 +869,37 @@ class SynnefoCI(object):
cmd = """
echo 'exec ssh -o "StrictHostKeyChecking no" \
-o "UserKnownHostsFile /dev/null" \
-q "$@"' > {4}
chmod u+x {4}
export GIT_SSH="{4}"
echo "{0}" | git push --quiet --mirror ssh://{1}@{2}:{3}/~/synnefo
rm -f {4}
-q "$@"' > {5}
chmod u+x {5}
export GIT_SSH="{5}"
echo "{0}" | git push -q --mirror ssh://{1}@{2}:{3}{4}/synnefo
#echo "{0}" | git push -q --mirror ssh://{1}@{2}:{3}/~/synnefo
rm -f {5}
""".format(fabric.env.password,
fabric.env.user,
fabric.env.host_string,
fabric.env.port,
work_dir,
temp_ssh_file)
os.system(cmd)
else:
# Clone Synnefo from remote repo
self.logger.debug("Clone synnefo from %s" % synnefo_repo)
self._git_clone(synnefo_repo, directory="synnefo")
self._git_clone(synnefo_repo, directory="%s/synnefo" % work_dir)
# Checkout the desired synnefo_branch
self.logger.debug("Checkout \"%s\" branch/commit" % synnefo_branch)
cmd = """
cd synnefo
cd %s/synnefo
# Squelch the error message about pushing to master.
# Keep default behaviour but hide the error message.
git config receive.denyCurrentBranch refuse
#cd synnefo
for branch in `git branch -a | grep remotes | grep -v HEAD`; do
git branch --track ${branch##*/} $branch
done
git checkout %s
""" % (synnefo_branch)
""" % (work_dir, synnefo_branch)
_run(cmd, False)
# Apply a Github pull request
......@@ -896,9 +907,9 @@ class SynnefoCI(object):
self.logger.debug("Apply patches from pull request %s",
pull_number)
cmd = """
cd synnefo
cd %s/synnefo
git pull --no-edit --no-rebase {0} {1}
""".format(pull_repo[0], pull_repo[1])
""".format(work_dir, pull_repo[0], pull_repo[1])
_run(cmd, False)
return synnefo_branch
......@@ -915,16 +926,17 @@ class SynnefoCI(object):
# Clone pithos-webclient from remote repo
self.logger.debug("Clone pithos-webclient from %s" %
pithos_webclient_repo)
self._git_clone(pithos_webclient_repo, directory="pithos-web-client")
self._git_clone(pithos_webclient_repo, directory="%s/pithos-web-client"
% work_dir)
# Track all pithos-webclient branches
cmd = """
cd pithos-web-client
cd %s/pithos-web-client
for branch in `git branch -a | grep remotes | grep -v HEAD`; do
git branch --track ${branch##*/} $branch > /dev/null 2>&1
done
git --no-pager branch --no-color
"""
""" % work_dir
webclient_branches = _run(cmd, False)
webclient_branches = webclient_branches.split()
......@@ -955,9 +967,9 @@ class SynnefoCI(object):
self.logger.debug("Checkout \"%s\" branch" %
_green(pithos_webclient_branch))
cmd = """
cd pithos-web-client
git checkout {0}
""".format(pithos_webclient_branch)
cd {0}/pithos-web-client
git checkout {1}
""".format(work_dir, pithos_webclient_branch)
_run(cmd, False)
def _git_clone(self, repo, directory=""):
......@@ -1012,9 +1024,9 @@ class SynnefoCI(object):
self.logger.info("Build Synnefo packages..")
cmd = """
devflow-autopkg snapshot -b ~/synnefo_build-area --no-sign
"""
with fabric.cd("synnefo"):
devflow-autopkg snapshot -b %s/synnefo_build-area --no-sign
""" % work_dir
with fabric.cd("%s/synnefo" % work_dir):
_run(cmd, True)
# Install snf-deploy package
......@@ -1024,15 +1036,15 @@ class SynnefoCI(object):
apt-get -f install --yes --force-yes
snf-deploy keygen
"""
with fabric.cd("synnefo_build-area"):
with fabric.cd("%s/synnefo_build-area" % work_dir):
with fabric.settings(warn_only=True):
_run(cmd, True)
# Setup synnefo packages for snf-deploy
self.logger.debug("Copy synnefo debs to snf-deploy packages dir")
cmd = """
cp ~/synnefo_build-area/*.deb /var/lib/snf-deploy/packages/
"""
cp %s/synnefo_build-area/*.deb /var/lib/snf-deploy/packages/
""" % work_dir
_run(cmd, False)
@_check_fabric
......@@ -1041,16 +1053,16 @@ class SynnefoCI(object):
self.logger.info("Build pithos-web-client packages..")
cmd = """
devflow-autopkg snapshot -b ~/webclient_build-area --no-sign
"""
with fabric.cd("pithos-web-client"):
devflow-autopkg snapshot -b %s/webclient_build-area --no-sign
""" % work_dir
with fabric.cd("%s/pithos-web-client" % work_dir):
_run(cmd, True)
# Setup pithos-web-client packages for snf-deploy
self.logger.debug("Copy webclient debs to snf-deploy packages dir")
cmd = """
cp ~/webclient_build-area/*.deb /var/lib/snf-deploy/packages/
"""
cp %s/webclient_build-area/*.deb /var/lib/snf-deploy/packages/
""" % work_dir
_run(cmd, False)
@_check_fabric
......@@ -1058,7 +1070,7 @@ class SynnefoCI(object):
"""Build Synnefo documentation"""
self.logger.info("Build Synnefo documentation..")
_run("pip install -U Sphinx", False)
with fabric.cd("synnefo"):
with fabric.cd("%s/synnefo" % work_dir):
_run("devflow-update-version; "
"./ci/make_docs.sh synnefo_documentation", False)
......@@ -1070,7 +1082,8 @@ class SynnefoCI(object):
dest = os.path.abspath(dest)
if not os.path.exists(dest):
os.makedirs(dest)
self.fetch_compressed("synnefo/synnefo_documentation", dest)
self.fetch_compressed("%s/synnefo/synnefo_documentation" % work_dir,
dest)
self.logger.info("Downloaded documentation to %s" %
_green(dest))
......@@ -1084,14 +1097,14 @@ class SynnefoCI(object):
self.logger.debug("Update schema files to server")
cmd = """
schema_dir="synnefo/ci/schemas/{0}"
schema_dir="{0}/synnefo/ci/schemas/{1}"
if [ -d "$schema_dir" ]; then
cp "$schema_dir"/* /etc/snf-deploy/
else
echo "$schema_dir" does not exist
exit 1
fi
""".format(schema)
""".format(work_dir, schema)
_run(cmd, False)
self.logger.debug("Change password in nodes.conf file")
......@@ -1114,7 +1127,7 @@ class SynnefoCI(object):
self.logger.debug("Install needed packages")
cmd = """
pip install -U mock factory_boy nose coverage
pip install -U funcsigs mock==1.1.2 factory_boy==2.4.1 nose coverage
"""
_run(cmd, False)
......@@ -1163,9 +1176,9 @@ class SynnefoCI(object):
cmd = """
cd %s
tar xzf %s
cp -r %s/* %s
cp -r %s/%s/* %s
rm -r %s
""" % (tmp_dir, tar_file, src, dest, tmp_dir)
""" % (tmp_dir, tar_file, tmp_dir, src, dest, tmp_dir)
os.system(cmd)
self.logger.info("Downloaded %s to %s" %
(src, _green(dest)))
......@@ -1178,9 +1191,9 @@ class SynnefoCI(object):
dest = os.path.abspath(os.path.expanduser(dest))
if not os.path.exists(dest):
os.makedirs(dest)
self.fetch_compressed("synnefo_build-area", dest)
self.fetch_compressed("%s/synnefo_build-area" % work_dir, dest)
if self.config.get("Global", "build_pithos_webclient") == "True":
self.fetch_compressed("webclient_build-area", dest)
self.fetch_compressed("%s/webclient_build-area" % work_dir, dest)
self.logger.info("Downloaded debian packages to %s" %
_green(dest))
......
......@@ -36,6 +36,12 @@ which serves Synnefo by default, needs read access to the configuration
files and we don't want it to run as root, it must run with group
``synnefo``.
.. warning:: If you want to add your own configuration file, do not forget to
declare the appropriate encoding by adding the line
``## -*- coding: utf-8 -*-`` at the beggining of the file.
Cyclades and Pithos talk to Archipelago over some named pipes under
``/dev/shm/posixfd``. This directory is created by Archipelago, owned by
the user/group that Archipelago runs as, and at the same time it must be
......@@ -175,10 +181,12 @@ Click Create an application.
Fill the necessary information and for callback URL give::
https://node1.example.com/ui/login/twitter/authenticated
https://node1.example.com/astakos/ui/login/twitter/authenticated
Finally, add 'twitter' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``
Edit ``/etc/synnefo/20-snf-astakos-app-settings.conf`` and set the
corresponding variables ``ASTAKOS_TWITTER_TOKEN`` and
``ASTAKOS_TWITTER_SECRET`` to reflect your newly created pair.
Finally, add 'twitter' in ``ASTAKOS_IM_MODULES`` list.
Google Authentication
~~~~~~~~~~~~~~~~~~~~~
......@@ -188,15 +196,14 @@ visit https://code.google.com/apis/console/.
Under API Access select Create another client ID, select Web application,
expand more options in Your site or hostname section and in Authorized
Redirect URIs add:
Fill the necessary information and for callback URL give::
Redirect URIs add::
https://node1.example.com/ui/login/google/authenticated
https://node1.example.com/astakos/ui/login/google/authenticated
Finally, add 'google' in ``ASTAKOS_IM_MODULES`` list. The variable resides
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``
Edit ``/etc/synnefo/20-snf-astakos-app-settings.conf`` and set the
corresponding variables ``ASTAKOS_GOOGLE_CLIENT_ID`` and
``ASTAKOS_GOOGLE_SECRET`` to reflect your newly created pair.
Finally, add 'google' in ``ASTAKOS_IM_MODULES`` list.
Working with Astakos
......@@ -311,7 +318,11 @@ Currently astakos supports the following identity providers:
(module name ``linkedin``)
To enable any of the above modules (by default only ``local`` accounts are
allowed), retrieve and set the required provider settings and append the
allowed) you have to install oauth2 package. To do so run::
apt-get install python-oauth2
Then retrieve and set the required provider settings and append the
module name in ``ASTAKOS_IM_MODULES``.
.. code-block:: python
......@@ -323,7 +334,6 @@ module name in ``ASTAKOS_IM_MODULES``.
# let users signup and login using their google account
ASTAKOS_IM_MODULES = ['local', 'google']
.. _auth_methods_policies:
Authentication method policies
......@@ -889,11 +899,22 @@ image to Cyclades:
Deletion of an image is done via `kamaki image unregister` command, which will
delete the Cyclades Images but will leave the Pithos file as is (unregister).
Apart from using `kamaki` to see and hangle the available images, the
Apart from using `kamaki` to see and handle the available images, the
administrator can use `snf-manage image-list` and `snf-manage image-show`
commands to list and inspect the available public images. Also, the `--user`
option can be used the see the images of a specific user.
Custom image listing sections
`````````````````````````````
Since Synnefo 0.16.2, the installation wizard supports custom image listing
sections. Images with the ``LISTING_SECTION`` image property set, and whose
owner uuid is listed in the ``UI_IMAGE_LISTING_USERS`` Cyclades setting (in
``/etc/synnefo/20-snf-cyclades-app-ui.conf``) will be displayed in a separate
section in the installation wizard. The name of the new section will be the
value of the ``LISTING_SECTION`` image property.
Virtual Servers
~~~~~~~~~~~~~~~
......@@ -2958,6 +2979,7 @@ Upgrade Notes
v0.14 -> v0.15 <upgrade/upgrade-0.15>
v0.15 -> v0.15.1 <upgrade/upgrade-0.15.1>
v0.15 -> v0.16 <upgrade/upgrade-0.16>
v0.16.1 -> v0.16.2 <upgrade/upgrade-0.16.2>
.. _changelog-news:
......@@ -2966,6 +2988,7 @@ Changelog, NEWS
===============
* v0.16.2 :ref:`Changelog <Changelog-0.16.2>`, :ref:`NEWS <NEWS-0.16.2>`
* v0.16.1 :ref:`Changelog <Changelog-0.16.1>`, :ref:`NEWS <NEWS-0.16.1>`
* v0.16 :ref:`Changelog <Changelog-0.16>`, :ref:`NEWS <NEWS-0.16>`
* v0.15.2 :ref:`Changelog <Changelog-0.15.1>`, :ref:`NEWS <NEWS-0.15.2>`
......
......@@ -8,7 +8,7 @@ reload(synnefo.versions)
from synnefo.versions.app import __version__
project = u'synnefo'
copyright = u'2012-2014, GRNET'
copyright = u'2012-2015, GRNET'
version = __version__
release = __version__
html_title = 'synnefo ' + version
......
......@@ -1405,7 +1405,7 @@ specific to Synnefo.
Kamaki Installation and Configuration
-------------------------------------
To install kamaki run:
To install Kamaki run:
.. code-block:: console
......@@ -1419,7 +1419,7 @@ log in and click on ``API access``. Scroll all the way to the bottom of the
page, click on the orange ``Download your .kamakirc`` button and save the file
as ``.kamakirc`` in your home directory.
That's all, kamaki is now configured and you can start using it. For a list of
That's all, Kamaki is now configured and you can start using it. For a list of
commands, see the `official documentantion <http://www.synnefo.org/docs/kamaki/latest/commands.html>`_.
Cyclades Prerequisites
......@@ -1713,7 +1713,7 @@ also supports three (3) different locations for the above Images to be stored:
* On a remote host (accessible via public URL e.g: http://... or ftp://...)
* On Pithos (accessible natively, not only by its public URL)
For the purpose of this guide, we will use the Debian Squeeze Base Image found
For the purpose of this guide, we will use the Debian Wheezy Base Image found
on the official `snf-image page
<http://www.synnefo.org/docs/snf-image/latest/usage.html#sample-images>`_. The
image is of type ``diskdump``. We will store it in our new Pithos installation.
......@@ -1723,14 +1723,14 @@ To do so, do the following:
a) Download the Image from the official snf-image page.
b) Upload the Image to your Pithos installation, either using the Pithos Web
UI or the command line client `kamaki
UI or the command line client `Kamaki
<http://www.synnefo.org/docs/kamaki/latest/index.html>`_.
To upload the file using kamaki, run:
To upload the file using Kamaki, run:
.. code-block:: console
# kamaki file upload debian_base-6.0-x86_64.diskdump pithos
# kamaki file upload debian_base-7.0-x86_64.diskdump pithos
Once the Image is uploaded successfully, download the Image's metadata file
from the official snf-image page. You will need it, for spawning a VM from
......@@ -1747,9 +1747,9 @@ Spawning a VM from a Pithos Image, using Ganeti
Now, it is time to test our installation so far. So, we have Astakos and
Pithos installed, we have a working Ganeti installation, the snf-image
definition installed on all VM-capable nodes, a Debian Squeeze Image on
Pithos and kamaki installed and configured. Make sure you also have the
`metadata file <http://cdn.synnefo.org/debian_base-6.0-x86_64.diskdump.meta>`_
definition installed on all VM-capable nodes, a Debian Wheezy Image on
Pithos and Kamaki installed and configured. Make sure you also have the
`metadata file <https://cdn.synnefo.org/debian_base-7.0-x86_64.diskdump.meta>`_
for this image.
To spawn a VM from a Pithos file, we need to know:
......@@ -1757,7 +1757,7 @@ To spawn a VM from a Pithos file, we need to know:
1) The mapfile name of the file
2) The size of the file
If you uploaded the file with kamaki as described above, run on the Astakos
If you uploaded the file with Kamaki as described above, run on the Astakos
node:
.. code-block:: console
......@@ -1777,9 +1777,9 @@ Then on the Pithos node run the following:
.. code-block:: console