Commit 97b36d82 authored by Christos Stavrakakis's avatar Christos Stavrakakis
Browse files

Merge branch 'release-0.14' into develop

Conflicts:
	snf-astakos-app/astakos/im/views/target/redirect.py
	snf-cyclades-app/synnefo/logic/backend.py
	version
parents 4e618544 3b04f900
......@@ -22,6 +22,7 @@ settings.d/*-local.conf
*.egg-info
dist
_build
.coverage
# pithos temp files
pithos/data
......
#!/bin/sh
set -e
export SYNNEFO_SETTINGS_DIR='/tmp/snf-test-settings'
export SYNNEFO_SETTINGS_DIR=/tmp/snf-test-settings
SNF_MANAGE=$(which snf-manage) ||
{ echo "Cannot find snf-manage in $PATH" 1>&2; exit 1; }
APPS="api db logic plankton quotas vmapi im quotaholder_app helpdesk"
TEST="$(which snf-manage) test $APPS --traceback --settings=synnefo.settings.test"
TEST="$SNF_MANAGE test $APPS --traceback --settings=synnefo.settings.test"
if coverage >/dev/null 2>&1; then
coverage run $TEST
coverage report --include=snf-*
else
echo "coverage not installed"
echo "WARNING: Cannot find coverage in path, skipping coverage tests" 1>&2
$TEST
fi
......@@ -560,20 +560,14 @@ For astakos specific configuration, edit the following options in
.. code-block:: console
ASTAKOS_DEFAULT_ADMIN_EMAIL = None
ASTAKOS_COOKIE_DOMAIN = '.example.com'
ASTAKOS_BASE_URL = 'https://node1.example.com/astakos'
The ``ASTAKOS_COOKIE_DOMAIN`` should be the base url of our domain (for all
services). ``ASTAKOS_BASE_URL`` is the astakos top-level URL.
``ASTAKOS_DEFAULT_ADMIN_EMAIL`` refers to the administrator's email.
Every time a new account is created a notification is sent to this email.
For this we need access to a running mail server, so we have disabled
it for now by setting its value to None. For more informations on this,
read the relative :ref:`section <mail-server>`.
services). ``ASTAKOS_BASE_URL`` is the astakos top-level URL. Appending an
extra path (``/astakos`` here) is recommended in order to distinguish
components, if more than one are installed on the same machine.
.. note:: For the purpose of this guide, we don't enable recaptcha authentication.
If you would like to enable it, you have to edit the following options:
......@@ -605,6 +599,80 @@ to point at node1 which is where we have installed Astakos.
If you are an advanced user and want to use the Shibboleth Authentication
method, read the relative :ref:`section <shibboleth-auth>`.
.. _email-configuration:
Email delivery configuration
----------------------------
Many of the ``astakos`` operations require server to notify service users and
administrators via email. e.g. right after the signup process the service sents
an email to the registered email address containing an email verification url,
after the user verifies the email address astakos once again needs to notify
administrators with a notice that a new account has just been verified.
More specifically astakos sends emails in the following cases
- An email containing a verification link after each signup process.
- An email to the people listed in ``ADMINS`` setting after each email
verification if ``ASTAKOS_MODERATION`` setting is ``True``. The email
notifies administrators that an additional action is required in order to
activate the user.
- A welcome email to the user email and an admin notification to ``ADMINS``
right after each account activation.
- Feedback messages submited from astakos contact view and astakos feedback
API endpoint are sent to contacts listed in ``HELPDESK`` setting.
- Project application request notifications to people included in ``HELPDESK``
and ``MANAGERS`` settings.
- Notifications after each project members action (join request, membership
accepted/declinde etc.) to project members or project owners.
Astakos uses the Django internal email delivering mechanism to send email
notifications. A simple configuration, using an external smtp server to
deliver messages, is shown below.
.. code-block:: python
# /etc/synnefo/10-snf-common-admins.conf
EMAIL_HOST = "mysmtp.server.synnefo.org"
EMAIL_HOST_USER = "<smtpuser>"
EMAIL_HOST_PASSWORD = "<smtppassword>"
# this gets appended in all email subjects
EMAIL_SUBJECT_PREFIX = "[example.synnefo.org] "
# Address to use for outgoing emails
DEFAULT_FROM_EMAIL = "server@example.synnefo.org"
# Email where users can contact for support. This is used in html/email
# templates.
CONTACT_EMAIL = "server@example.synnefo.org"
# The email address that error messages come from
SERVER_EMAIL = "server-errors@example.synnefo.org"
Notice that since email settings might be required by applications other than
astakos they are defined in a different configuration file than the one
previously used to set astakos specific settings.
Refer to
`Django documentation <https://docs.djangoproject.com/en/1.2/topics/email/>`_
for additional information on available email settings.
As refered in the previous section, based on the operation that triggers
an email notification, the recipients list differs. Specifically for
emails whose recipients include contacts from your service team
(administrators, managers, helpdesk etc) synnefo provides the following
settings located in ``10-snf-common-admins.conf``:
.. code-block:: python
ADMINS = (('Admin name', 'admin@example.synnefo.org'),
('Admin2 name', 'admin2@example.synnefo.org))
MANAGERS = (('Manager name', 'manager@example.synnefo.org'),)
HELPDESK = (('Helpdesk user name', 'helpdesk@example.synnefo.org'),)
Enable Pooling
--------------
......@@ -700,9 +768,21 @@ When the database is ready, we need to register the services. The following
command will ask you to register the standard Synnefo components (astakos,
cyclades, and pithos) along with the services they provide. Note that you
have to register at least astakos in order to have a usable authentication
system. For each component, you will be asked to provide its base
installation URL as well as the UI URL (to appear in the Cloudbar).
Moreover, the command will automatically register the resource definitions
system. For each component, you will be asked to provide two URLs: its base
URL and its UI URL.
The former is the location where the component resides; it should equal
the ``<component_name>_BASE_URL`` as specified in the respective component
settings. For example, the base URL for astakos would be
``https://node1.example.com/astakos``.
The latter is the URL that appears in the Cloudbar and leads to the
component UI. If you want to follow the default setup, set
the UI URL to ``<base_url>/ui/`` where ``base_url`` the component's base
URL as explained before. (You can later change the UI URL with
``snf-manage component-modify <component_name> --url new_ui_url``.)
The command will also register automatically the resource definitions
offered by the services.
.. code-block:: console
......@@ -850,7 +930,7 @@ this options:
PITHOS_SERVICE_TOKEN = 'pithos_service_token22w'
# Set to False if astakos & pithos are on the same host
#PITHOS_PROXY_USER_SERVICES = True
PITHOS_PROXY_USER_SERVICES = True
The ``PITHOS_BACKEND_DB_CONNECTION`` option tells to the Pithos app where to
......@@ -879,19 +959,6 @@ It can be retrieved by running on the Astakos node (node1 in our case):
The token has been generated automatically during the :ref:`Pithos service
registration <services-reg>`.
Then we need to setup the web UI and connect it to astakos. To do so, edit
``/etc/synnefo/20-snf-pithos-webclient-settings.conf``:
.. code-block:: console
PITHOS_UI_LOGIN_URL = "https://node1.example.com/ui/login?next="
PITHOS_UI_FEEDBACK_URL = "https://node2.example.com/feedback"
The ``PITHOS_UI_LOGIN_URL`` option tells the client where to redirect you, if
you are not logged in. The ``PITHOS_UI_FEEDBACK_URL`` option points at the
Pithos feedback form. Astakos already provides a generic feedback form for all
services, so we use this one.
The ``PITHOS_UPDATE_MD5`` option by default disables the computation of the
object checksums. This results to improved performance during object uploading.
However, if compatibility with the OpenStack Object Storage API is important
......@@ -1036,7 +1103,7 @@ Ganeti
`Ganeti <http://code.google.com/p/ganeti/>`_ handles the low level VM management
for Cyclades, so Cyclades requires a working Ganeti installation at the backend.
Please refer to the
`ganeti documentation <http://docs.ganeti.org/ganeti/2.5/html>`_ for all the
`ganeti documentation <http://docs.ganeti.org/ganeti/2.6/html>`_ for all the
gory details. A successful Ganeti installation concludes with a working
:ref:`GANETI-MASTER <GANETI_NODES>` and a number of :ref:`GANETI-NODEs
<GANETI_NODES>`.
......@@ -1090,7 +1157,7 @@ Then run on node1:
root@node1:~ # gnt-group modify --disk-parameters=drbd:metavg=ganeti default
For any problems you may stumble upon installing Ganeti, please refer to the
`official documentation <http://docs.ganeti.org/ganeti/2.5/html>`_. Installation
`official documentation <http://docs.ganeti.org/ganeti/2.6/html>`_. Installation
of Ganeti is out of the scope of this guide.
.. _cyclades-install-snfimage:
......@@ -1618,7 +1685,7 @@ and then putting the output in ``/var/lib/ganeti/rapi/users`` as follows:
cyclades {HA1}55aec7050aa4e4b111ca43cb505a61a0 write
More about Ganeti's RAPI users `here.
<http://docs.ganeti.org/ganeti/2.5/html/rapi.html#introduction>`_
<http://docs.ganeti.org/ganeti/2.6/html/rapi.html#introduction>`_
You have now finished with all needed Prerequisites for Cyclades. Let's move on
to the actual Cyclades installation.
......@@ -1679,6 +1746,10 @@ Since our Astakos, Cyclades, and Pithos installations belong together,
they should all have identical ``ASTAKOS_BASE_URL`` setting
(see also, :ref:`previously <conf-pithos>`).
The ``CYCLADES_BASE_URL`` setting must point to the top-level Cyclades URL.
Appending an extra path (``/cyclades`` here) is recommended in order to
distinguish components, if more than one are installed on the same machine.
The ``CYCLADES_SERVICE_TOKEN`` is the token used for authentication with astakos.
It can be retrieved by running on the Astakos node (node1 in our case):
......
......@@ -94,7 +94,6 @@ In `/etc/synnefo/cyclades.conf` add:
}
CACHE_BACKEND = 'memcached://127.0.0.1:11211/'
VMAPI_BASE_URL = 'https://cyclades.example.com/'
CYCLADES_SERVICE_TOKEN = "XXXXXXXXXX"
......
......@@ -82,11 +82,6 @@ In `/etc/synnefo/webclient.conf` add:
CLOUDBAR_SERVICES_URL = 'https://accounts.example.com/ui/get_services'
CLOUDBAR_MENU_URL = 'https://accounts.example.com/ui/get_menu'
PITHOS_UI_LOGIN_URL = "https://accounts.example.com/ui/login?next="
PITHOS_UI_FEEDBACK_URL = "/feedback"
XXXXXXXXXXXXXX should be the Pithos token and id found on astakos node by running:
.. code-block:: console
......
......@@ -9,6 +9,8 @@ The upgrade to v0.14 consists in three steps:
3. Register services to astakos and perform a quota-related data migration.
4. Bring up all services.
.. warning::
It is strongly suggested that you keep separate database backups
......@@ -22,7 +24,7 @@ The upgrade to v0.14 consists in three steps:
$ service gunicorn stop
$ service snf-dispatcher stop
etc.
$ service snf-ganeti-eventd stop
2. Backup databases for recovery to a pre-migration state.
......@@ -73,12 +75,20 @@ The upgrade to v0.14 consists in three steps:
snf-cyclades-gtools \
snf-pithos-backend
.. note::
Make sure `snf-webproject' has the same version with snf-common
.. note::
Package `kamaki', installed on all nodes in version 0.13, is not required
any more and can safely be uninstalled.
.. note::
Installing the packages will cause services to start. Make sure you bring
them down again (at least ``gunicorn``, ``snf-dispatcher``)
2.2 Sync and migrate the database
---------------------------------
......@@ -125,21 +135,37 @@ setting was renamed to this. Therefore:
where ``ASTAKOS_BASE_URL`` as above. Similarly, set
``CLOUDBAR_MENU_URL`` to ``ASTAKOS_BASE_URL/ui/get_menu``.
If two or more services are installed on the same machine, make sure their
base URLs do not clash. You can distinguish them with a suffix, e.g.
``ASTAKOS_BASE_URL = "https://node1.example.com/astakos"`` and
``CYCLADES_BASE_URL = "https://node1.example.com/cyclades"``.
3 Register services and migrate quota
=====================================
You need to register astakos as a component. Moreover you need to register
all services provided by cyclades and pithos.
Running the following script you will be asked to provide the base
installation URL for each component. You will also need to specify the URL
where the astakos UI resides::
installation URL for each component. You will also need to specify the UI
URL for astakos.
The former is the location where each component resides; it should equal
the ``<component_name>_BASE_URL`` as specified in the respective component
settings (see above).
The latter is the URL that appears in the Cloudbar and leads to the
component UI. If you want to follow the default setup, set
the UI URL to ``<base_url>/ui/`` where ``base_url`` is the component's base
URL as explained before.
For example, for Astakos, if
``BASE_URL = https://accounts.example.synnefo.org/astakos``,
then ``UI_URL = https://accounts.example.synnefo.org/astakos/ui/``)::
astakos-host$ snf-component-register
(proceed to the next step without running ``snf-manage resource-modify``
suggested at the end of this command)
(ATTENTION: make sure to go to the next step *WITHOUT* running
``snf-manage resource-modify``, suggested at the end of this command)
.. note::
......@@ -159,12 +185,46 @@ suggested at the end of this command)
# copy the file to astakos-host
astakos-host$ snf-manage service-import --json pithos.json
The limit on pending project applications is since 0.14 handled as an
Astakos resource, rather than a custom setting. Command::
Run::
astakos-host$ snf-manage component-list
to make sure that all UI URLs are set to the correct value (``<base_url>/ui/``
as explained above). If you have changed some ``<component_name>_BASE_URL``
in the previous step, you will need to update the UI URL with::
snf-manage component-modify <component_name> --url new_ui_url
The limit on the pending project applications is since 0.14 handled as an
Astakos resource, rather than a custom setting. So, as a last step we need
to run::
astakos-host$ astakos-migrate-0.14
will prompt you to set this limit (replacing setting
ASTAKOS_PENDING_APPLICATION_LIMIT) and then automatically migrate the
This will prompt you to set this limit (replacing setting
``ASTAKOS_PENDING_APPLICATION_LIMIT``) and then automatically migrate the
user-specific base quota for the new resource ``astakos.pending_app`` using
the deprecated user setting.
You are now done migrating from Synnefo v0.13 to v0.14. Please test your
installation to make sure everything works as expected.
4. Update astakos email notification settings
=============================================
Make sure to update your configuration to include settings refered in the
updated :ref:`Email delivery configuration <email-configuration>` section
of the quick installation admin guide.
5. Bring all services up
========================
After the upgrade is finished, we bring up all services:
.. code-block:: console
astakos.host # service gunicorn start
cyclades.host # service gunicorn start
pithos.host # service gunicorn start
cyclades.host # service snf-dispatcher start
......@@ -120,6 +120,7 @@ class LocalUserCreationForm(UserCreationForm, StoreUserMixin):
if not provider == 'local':
raise Exception('Invalid provider')
self.ip = None
if request:
self.ip = request.META.get('REMOTE_ADDR',
request.META.get('HTTP_X_REAL_IP',
......@@ -244,6 +245,7 @@ class ThirdPartyUserCreationForm(forms.ModelForm, StoreUserMixin):
"""
self.provider = kwargs.pop('provider', None)
self.request = kwargs.pop('request', None)
if not self.provider or self.provider == 'local':
raise Exception('Invalid provider, %r' % self.provider)
......@@ -530,7 +532,7 @@ class ExtendedPasswordResetForm(PasswordResetForm):
"""
for user in self.users_cache:
url = user.astakosuser.get_password_reset_url(token_generator)
url = join_urls(settings.BASE_URL, url)
url = join_urls(settings.BASE_HOST, url)
c = {
'email': user.email,
'url': url,
......
......@@ -199,3 +199,7 @@ API_CLIENT_URL = getattr(settings, 'ASTAKOS_API_CLIENT_URL',
KAMAKI_CONFIG_CLOUD_NAME = getattr(settings,
'ASTAKOS_KAMAKI_CONFIG_CLOUD_NAME',
None)
REDIRECT_ALLOWED_SCHEMES = getattr(settings,
'ASTAKOS_REDIRECT_ALLOWED_SCHEMES',
('pithos', 'pithosdev'))
......@@ -6,9 +6,9 @@ Dear {{ user.realname }},
We received a request for associating the specific email address with a shibboleth account for {{ BRANDING_COMPANY_NAME }}'s {{ BRANDING_SERVICE_NAME }} service.
However this email is already associated with another local account.
In order to switch the existing account to a shibboleth one, use the link: {{ url }}.
In order to switch the existing account to a shibboleth one, use the link: {{ url }}
Otherwise, just ignore this email.
{% endblock %}
{% block note %} {% endblock%}
\ No newline at end of file
{% block note %} {% endblock%}
{% extends "im/email.txt" %}
{% block content %}
To reset your password for {{ BRANDING_COMPANY_NAME|upper }}'s {{ BRANDING_SERVICE_NAME }}, you can use the link: {{ url }}.
To reset your password for {{ BRANDING_COMPANY_NAME|upper }}'s {{ BRANDING_SERVICE_NAME }}, please use the following link:
{{ url }}
{% endblock %}
......@@ -453,7 +453,7 @@ def signup(request, template_name='im/signup.html', on_success='index',
if not activation_backend:
activation_backend = activation_backends.get_backend()
form_kwargs = {'instance': instance}
form_kwargs = {'instance': instance, 'request': request}
if third_party_token:
form_kwargs['third_party_token'] = third_party_token
......
......@@ -47,6 +47,7 @@ from astakos.im.functions import login as auth_login, logout
from astakos.im.views.decorators import cookie_fix
import astakos.im.messages as astakos_messages
from astakos.im.settings import REDIRECT_ALLOWED_SCHEMES
import logging
......@@ -69,7 +70,7 @@ def login(request):
if not next:
raise HttpResponseBadRequest('Missing next parameter')
if not restrict_next(next, allowed_schemes=('pithos', 'pithosdev')):
if not restrict_next(next, allowed_schemes=REDIRECT_ALLOWED_SCHEMES):
return HttpResponseForbidden(_(
astakos_messages.NOT_ALLOWED_NEXT_PARAM))
force = request.GET.get('force', None)
......
......@@ -2,4 +2,3 @@
#VMAPI_CACHE_BACKEND = "locmem://"
#VMAPI_CACHE_KEY_PREFIX = "vmapi"
#VMAPI_RESET_PARAMS = True
#VMAPI_BASE_URL = "https://cyclades.synnefo.org/"
......@@ -105,7 +105,7 @@ def get_available_backends():
def filter_archipelagos_backends(available_backends, disk_template):
if disk_template.startswith("ext_"):
if disk_template == "ext":
available_backends = filter(lambda x: x.id in ARCHIPELAGO_BACKENDS,
available_backends)
else:
......
......@@ -79,6 +79,17 @@ from synnefo.logic import utils
log = logging.getLogger()
try:
CHECK_INTERVAL = settings.RECONCILIATION_CHECK_INTERVAL
except AttributeError:
CHECK_INTERVAL = 60
def needs_reconciliation(vm):
now = datetime.now()
return (now > vm.updated + timedelta(seconds=CHECK_INTERVAL)) or\
(now > vm.backendtime + timedelta(seconds=2*CHECK_INTERVAL))
def stale_servers_in_db(D, G):
idD = set(D.keys())
......@@ -88,8 +99,7 @@ def stale_servers_in_db(D, G):
for i in idD - idG:
if D[i] == 'BUILD':
vm = VirtualMachine.objects.get(id=i)
# Check time to avoid many rapi calls
if datetime.now() > vm.backendtime + timedelta(seconds=5):
if needs_reconciliation(vm):
with pooled_rapi_client(vm) as c:
try:
job_status = c.GetJobStatus(vm.backendjobid)['status']
......@@ -127,8 +137,7 @@ def unsynced_operstate(D, G):
unsynced.add((i, D[i], G[i]))
if not G[i] and D[i] == 'BUILD':
vm = VirtualMachine.objects.get(id=i)
# Check time to avoid many rapi calls
if datetime.now() > vm.backendtime + timedelta(seconds=5):
if needs_reconciliation(vm):
with pooled_rapi_client(vm) as c:
try:
job_info = c.GetJobStatus(job_id=vm.backendjobid)
......@@ -153,7 +162,7 @@ def instances_with_build_errors(D, G):
# If a job has not been enqueued after 2 minutues, then
# it must be a stale entry..
failed.add(i)
elif datetime.now() > vm.backendtime + timedelta(seconds=30):
elif needs_reconciliation(vm):
# Check time to avoid many rapi calls
with pooled_rapi_client(vm) as c:
try:
......
......@@ -20,10 +20,13 @@ body {
background: url("../images/body-bg.png") repeat-x #FFFFFF;
text-align:center;
font-size: 16px;
background-position: 0px 34px;
overflow-y: scroll;
}
body#app {
background-position: 0px 34px;
}
a, a:hover, a:active, a:focus, div:focus, span:focus, li:focus {
outline: none;
}
......
......@@ -32,8 +32,10 @@
# or implied, of GRNET S.A.
from django.conf.urls.defaults import patterns, url
from snf_django.lib.api import api_endpoint_not_found
urlpatterns = patterns('synnefo.vmapi.views',
url(r'^v1.0/server-params/(?P<uuid>.*)$',
'server_params',
name="vmapi_server_params"),)
name="vmapi_server_params"),
(r'^.*', api_endpoint_not_found),)
......@@ -76,7 +76,7 @@ def api_method(http_method=None, token_required=True, user_required=True,
try:
# Get the requested serialization format
serialization = get_serialization(
request, format_allowed, 'json')
request, format_allowed, serializations[0])
# If guessed serialization is not supported, fallback to
# the default serialization or return an API error in case
......
......@@ -37,12 +37,12 @@
# identity management. WARNING: Setting to True will break your installation.
# PITHOS_TRANSLATE_UUIDS = False
# Set PROXY_USER_SERVICES to True to have snf-pithos-app handle all Astakos
# user-visible services (feedback, login, etc.) by proxying them to a running
# Astakos.
# Set PITHOS_PROXY_USER_SERVICES to True to have snf-pithos-app handle all
# Astakos user-visible services (feedback, login, etc.) by proxying them to
# a running Astakos.
# Set to False if snf astakos-app is running on the same machine, so it handles
# the requests on its own.
#PROXY_USER_SERVICES = True
#PITHOS_PROXY_USER_SERVICES = True
# Tune the size of the http pool for the astakos client.
# It limits the maximum number of quota changing requests
......
......@@ -617,7 +617,7 @@ class SpawnServerTestCase(unittest.TestCase):
def _ping_once(self, ipv6, ip):
"""Test server responds to a single IPv4 or IPv6 ping"""
cmd = "ping%s -c 2 -w 3 %s" % ("6" if ipv6 else "", ip)
cmd = "ping%s -c 7 -w 20 %s" % ("6" if ipv6 else "", ip)
ping = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = ping.communicate()
......@@ -1073,7 +1073,7 @@ class NetworkTestCase(unittest.TestCase):
def _ping_once(self, ip):
"""Test server responds to a single IPv4 or IPv6 ping"""
cmd = "ping -c 2 -w 3 %s" % (ip)
cmd = "ping -c 7 -w 20 %s" % (ip)
ping = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = ping.communicate()
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment