Commit f34a03d8 authored by Christos Stavrakakis's avatar Christos Stavrakakis

Merge branch 'hotfix-0.14.6'

parents 0e4ef554 bb7b02a3
......@@ -6,6 +6,33 @@ Unified Changelog file for Synnefo versions >= 0.13
Since v0.13 most of the Synnefo components have been merged into a single
repository and have aligned versions.
.. _Changelog-0.14.6:
v0.14.6
=======
Released: Wed Sep 18 16:18:58 EEST 2013
Pithos
------
* Substitute the PITHOS_BACKEND_QUOTA setting with two
distinct settings: PITHOS_BACKEND_ACCOUNT_QUOTA &
PITHOS_BACKEND_CONTAINER_QUOTA
* Set PITHOS_BACKEND_CONTAINER_QUOTA default value to 0 (unlimited)
* Fix bug that resulted in DB deadlocks.
Cyclades
--------
* Fix bug in snf-dispatcher that resulted in servers to be deleted from the
DB even if the corresponding Ganeti job failed.
Branding
--------
* Add new BRANDING_FOOTER_EXTRA_MESSAGE setting.
.. _Changelog-0.14.5:
v0.14.5
......
......@@ -5,6 +5,15 @@ Unified NEWS file for Synnefo versions >= 0.13
Since v0.13 all Synnefo components have been merged into a single repository.
.. _NEWS-0.14.6:
v0.14.6
=======
Released: Wed Sep 18 16:18:58 EEST 2013
* Bug fix version
.. _NEWS-0.14.5:
v0.14.5
......
......@@ -1301,7 +1301,7 @@ uncommenting and setting the following:
BRANDING_COMPANY_URL = 'https://www.company-ltd.synnefo.org/'
**Copyright options:**
**Copyright and footer options:**
By default, no Copyright message is shown in the UI footer. If you want to make
it visible in the footer of Astakos, Pithos and Cyclades UI, you can uncomment
......@@ -1319,6 +1319,15 @@ setting the following option:
BRANDING_COPYRIGHT_MESSAGE = 'Copyright (c) 2011-2013 GRNET'
If you want to include a custom message in the footer, you can uncomment and
set the ``BRANDING_FOOTER_EXTRA_MESSAGE`` setting. You can use html markup.
Your custom message will appear above Copyright message at the Compute
templates and the Dashboard UI.
.. code-block:: python
#BRANDING_FOOTER_EXTRA_MESSAGE = ''
**Images:**
......@@ -1803,12 +1812,14 @@ Upgrade Notes
v0.12 -> v0.13 <upgrade/upgrade-0.13>
v0.13 -> v0.14 <upgrade/upgrade-0.14>
v0.14 -> v0.14.2 <upgrade/upgrade-0.14.2>
v0.14.2 -> v0.14.6 <upgrade/upgrade-0.14.6>
Changelog, NEWS
===============
* v0.14.6 :ref:`Changelog <Changelog-0.14.6>`, :ref:`NEWS <NEWS-0.14.6>`
* v0.14.5 :ref:`Changelog <Changelog-0.14.5>`, :ref:`NEWS <NEWS-0.14.5>`
* v0.14.4 :ref:`Changelog <Changelog-0.14.4>`, :ref:`NEWS <NEWS-0.14.4>`
* v0.14.3 :ref:`Changelog <Changelog-0.14.3>`, :ref:`NEWS <NEWS-0.14.3>`
......
Upgrade to Synnefo v0.14.6
^^^^^^^^^^^^^^^^^^^^^^^^^^
The upgrade from v0.14.2 to v0.14.6 consists of the following step:
1. Set default container quota policy to unlimited for the containers
created prior to 0.14
2. Re-register services in astakos
1. Set default container quota policy to unlimited in old containers
====================================================================
1. In 0.14 has changed the default container quota policy and the containers
by default have no limits in their quota. However this affects only the
neawly created containers.
In order to massively change the quota of ``pithos`` container
(in all the accounts)::
$ pithos-manage-accounts set-container-quota pithos 0
In order to massively change the quota of ``trash`` container
(in all the accounts)::
$ pithos-manage-accounts set-container-quota trash 0
In order to massively change the quota of ``images`` container
(in all the accounts)::
$ pithos-manage-accounts set-container-quota images 0
2. Re-register services in astakos
==================================
Service definitions have changed; you will thus need to register their new
version. In astakos node, run::
astakos-host$ snf-component-register
This will detect that the Synnefo components are already registered and ask
to update the registered services. Answer positively. You need to enter the
base URL for each component; give the same value as in the initial
registration.
{% block footer_content %}
{% if BRANDING_FOOTER_EXTRA_MESSAGE %}
<div class="extra-msg">{{ BRANDING_FOOTER_EXTRA_MESSAGE|safe }}</div>
{% endif %}
<p class="termslink" style="float:right"><a href="{% url latest_terms %}">Terms of service</a></p>
{% if BRANDING_SHOW_COPYRIGHT%}
......
......@@ -119,7 +119,7 @@ cyclades_services = {
},
'cyclades_vmapi': {
'type': 'cyclades_vmapi',
'type': 'vmapi',
'component': 'cyclades',
'prefix': 'vmapi',
'public': True,
......@@ -164,6 +164,18 @@ cyclades_services = {
],
'resources': {},
},
'cyclades_admin': {
'type': 'admin',
'component': 'cyclades',
'prefix': 'admin',
'public': True,
'endpoints': [
{'versionId': '',
'publicURL': None},
],
'resources': {},
},
}
pithos_services = {
......@@ -188,12 +200,12 @@ pithos_services = {
},
'pithos_public': {
'type': 'public',
'type': 'pithos_public',
'component': 'pithos',
'prefix': 'public',
'public': False,
'endpoints': [
{'versionId': 'v1.0',
{'versionId': '',
'publicURL': None},
],
'resources': {},
......
......@@ -64,14 +64,14 @@ context_processors = [
'astakos.im.context_processors.custom_messages',
'astakos.im.context_processors.last_login_method',
'astakos.im.context_processors.membership_policies',
'synnefo.lib.context_processors.cloudbar'
'synnefo.webproject.context_processors.cloudbar'
]
middlware_classes = [
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'synnefo.lib.middleware.LoggingConfigMiddleware',
'synnefo.lib.middleware.SecureMiddleware',
'synnefo.webproject.middleware.LoggingConfigMiddleware',
'synnefo.webproject.middleware.SecureMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
# 'debug_toolbar.middleware.DebugToolbarMiddleware',
......
......@@ -36,8 +36,15 @@ from synnefo.lib import join_urls
from astakos.im.settings import (
BASE_PATH, ACCOUNTS_PREFIX, VIEWS_PREFIX, KEYSTONE_PREFIX, WEBLOGIN_PREFIX)
from snf_django.lib.api.utils import prefix_pattern
from snf_django.utils.urls import extend_with_root_redirects
from astakos.im import settings
from snf_django.utils.urls import \
extend_with_root_redirects, extend_endpoint_with_slash
from astakos.im.settings import astakos_services
urlpatterns = []
# Redirects should be first, otherwise they may get overridden by wildcards
extend_endpoint_with_slash(urlpatterns, astakos_services, 'astakos_ui')
extend_endpoint_with_slash(urlpatterns, astakos_services, 'astakos_weblogin')
astakos_patterns = patterns(
'',
......@@ -48,11 +55,11 @@ astakos_patterns = patterns(
)
urlpatterns = patterns(
urlpatterns += patterns(
'',
(prefix_pattern(BASE_PATH), include(astakos_patterns)),
)
# set utility redirects
extend_with_root_redirects(urlpatterns, settings.astakos_services,
extend_with_root_redirects(urlpatterns, astakos_services,
'astakos_ui', BASE_PATH)
......@@ -32,7 +32,7 @@
#Logo used in Storage pages (Pithos)
#BRANDING_STORAGE_LOGO_URL = '<path_to_image_folder>/storage_logo.png'
## Copyright options
## Copyright and footer options
######################
# If True, Copyright message will appear at the footer of the Compute and
......@@ -41,3 +41,7 @@
# Defaults to 'Copyright (c) 2011-<current_year> GRNET.'
#BRANDING_COPYRIGHT_MESSAGE = 'Copyright (c) 2011-<current_year> GRNET S.A.'
# Footer message appears above Copyright message at the Compute templates
# and the Dashboard UI. Accepts html tags
#BRANDING_FOOTER_EXTRA_MESSAGE = ''
......@@ -6,47 +6,50 @@ import datetime
######################
SERVICE_NAME = getattr(settings, 'BRANDING_SERVICE_NAME', 'Synnefo')
SERVICE_URL = getattr(settings, 'BRANDING_SERVICE_URL',
'http://www.synnefo.org/')
SERVICE_URL = getattr(settings, 'BRANDING_SERVICE_URL',
'http://www.synnefo.org/')
COMPANY_NAME = getattr(settings, 'BRANDING_COMPANY_NAME', 'GRNET')
COMPANY_URL = getattr(settings, 'BRANDING_COMPANY_URL',
'https://www.grnet.gr/en/')
COMPANY_URL = getattr(settings, 'BRANDING_COMPANY_URL',
'https://www.grnet.gr/en/')
## Images
######################
# The default path to the folder that contains all branding images
IMAGE_MEDIA_URL = getattr(settings, 'BRANDING_IMAGE_MEDIA_URL',
settings.MEDIA_URL+'branding/images/')
IMAGE_MEDIA_URL = getattr(settings, 'BRANDING_IMAGE_MEDIA_URL',
settings.MEDIA_URL+'branding/images/')
# The service favicon
# The service favicon
FAVICON_URL = getattr(settings, 'BRANDING_FAVICON_URL',
IMAGE_MEDIA_URL+'favicon.ico')
IMAGE_MEDIA_URL+'favicon.ico')
# Logo used in Dashboard pages (Astakos)
DASHBOARD_LOGO_URL = getattr(settings, 'BRANDING_DASHBOARD_LOGO_URL',
IMAGE_MEDIA_URL+'dashboard_logo.png')
DASHBOARD_LOGO_URL = getattr(settings, 'BRANDING_DASHBOARD_LOGO_URL',
IMAGE_MEDIA_URL+'dashboard_logo.png')
# Logo used in Compute pages (Cyclades)
COMPUTE_LOGO_URL = getattr(settings, 'BRANDING_COMPUTE_LOGO_URL',
IMAGE_MEDIA_URL+'compute_logo.png')
IMAGE_MEDIA_URL+'compute_logo.png')
# Logo used in Console page for VM (Cyclades)
CONSOLE_LOGO_URL = getattr(settings, 'BRANDING_CONSOLE_LOGO_URL',
IMAGE_MEDIA_URL+'console_logo.png')
IMAGE_MEDIA_URL+'console_logo.png')
# Logo used in Storage pages (Pithos)
STORAGE_LOGO_URL = getattr(settings, 'BRANDING_STORAGE_LOGO_URL',
IMAGE_MEDIA_URL+'storage_logo.png')
IMAGE_MEDIA_URL+'storage_logo.png')
## Copyright options
## Copyright and footer options
######################
# If True, Copyright message will appear at the footer of the Compute and
# Dashboard UI
SHOW_COPYRIGHT = getattr(settings, 'BRANDING_SHOW_COPYRIGHT', False)
# If True, Copyright message will appear at the footer of the Compute and
# Dashboard UI
SHOW_COPYRIGHT = getattr(settings, 'BRANDING_SHOW_COPYRIGHT', True)
copyright_period_default = '2011-%s' % (datetime.datetime.now().year)
copyright_message_default = 'Copyright (c) %s %s' % (copyright_period_default,
COMPANY_NAME)
copyright_message_default = 'Copyright (c) %s %s' % (copyright_period_default,
COMPANY_NAME)
# Defaults to Copyright (c) 2011-<current_year> GRNET.
COPYRIGHT_MESSAGE = getattr(settings, 'BRANDING_COPYRIGHT_MESSAGE',
copyright_message_default)
COPYRIGHT_MESSAGE = getattr(settings, 'BRANDING_COPYRIGHT_MESSAGE',
copyright_message_default)
SYNNEFO_VERSION = get_component_version('common')
# Footer message appears above Copyright message at the Compute templates
# and the Dashboard UI. Accepts html tags
FOOTER_EXTRA_MESSAGE = getattr(settings, 'BRANDING_FOOTER_EXTRA_MESSAGE', '')
......@@ -59,6 +59,3 @@ settings:
.. literalinclude:: ../synnefo/settings/default/admins.py
:lines: 4-
.. literalinclude:: ../synnefo/settings/default/logging.py
:lines: 4-
from log import LoggingConfigMiddleware
from secure import SecureMiddleware
from remoteaddr import RemoteAddrMiddleware
from cleanse import CleanseSettingsMiddleware
......@@ -47,7 +47,7 @@ def fill_endpoints(services, base_url):
if publicURL is not None:
continue
publicURL = join_urls(base_url, prefix, version).rstrip('/') + '/'
publicURL = join_urls(base_url, prefix, version).rstrip('/')
set_path(endpoint, 'publicURL', publicURL)
......@@ -84,4 +84,4 @@ def get_public_endpoint(services, service_type, version=None):
def get_service_path(services, service_type, version=None):
service_url = get_public_endpoint(services, service_type, version=version)
return urlparse(service_url).path.rstrip('/') + '/'
return urlparse(service_url).path.rstrip('/')
......@@ -63,4 +63,4 @@ class Command(BaseCommand):
delete_network(network, bnet.backend)
self.stdout.write("Successfully submitted Ganeti jobs to"
" remove network %s" % network.backend_id)
" remove network %s\n" % network.backend_id)
......@@ -97,6 +97,8 @@ class Command(BaseCommand):
image = common.get_image(image_id, user_id)
if backend_id:
backend = common.get_backend(backend_id)
else:
backend = None
do_create_server(user_id, name, password, flavor, image,
backend=backend)
......@@ -31,6 +31,7 @@
# interpreted as representing official policies, either expressed
# or implied, of GRNET S.A.
import datetime
from django import dispatch
from django.conf import settings
from django.conf.urls.defaults import patterns
......@@ -45,9 +46,9 @@ from synnefo.api import util
from synnefo.api.actions import server_actions
from synnefo.db.models import (VirtualMachine, VirtualMachineMetadata,
NetworkInterface)
from synnefo.logic.backend import create_instance, delete_instance
from synnefo.logic.backend import (create_instance, delete_instance,
process_op_status)
from synnefo.logic.utils import get_rsapi_state
from synnefo.logic.rapi import GanetiApiError
from synnefo.logic.backend_allocator import BackendAllocator
from synnefo import quotas
......@@ -359,23 +360,14 @@ def do_create_server(userid, name, password, flavor, image, metadata={},
flavor=flavor,
action="CREATE")
log.info("Created entry in DB for VM '%s'", vm)
# Create VM's public NIC. Do not wait notification form ganeti hooks to
# create this NIC, because if the hooks never run (e.g. building error)
# the VM's public IP address will never be released!
NetworkInterface.objects.create(machine=vm, network=network, index=0,
ipv4=address, state="BUILDING")
log.info("Created entry in DB for VM '%s'", vm)
# dispatch server created signal
server_created.send(sender=vm, created_vm_params={
'img_id': image['backend_id'],
'img_passwd': password,
'img_format': str(image['format']),
'img_personality': json.dumps(personality),
'img_properties': json.dumps(image['metadata']),
})
# Also we must create the VM metadata in the same transaction.
for key, val in metadata.items():
VirtualMachineMetadata.objects.create(
......@@ -393,6 +385,18 @@ def do_create_server(userid, name, password, flavor, image, metadata={},
transaction.commit()
try:
vm = VirtualMachine.objects.select_for_update().get(id=vm.id)
# dispatch server created signal needed to trigger the 'vmapi', which
# enriches the vm object with the 'config_url' attribute which must be
# passed to the Ganeti job.
server_created.send(sender=vm, created_vm_params={
'img_id': image['backend_id'],
'img_passwd': password,
'img_format': str(image['format']),
'img_personality': json.dumps(personality),
'img_properties': json.dumps(image['metadata']),
})
jobID = create_instance(vm, nic, flavor, image)
# At this point the job is enqueued in the Ganeti backend
vm.backendjobid = jobID
......@@ -400,19 +404,16 @@ def do_create_server(userid, name, password, flavor, image, metadata={},
transaction.commit()
log.info("User %s created VM %s, NIC %s, Backend %s, JobID %s",
userid, vm, nic, backend, str(jobID))
except GanetiApiError as e:
log.exception("Can not communicate to backend %s: %s.",
backend, e)
# Failed while enqueuing OP_INSTANCE_CREATE to backend. Restore
# already reserved quotas by issuing a negative commission
vm.operstate = "ERROR"
vm.backendlogmsg = "Can not communicate to backend."
vm.deleted = True
vm.save()
quotas.issue_and_accept_commission(vm, delete=True)
raise
except:
transaction.rollback()
# If an exception is raised, then the user will never get the VM id.
# In order to delete it from DB and release it's resources, we
# mock a successful OP_INSTANCE_REMOVE job.
process_op_status(vm=vm,
etime=datetime.datetime.now(),
jobid=-0,
opcode="OP_INSTANCE_REMOVE",
status="success",
logmsg="Reconciled eventd: VM creation failed.")
raise
return vm
......
......@@ -40,6 +40,7 @@ from synnefo.logic.utils import get_rsapi_state
from synnefo.cyclades_settings import cyclades_services
from synnefo.lib.services import get_service_path
from synnefo.lib import join_urls
from synnefo.logic.rapi import GanetiApiError
from mock import patch
......@@ -303,6 +304,45 @@ class ServerCreateAPITest(ComputeAPITest):
json.dumps(request), 'json')
self.assertItemNotFound(response)
def test_create_server_error(self, mrapi, mimage):
"""Test if the create server call returns the expected response
if a valid request has been speficied."""
mimage.return_value = {'location': 'pithos://foo',
'checksum': '1234',
"id": 1,
"name": "test_image",
'disk_format': 'diskdump'}
mrapi().CreateInstance.side_effect = GanetiApiError("..ganeti is down")
flavor = mfactory.FlavorFactory()
# Create public network and backend
network = mfactory.NetworkFactory(public=True)
backend = mfactory.BackendFactory()
mfactory.BackendNetworkFactory(network=network, backend=backend)
request = {
"server": {
"name": "new-server-test",
"userid": "test_user",
"imageRef": 1,
"flavorRef": flavor.id,
"metadata": {
"My Server Name": "Apache1"
},
"personality": []
}
}
with mocked_quotaholder():
response = self.mypost('servers', 'test_user',
json.dumps(request), 'json')
self.assertEqual(response.status_code, 500)
mrapi().CreateInstance.assert_called_once()
vm = VirtualMachine.objects.get()
# The VM has been deleted
self.assertTrue(vm.deleted)
# and it has no nics
self.assertEqual(len(vm.nics.all()), 0)
self.assertEqual(vm.backendjobid, 0)
@patch('synnefo.logic.rapi_pool.GanetiRapiClient')
class ServerDestroyAPITest(ComputeAPITest):
......
......@@ -45,7 +45,7 @@ class APITest(TestCase):
path = get_service_path(cyclades_services,
'compute', version='v2.0')
with astakos_user('user'):
response = self.client.get(path)
response = self.client.get(path.rstrip('/') + '/')
self.assertEqual(response.status_code, 200)
api_version = json.loads(response.content)['version']
self.assertEqual(api_version['id'], 'v2.0')
......
......@@ -11,7 +11,8 @@ synnefo_web_apps = [
]
synnefo_web_middleware = []
synnefo_web_context_processors = ['synnefo.lib.context_processors.cloudbar']
synnefo_web_context_processors = \
['synnefo.webproject.context_processors.cloudbar']
synnefo_static_files = {
'synnefo.ui': 'ui/static',
......
......@@ -35,7 +35,8 @@ from django.conf.urls.defaults import *
from django.conf import settings
from snf_django.lib.api.proxy import proxy
from snf_django.lib.api.utils import prefix_pattern
from snf_django.utils.urls import extend_with_root_redirects
from snf_django.utils.urls import \
extend_with_root_redirects, extend_endpoint_with_slash
from snf_django.lib.api.urls import api_patterns
from synnefo.cyclades_settings import (
BASE_URL, BASE_HOST, BASE_PATH, COMPUTE_PREFIX, VMAPI_PREFIX,
......@@ -47,6 +48,14 @@ from synnefo.cyclades_settings import (
from functools import partial
urlpatterns = []
# Redirects should be first, otherwise they may get overridden by wildcards
extend_endpoint_with_slash(urlpatterns, cyclades_services, 'cyclades_ui')
extend_endpoint_with_slash(urlpatterns, cyclades_services, 'cyclades_helpdesk')
extend_endpoint_with_slash(urlpatterns, cyclades_services, 'admin')
extend_endpoint_with_slash(urlpatterns, cyclades_services, 'cyclades_userdata')
astakos_proxy = partial(proxy, proxy_base=BASE_ASTAKOS_PROXY_PATH,
target_base=ASTAKOS_BASE_URL)
......@@ -63,7 +72,7 @@ cyclades_patterns += patterns('',
(prefix_pattern(HELPDESK_PREFIX), include('synnefo.helpdesk.urls')),
)
urlpatterns = patterns(
urlpatterns += patterns(
'',
(prefix_pattern(BASE_PATH), include(cyclades_patterns)),
)
......
......@@ -743,7 +743,7 @@ class IPPoolTable(PoolTable):
@contextmanager
def pooled_rapi_client(obj):
if isinstance(obj, VirtualMachine):
if isinstance(obj, (VirtualMachine, BackendNetwork)):
backend = obj.backend
else:
backend = obj
......
......@@ -185,7 +185,7 @@ def account(request, search_query):
if is_ip:
try:
nic = NetworkInterface.objects.get(ipv4=search_query)
nic = NetworkInterface.objects.get(ipv4=search_query).exclude(machine__deleted=True)
search_query = nic.machine.userid
is_uuid = True
except NetworkInterface.DoesNotExist:
......
# Copyright 2011 GRNET S.A. All rights reserved.
# Copyright 2011-2013 GRNET S.A. All rights reserved.
#
# Redistribution and use in source and binary forms, with or
# without modification, are permitted provided that the following
......@@ -41,6 +41,7 @@ from synnefo.logic import utils
from synnefo import quotas
from synnefo.api.util import release_resource
from synnefo.util.mac2eui64 import mac2eui64
from synnefo.logic.rapi import GanetiApiError
from logging import getLogger
log = getLogger(__name__)
......@@ -73,6 +74,13 @@ def process_op_status(vm, etime, jobid, opcode, status, logmsg, nics=None):
vm.backendopcode = opcode
vm.backendlogmsg = logmsg
# Update backendtime only for jobs that have been successfully completed,
# since only these jobs update the state of the VM. Else a "race condition"
# may occur when a successful job (e.g. OP_INSTANCE_REMOVE) completes
# before an error job and messages arrive in reversed order.
if status == 'success':
vm.backendtime = etime
# Notifications of success change the operating state
state_for_success = VirtualMachine.OPER_STATE_FROM_OPCODE.get(opcode, None)
if status == 'success' and state_for_success is not None:
......@@ -87,27 +95,21 @@ def process_op_status(vm, etime, jobid, opcode, status, logmsg, nics=None):
vm.operstate = 'ERROR'
vm.backendtime = etime
elif opcode == 'OP_INSTANCE_REMOVE':
# Set the deleted flag explicitly, cater for admin-initiated removals
# Special case: OP_INSTANCE_REMOVE fails for machines in ERROR,
# when no instance exists at the Ganeti backend.
# See ticket #799 for all the details.
#
if (status == 'success' or
(status == 'error' and (vm.operstate == 'ERROR' or
vm.action == 'DESTROY'))):
if status == "success" or (status == "error" and