Skip to content

GitLab

Commit c7b2f023 authored by Kostas Papadimitriou's avatar Kostas Papadimitriou
Browse files

Merge remote-tracking branch 'origin/master' into ui-refactor

parents b6169371 893a4ebc
Hide whitespace changes
Inline Side-by-side
README.deploy
View file @ c7b2f023
......@@ -4,7 +4,7 @@ This document describes the basic steps to obtain a basic, working Synnefo
deployment. It begins by examining the different node roles, then moves to the
installation and setup of distinct software components.
It is current as of Synnefo v0.5.2.
It is current as of Synnefo v0.7.
Node types
......@@ -430,60 +430,19 @@ Synnefo deployment.
sudo pip install -e git+https://code.grnet.gr/git/vncauthproxy@5a196d8481e171a#egg=vncauthproxy
13. Installation of the customized Ganeti Instance Image for image deployment:
13. Installation of the snf-image Ganeti OS provider for image deployment:
For Synnefo to be able to launch VMs from specified Images, you need
the gnt-instance-image OS Provider installed on *all* Ganeti nodes.
There are 2 different ways to install gnt-instance-image on a node:
1.As a debian package (recommended)
Download and install the debian package found here:
https://code.grnet.gr/attachments/download/351/ganeti-instance-image_0.5.1-1-snf1_all.deb
You can do this by running the following:
$ wget https://code.grnet.gr/attachments/download/351/ganeti-instance-image_0.5.1-1-snf1_all.deb
# dpkg -i ganeti-instance-image_0.5.1-1-snf1_all.deb
Make any additional configuration changes in
/etc/default/ganeti-instance-image.
It is recommended to change the default IMAGE_DIR from
/var/cache/ganeti-instance-image to something like /srv/example_images_repo
Be sure to have all the package's needed dependencies installed
on your system.
2.From source
Download and install gnt-instance-image from its own repository,
at https://code.grnet.gr/git/gnt-instance-image.
[FIXME: Must be updated to include information on installation of snf-image]
Make sure to enable progress monitoring, using the --with-progress-monitor
argument to configure. This requires the snf-progress-monitor tool,
provided in snf-ganeti-tools/ and also as part of the snf-ganeti-tools
Debian package.
After installing gnt-instance-image do the following:
1. $ cd /path-to-repo
# cp ./defaults /etc/default/ganeti-instance-image
2. In /etc/ganeti/instance-image/hooks, make sure the hooks you want to
run during the instance creation process have execute permission.
For linux you will need at lease `grub' and `root_passwd' to make the
instance usable:
chmod +x /etc/ganeti/instance-image/hooks/linux/{grub,root_passwd}
For security reasons make sure `ssh' hook is also enabled.
For windows you will need `mbr' and `admin_passwd':
chmod +x /etc/ganeti/instance-image/hooks/windows/{mbr,admin_passwd}
For both architectures it is also highly recommended to enable the
`hostname' hook too:
chmod +x /et/ganeti/instance-image/hooks/{linux,windows}/hostname
Your custom Images should be stored in a dump format under
/var/cache/ganeti-instance-image (default) or a different directory of your
choice, accordingly set in /etc/default/ganeti-instance-image. The latter
is recommended. Their filenames should have the following format:
{backend_id}-x86_64-root.dump
e.g., debian-6.0.1a-x86_64-root.dump (backend_id = "debian-6.0.1a")
the snf-image OS Provider installed on *all* Ganeti nodes.
Please see https://code.grnet.gr/projects/snf-image/wiki
for installation instructions and documentation on the design
and implementation of snf-image.
Please see https://code.grnet.gr/projects/snf-image/files
for the latest packages.
Images should be stored under extdump format in a directory
of your choice, configurable as IMAGE_DIR in /etc/default/snf-image.
14. Setup Synnefo-specific networking on the Ganeti backend:
......
README.upgrade
View file @ c7b2f023
......@@ -5,7 +5,7 @@ This file documents the upgrade to newer versions of the Synnefo software.
For more information, please see README.deploy.
v0.6.2 -> TBD
v0.6.2 -> v0.7
HTML TEMPLATES
* Included a generic service unavailable template based on
generic_info_tpl.html, as ui/static/service_unavailable.html.
......@@ -14,8 +14,17 @@ NEW DEPENDENCIES
* python-unittest2, python-paramiko needed by the new integration
test suite snf-tools/snf-test. Paramiko shipped with Squeeze uses
broken RandomPool, produces warnings, see #576697.
* snf-image must be installed on all Ganeti hosts, see README.deploy,
Step 13.
NEW SETTINGS
COMPONENTS
* snf-image replaces snf-ganeti-instance-image as the Ganeti OS provider
used by Synnefo, and can live alongside snf-ganeti-instance-image.
Once snf-image has been deployed on all Ganeti nodes, be sure to modify
the value of settings.d/10-backend.conf:GANETI_CREATEINSTANCE_KWARGS
to use 'snf-image+default'.
NEW/UPDATED SETTINGS
* EMAIL_SUBJECT_PREFIX:
Prepended to automated emails, set to someting that uniquely identifies
the deployment.
......@@ -38,6 +47,8 @@ NEW SETTINGS
error overlay.
* UI_HANDLE_WINDOW_EXCEPTIONS in 30-ui.conf:
Whether UI should display error overlay for all Javascript exceptions.
* GANETI_CREATEINSTANCE_KWARGS in 10-backend.conf:
Must be updated to use snf-image as the Ganeti OS provider.
DB MIGRATION
* Uniqueness constraints have been added to the metadata models.
......
invitations/invitations.py
View file @ c7b2f023
......@@ -50,11 +50,11 @@ from synnefo.logic.email_send import send_async, send
from synnefo.api.common import method_not_allowed
from synnefo.db.models import Invitations, SynnefoUser
from synnefo.logic import users
from synnefo.util import log
from synnefo.util.log import getLogger
from Crypto.Cipher import AES
_logger = log.getLogger("synnefo.invitations")
log = getLogger('synnefo.invitations')
def process_form(request):
......@@ -82,7 +82,7 @@ def process_form(request):
(name, email, e.messages[0])]
except Exception as e:
remove_invitation(invitation)
_logger.exception(e)
log.exception(e)
errors += ["Invitation to %s <%s> could not be sent. An unexpected"
" error occurred. Please try again later." %
(name, email)]
......@@ -98,8 +98,8 @@ def process_form(request):
get_invitations_left(request.user)},
context_instance=RequestContext(request))
response = HttpResponse(data)
_logger.warn("Error adding invitation %s -> %s: %s" %
(request.user.uniq, email, errors))
log.warn("Error adding invitation %s -> %s: %s",
request.user.uniq, email, errors)
else:
# form submitted
data = render_to_string('invitations.html',
......@@ -109,7 +109,7 @@ def process_form(request):
get_invitations_left(request.user)},
context_instance=RequestContext(request))
response = HttpResponse(data)
_logger.info("Added invitation %s -> %s" % (request.user.uniq, email))
log.info("Added invitation %s -> %s", request.user.uniq, email)
return response
......@@ -215,7 +215,7 @@ def login(request):
inv.accepted = True
inv.save()
_logger.info("Invited user %s logged in", inv.target.uniq)
log.info("Invited user %s logged in", inv.target.uniq)
data = dict()
data['user'] = user.realname
......@@ -256,7 +256,7 @@ def send_invitation(invitation):
data = render_to_string('invitation.txt', {'email': email})
_logger.debug("Invitation URL: %s" % email['url'])
log.debug("Invitation URL: %s", email['url'])
# send_async(
# frm = "%s"%(settings.DEFAULT_FROM_EMAIL),
......@@ -311,7 +311,7 @@ def resend(request):
try:
send_invitation(inv)
except Exception as e:
_logger.exception(e)
log.exception(e)
return HttpResponseServerError("Error sending invitation email")
return HttpResponse("Invitation has been resent")
......
logic/backend.py
View file @ c7b2f023
......@@ -236,11 +236,11 @@ def create_instance(vm, flavor, image, password, personality):
kw = settings.GANETI_CREATEINSTANCE_KWARGS
kw['mode'] = 'create'
kw['name'] = vm.backend_id
# Defined in settings.GANETI_CREATE_INSTANCE_KWARGS
# Defined in settings.GANETI_CREATEINSTANCE_KWARGS
# kw['disk_template'] = settings.GANETI_DISK_TEMPLATE
kw['disks'] = [{"size": sz}]
kw['nics'] = [nic]
# Defined in settings.GANETI_CREATE_INSTANCE_KWARGS
# Defined in settings.GANETI_CREATEINSTANCE_KWARGS
# kw['os'] = settings.GANETI_OS_PROVIDER
kw['ip_check'] = False
kw['name_check'] = False
......@@ -262,7 +262,7 @@ def create_instance(vm, flavor, image, password, personality):
if personality:
kw['osparams']['img_personality'] = json.dumps(personality)
# Defined in settings.GANETI_CREATE_INSTANCE_KWARGS
# Defined in settings.GANETI_CREATEINSTANCE_KWARGS
# kw['hvparams'] = dict(serial_console=False)
return rapi.CreateInstance(**kw)
......
logic/callbacks.py
View file @ c7b2f023
......@@ -30,26 +30,28 @@
# Callback functions used by the dispatcher to process incoming notifications
# from AMQP queues.
import logging
import socket
import traceback
import json
import sys
from synnefo.db.models import VirtualMachine
from synnefo.logic import utils, backend, email_send, log
from synnefo.logic import utils, backend, email_send
_logger = log.get_logger("synnefo.dispatcher")
log = logging.getLogger()
def update_db(message):
"""Process a notification of type 'ganeti-op-status'"""
_logger.debug("Processing ganeti-op-status msg: %s", message.body)
log.debug("Processing ganeti-op-status msg: %s", message.body)
msg = None
try:
msg = json.loads(message.body)
if msg["type"] != "ganeti-op-status":
_logger.error("Message is of unknown type %s.", msg["type"])
log.error("Message is of unknown type %s.", msg["type"])
return
if msg["operation"] == "OP_INSTANCE_QUERY_DATA":
......@@ -60,53 +62,51 @@ def update_db(message):
backend.process_op_status(vm, msg["jobId"], msg["operation"],
msg["status"], msg["logmsg"])
_logger.debug("Done processing ganeti-op-status msg for vm %s.",
log.debug("Done processing ganeti-op-status msg for vm %s.",
msg["instance"])
message.channel.basic_ack(message.delivery_tag)
except KeyError:
_logger.error("Malformed incoming JSON, missing attributes: %s",
log.error("Malformed incoming JSON, missing attributes: %s",
message.body)
except VirtualMachine.InvalidBackendIdError:
_logger.debug("Ignoring msg for unknown instance %s.",
msg["instance"])
log.debug("Ignoring msg for unknown instance %s.", msg["instance"])
except VirtualMachine.InvalidBackendMsgError, e:
_logger.debug("Ignoring msg of unknown type: %s.", e)
log.debug("Ignoring msg of unknown type: %s.", e)
except VirtualMachine.DoesNotExist:
_logger.error("VM for instance %s with id %d not found in DB.",
log.error("VM for instance %s with id %d not found in DB.",
msg["instance"], vmid)
except Exception as e:
_logger.exception("Unexpected error, msg: %s", msg)
log.exception("Unexpected error, msg: %s", msg)
def update_net(message):
"""Process a notification of type 'ganeti-net-status'"""
_logger.debug("Processing ganeti-net-status msg: %s", message.body)
log.debug("Processing ganeti-net-status msg: %s", message.body)
msg = None
try:
msg = json.loads(message.body)
if msg["type"] != "ganeti-net-status":
_logger.error("Message is of unknown type %s", msg["type"])
log.error("Message is of unknown type %s", msg["type"])
return
vmid = utils.id_from_instance_name(msg["instance"])
vm = VirtualMachine.objects.get(id=vmid)
backend.process_net_status(vm, msg["nics"])
_logger.debug("Done processing ganeti-net-status msg for vm %s.",
log.debug("Done processing ganeti-net-status msg for vm %s.",
msg["instance"])
message.channel.basic_ack(message.delivery_tag)
except KeyError:
_logger.error("Malformed incoming JSON, missing attributes: %s",
log.error("Malformed incoming JSON, missing attributes: %s",
message.body)
except VirtualMachine.InvalidBackendIdError:
_logger.debug("Ignoring msg for unknown instance %s.",
msg["instance"])
log.debug("Ignoring msg for unknown instance %s.", msg["instance"])
except VirtualMachine.DoesNotExist:
_logger.error("VM for instance %s with id %d not found in DB.",
log.error("VM for instance %s with id %d not found in DB.",
msg["instance"], vmid)
except Exception as e:
_logger.exception("Unexpected error, msg: %s", msg)
log.exception("Unexpected error, msg: %s", msg)
def send_email(message):
......@@ -119,33 +119,33 @@ def send_email(message):
body=msg['body'], subject=msg['subject'])
if not sent:
_logger.warn("Failed to send email to %s", msg['to'])
log.warn("Failed to send email to %s", msg['to'])
else:
message.channel.basic_ack(message.delivery_tag)
except KeyError:
_logger.error("Malformed incoming JSON, missing attributes: %s",
log.error("Malformed incoming JSON, missing attributes: %s",
message.body)
except socket.error as e:
_logger.error("Cannot connect to SMTP server:%s\n", e)
log.error("Cannot connect to SMTP server:%s\n", e)
except Exception as e:
_logger.exception("Unexpected error, msg: %s", msg)
log.exception("Unexpected error, msg: %s", msg)
raise
def update_credits(message):
_logger.debug("Request to update credits")
log.debug("Request to update credits")
message.channel.basic_ack(message.delivery_tag)
def update_build_progress(message):
"""Process a create progress message"""
_logger.debug("Processing ganeti-create-progress msg: %s", message.body)
log.debug("Processing ganeti-create-progress msg: %s", message.body)
msg = None
try:
msg = json.loads(message.body)
if msg['type'] != "ganeti-create-progress":
_logger.error("Message is of unknown type %s", msg["type"])
log.error("Message is of unknown type %s", msg["type"])
return
# XXX: The following assumes names like snf-12
......@@ -153,30 +153,30 @@ def update_build_progress(message):
vm = VirtualMachine.objects.get(id=vmid)
backend.process_create_progress(vm, msg['rprogress'], None)
_logger.debug("Done processing ganeti-create-progress msg for vm %s.",
log.debug("Done processing ganeti-create-progress msg for vm %s.",
msg["instance"])
message.channel.basic_ack(message.delivery_tag)
except KeyError:
_logger.error("Malformed incoming JSON, missing attributes: %s",
log.error("Malformed incoming JSON, missing attributes: %s",
message.body)
except Exception as e:
_logger.exception("Unexpected error, msg: %s", msg)
log.exception("Unexpected error, msg: %s", msg)
raise
def trigger_status_update(message):
"""Triggers a status update job for a specific VM id"""
_logger.debug("Request to trigger status update: %s", message.body)
log.debug("Request to trigger status update: %s", message.body)
msg = None
try:
msg = json.loads(message.body)
if msg["type"] != "reconcile":
_logger.error("Message is of unknown type %s", msg["type"])
log.error("Message is of unknown type %s", msg["type"])
return
if msg["vmid"] == "":
_logger.error("Reconciliation message does not specify a VM id")
log.error("Reconciliation message does not specify a VM id")
return
vm = VirtualMachine.objects.get(id=msg["vmid"])
......@@ -184,9 +184,9 @@ def trigger_status_update(message):
message.channel.basic_ack(message.delivery_tag)
except KeyError as k:
_logger.error("Malformed incoming JSON, missing attributes: %s", k)
log.error("Malformed incoming JSON, missing attributes: %s", k)
except Exception as e:
_logger.exception("Unexpected error, msg: %s", msg)
log.exception("Unexpected error, msg: %s", msg)
def status_job_finished(message):
......@@ -196,21 +196,22 @@ def status_job_finished(message):
msg = json.loads(message.body)
if msg["operation"] != 'OP_INSTANCE_QUERY_DATA':
_logger.error("Message is of unknown type %s", msg["operation"])
log.error("Message is of unknown type %s", msg["operation"])
return
if msg["status"] != "success":
_logger.warn("Ignoring non-success status update from job %d on VM %s",
log.warn("Ignoring non-success status update from job %d on VM %s",
msg['jobId'], msg['instance'])
message.channel.basic_ack(message.delivery_tag)
return
status = backend.get_job_status(msg['jobId'])
_logger.debug("Node status job result: %s" % status)
log.debug("Node status job result: %s", status)
if status['summary'][0] != u'INSTANCE_QUERY_DATA':
_logger.error("Status update is of unknown type %s", status['summary'])
log.error("Status update is of unknown type %s",
status['summary'])
return
conf_state = status['opresult'][0][msg['instance']]['config_state']
......@@ -232,15 +233,15 @@ def status_job_finished(message):
message.channel.basic_ack(message.delivery_tag)
except KeyError as k:
_logger.error("Malformed incoming JSON, missing attributes: %s", k)
log.error("Malformed incoming JSON, missing attributes: %s", k)
except Exception as e:
_logger.exception("Unexpected error, msg: %s", msg)
log.exception("Unexpected error, msg: %s", msg)
def dummy_proc(message):
try:
_logger.debug("Msg: %s", message.body)
log.debug("Msg: %s", message.body)
message.channel.basic_ack(message.delivery_tag)
except Exception as e:
_logger.exception("Could not receive message")
log.exception("Could not receive message")
pass
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment