Commit 50795600 authored by Nikos Skalkotos's avatar Nikos Skalkotos

Merge branch 'develop'

parents 1cd84cb6 9228aff8
#!/usr/bin/env sh
set -e
PACKAGES_DIR=$1
shift
TEMP_DIR=$(mktemp -d /tmp/devflow_autopkg_XXXXXXX)
# Create the packages
devflow-autopkg snapshot -b $TEMP_DIR $@
# MOVE the packages
mkdir -p $PACKAGES_DIR
mv -n $TEMP_DIR/* $PACKAGES_DIR
echo "Moved packages to: $(pwd)/$PACKAGES_DIR"
#!/bin/sh
set -e
DOCS_DIR=$1
cd docs
make html
cd -
mkdir -p $DOCS_DIR
mv -n docs/_build/html/* $DOCS_DIR
echo "Moved docs to to: $(pwd)/$DOCS_DIR"
#!/bin/sh
pep8 image_creator
!/bin/sh
pylint image_creator
......@@ -32,11 +32,41 @@ method you choose. There are two installation methods available:
Install snf-image-creator using packages
========================================
Debian
------
For *Debian 7.0 (wheezy)* you can use our official packages found in our
development repository.
Add the following line to */etc/apt/sources.list*:
.. code-block:: console
deb http://apt.dev.grnet.gr wheezy/
And resynchronize the package index files from their sources:
.. code-block:: console
$ sudo apt-get update
You should be able to list the package by calling:
.. code-block:: console
$ apt-cache showpkg snf-image-creator
And install the package with this command:
.. code-block:: console
$ apt-get install snf-image-creator
Ubuntu
------
For *Ubuntu 12.04 LTS* and *12.10* systems, you can use our official packages
found in *grnet/synnefo* Lauchpad PPA.
For *Ubuntu 12.04 LTS*, *12.10* and *13.04* systems, you can use our official
packages found in *grnet/synnefo* Lauchpad PPA.
Add the synnefo PPA in your system:
......@@ -47,7 +77,7 @@ Add the synnefo PPA in your system:
If *apt-add-repository* is missing, first install:
*software-properties-common* (Ubuntu 12.10):
*software-properties-common* (Ubuntu 12.10 & 13.04):
.. code-block:: console
......@@ -76,14 +106,6 @@ Install the package by issuing:
If you are asked during the installation to create/update a
"supermin appliance", choose "Yes".
.. warning::
In *Ubuntu 12.10* the current package of libguestfs (1.18-2) is broken. Take
a look at the open `bug report <https://bugs.launchpad.net/ubuntu/quantal/+source/libguestfs/+bug/1086974>`_.
Until version 1.18-2ubunut1 is out, you may workaround this problem by
creating a symlink like this:
*sudo ln -s /usr/lib/guestfs /usr/lib/x86_64-linux-gnu/guestfs*
Fedora
------
......@@ -113,7 +135,7 @@ CentOS
------
For *CentOS 6* you can use our official packages hosted at the *synnefo*
repository of the openSUSE Build Service.
repository of the OpenSUSE Build Service.
Add the *synnefo* repository for *CentOS 6* to the yum repositories list:
......@@ -124,6 +146,38 @@ Add the *synnefo* repository for *CentOS 6* to the yum repositories list:
Check the `Fedora <#fedora>`_ instructions on how to install the software.
OpenSUSE
--------
For *OpenSUSE 12.3* you can use our official packages hosted at the *synnefo*
repository of the OpenSUSE Build Service.
Add the *Virtualization* repository for *OpenSUSE 12.3* to *YaST* with the
*Zypper* package manager:
.. code-block:: console
$ zypper ar -f http://download.opensuse.org/repositories/Virtualization/openSUSE_12.3/Virtualization.repo
Add the *synnefo* repository:
.. code-block:: console
$ zypper ar -f http://download.opensuse.org/repositories/home:/GRNET:/synnefo/openSUSE_12.3/home:GRNET:synnefo.repo
To list the *snf-image-creator* package use the following command:
.. code-block:: console
$ zypper se snf-image-creator
Install the package by issuing:
.. code-block:: console
$ zypper in snf-image-creator
Arch Linux
----------
......
......@@ -35,6 +35,7 @@ deployment as private or public image.
Image Format
============
The extracted images are in diskdump format. This is the recommended format for
The extracted images are in diskdump format, which is a raw dump of a disk
device (or file). This is the recommended format for
`snf-image <https://code.grnet.gr/projects/snf-image>`_, the Ganeti OS
Definition used by `Synnefo <https://code.grnet.gr/projects/synnefo>`_.
......@@ -17,42 +17,47 @@ snf-image-creator receives the following options:
.. code-block:: console
$ snf-image-creator --help
Usage: snf-image-creator [options] <input_media>
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-o FILE, --outfile=FILE
dump image to FILE
-f, --force overwrite output files if they exist
-s, --silent output only errors
-u FILENAME, --upload=FILENAME
upload the image to the storage service with name FILENAME
-r IMAGENAME, --register=IMAGENAME
register the image with the compute service as IMAGENAME
-m KEY=VALUE, --metadata=KEY=VALUE
add custom KEY=VALUE metadata to the image
-t TOKEN, --token=TOKEN
use this authentication token when
uploading/registering images
-a URL, --authentication-url=URL
use this authentication URL when uploading/registering
images
-c CLOUD, --cloud=CLOUD
use this saved cloud account to authenticate against a
cloud when uploading/registering images
--print-sysprep print the enabled and disabled system preparation
operations for this input media
--enable-sysprep=SYSPREP
run SYSPREP operation on the input media
--disable-sysprep=SYSPREP
prevent SYSPREP operation from running on the input
media
--no-sysprep don't perform any system preparation operation
--no-shrink don't shrink any partition
--public register image with the compute service as public
--tmpdir=DIR create large temporary image files under DIR
$ snf-image-creator --help
Usage: snf-image-creator [options] <input_media>
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-o FILE, --outfile=FILE
dump image to FILE
-f, --force overwrite output files if they exist
-s, --silent output only errors
-u FILENAME, --upload=FILENAME
upload the image to the cloud with name FILENAME
-r IMAGENAME, --register=IMAGENAME
register the image with a cloud as IMAGENAME
-m KEY=VALUE, --metadata=KEY=VALUE
add custom KEY=VALUE metadata to the image
-t TOKEN, --token=TOKEN
use this authentication token when
uploading/registering images
-a URL, --authentication-url=URL
use this authentication URL when uploading/registering
images
-c CLOUD, --cloud=CLOUD
use this saved cloud account to authenticate against a
cloud when uploading/registering images
--print-syspreps print the enabled and disabled system preparation
operations for this input media
--enable-sysprep=SYSPREP
run SYSPREP operation on the input media
--disable-sysprep=SYSPREP
prevent SYSPREP operation from running on the input
media
--print-sysprep-params
print the needed sysprep parameters for this input
media
--sysprep-param=SYSPREP_PARAMS
Add KEY=VALUE system preparation parameter
--no-sysprep don't perform any system preparation operation
--no-shrink don't shrink any partition
--public register image with the cloud as public
--tmpdir=DIR create large temporary image files under DIR
Most input options are self-describing. If you want to save a local copy of
the image you create, provide a filename using the *-o* option. To upload the
......@@ -298,10 +303,11 @@ Limitations
Supported operating systems
---------------------------
*snf-image-creator* can only fully function on input media hosting *Linux*
systems. The program will detect the needed metadata and you may use it to
upload and register other *Unix* or *Windows* images, but you cannot use it to
shrink them or perform system preparation operations.
*snf-image-creator* can only fully function on input media hosting *Linux*,
*FreeBSD* (tested on version 9.1) and *Windows* (Server 2008 R2 and Server
2012) systems. The program will detect the needed metadata and you may use it
to upload and register other *Unix* images, but you cannot use it to shrink
them or perform system preparation operations.
Logical Volumes
---------------
......@@ -330,6 +336,13 @@ if a system can boot with para-virtualized disk controller by launching it with
kvm using the *if=virtio* option (see the kvm command in the
`Creating a new image`_ section).
For Windows and FreeBSD systems, the needed drivers need to be manually
downloaded and installed on the media before the image creation process takes
place. For *FreeBSD* the virtio drivers can be found
`here <http://people.freebsd.org/~kuriyama/virtio/>`_. For Windows the drivers
are hosted by the
`Fedora Project <http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/>`_.
Some caveats on image creation
==============================
......
......@@ -267,8 +267,11 @@ class BundleVolume(object):
name = os.path.basename(dev) + "_" + uuid.uuid4().hex
tablefd, table = tempfile.mkstemp()
try:
size = end - start + 1
os.write(tablefd, "0 %d linear %s %d" % (size, dev, start))
try:
size = end - start + 1
os.write(tablefd, "0 %d linear %s %d" % (size, dev, start))
finally:
os.close(tablefd)
dmsetup('create', "%sp%d" % (name, num), table)
finally:
os.unlink(table)
......@@ -301,7 +304,7 @@ class BundleVolume(object):
mpoints = []
for entry in self._read_fstable('/proc/mounts'):
if entry.mpoint.startswith(os.path.abspath(target)):
mpoints.append(entry.mpoint)
mpoints.append(entry.mpoint)
mpoints.sort()
for mpoint in reversed(mpoints):
......@@ -333,10 +336,9 @@ class BundleVolume(object):
continue
dirname = mpoint
basename = ''
found_ancestor = False
while dirname != '/':
(dirname, basename) = os.path.split(dirname)
(dirname, _) = os.path.split(dirname)
if dirname in excluded:
found_ancestor = True
break
......
......@@ -68,6 +68,8 @@ CONFIGURATION_TASKS = [
("File injection", ["EnforcePersonality"], ["windows", "linux"])
]
SYSPREP_PARAM_MAXLEN = 20
class MetadataMonitor(object):
"""Monitors image metadata chages"""
......@@ -157,7 +159,7 @@ def upload_image(session):
session['upload'] = filename
break
gauge = GaugeOutput(d, "Image Upload", "Uploading...")
gauge = GaugeOutput(d, "Image Upload", "Uploading ...")
try:
out = image.out
out.add(gauge)
......@@ -175,7 +177,7 @@ def upload_image(session):
"Calculating block hashes",
"Uploading missing blocks")
# Upload md5sum file
out.output("Uploading md5sum file...")
out.output("Uploading md5sum file ...")
md5str = "%s %s\n" % (session['checksum'], filename)
kamaki.upload(StringIO.StringIO(md5str), size=len(md5str),
remote_path="%s.md5sum" % filename)
......@@ -243,26 +245,27 @@ def register_image(session):
metadata[key] = 'yes'
img_type = "public" if is_public else "private"
gauge = GaugeOutput(d, "Image Registration", "Registering image...")
gauge = GaugeOutput(d, "Image Registration", "Registering image ...")
try:
out = session['image'].out
out.add(gauge)
try:
try:
out.output("Registering %s image with the cloud..." % img_type)
out.output("Registering %s image with the cloud ..." %
img_type)
kamaki = Kamaki(session['account'], out)
result = kamaki.register(name, session['pithos_uri'], metadata,
is_public)
out.success('done')
# Upload metadata file
out.output("Uploading metadata file...")
out.output("Uploading metadata file ...")
metastring = unicode(json.dumps(result, ensure_ascii=False))
kamaki.upload(StringIO.StringIO(metastring),
size=len(metastring),
remote_path="%s.meta" % session['upload'])
out.success("done")
if is_public:
out.output("Sharing metadata and md5sum files...")
out.output("Sharing metadata and md5sum files ...")
kamaki.share("%s.meta" % session['upload'])
kamaki.share("%s.md5sum" % session['upload'])
out.success('done')
......@@ -396,9 +399,9 @@ def kamaki_menu(session):
if len(Kamaki.get_clouds()):
default_item = "Cloud"
else:
default_time = "Add/Edit"
default_item = "Add/Edit"
else:
default_time = "Delete"
default_item = "Delete"
elif choice == "Cloud":
default_item = "Cloud"
clouds = Kamaki.get_clouds()
......@@ -618,6 +621,75 @@ def exclude_tasks(session):
return True
def sysprep_params(session):
"""Collect the needed sysprep parameters"""
d = session['dialog']
image = session['image']
available = image.os.sysprep_params
needed = image.os.needed_sysprep_params
if len(needed) == 0:
return True
def print_form(names, extra_button=False):
"""print the dialog form providing sysprep_params"""
fields = []
for name in names:
param = needed[name]
default = str(available[name]) if name in available else ""
fields.append(("%s: " % param.description, default,
SYSPREP_PARAM_MAXLEN))
kwargs = {}
if extra_button:
kwargs['extra_button'] = 1
kwargs['extra_label'] = "Advanced"
txt = "Please provide the following system preparation parameters:"
return d.form(txt, height=13, width=WIDTH, form_height=len(fields),
fields=fields, **kwargs)
def check_params(names, values):
"""check if the provided sysprep parameters have leagal values"""
for i in range(len(names)):
param = needed[names[i]]
try:
normalized = param.type(values[i])
if param.validate(normalized):
image.os.sysprep_params[names[i]] = normalized
continue
except ValueError:
pass
d.msgbox("Invalid value for parameter: `%s'" % names[i],
width=SMALL_WIDTH)
return False
return True
simple_names = [k for k, v in needed.items() if v.default is None]
advanced_names = [k for k, v in needed.items() if v.default is not None]
while 1:
code, output = print_form(simple_names, extra_button=True)
if code in (d.DIALOG_CANCEL, d.DIALOG_ESC):
return False
if code == d.DIALOG_EXTRA:
while 1:
code, output = print_form(advanced_names)
if code in (d.DIALOG_CANCEL, d.DIALOG_ESC):
break
if check_params(advanced_names, output):
break
continue
if check_params(simple_names, output):
break
return True
def sysprep(session):
"""Perform various system preperation tasks on the image"""
d = session['dialog']
......@@ -634,9 +706,6 @@ def sysprep(session):
wrapper = textwrap.TextWrapper(width=WIDTH - 5)
help_title = "System Preperation Tasks"
sysprep_help = "%s\n%s\n\n" % (help_title, '=' * len(help_title))
syspreps = image.os.list_syspreps()
if len(syspreps) == 0:
......@@ -647,6 +716,10 @@ def sysprep(session):
while 1:
choices = []
index = 0
help_title = "System Preperation Tasks"
sysprep_help = "%s\n%s\n\n" % (help_title, '=' * len(help_title))
for sysprep in syspreps:
name, descr = image.os.sysprep_info(sysprep)
display_name = name.replace('-', ' ').capitalize()
......@@ -681,6 +754,9 @@ def sysprep(session):
title="System Preperation", width=SMALL_WIDTH)
continue
if not sysprep_params(session):
continue
infobox = InfoBoxOutput(d, "Image Configuration")
try:
image.out.add(infobox)
......@@ -804,7 +880,7 @@ def main_menu(session):
break
elif choice == "Reset":
if confirm_reset(d):
d.infobox("Resetting snf-image-creator. Please wait...",
d.infobox("Resetting snf-image-creator. Please wait ...",
width=SMALL_WIDTH)
raise Reset
elif choice == "Help":
......
......@@ -49,6 +49,7 @@ from image_creator.dialog_util import extract_image, update_background_title, \
PAGE_WIDTH = 70
PAGE_HEIGHT = 10
SYSPREP_PARAM_MAXLEN = 20
class WizardExit(Exception):
......@@ -208,6 +209,36 @@ class WizardInputPage(WizardPage):
return self.NEXT
class WizardFormPage(WizardPage):
"""Represents a Form in a wizard"""
def __init__(self, name, display_name, text, fields, **kargs):
super(WizardFormPage, self).__init__(name, display_name, text, **kargs)
self.fields = fields
def run(self, session, title):
d = session['dialog']
w = session['wizard']
field_lenght = len(self.fields())
form_height = field_lenght if field_lenght < PAGE_HEIGHT - 4 \
else PAGE_HEIGHT - 4
(code, output) = d.form(
self.text, width=PAGE_WIDTH, height=PAGE_HEIGHT,
form_height=form_height, ok_label="Next", cancel="Back",
fields=self.fields(), title=title)
if code in (d.DIALOG_CANCEL, d.DIALOG_ESC):
return self.PREV
w[self.name] = self.validate(output)
self.default = output
self.info = "%s: %s" % (self.display_name, self.display(w[self.name]))
return self.NEXT
class WizardMenuPage(WizardPageWthChoices):
"""Represents a menu dialog with available choices in a wizard"""
......@@ -250,9 +281,11 @@ class WizardMenuPage(WizardPageWthChoices):
def start_wizard(session):
"""Run the image creation wizard"""
distro = session['image'].distro
ostype = session['image'].ostype
image = session['image']
distro = image.distro
ostype = image.ostype
# Create Cloud Wizard Page
def cloud_choices():
choices = []
for (name, cloud) in Kamaki.get_clouds().items():
......@@ -279,7 +312,7 @@ def start_wizard(session):
if edit_cloud(session, cloud):
return cloud
raise WizardInvalidData
raise WizardReloadPage
return cloud
......@@ -289,16 +322,59 @@ def start_wizard(session):
choices=cloud_choices, extra_label="Add", extra=cloud_add,
title="Clouds", validate=cloud_validate, fallback=cloud_none_available)
# Create Image Name Wizard Page
name = WizardInputPage(
"ImageName", "Image Name", "Please provide a name for the image:",
title="Image Name", default=ostype if distro == "unknown" else distro)
# Create Image Description Wizard Page
descr = WizardInputPage(
"ImageDescription", "Image Description",
"Please provide a description for the image:",
title="Image Description", default=session['metadata']['DESCRIPTION']
if 'DESCRIPTION' in session['metadata'] else '')
# Create Sysprep Params Wizard Page
needed = image.os.needed_sysprep_params
# Only show the parameters that don't have default values
param_names = [param for param in needed if needed[param].default is None]
def sysprep_params_fields():
fields = []
available = image.os.sysprep_params
for name in param_names:
text = needed[name].description
default = str(available[name]) if name in available else ""
fields.append(("%s: " % text, default, SYSPREP_PARAM_MAXLEN))
return fields
def sysprep_params_validate(answer):
params = {}
for i in range(len(answer)):
try:
value = needed[param_names[i]].type(answer[i])
if needed[param_names[i]].validate(value):
params[param_names[i]] = value
continue
except ValueError:
pass
session['dialog'].msgbox("Invalid value for parameter `%s'" %
param_names[i])
raise WizardReloadPage
return params
def sysprep_params_display(params):
return ",".join(["%s=%s" % (key, val) for key, val in params.items()])
sysprep_params = WizardFormPage(
"SysprepParams", "Sysprep Parameters",
"Prease fill in the following system preparation parameters:",
title="System Preparation Parameters", fields=sysprep_params_fields,
display=sysprep_params_display, validate=sysprep_params_validate
) if len(needed) != 0 else None
# Create Image Registration Wizard Page
def registration_choices():
return [("Private", "Image is accessible only by this user"),
("Public", "Everyone can create VMs from this image")]
......@@ -313,6 +389,8 @@ def start_wizard(session):
w.add_page(cloud)
w.add_page(name)
w.add_page(descr)
if sysprep_params is not None:
w.add_page(sysprep_params)
w.add_page(registration)
if w.run():
......@@ -336,6 +414,7 @@ def create_image(session):
out.clear()
#Sysprep
image.os.sysprep_params.update(wizard['SysprepParams'])
image.os.do_sysprep()
metadata = image.os.meta
......
......@@ -174,20 +174,23 @@ class Disk(object):
snapshot = uuid.uuid4().hex
tablefd, table = tempfile.mkstemp()
try:
os.write(tablefd, "0 %d snapshot %s %s n 8" %
(int(size), sourcedev, cowdev))
try:
os.write(tablefd, "0 %d snapshot %s %s n 8" %
(int(size), sourcedev, cowdev))
finally:
os.close(tablefd)
dmsetup('create', snapshot, table)
self._add_cleanup(try_fail_repeat, dmsetup, 'remove', snapshot)
finally:
os.unlink(table)
self.out.success('done')
return "/dev/mapper/%s" % snapshot
def get_image(self, media):
def get_image(self, media, **kargs):
"""Returns a newly created Image instance."""
image = Image(media, self.out)
image = Image(media, self.out, **kargs)
self._images.append(image)
image.enable()
return image
......
......@@ -45,53 +45,46 @@ from sendfile import sendfile
class Image(object):
"""The instances of this class can create images out of block devices."""
def __init__(self, device, output, bootable=True, meta={}):
def __init__(self, device, output, **kargs):
"""Create a new Image instance"""
self.device = device
self.out = output
self.bootable = bootable
self.meta = meta
self.meta = kargs['meta'] if 'meta' in kargs else {}
self.sysprep_params = \
kargs['sysprep_params'] if 'sysprep_params' in kargs else {}
self.progress_bar = None
self.guestfs_device = None
self.size = 0
self.g = guestfs.GuestFS()
self.g.add_drive_opts(self.device, readonly=0, format="raw")
self.guestfs_enabled = False
self.guestfs_version = self.g.version()
# Before version 1.17.14 the recovery process, which is a fork of the
# original process that called libguestfs, did not close its inherited
# file descriptors. This can cause problems especially if the parent
# process has opened pipes. Since the recovery process is an optional
# feature of libguestfs, it's better to disable it.
self.g.set_recovery_proc(0)
version = self.g.version()
if version['major'] > 1 or \
(version['major'] == 1 and (version['minor'] >= 18 or
(version['minor'] == 17 and
version['release'] >= 14))):
self.g.set_recovery_proc(1)
self.out.output("Enabling recovery proc")
def check_guestfs_version(self, major, minor, release):
"""Checks if the version of the used libguestfs is smaller, equal or
greater than the one specified by the major, minor and release triplet
#self.g.set_trace(1)
#self.g.set_verbose(1)
Returns:
< 0 if the installed version is smaller than the specified one
= 0 if they are equal
> 0 if the installed one is greater than the specified one
"""
self.guestfs_enabled = False
for (a, b) in (self.guestfs_version['major'], major), \
(self.guestfs_version['minor'], minor), \
(self.guestfs_version['release'], release):
if a != b:
return a - b
return 0
def enable(self):
"""Enable a newly created Image instance"""
self.out.output('Launching helper VM (may take a while) ...', False)
# self.progressbar = self.out.Progress(100, "Launching helper VM",
# "percent")
# eh = self.g.set_event_callback(self.progress_callback,
# guestfs.EVENT_PROGRESS)
self.g.launch()
self.guestfs_enabled = True
# self.g.delete_event_callback(eh)
# self.progressbar.success('done')