Skip to content

GitLab

S
snf-image-creator
Commit 96d50cb1 authored by Constantinos Venetsanopoulos's avatar Constantinos Venetsanopoulos
Browse files
Options

Merge pull request #20 from skalkoto/feature-image-format-support 

Feature image format support
parents 08d71238 cb7cd335
Hide whitespace changes
Inline Side-by-side
Showing with 303 additions and 244 deletions
docs/overview.rst
View file @ 96d50cb1
......@@ -2,8 +2,12 @@ Overview
^^^^^^^^
snf-image-creator is a simple command-line tool for creating OS images. The
original media the image is created from, can be a block device, a regular
file that represents a hard disk or the host system itself.
original media, the image is created from, can be:
* a block device, representing a hard disk
* a disk image file, representing a hard disk (supports all image file formats
supported by QEMU)
* the host system itself
Snapshotting
============
......
docs/usage.rst
View file @ 96d50cb1
......@@ -318,102 +318,17 @@ Choosing *YES* will create and upload the image to your cloud account.
Working with different image formats
====================================
*snf-image-creator* works on raw image files. If you have an image file with a
different image format you can either convert it to raw using
*qemu-img convert* command or use the *blktap* toolkit that provides a
user-level disk I/O interface and use the exposed *tapdev* block device as
input on *snf-image-creator*.
*snf-image-creator* is able to work with the most popular disk image formats.
It has been successfully tested with:
Converting images to raw
------------------------
Converting between images with *qemu-img convert* is generally straightforward.
All you need to provide is the output format (*-O raw*) and an output filename.
You may use the *-f* option to define the input format, but in most cases this
is guessed automatically. The table below shows a list of supported image
formats and the equivalent argument you may pass to the *-f* flag.
+--------------------------+-----------+
|Image Format |-f argument|
+==========================+===========+
|qcow2 (QEMU Copy On Write)|qcow2 |
+--------------------------+-----------+
|VHD (Mircosoft Hyper-V) |vpc |
+--------------------------+-----------+
|VMDK (VMware) |vmdk |
+--------------------------+-----------+
With the following commands we demonstrate how to download and convert an
official Ubuntu 14.04 *qcow2* image to raw:
.. code-block:: console
$ wget http://uec-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
$ qemu-img convert -f qcow2 -O raw trusty-server-cloudimg-amd64-disk1.img ubuntu.raw
Working on .vhd disk images using blktap/tapdisk
------------------------------------------------
If the source image format is *Microsoft VHD* [#f1]_, we can use blktap/tapdisk
to connect it to a block device and use this block device as input in
*snf-image-creator* without having to convert the image to raw format.
Assuming that you work on a recent Debian GNU/Linux, you can install the
needed tools by giving the following command:
.. code-block:: console
# apt-get install blktap-utils
# modprobe blktap
Please refer to your distribution's documentation on how to install the blktap
user-space tools and the corresponding kernel module.
After you have successfully installed blktap, do the following to attack the
source image (*/tmp/Centos-6.2-x86_64-minimal-dist.vhd*) to a block device:
Allocate a minor number in the kernel:
.. code-block:: console
# tap-ctl allocate
/dev/xen/blktap-2/tapdev0
Then, spawn a tapdisk process:
.. code-block:: console
# tap-ctl spawn
tapdisk spawned with pid 14879
Now, attach them together:
.. code-block:: console
# tap-ctl attach -m 0 -p 14879
And finally, open the VHD image:
.. code-block:: console
# tap-clt open -m 0 -p 14879 -a vhd:/tmp/Centos-6.2-x86_64-minimal-dist.vhd
Now you can open the associated block device with *snf-image-creator* like
this:
.. code-block:: console
# snf-image-creator /dev/xen/blktap-2/tapdev
When done, you may release the allocated resources by giving the following
commands:
.. code-block:: console
# tap-ctl close -m 0 -p 14879
# tap-ctl detach -m 0 -p 14879
# tap-ctl free -m 0
* Raw disk images
* VMDK (VMware)
* VHD (Microsoft Hyper-V)
* VDI (VirtualBox)
* qcow2 (QEMU)
It can support any image format QEMU supports as long as it represents a
bootable hard drive.
Limitations
===========
......
image_creator/dialog_main.py
View file @ 96d50cb1
......@@ -46,7 +46,7 @@ from image_creator.dialog_util import WIDTH, confirm_exit, Reset, \
PROGNAME = os.path.basename(sys.argv[0])
def create_image(d, media, out, tmp):
def create_image(d, media, out, tmp, snapshot):
"""Create an image out of `media'"""
d.setBackgroundTitle('snf-image-creator')
......@@ -61,9 +61,8 @@ def create_image(d, media, out, tmp):
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
try:
# There is no need to snapshot the media if it was created by the Disk
# instance as a temporary object.
device = disk.device if disk.source == '/' else disk.snapshot()
device = disk.file if not snapshot else disk.snapshot()
image = disk.get_image(device)
......@@ -172,7 +171,7 @@ def _dialog_form(self, text, height=20, width=60, form_height=15, fields=[],
return (code, output.splitlines())
def dialog_main(media, logfile, tmpdir):
def dialog_main(media, logfile, tmpdir, snapshot):
# In openSUSE dialog is buggy under xterm
if os.environ['TERM'] == 'xterm':
......@@ -214,7 +213,7 @@ def dialog_main(media, logfile, tmpdir):
try:
out = CompositeOutput([log])
out.output("Starting %s v%s ..." % (PROGNAME, version))
return create_image(d, media, out, tmpdir)
return create_image(d, media, out, tmpdir, snapshot)
except Reset:
log.output("Resetting everything ...")
continue
......@@ -235,6 +234,10 @@ def main():
parser.add_option("-l", "--logfile", type="string", dest="logfile",
default=None, help="log all messages to FILE",
metavar="FILE")
parser.add_option("--no-snapshot", dest="snapshot", default=True,
help="don't snapshot the input media. (THIS IS "
"DANGEROUS AS IT WILL ALTER THE ORIGINAL MEDIA!!!)",
action="store_false")
parser.add_option("--tmpdir", type="string", dest="tmp", default=None,
help="create large temporary image files under DIR",
metavar="DIR")
......@@ -260,7 +263,7 @@ def main():
# Save the terminal attributes
attr = termios.tcgetattr(sys.stdin.fileno())
try:
ret = dialog_main(media, logfile, opts.tmp)
ret = dialog_main(media, logfile, opts.tmp, opts.snapshot)
finally:
# Restore the terminal attributes. If an error occurs make sure
# that the terminal turns back to normal.
......
image_creator/dialog_menu.py
View file @ 96d50cb1
......@@ -27,7 +27,7 @@ import re
import time
from image_creator import __version__ as version
from image_creator.util import MD5, FatalError, virtio_versions
from image_creator.util import FatalError, virtio_versions
from image_creator.output.dialog import GaugeOutput, InfoBoxOutput
from image_creator.kamaki_wrapper import Kamaki, ClientError
from image_creator.help import get_help_file
......@@ -149,16 +149,16 @@ def upload_image(session):
kamaki.out = out
try:
if 'checksum' not in session:
md5 = MD5(out)
session['checksum'] = md5.compute(image.device, image.size)
session['checksum'] = image.md5()
try:
# Upload image file
with open(image.device, 'rb') as f:
session["pithos_uri"] = \
kamaki.upload(f, image.size, filename,
"Calculating block hashes",
"Uploading missing blocks")
with image.raw_device() as raw:
with open(raw, 'rb') as f:
session["pithos_uri"] = \
kamaki.upload(f, image.size, filename,
"Calculating block hashes",
"Uploading missing blocks")
# Upload md5sum file
out.output("Uploading md5sum file ...")
md5str = "%s %s\n" % (session['checksum'], filename)
......
image_creator/dialog_util.py
View file @ 96d50cb1
......@@ -24,7 +24,6 @@ import stat
import re
import json
from image_creator.output.dialog import GaugeOutput
from image_creator.util import MD5
from image_creator.kamaki_wrapper import Kamaki
SMALL_WIDTH = 60
......@@ -198,8 +197,7 @@ def extract_image(session):
out.add(gauge)
try:
if "checksum" not in session:
md5 = MD5(out)
session['checksum'] = md5.compute(image.device, image.size)
session['checksum'] = image.md5()
# Extract image file
image.dump(path)
......
image_creator/dialog_wizard.py
View file @ 96d50cb1
......@@ -25,7 +25,7 @@ import json
import re
from image_creator.kamaki_wrapper import Kamaki, ClientError
from image_creator.util import MD5, FatalError, virtio_versions
from image_creator.util import FatalError, virtio_versions
from image_creator.output.cli import OutputWthProgress
from image_creator.dialog_util import extract_image, update_background_title, \
add_cloud, edit_cloud, update_sysprep_param
......@@ -461,7 +461,7 @@ def create_image(session, answers):
metadata['DESCRIPTION'] = answers['ImageDescription']
# MD5
session['checksum'] = MD5(image.out).compute(image.device, image.size)
session['checksum'] = image.md5()
image.out.output()
try:
......@@ -472,10 +472,11 @@ def create_image(session, answers):
name = "%s-%s.diskdump" % (answers['ImageName'],
time.strftime("%Y%m%d%H%M"))
with open(image.device, 'rb') as device:
remote = kamaki.upload(device, image.size, name,
"(1/3) Calculating block hashes",
"(2/3) Uploading image blocks")
with image.raw_device() as raw:
with open(raw, 'rb') as device:
remote = kamaki.upload(device, image.size, name,
"(1/3) Calculating block hashes",
"(2/3) Uploading image blocks")
image.out.output("(3/3) Uploading md5sum file ...", False)
md5sumstr = '%s %s\n' % (session['checksum'], name)
......
image_creator/disk.py
View file @ 96d50cb1
......@@ -17,10 +17,8 @@
"""Module hosting the Disk class."""
from image_creator.util import get_command
from image_creator.util import try_fail_repeat
from image_creator.util import free_space
from image_creator.util import FatalError
from image_creator.util import get_command, try_fail_repeat, free_space, \
FatalError, create_snapshot
from image_creator.bundle_volume import BundleVolume
from image_creator.image import Image
......@@ -72,7 +70,7 @@ class Disk(object):
"""
self._cleanup_jobs = []
self._images = []
self._device = None
self._file = None
self.source = source
self.out = output
self.meta = {}
......@@ -98,15 +96,16 @@ class Disk(object):
"""Create a disk out of a directory"""
if self.source == '/':
bundle = BundleVolume(self.out, self.meta)
image = '%s/%s.diskdump' % (self.tmp, uuid.uuid4().hex)
image = '%s/%s.raw' % (self.tmp, uuid.uuid4().hex)
def check_unlink(path):
"""Unlinks file if exists"""
if os.path.exists(path):
os.unlink(path)
self._add_cleanup(check_unlink, image)
bundle.create_image(image)
return self._losetup(image)
return image
raise FatalError("Using a directory as media source is supported")
def cleanup(self):
......@@ -125,49 +124,63 @@ class Disk(object):
job(*args)
@property
def device(self):
"""Convert the source media into a block device"""
def file(self):
"""Convert the source media into a file"""
if self._device is not None:
return self._device
if self._file is not None:
return self._file
self.out.output("Examining source media `%s' ..." % self.source, False)
mode = os.stat(self.source).st_mode
if stat.S_ISDIR(mode):
self.out.success('looks like a directory')
self._device = self._dir_to_disk()
self._file = self._dir_to_disk()
elif stat.S_ISREG(mode):
self.out.success('looks like an image file')
self._device = self._losetup(self.source)
self._file = self.source
elif not stat.S_ISBLK(mode):
raise FatalError("Invalid media source. Only block devices, "
"regular files and directories are supported.")
else:
self.out.success('looks like a block device')
self._device = self.source
self._file = self.source
return self._device
return self._file
def snapshot(self):
"""Creates a snapshot of the original source media of the Disk
instance.
"""
size = blockdev('--getsz', self.device)
if self.source == '/':
self.out.warn("Snapshotting ignored for host bundling mode.")
return self.file
self.out.output("Snapshotting media source ...", False)
# Create a qcow2 snapshot for image files
if not stat.S_ISBLK(os.stat(self.file).st_mode):
snapshot = create_snapshot(self.file, self.tmp)
self._add_cleanup(os.unlink, snapshot)
self.out.success('done')
return snapshot
# Create a device-mapper snapshot for block devices
size = int(blockdev('--getsz', self.file))
cowfd, cow = tempfile.mkstemp(dir=self.tmp)
os.close(cowfd)
self._add_cleanup(os.unlink, cow)
# Create cow sparse file
dd('if=/dev/null', 'of=%s' % cow, 'bs=512', 'seek=%d' % int(size))
dd('if=/dev/null', 'of=%s' % cow, 'bs=512', 'seek=%d' % size)
cowdev = self._losetup(cow)
snapshot = uuid.uuid4().hex
snapshot = 'snf-image-creator-snapshot-%s' % uuid.uuid4().hex
tablefd, table = tempfile.mkstemp()
try:
try:
os.write(tablefd,
"0 %d snapshot %s %s n 8" %
(int(size), self.device, cowdev))
os.write(tablefd, "0 %d snapshot %s %s n 8\n" %
(size, self.file, cowdev))
finally:
os.close(tablefd)
......
image_creator/image.py
View file @ 96d50cb1
......@@ -15,12 +15,13 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
from image_creator.util import FatalError
from image_creator.util import FatalError, QemuNBD, image_info
from image_creator.gpt import GPTPartitionTable
from image_creator.os_type import os_cls
import re
import guestfs
import hashlib
from sendfile import sendfile
......@@ -32,6 +33,7 @@ class Image(object):
self.device = device
self.out = output
self.info = image_info(device)
self.meta = kargs['meta'] if 'meta' in kargs else {}
self.sysprep_params = \
......@@ -45,6 +47,9 @@ class Image(object):
self.guestfs_enabled = False
self.guestfs_version = self.g.version()
# This is needed if the image format is not raw
self.nbd = QemuNBD(device)
def check_guestfs_version(self, major, minor, release):
"""Checks if the version of the used libguestfs is smaller, equal or
greater than the one specified by the major, minor and release triplet
......@@ -124,7 +129,7 @@ class Image(object):
if self.check_guestfs_version(1, 18, 4) < 0:
self.g = guestfs.GuestFS()
self.g.add_drive_opts(self.device, readonly=0, format="raw")
self.g.add_drive_opts(self.device, readonly=0)
# Before version 1.17.14 the recovery process, which is a fork of the
# original process that called libguestfs, did not close its inherited
......@@ -201,6 +206,31 @@ class Image(object):
return self._os
def raw_device(self, readonly=True):
"""Returns a context manager that exports the raw image device. If
readonly is true, the block device that is returned is read only.
"""
if self.guestfs_enabled:
self.g.umount_all()
self.g.sync()
self.g.drop_caches(3) # drop everything
# Self gets overwritten
img = self
class RawImage:
"""The RawImage context manager"""
def __enter__(self):
return img.device if img.info['format'] == 'raw' else \
img.nbd.connect(readonly)
def __exit__(self, exc_type, exc_value, traceback):
if img.info['format'] != 'raw':
img.nbd.disconnect()
return RawImage()
def destroy(self):
"""Destroy this Image instance."""
......@@ -373,8 +403,9 @@ class Image(object):
assert (new_size <= self.size)
if self.meta['PARTITION_TABLE'] == 'gpt':
ptable = GPTPartitionTable(self.device)
self.size = ptable.shrink(new_size, self.size)
with self.raw_device(readonly=False) as raw:
ptable = GPTPartitionTable(raw)
self.size = ptable.shrink(new_size, self.size)
else:
self.size = min(new_size + 2048 * sector_size, self.size)
......@@ -391,30 +422,57 @@ class Image(object):
partition table. Empty space in the end of the device will be ignored.
"""
MB = 2 ** 20
blocksize = 4 * MB # 4MB
size = self.size
progr_size = (size + MB - 1) // MB # in MB
blocksize = 2 ** 22 # 4MB
progr_size = (self.size + MB - 1) // MB # in MB
progressbar = self.out.Progress(progr_size, "Dumping image file", 'mb')
with open(self.device, 'r') as src:
with open(outfile, "w") as dst:
left = size
offset = 0
progressbar.next()
with self.raw_device() as raw:
with open(raw, 'rb') as src:
with open(outfile, "wb") as dst:
left = self.size
offset = 0
progressbar.next()
while left > 0:
length = min(left, blocksize)
sent = sendfile(dst.fileno(), src.fileno(), offset,
length)
# Workaround for python-sendfile API change. In
# python-sendfile 1.2.x (py-sendfile) the returning
# value of sendfile is a tuple, where in version 2.x
# (pysendfile) it is just a single integer.
if isinstance(sent, tuple):
sent = sent[1]
offset += sent
left -= sent
progressbar.goto((self.size - left) // MB)
progressbar.success('image file %s was successfully created' % outfile)
def md5(self):
"""Computes the MD5 checksum of the image"""
MB = 2 ** 20
blocksize = 2 ** 22 # 4MB
progr_size = ((self.size + MB - 1) // MB) # in MB
progressbar = self.out.Progress(progr_size, "Calculating md5sum", 'mb')
md5 = hashlib.md5()
with self.raw_device() as raw:
with open(raw, "rb") as src:
left = self.size
while left > 0:
length = min(left, blocksize)
sent = sendfile(dst.fileno(), src.fileno(), offset, length)
# Workaround for python-sendfile API change. In
# python-sendfile 1.2.x (py-sendfile) the returning value
# of sendfile is a tuple, where in version 2.x (pysendfile)
# it is just a single integer.
if isinstance(sent, tuple):
sent = sent[1]
offset += sent
left -= sent
progressbar.goto((size - left) // MB)
progressbar.success('image file %s was successfully created' % outfile)
data = src.read(length)
md5.update(data)
left -= length
progressbar.goto((self.size - left) // MB)
checksum = md5.hexdigest()
progressbar.success(checksum)
return checksum
# vim: set sta sts=4 shiftwidth=4 sw=4 et ai :
image_creator/main.py
View file @ 96d50cb1
......@@ -22,7 +22,7 @@ snf-image-creator program.
from image_creator import __version__ as version
from image_creator.disk import Disk
from image_creator.util import FatalError, MD5
from image_creator.util import FatalError
from image_creator.output.cli import SilentOutput, SimpleOutput, \
OutputWthProgress
from image_creator.kamaki_wrapper import Kamaki, ClientError
......@@ -120,6 +120,11 @@ def parse_options(input_args):
help="don't perform any system preparation operation",
action="store_false")
parser.add_option("--no-snapshot", dest="snapshot", default=True,
help="don't snapshot the input media. (THIS IS "
"DANGEROUS AS IT WILL ALTER THE ORIGINAL MEDIA!!!)",
action="store_false")
parser.add_option("--public", dest="public", default=False,
help="register image with the cloud as public",
action="store_true")
......@@ -263,7 +268,7 @@ def image_creator():
try:
# There is no need to snapshot the media if it was created by the Disk
# instance as a temporary object.
device = disk.device if disk.source == '/' else disk.snapshot()
device = disk.file if not options.snapshot else disk.snapshot()
image = disk.get_image(device, sysprep_params=options.sysprep_params)
if image.is_unsupported() and not options.allow_unsupported:
......@@ -309,8 +314,7 @@ def image_creator():
# Add command line metadata to the collected ones...
metadata.update(options.metadata)
md5 = MD5(out)
checksum = md5.compute(image.device, image.size)
checksum = image.md5()
metastring = unicode(json.dumps(
{'properties': metadata,
......@@ -330,19 +334,17 @@ def image_creator():
os.path.basename(options.outfile)))
out.success('done')
# Destroy the image instance. We only need the disk device from now on
disk.destroy_image(image)
out.output()
try:
uploaded_obj = ""
if options.upload:
out.output("Uploading image to the storage service:")
with open(device, 'rb') as f:
uploaded_obj = kamaki.upload(
f, image.size, options.upload,
"(1/3) Calculating block hashes",
"(2/3) Uploading missing blocks")
with image.raw_device() as raw:
with open(raw, 'rb') as f:
remote = kamaki.upload(
f, image.size, options.upload,
"(1/3) Calculating block hashes",
"(2/3) Uploading missing blocks")
out.output("(3/3) Uploading md5sum file ...", False)
md5sumstr = '%s %s\n' % (checksum,
os.path.basename(options.upload))
......@@ -356,7 +358,7 @@ def image_creator():
img_type = 'public' if options.public else 'private'
out.output('Registering %s image with the compute service ...'
% img_type, False)
result = kamaki.register(options.register, uploaded_obj,
result = kamaki.register(options.register, remote,
metadata, options.public)
out.success('done')
out.output("Uploading metadata file ...", False)
......
image_creator/os_type/__init__.py
View file @ 96d50cb1
......@@ -420,7 +420,7 @@ class OSBase(object):
self.out.output()
@sysprep('Shrinking image', nomount=True)
@sysprep('Shrinking image (may take a while)', nomount=True)
def _shrink(self):
"""Shrink the last file system and update the partition table"""
self.image.shrink()
......
image_creator/os_type/windows/__init__.py
View file @ 96d50cb1
......@@ -701,12 +701,16 @@ class Windows(OSBase):
"""Check if winexe works on the Windows VM"""
retries = self.sysprep_params['connection_retries'].value
timeout = [2]
for i in xrange(1, retries - 1):
timeout.insert(0, timeout[0] * 2)
# If the connection_retries parameter is set to 0 disable the
# connectivity check
if retries == 0:
return True
for i in range(retries):
for i in xrange(retries):
(stdout, stderr, rc) = self.vm.rexec('cmd /C', fatal=False,
debug=True)
if rc == 0:
......@@ -721,6 +725,7 @@ class Windows(OSBase):
self.out.output("failed! See: `%s' for the full output" % log.name)
if i < retries - 1:
self.out.output("retrying ...", False)
time.sleep(timeout.pop())
raise FatalError("Connection to the Windows VM failed after %d retries"
% retries)
......@@ -866,8 +871,17 @@ class Windows(OSBase):
tmp = uuid.uuid4().hex
self.image.g.mkdir_p("%s/%s" % (self.systemroot, tmp))
self._add_cleanup('virtio', self.image.g.rm_rf,
"%s/%s" % (self.systemroot, tmp))
# This is a hack. We create a function here and pass it to
# _add_cleanup because self.image.g may change and the _add_cleanup
# will cache it which is wrong. For older versions of the guestfs
# library we recreate the g handler in enable_guestfs() and the
# program will crash if cleanup retains an older value for the
# guestfs handler.
def remove_tmp():
self.image.g.rm_rf("%s/%s" % (self.systemroot, tmp))
self._add_cleanup('virtio', remove_tmp)
for fname in os.listdir(dirname):
full_path = os.path.join(dirname, fname)
......@@ -929,38 +943,43 @@ class Windows(OSBase):
def _boot_virtio_vm(self):
"""Boot the media and install the VirtIO drivers"""
timeout = self.sysprep_params['boot_timeout'].value
shutdown_timeout = self.sysprep_params['shutdown_timeout'].value
virtio_timeout =