Commit 96d50cb1 authored by Constantinos Venetsanopoulos's avatar Constantinos Venetsanopoulos

Merge pull request #20 from skalkoto/feature-image-format-support

Feature image format support
parents 08d71238 cb7cd335
...@@ -2,8 +2,12 @@ Overview ...@@ -2,8 +2,12 @@ Overview
^^^^^^^^ ^^^^^^^^
snf-image-creator is a simple command-line tool for creating OS images. The snf-image-creator is a simple command-line tool for creating OS images. The
original media the image is created from, can be a block device, a regular original media, the image is created from, can be:
file that represents a hard disk or the host system itself.
* a block device, representing a hard disk
* a disk image file, representing a hard disk (supports all image file formats
supported by QEMU)
* the host system itself
Snapshotting Snapshotting
============ ============
......
...@@ -318,102 +318,17 @@ Choosing *YES* will create and upload the image to your cloud account. ...@@ -318,102 +318,17 @@ Choosing *YES* will create and upload the image to your cloud account.
Working with different image formats Working with different image formats
==================================== ====================================
*snf-image-creator* works on raw image files. If you have an image file with a *snf-image-creator* is able to work with the most popular disk image formats.
different image format you can either convert it to raw using It has been successfully tested with:
*qemu-img convert* command or use the *blktap* toolkit that provides a
user-level disk I/O interface and use the exposed *tapdev* block device as
input on *snf-image-creator*.
Converting images to raw * Raw disk images
------------------------ * VMDK (VMware)
* VHD (Microsoft Hyper-V)
Converting between images with *qemu-img convert* is generally straightforward. * VDI (VirtualBox)
All you need to provide is the output format (*-O raw*) and an output filename. * qcow2 (QEMU)
You may use the *-f* option to define the input format, but in most cases this
is guessed automatically. The table below shows a list of supported image
formats and the equivalent argument you may pass to the *-f* flag.
+--------------------------+-----------+
|Image Format |-f argument|
+==========================+===========+
|qcow2 (QEMU Copy On Write)|qcow2 |
+--------------------------+-----------+
|VHD (Mircosoft Hyper-V) |vpc |
+--------------------------+-----------+
|VMDK (VMware) |vmdk |
+--------------------------+-----------+
With the following commands we demonstrate how to download and convert an
official Ubuntu 14.04 *qcow2* image to raw:
.. code-block:: console
$ wget http://uec-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
$ qemu-img convert -f qcow2 -O raw trusty-server-cloudimg-amd64-disk1.img ubuntu.raw
Working on .vhd disk images using blktap/tapdisk
------------------------------------------------
If the source image format is *Microsoft VHD* [#f1]_, we can use blktap/tapdisk
to connect it to a block device and use this block device as input in
*snf-image-creator* without having to convert the image to raw format.
Assuming that you work on a recent Debian GNU/Linux, you can install the
needed tools by giving the following command:
.. code-block:: console
# apt-get install blktap-utils
# modprobe blktap
Please refer to your distribution's documentation on how to install the blktap
user-space tools and the corresponding kernel module.
After you have successfully installed blktap, do the following to attack the
source image (*/tmp/Centos-6.2-x86_64-minimal-dist.vhd*) to a block device:
Allocate a minor number in the kernel:
.. code-block:: console
# tap-ctl allocate
/dev/xen/blktap-2/tapdev0
Then, spawn a tapdisk process:
.. code-block:: console
# tap-ctl spawn
tapdisk spawned with pid 14879
Now, attach them together:
.. code-block:: console
# tap-ctl attach -m 0 -p 14879
And finally, open the VHD image:
.. code-block:: console
# tap-clt open -m 0 -p 14879 -a vhd:/tmp/Centos-6.2-x86_64-minimal-dist.vhd
Now you can open the associated block device with *snf-image-creator* like
this:
.. code-block:: console
# snf-image-creator /dev/xen/blktap-2/tapdev
When done, you may release the allocated resources by giving the following
commands:
.. code-block:: console
# tap-ctl close -m 0 -p 14879
# tap-ctl detach -m 0 -p 14879
# tap-ctl free -m 0
It can support any image format QEMU supports as long as it represents a
bootable hard drive.
Limitations Limitations
=========== ===========
......
...@@ -46,7 +46,7 @@ from image_creator.dialog_util import WIDTH, confirm_exit, Reset, \ ...@@ -46,7 +46,7 @@ from image_creator.dialog_util import WIDTH, confirm_exit, Reset, \
PROGNAME = os.path.basename(sys.argv[0]) PROGNAME = os.path.basename(sys.argv[0])
def create_image(d, media, out, tmp): def create_image(d, media, out, tmp, snapshot):
"""Create an image out of `media'""" """Create an image out of `media'"""
d.setBackgroundTitle('snf-image-creator') d.setBackgroundTitle('snf-image-creator')
...@@ -61,9 +61,8 @@ def create_image(d, media, out, tmp): ...@@ -61,9 +61,8 @@ def create_image(d, media, out, tmp):
signal.signal(signal.SIGINT, signal_handler) signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler) signal.signal(signal.SIGTERM, signal_handler)
try: try:
# There is no need to snapshot the media if it was created by the Disk
# instance as a temporary object. device = disk.file if not snapshot else disk.snapshot()
device = disk.device if disk.source == '/' else disk.snapshot()
image = disk.get_image(device) image = disk.get_image(device)
...@@ -172,7 +171,7 @@ def _dialog_form(self, text, height=20, width=60, form_height=15, fields=[], ...@@ -172,7 +171,7 @@ def _dialog_form(self, text, height=20, width=60, form_height=15, fields=[],
return (code, output.splitlines()) return (code, output.splitlines())
def dialog_main(media, logfile, tmpdir): def dialog_main(media, logfile, tmpdir, snapshot):
# In openSUSE dialog is buggy under xterm # In openSUSE dialog is buggy under xterm
if os.environ['TERM'] == 'xterm': if os.environ['TERM'] == 'xterm':
...@@ -214,7 +213,7 @@ def dialog_main(media, logfile, tmpdir): ...@@ -214,7 +213,7 @@ def dialog_main(media, logfile, tmpdir):
try: try:
out = CompositeOutput([log]) out = CompositeOutput([log])
out.output("Starting %s v%s ..." % (PROGNAME, version)) out.output("Starting %s v%s ..." % (PROGNAME, version))
return create_image(d, media, out, tmpdir) return create_image(d, media, out, tmpdir, snapshot)
except Reset: except Reset:
log.output("Resetting everything ...") log.output("Resetting everything ...")
continue continue
...@@ -235,6 +234,10 @@ def main(): ...@@ -235,6 +234,10 @@ def main():
parser.add_option("-l", "--logfile", type="string", dest="logfile", parser.add_option("-l", "--logfile", type="string", dest="logfile",
default=None, help="log all messages to FILE", default=None, help="log all messages to FILE",
metavar="FILE") metavar="FILE")
parser.add_option("--no-snapshot", dest="snapshot", default=True,
help="don't snapshot the input media. (THIS IS "
"DANGEROUS AS IT WILL ALTER THE ORIGINAL MEDIA!!!)",
action="store_false")
parser.add_option("--tmpdir", type="string", dest="tmp", default=None, parser.add_option("--tmpdir", type="string", dest="tmp", default=None,
help="create large temporary image files under DIR", help="create large temporary image files under DIR",
metavar="DIR") metavar="DIR")
...@@ -260,7 +263,7 @@ def main(): ...@@ -260,7 +263,7 @@ def main():
# Save the terminal attributes # Save the terminal attributes
attr = termios.tcgetattr(sys.stdin.fileno()) attr = termios.tcgetattr(sys.stdin.fileno())
try: try:
ret = dialog_main(media, logfile, opts.tmp) ret = dialog_main(media, logfile, opts.tmp, opts.snapshot)
finally: finally:
# Restore the terminal attributes. If an error occurs make sure # Restore the terminal attributes. If an error occurs make sure
# that the terminal turns back to normal. # that the terminal turns back to normal.
......
...@@ -27,7 +27,7 @@ import re ...@@ -27,7 +27,7 @@ import re
import time import time
from image_creator import __version__ as version from image_creator import __version__ as version
from image_creator.util import MD5, FatalError, virtio_versions from image_creator.util import FatalError, virtio_versions
from image_creator.output.dialog import GaugeOutput, InfoBoxOutput from image_creator.output.dialog import GaugeOutput, InfoBoxOutput
from image_creator.kamaki_wrapper import Kamaki, ClientError from image_creator.kamaki_wrapper import Kamaki, ClientError
from image_creator.help import get_help_file from image_creator.help import get_help_file
...@@ -149,12 +149,12 @@ def upload_image(session): ...@@ -149,12 +149,12 @@ def upload_image(session):
kamaki.out = out kamaki.out = out
try: try:
if 'checksum' not in session: if 'checksum' not in session:
md5 = MD5(out) session['checksum'] = image.md5()
session['checksum'] = md5.compute(image.device, image.size)
try: try:
# Upload image file # Upload image file
with open(image.device, 'rb') as f: with image.raw_device() as raw:
with open(raw, 'rb') as f:
session["pithos_uri"] = \ session["pithos_uri"] = \
kamaki.upload(f, image.size, filename, kamaki.upload(f, image.size, filename,
"Calculating block hashes", "Calculating block hashes",
......
...@@ -24,7 +24,6 @@ import stat ...@@ -24,7 +24,6 @@ import stat
import re import re
import json import json
from image_creator.output.dialog import GaugeOutput from image_creator.output.dialog import GaugeOutput
from image_creator.util import MD5
from image_creator.kamaki_wrapper import Kamaki from image_creator.kamaki_wrapper import Kamaki
SMALL_WIDTH = 60 SMALL_WIDTH = 60
...@@ -198,8 +197,7 @@ def extract_image(session): ...@@ -198,8 +197,7 @@ def extract_image(session):
out.add(gauge) out.add(gauge)
try: try:
if "checksum" not in session: if "checksum" not in session:
md5 = MD5(out) session['checksum'] = image.md5()
session['checksum'] = md5.compute(image.device, image.size)
# Extract image file # Extract image file
image.dump(path) image.dump(path)
......
...@@ -25,7 +25,7 @@ import json ...@@ -25,7 +25,7 @@ import json
import re import re
from image_creator.kamaki_wrapper import Kamaki, ClientError from image_creator.kamaki_wrapper import Kamaki, ClientError
from image_creator.util import MD5, FatalError, virtio_versions from image_creator.util import FatalError, virtio_versions
from image_creator.output.cli import OutputWthProgress from image_creator.output.cli import OutputWthProgress
from image_creator.dialog_util import extract_image, update_background_title, \ from image_creator.dialog_util import extract_image, update_background_title, \
add_cloud, edit_cloud, update_sysprep_param add_cloud, edit_cloud, update_sysprep_param
...@@ -461,7 +461,7 @@ def create_image(session, answers): ...@@ -461,7 +461,7 @@ def create_image(session, answers):
metadata['DESCRIPTION'] = answers['ImageDescription'] metadata['DESCRIPTION'] = answers['ImageDescription']
# MD5 # MD5
session['checksum'] = MD5(image.out).compute(image.device, image.size) session['checksum'] = image.md5()
image.out.output() image.out.output()
try: try:
...@@ -472,7 +472,8 @@ def create_image(session, answers): ...@@ -472,7 +472,8 @@ def create_image(session, answers):
name = "%s-%s.diskdump" % (answers['ImageName'], name = "%s-%s.diskdump" % (answers['ImageName'],
time.strftime("%Y%m%d%H%M")) time.strftime("%Y%m%d%H%M"))
with open(image.device, 'rb') as device: with image.raw_device() as raw:
with open(raw, 'rb') as device:
remote = kamaki.upload(device, image.size, name, remote = kamaki.upload(device, image.size, name,
"(1/3) Calculating block hashes", "(1/3) Calculating block hashes",
"(2/3) Uploading image blocks") "(2/3) Uploading image blocks")
......
...@@ -17,10 +17,8 @@ ...@@ -17,10 +17,8 @@
"""Module hosting the Disk class.""" """Module hosting the Disk class."""
from image_creator.util import get_command from image_creator.util import get_command, try_fail_repeat, free_space, \
from image_creator.util import try_fail_repeat FatalError, create_snapshot
from image_creator.util import free_space
from image_creator.util import FatalError
from image_creator.bundle_volume import BundleVolume from image_creator.bundle_volume import BundleVolume
from image_creator.image import Image from image_creator.image import Image
...@@ -72,7 +70,7 @@ class Disk(object): ...@@ -72,7 +70,7 @@ class Disk(object):
""" """
self._cleanup_jobs = [] self._cleanup_jobs = []
self._images = [] self._images = []
self._device = None self._file = None
self.source = source self.source = source
self.out = output self.out = output
self.meta = {} self.meta = {}
...@@ -98,15 +96,16 @@ class Disk(object): ...@@ -98,15 +96,16 @@ class Disk(object):
"""Create a disk out of a directory""" """Create a disk out of a directory"""
if self.source == '/': if self.source == '/':
bundle = BundleVolume(self.out, self.meta) bundle = BundleVolume(self.out, self.meta)
image = '%s/%s.diskdump' % (self.tmp, uuid.uuid4().hex) image = '%s/%s.raw' % (self.tmp, uuid.uuid4().hex)
def check_unlink(path): def check_unlink(path):
"""Unlinks file if exists"""
if os.path.exists(path): if os.path.exists(path):
os.unlink(path) os.unlink(path)
self._add_cleanup(check_unlink, image) self._add_cleanup(check_unlink, image)
bundle.create_image(image) bundle.create_image(image)
return self._losetup(image) return image
raise FatalError("Using a directory as media source is supported") raise FatalError("Using a directory as media source is supported")
def cleanup(self): def cleanup(self):
...@@ -125,49 +124,63 @@ class Disk(object): ...@@ -125,49 +124,63 @@ class Disk(object):
job(*args) job(*args)
@property @property
def device(self): def file(self):
"""Convert the source media into a block device""" """Convert the source media into a file"""
if self._device is not None: if self._file is not None:
return self._device return self._file
self.out.output("Examining source media `%s' ..." % self.source, False) self.out.output("Examining source media `%s' ..." % self.source, False)
mode = os.stat(self.source).st_mode mode = os.stat(self.source).st_mode
if stat.S_ISDIR(mode): if stat.S_ISDIR(mode):
self.out.success('looks like a directory') self.out.success('looks like a directory')
self._device = self._dir_to_disk() self._file = self._dir_to_disk()
elif stat.S_ISREG(mode): elif stat.S_ISREG(mode):
self.out.success('looks like an image file') self.out.success('looks like an image file')
self._device = self._losetup(self.source) self._file = self.source
elif not stat.S_ISBLK(mode): elif not stat.S_ISBLK(mode):
raise FatalError("Invalid media source. Only block devices, " raise FatalError("Invalid media source. Only block devices, "
"regular files and directories are supported.") "regular files and directories are supported.")
else: else:
self.out.success('looks like a block device') self.out.success('looks like a block device')
self._device = self.source self._file = self.source
return self._device return self._file
def snapshot(self): def snapshot(self):
"""Creates a snapshot of the original source media of the Disk """Creates a snapshot of the original source media of the Disk
instance. instance.
""" """
size = blockdev('--getsz', self.device)
if self.source == '/':
self.out.warn("Snapshotting ignored for host bundling mode.")
return self.file
self.out.output("Snapshotting media source ...", False) self.out.output("Snapshotting media source ...", False)
# Create a qcow2 snapshot for image files
if not stat.S_ISBLK(os.stat(self.file).st_mode):
snapshot = create_snapshot(self.file, self.tmp)
self._add_cleanup(os.unlink, snapshot)
self.out.success('done')
return snapshot
# Create a device-mapper snapshot for block devices
size = int(blockdev('--getsz', self.file))
cowfd, cow = tempfile.mkstemp(dir=self.tmp) cowfd, cow = tempfile.mkstemp(dir=self.tmp)
os.close(cowfd) os.close(cowfd)
self._add_cleanup(os.unlink, cow) self._add_cleanup(os.unlink, cow)
# Create cow sparse file # Create cow sparse file
dd('if=/dev/null', 'of=%s' % cow, 'bs=512', 'seek=%d' % int(size)) dd('if=/dev/null', 'of=%s' % cow, 'bs=512', 'seek=%d' % size)
cowdev = self._losetup(cow) cowdev = self._losetup(cow)
snapshot = uuid.uuid4().hex snapshot = 'snf-image-creator-snapshot-%s' % uuid.uuid4().hex
tablefd, table = tempfile.mkstemp() tablefd, table = tempfile.mkstemp()
try: try:
try: try:
os.write(tablefd, os.write(tablefd, "0 %d snapshot %s %s n 8\n" %
"0 %d snapshot %s %s n 8" % (size, self.file, cowdev))
(int(size), self.device, cowdev))
finally: finally:
os.close(tablefd) os.close(tablefd)
......
...@@ -15,12 +15,13 @@ ...@@ -15,12 +15,13 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
from image_creator.util import FatalError from image_creator.util import FatalError, QemuNBD, image_info
from image_creator.gpt import GPTPartitionTable from image_creator.gpt import GPTPartitionTable
from image_creator.os_type import os_cls from image_creator.os_type import os_cls
import re import re
import guestfs import guestfs
import hashlib
from sendfile import sendfile from sendfile import sendfile
...@@ -32,6 +33,7 @@ class Image(object): ...@@ -32,6 +33,7 @@ class Image(object):
self.device = device self.device = device
self.out = output self.out = output
self.info = image_info(device)
self.meta = kargs['meta'] if 'meta' in kargs else {} self.meta = kargs['meta'] if 'meta' in kargs else {}
self.sysprep_params = \ self.sysprep_params = \
...@@ -45,6 +47,9 @@ class Image(object): ...@@ -45,6 +47,9 @@ class Image(object):
self.guestfs_enabled = False self.guestfs_enabled = False
self.guestfs_version = self.g.version() self.guestfs_version = self.g.version()
# This is needed if the image format is not raw
self.nbd = QemuNBD(device)
def check_guestfs_version(self, major, minor, release): def check_guestfs_version(self, major, minor, release):
"""Checks if the version of the used libguestfs is smaller, equal or """Checks if the version of the used libguestfs is smaller, equal or
greater than the one specified by the major, minor and release triplet greater than the one specified by the major, minor and release triplet
...@@ -124,7 +129,7 @@ class Image(object): ...@@ -124,7 +129,7 @@ class Image(object):
if self.check_guestfs_version(1, 18, 4) < 0: if self.check_guestfs_version(1, 18, 4) < 0:
self.g = guestfs.GuestFS() self.g = guestfs.GuestFS()
self.g.add_drive_opts(self.device, readonly=0, format="raw") self.g.add_drive_opts(self.device, readonly=0)
# Before version 1.17.14 the recovery process, which is a fork of the # Before version 1.17.14 the recovery process, which is a fork of the
# original process that called libguestfs, did not close its inherited # original process that called libguestfs, did not close its inherited
...@@ -201,6 +206,31 @@ class Image(object): ...@@ -201,6 +206,31 @@ class Image(object):
return self._os return self._os
def raw_device(self, readonly=True):
"""Returns a context manager that exports the raw image device. If
readonly is true, the block device that is returned is read only.
"""
if self.guestfs_enabled:
self.g.umount_all()
self.g.sync()
self.g.drop_caches(3) # drop everything
# Self gets overwritten
img = self
class RawImage:
"""The RawImage context manager"""
def __enter__(self):
return img.device if img.info['format'] == 'raw' else \
img.nbd.connect(readonly)
def __exit__(self, exc_type, exc_value, traceback):
if img.info['format'] != 'raw':
img.nbd.disconnect()
return RawImage()
def destroy(self): def destroy(self):
"""Destroy this Image instance.""" """Destroy this Image instance."""
...@@ -373,7 +403,8 @@ class Image(object): ...@@ -373,7 +403,8 @@ class Image(object):
assert (new_size <= self.size) assert (new_size <= self.size)
if self.meta['PARTITION_TABLE'] == 'gpt': if self.meta['PARTITION_TABLE'] == 'gpt':
ptable = GPTPartitionTable(self.device) with self.raw_device(readonly=False) as raw:
ptable = GPTPartitionTable(raw)
self.size = ptable.shrink(new_size, self.size) self.size = ptable.shrink(new_size, self.size)
else: else:
self.size = min(new_size + 2048 * sector_size, self.size) self.size = min(new_size + 2048 * sector_size, self.size)
...@@ -391,30 +422,57 @@ class Image(object): ...@@ -391,30 +422,57 @@ class Image(object):
partition table. Empty space in the end of the device will be ignored. partition table. Empty space in the end of the device will be ignored.
""" """
MB = 2 ** 20 MB = 2 ** 20
blocksize = 4 * MB # 4MB blocksize = 2 ** 22 # 4MB
size = self.size progr_size = (self.size + MB - 1) // MB # in MB
progr_size = (size + MB - 1) // MB # in MB
progressbar = self.out.Progress(progr_size, "Dumping image file", 'mb') progressbar = self.out.Progress(progr_size, "Dumping image file", 'mb')
with open(self.device, 'r') as src: with self.raw_device() as raw:
with open(outfile, "w") as dst: with open(raw, 'rb') as src:
left = size with open(outfile, "wb") as dst:
left = self.size
offset = 0 offset = 0
progressbar.next() progressbar.next()
while left > 0: while left > 0:
length = min(left, blocksize) length = min(left, blocksize)
sent = sendfile(dst.fileno(), src.fileno(), offset, length) sent = sendfile(dst.fileno(), src.fileno(), offset,
length)
# Workaround for python-sendfile API change. In # Workaround for python-sendfile API change. In
# python-sendfile 1.2.x (py-sendfile) the returning value # python-sendfile 1.2.x (py-sendfile) the returning
# of sendfile is a tuple, where in version 2.x (pysendfile) # value of sendfile is a tuple, where in version 2.x
# it is just a single integer.