Merge pull request #26 from skalkoto/hotfix-0.7.1

Bug fixes and documentation updates

ChangeLog

+2014-10-06, v0.7.1
+	* Fix a bug in host bundling operation where the file system UUIDs and
+	  Labels where not preserved
+	* Fix a few bugs in the Windows deployment code
+	* Successfully detect Lubuntu and Xubuntu systems
+	* Update the documentation
+
 2014-09-26, v0.7
 	* Support all QEMU supported disk image formats as input media
 	* Detect if a Windows input media is sysprepped

docs/install.rst

@@ -123,7 +123,7 @@ Add the *synnefo* repository for *Fedora 20* to *yum*:
 .. code-block:: console
 
    # cd /etc/yum.repos.d
-   # wget http://download.opensuse.org/repositories/home:/GRNET:/synnefo/Fedora_20/home:GRNET:synnefo.repo
+   # curl -O http://download.opensuse.org/repositories/home:/GRNET:/synnefo/Fedora_20/home:GRNET:synnefo.repo
 
 To list the *snf-image-creator* package use the following command:
 
@@ -140,15 +140,24 @@ Install the package by issuing:
 CentOS
 ------
 
-For *CentOS 6* you can use our official packages hosted at the *synnefo*
-repository of the OpenSUSE Build Service.
+For *CentOS 6* and *CentOS 7* you can use our official packages hosted at the
+*synnefo* repository of the OpenSUSE Build Service.
 
 Add the *synnefo* repository for *CentOS 6* to the yum repositories list:
 
+For *CentOS 6*:
+
+.. code-block:: console
+
+   # cd /etc/yum.repos.d
+   # curl -O http://download.opensuse.org/repositories/home:/GRNET:/synnefo/CentOS_CentOS-6/home:GRNET:synnefo.repo
+
+For *CentOS 7*:
+
 .. code-block:: console
 
    # cd /etc/yum.repos.d
-   # wget http://download.opensuse.org/repositories/home:/GRNET:/synnefo/CentOS_CentOS-6/home:GRNET:synnefo.repo
+   # curl -O http://download.opensuse.org/repositories/home:/GRNET:/synnefo/CentOS_7/home:GRNET:synnefo.repo
 
 Check the `Fedora <#fedora>`_ instructions on how to install the software.
 

docs/usage.rst

@@ -412,8 +412,80 @@ the one with the most available space. The user may overwrite this behavior and
 indicate a different directory using the *tmpdir* option. This option is
 supported by both *snf-image-creator* and *snf-mkimage*.
 
+Troubleshooting
+===============
+
+Failures in launching libguestfs's helper VM
+--------------------------------------------
+
+The most common error you may get when using *snf-image-creator* is a failure
+when launching *libguestfs*'s helper VM. *libguestfs* [#f4]_ is a library
+for manipulating disk images and *snf-image-creator* makes heavy use of it.
+Most of the time those errors have to do with the installation of this
+library and not with *snf-image-creator* itself.
+
+The first thing you should do when troubleshooting this is to run the
+``liguestfs-test-tool`` diagnostic tool. This tool gets shipped with the
+library to test if *libguestfs* works as expected. If it runs to completion
+successfully, you will see this near the end:
+
+.. code-block:: console
+
+    ===== TEST FINISHED OK =====
+
+and the test tool will exit with code 0.
+
+If you get errors like this:
+
+.. code-block:: console
+
+   libguestfs: launch: backend=libvirt
+   libguestfs: launch: tmpdir=/tmp/libguestfseKwXgq
+   libguestfs: launch: umask=0022
+   libguestfs: launch: euid=0
+   libguestfs: libvirt version = 1001001 (1.1.1)
+   libguestfs: [00012ms] connect to libvirt
+   libguestfs: opening libvirt handle: URI = NULL, auth = virConnectAuthPtrDefault, flags = 0
+   libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
+   libguestfs: error: could not connect to libvirt (URI = NULL): Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory [code=38 domain=7]
+   libguestfs-test-tool: failed to launch appliance
+   libguestfs: closing guestfs handle 0x7ff0d44f8bb0 (state 0)
+   libguestfs: command: run: rm
+   libguestfs: command: run: \ -rf /tmp/libguestfseKwXgq
+
+it means that *libguestfs* is configured to use *libvirt* backend by default
+but the libvirt deamon is not running. You can either start libvirt deamon
+(providing instructions on how to do this is out of the scope of this
+tutorial) or change the default backend to *direct* by defining the
+**LIBGUESTFS_BACKEND** variable like this:
+
+.. code-block:: console
+
+   # export LIBGUESTFS_BACKEND=direct
+
+If you run the ``libguestfs-test-tool``, the command should finish without
+errors. Do the same every time before running *snf-image-creator*.
+
+If you get errors on *febootstrap-supermin-helper* like this one:
+
+.. code-block:: console
+
+   febootstrap-supermin-helper: ext2: parent directory not found: /lib:
+   File not found by ext2_lookup
+   libguestfs: error: external command failed, see earlier error messages
+   libguestfs-test-tool: failed to launch appliance
+   libguestfs: closing guestfs handle 0x7b3160 (state 0)
+
+you probably need to update the supermin appliance (just once). On Debian
+and Ubuntu systems you can do it using the command below:
+
+.. code-block:: console
+
+   # update-guestfs-appliance
+
 .. rubric:: Footnotes
 
 .. [#f1] http://technet.microsoft.com/en-us/library/bb676673.aspx
 .. [#f2] http://sourceware.org/lvm2/
 .. [#f3] http://mirrors.slackware.com/slackware/slackware-14.0/README.initrd
+.. [#f4] http://libguestfs.org/

image_creator/bundle_volume.py

@@ -44,17 +44,52 @@ umount = get_command('umount')
 blkid = get_command('blkid')
 tune2fs = get_command('tune2fs')
 
-MKFS_OPTS = {'ext2': ['-F'],
-             'ext3': ['-F'],
-             'ext4': ['-F'],
-             'reiserfs': ['-ff'],
-             'btrfs': [],
-             'minix': [],
-             'xfs': ['-f'],
-             'jfs': ['-f'],
-             'ntfs': ['-F'],
-             'msdos': [],
-             'vfat': []}
+MKFS_OPTS = {'ext2': {'force': '-F', 'uuid': '-U', 'label': '-L'},
+             'ext3': {'force': '-F', 'uuid': '-U', 'label': '-L'},
+             'ext4': {'force': '-F', 'uuid': '-U', 'label': '-L'},
+             'reiserfs': {'force': '-ff', 'uuid': '-u', 'label': '-l'},
+             'btrfs':  {'force': '-f', 'label': '-L'},
+             'minix': {},
+             'xfs': {'force': '-f', 'label': '-L'},
+             'jfs': {'force': '-f', 'label': '-L'},
+             'ntfs': {'force': '-F', 'label': '-L'},
+             'msdos': {'uuid': '-i'},
+             'vfat': {'uuid': '-i'}}
+
+UUID_UPDATE = {
+    'ext2': lambda d, u: tune2fs('-U', u, d),
+    'ext3': lambda d, u: tune2fs('-U', u, d),
+    'ext4': lambda d, u: tune2fs('-U', u, d),
+    'reiserfs': lambda d, u: get_command('reiserfstune')('-u', u, d),
+    'xfs': lambda d, u: get_command('xfs_admin')('-U', u, d),
+    'jfs': lambda d, u: get_command('jfstune')('-U', u, d),
+    'ntfs': lambda d, u: get_command('ntfslable')('--new-serial=%s' % u, d)}
+
+
+def mkfs(fs, device, uuid=None, label=None):
+    """Create a filesystem on the device"""
+
+    mkfs = get_command('mkfs.%s' % fs)
+
+    args = []
+
+    if 'force' in MKFS_OPTS[fs]:
+        args.append(MKFS_OPTS[fs]['force'])
+
+    if label:
+        args.append(MKFS_OPTS[fs]['label'])
+        args.append(label)
+
+    if 'uuid' in MKFS_OPTS[fs] and uuid:
+        args.append(MKFS_OPTS[fs]['uuid'])
+        args.append(uuid)
+
+    args.append(device)
+
+    mkfs(*args)
+
+    if 'uuid' not in MKFS_OPTS[fs] and 'uuid':
+        UUID_UPDATE[fs](device, uuid)
 
 
 class BundleVolume(object):
@@ -329,34 +364,6 @@ class BundleVolume(object):
 
         return excluded
 
-    def _replace_uuids(self, target, new_uuid):
-        """Replace UUID references in various files. This is needed after
-        copying system files of the host into a new file system
-        """
-
-        files = ['/etc/fstab',
-                 '/boot/grub/grub.cfg',
-                 '/boot/grub/menu.lst',
-                 '/boot/grub/grub.conf']
-
-        orig = {}
-        for p in self.disk.partitions:
-            if p.number in new_uuid.keys():
-                orig[p.number] = \
-                    blkid('-s', 'UUID', '-o', 'value', p.path).stdout.strip()
-
-        for f in map(lambda f: target + f, files):
-            if not os.path.exists(f):
-                continue
-
-            with open(f, 'r') as src:
-                lines = src.readlines()
-            with open(f, 'w') as dest:
-                for line in lines:
-                    for i, new in new_uuid.items():
-                        line = re.sub(orig[i], new, line)
-                    dest.write(line)
-
     def _create_filesystems(self, image, partitions):
         """Fill the image with data. Host file systems that are not currently
         mounted are binary copied into the image. For mounted file systems, a
@@ -393,10 +400,15 @@ class BundleVolume(object):
             new_uuid = {}
             # Create the file systems
             for i, dev in mapped.iteritems():
+                uuid = blkid(
+                    '-s', 'UUID', '-o', 'value', orig_dev[i]).stdout.strip()
+                label = blkid(
+                    '-s', 'LABEL', '-o', 'value', orig_dev[i]).stdout.strip()
                 fs = filesystem[i].fs
+
                 self.out.output('Creating %s file system on partition %d ... '
                                 % (fs, i), False)
-                get_command('mkfs.%s' % fs)(*(MKFS_OPTS[fs] + [dev]))
+                mkfs(fs, dev, uuid=uuid, label=label)
 
                 # For ext[234] enable the default mount options
                 if re.match('^ext[234]$', fs):
@@ -457,11 +469,6 @@ class BundleVolume(object):
                         os.chmod(target + excl, stat.st_mode)
                         os.chown(target + excl, stat.st_uid, stat.st_gid)
 
-                # We need to replace the old UUID references with the new ones
-                # in GRUB configuration files and /etc/fstab for file systems
-                # that have been recreated.
-                self._replace_uuids(target, new_uuid)
-
             finally:
                 self._umount_all(target)
                 os.rmdir(target)

image_creator/os_type/ubuntu.py

@@ -17,6 +17,8 @@
 
 """This module hosts OS-specific code for Ubuntu Linux"""
 
+import re
+
 from image_creator.os_type.linux import Linux
 
 
@@ -27,12 +29,19 @@ class Ubuntu(Linux):
         """Collect metadata about the OS"""
 
         super(Ubuntu, self)._do_collect_metadata()
-        apps = self.image.g.inspect_list_applications(self.root)
-        for app in apps:
-            if app['app_name'] == 'kubuntu-desktop':
-                self.meta['OS'] = 'kubuntu'
-                descr = self.meta['DESCRIPTION'].replace('Ubuntu', 'Kubuntu')
-                self.meta['DESCRIPTION'] = descr
+
+        regexp = re.compile('^(k|l|x)?ubuntu-desktop$')
+        variant = ""
+        for app in self.image.g.inspect_list_applications(self.root):
+            match = regexp.match(app['app_name'])
+            if match:
+                variant = match.group(1) + 'ubuntu'
                 break
 
+        if variant:
+            self.meta['OS'] = variant
+            descr = self.meta['DESCRIPTION'].replace('Ubuntu',
+                                                     variant.capitalize())
+            self.meta['DESCRIPTION'] = descr
+
 # vim: set sta sts=4 shiftwidth=4 sw=4 et ai :

image_creator/os_type/windows/__init__.py

@@ -407,20 +407,20 @@ class Windows(OSBase):
         stdout, stderr, rc = self.vm.rexec(cmd)
 
         querymax = None
+        expr = re.compile(
+            r'.+:\s*(\d+)\s*([KMGT]?)B\s*(?:\((\d+)\s*([KMGT]?)B\))?\s*$')
         for line in stdout.splitlines():
             # diskpart will return something like this:
             #
-            #   The maximum number of reclaimable bytes is: xxxx MB
-            #
-            if line.find('reclaimable') >= 0:
-                answer = line.split(':')[1].strip()
-                m = re.search(r'(\d+)\s*([KMGT]?)B', answer, re.IGNORECASE)
-                if m:
-                    querymax = m.group(1)
-                    unit = m.group(2)
-                else:
-                    raise FatalError("Unexpected output for `shrink querymax' "
-                                     "command: %s" % line)
+            # a) The maximum number of reclaimable bytes is: xxxx MB
+            # b) The maximum number of reclaimable bytes is: xxxx GB (xxxx MB)
+
+            match = expr.match(line)
+            if match:
+                offset = 0 if match.group(3) is None else 2
+                querymax = match.group(offset+1)
+                unit = match.group(offset+2)
+                break
 
         if querymax is None:
             raise FatalError("Error in shrinking! Couldn't find the max "
@@ -472,7 +472,7 @@ class Windows(OSBase):
                 "Shrinking failed. Please make sure the media is defragged.")
 
         for line in stdout.splitlines():
-            if line.find('shrunk') >= 0:
+            if line.find("%d" % querymax) >= 0:
                 self.out.output(" %s" % line)
 
         self.shrinked = True

image_creator/os_type/windows/vm.py

@@ -116,8 +116,7 @@ class VM(object):
         if 'mem' in self.params:
             args.extend(['-m', str(self.params['mem'].value)])
 
-        args.extend(['-drive',
-                     'file=%s,format=raw,cache=unsafe,if=%s' %
+        args.extend(['-drive', 'file=%s,cache=unsafe,if=%s' %
                      (self.disk, self.interface)])
 
         args.extend(

image_creator/version.py

 
-__version__ = "0.7"
+__version__ = "0.7.1"
 __version_vcs_info__ = {
-    'branch': 'master',
-    'revid': 'ab4d13b',
-    'revno': 571}
+    'branch': 'hotfix-0.7.1',
+    'revid': '080bf9c',
+    'revno': 578}
 __version_user_email__ = "skalkoto@grnet.gr"
 __version_user_name__ = "Nikos Skalkotos"

version

-0.7
+0.7.1