Skip to content

GitLab

S
snf-image-creator
Commit ce3a618a authored by Nikos Skalkotos's avatar Nikos Skalkotos
Browse files
Options

Merge branch 'release-0.7' into debian-release-0.7

parents e9f9606d 5b284039
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 4702 additions and 2718 deletions
COPYRIGHT
View file @ ce3a618a
This diff is collapsed.
MANIFEST.in
View file @ ce3a618a
include image_creator/help/*.rst
include README COPYRIGHT AUTHORS ChangeLog
include README.md COPYRIGHT AUTHORS ChangeLog
recursive-include docs *
prune docs/_build
README deleted 100644 → 0
View file @ e9f9606d
snf-image-creator
=================
snf-image-creator is a command line tool for creating OS images to be used with
synnefo.
It comes in 2 variants:
* snf-image-creator: A user-friendly dialog-based program
* snf-mkimage: A non-interactive command line program
README.md 0 → 100644
View file @ ce3a618a
snf-image-creator
=================
Overview
--------
This is snf-image-creator, a command-line tool for creating OS Images to be
used with Synnefo.
It comes in two variants:
* snf-image-creator: A user-friendly dialog-based program
* snf-mkimage: A non-interactive command-line program
Project Page
------------
Please see the [official Synnefo site](http://www.synnefo.org) and the
[latest snf-image-creator docs](http://www.synnefo.org/docs/snf-image-creator/latest/index.html)
for more information.
Copyright and license
=====================
Copyright (C) 2011-2014 GRNET S.A.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
docs/index.rst
View file @ ce3a618a
......@@ -6,7 +6,12 @@
Welcome to snf-image-creator's documentation!
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
snf-image-creator is an image creation tool for `Synnefo <https://code.grnet.gr/projects/synnefo>`_
This is snf-image-creator, a command-line tool for creating OS Images to be
used with `Synnefo <http://www.synnefo.org/>`_.
It comes in two variants:
* snf-image-creator: A user-friendly dialog-based program
* snf-mkimage: A non-interactive command-line program
Contents:
......
docs/install.rst
View file @ ce3a618a
......@@ -44,31 +44,37 @@ Add the following line to */etc/apt/sources.list*:
deb http://apt.dev.grnet.gr wheezy/
Add the apt-dev GPG key to the list of trusted keys:
.. code-block:: console
# curl https://dev.grnet.gr/files/apt-grnetdev.pub | apt-key add -
And resynchronize the package index files from their sources:
.. code-block:: console
$ sudo apt-get update
# apt-get update
You should be able to list the package by calling:
.. code-block:: console
$ apt-cache showpkg snf-image-creator
# apt-cache showpkg snf-image-creator
And install the package with this command:
.. code-block:: console
$ apt-get install snf-image-creator
# apt-get install snf-image-creator
Ubuntu
------
For *Ubuntu 12.04 LTS*, *12.10* and *13.04* systems, you can use our official
For *Ubuntu 12.04 LTS* and *14.04 LTS* systems, you can use our official
packages found in *grnet/synnefo* Lauchpad PPA.
Add the synnefo PPA in your system:
Add the Synnefo PPA in your system:
.. code-block:: console
......@@ -77,7 +83,7 @@ Add the synnefo PPA in your system:
If *apt-add-repository* is missing, first install:
*software-properties-common* (Ubuntu 12.10 & 13.04):
*software-properties-common* (Ubuntu 14.04):
.. code-block:: console
......@@ -89,7 +95,7 @@ Or *python-software-properties* (Ubuntu 12.04):
$ sudo apt-get install python-software-properties
After the synnefo repository is set up, you should be able to list
After the Synnefo repository is set up, you should be able to list
snf-image-creator by calling:
.. code-block:: console
......@@ -109,27 +115,27 @@ Install the package by issuing:
Fedora
------
For *Fedora 17* you can use our official packages hosted at the *synnefo*
For *Fedora 20* you can use our official packages hosted at the *synnefo*
repository of the openSUSE Build Service.
Add the *synnefo* repository for *Fedora 17* to *yum*:
Add the *synnefo* repository for *Fedora 20* to *yum*:
.. code-block:: console
$ cd /etc/yum.repos.d
$ wget http://download.opensuse.org/repositories/home:/GRNET:/synnefo/Fedora_17/home:GRNET:synnefo.repo
# cd /etc/yum.repos.d
# wget http://download.opensuse.org/repositories/home:/GRNET:/synnefo/Fedora_20/home:GRNET:synnefo.repo
To list the *snf-image-creator* package use the following command:
.. code-block:: console
$ yum info snf-image-creator
# yum info snf-image-creator
Install the package by issuing:
.. code-block:: console
$ yum install snf-image-creator
# yum install snf-image-creator
CentOS
------
......@@ -141,41 +147,41 @@ Add the *synnefo* repository for *CentOS 6* to the yum repositories list:
.. code-block:: console
$ cd /etc/yum.repos.d
$ wget http://download.opensuse.org/repositories/home:/GRNET:/synnefo/CentOS_CentOS-6/home:GRNET:synnefo.repo
# cd /etc/yum.repos.d
# wget http://download.opensuse.org/repositories/home:/GRNET:/synnefo/CentOS_CentOS-6/home:GRNET:synnefo.repo
Check the `Fedora <#fedora>`_ instructions on how to install the software.
OpenSUSE
openSUSE
--------
For *OpenSUSE 12.3* you can use our official packages hosted at the *synnefo*
repository of the OpenSUSE Build Service.
For *openSUSE 13.1* you can use our official packages hosted at the *Synnefo*
repository of the openSUSE Build Service.
Add the *Virtualization* repository for *OpenSUSE 12.3* to *YaST* with the
Add the *Virtualization* repository for *openSUSE 13.1* to *YaST* with the
*Zypper* package manager:
.. code-block:: console
$ zypper ar -f http://download.opensuse.org/repositories/Virtualization/openSUSE_12.3/Virtualization.repo
# zypper ar -f http://download.opensuse.org/repositories/Virtualization/openSUSE_13.1/Virtualization.repo
Add the *synnefo* repository:
Add the *Synnefo* repository:
.. code-block:: console
$ zypper ar -f http://download.opensuse.org/repositories/home:/GRNET:/synnefo/openSUSE_12.3/home:GRNET:synnefo.repo
# zypper ar -f http://download.opensuse.org/repositories/home:/GRNET:/synnefo/openSUSE_13.1/home:GRNET:synnefo.repo
To list the *snf-image-creator* package use the following command:
.. code-block:: console
$ zypper se snf-image-creator
# zypper se snf-image-creator
Install the package by issuing:
.. code-block:: console
$ zypper in snf-image-creator
# zypper in snf-image-creator
Arch Linux
......@@ -200,13 +206,13 @@ the *yaourt* package:
$ tar -xvf package-query.tar.gz
$ cd package-query
$ makepkg -s
$ pacman -U package-query-<VERSION>-<ARCH>.pkg.tar.xz
$ su -c 'pacman -U package-query-<VERSION>-<ARCH>.pkg.tar.xz'
$ cd ..
$ wget https://aur.archlinux.org/packages/ya/yaourt/yaourt.tar.gz
$ tar -xvf yaourt.tar.gz
$ cd yaourt
$ makepkg -s
$ pacman -U yaourt-<VERSION>-<ARCH>.pkg.tar.xz
$ su -c 'pacman -U yaourt-<VERSION>-<ARCH>.pkg.tar.xz'
Install *snf-image-creator* using yaourt:
......@@ -282,7 +288,7 @@ kamaki Installation
-------------------
Refer to `./kamaki documentation <http://docs.dev.grnet.gr/kamaki/latest/installation.html>`_
for instructions. You may install kamaki from source inside the virtualenv
for instructions. You may install *./kamaki* from source inside the virtualenv
you've created above or by using binary packages if they are available for your
distribution.
......@@ -309,7 +315,7 @@ cloning its git repository:
$ python ./setup.py install
To do the latter, you'll need to have git (http://git-scm.com/) installed.
For ubuntu this can be done using:
For Ubuntu this can be done using:
.. code-block:: console
......
docs/overview.rst
View file @ ce3a618a
......@@ -2,8 +2,12 @@ Overview
^^^^^^^^
snf-image-creator is a simple command-line tool for creating OS images. The
original media the image is created from, can be a block device, a regular
file that represents a hard disk or the host system itself.
original media, the image is created from, can be:
* a block device, representing a hard disk
* a disk image file, representing a hard disk (supports all image file formats
supported by QEMU)
* the host system itself
Snapshotting
============
......@@ -22,7 +26,7 @@ Those operations will:
* Shrink the image
* Clear out sensitive user data (passwords, ssh keys, history files, etc.)
* Prepare the guest OS for being deployed on a virtual environment (change
device names, remove persistent net rules, etc.)
device names, remove persistent net rules, install VirtIO drivers, etc.)
Creation
========
......
docs/usage.rst
View file @ ce3a618a
......@@ -17,7 +17,7 @@ snf-mkimage receives the following options:
.. code-block:: console
$ snf-mkimage --help
# snf-mkimage --help
Usage: snf-mkimage [options] <input_media>
Options:
......@@ -49,14 +49,16 @@ snf-mkimage receives the following options:
--disable-sysprep=SYSPREP
prevent SYSPREP operation from running on the input
media
--install-virtio=DIR install VirtIO drivers hosted under DIR (Windows only)
--print-sysprep-params
print the needed sysprep parameters for this input
media
print the defined system preparation parameters for
this input media
--sysprep-param=SYSPREP_PARAMS
Add KEY=VALUE system preparation parameter
add KEY=VALUE system preparation parameter
--no-sysprep don't perform any system preparation operation
--no-shrink don't shrink any partition
--public register image with the cloud as public
--allow-unsupported proceed with the image creation even if the media is
not supported
--tmpdir=DIR create large temporary image files under DIR
Most input options are self-describing. If you want to save a local copy of
......@@ -65,10 +67,9 @@ image to the storage service of a cloud, provide valid cloud API access info
(by either using a token and a URL with *-t* and *-a* respectively, or a cloud
name with *-c*) and a remote filename using *-u*. If you also want to register
the image with the compute service of the cloud, in addition to *-u* provide a
registration name using *-r*. All images are
registered as *private*. Only the user that registers the image can create
VM's out of it. If you want the image to be visible by other user too, use the
*--public* option.
registration name using *-r*. All images are registered as *private*. Only the
user that registers the image can create VM's out of it. If you want the image
to be visible by other user too, use the *--public* option.
By default, before extracting the image, snf-mkimage will perform a
number of system preparation operations on the snapshot of the media and will
......@@ -81,69 +82,74 @@ input media. The user can enable or disable specific *syspreps*, using
*-{enable,disable}-sysprep* options. The user may specify those options
multiple times.
Running *snf-mkimage* with *--print-sysprep* on a raw file that hosts a
debian system, we print the following output:
Running *snf-mkimage* with *--print-sysprep* on a raw file that hosts an
Ubuntu system, will print the following output:
.. _sysprep:
.. code-block:: console
$ snf-mkimage --print-syspreps ubuntu.raw
snf-image-creator 0.6
=====================
Examining source media `ubuntu_hd.raw' ... looks like an image file
Snapshotting media source ... done
Enabling recovery proc
Launching helper VM (may take a while) ... done
Inspecting Operating System ... ubuntu
Mounting the media read-only ... done
Collecting image metadata ... done
Umounting the media ... done
Enabled system preparation operations:
cleanup-cache:
Remove all regular files under /var/cache
cleanup-log:
Empty all files under /var/log
cleanup-passwords:
Remove all passwords and lock all user accounts
cleanup-tmp:
Remove all files under /tmp and /var/tmp
cleanup-userdata:
Delete sensitive userdata
fix-acpid:
Replace acpid powerdown action scripts to immediately shutdown the
system without checking if a GUI is running.
remove-persistent-net-rules:
Remove udev rules that will keep network interface names persistent
after hardware changes and reboots. Those rules will be created again
the next time the image runs.
remove-swap-entry:
Remove swap entry from /etc/fstab. If swap is the last partition
then the partition will be removed when shrinking is performed. If the
swap partition is not the last partition in the disk or if you are not
going to shrink the image you should probably disable this.
use-persistent-block-device-names:
Scan fstab & grub configuration files and replace all non-persistent
device references with UUIDs.
Disabled system preparation operations:
cleanup-mail:
Remove all files under /var/mail and /var/spool/mail
remove-user-accounts:
Remove all user accounts with id greater than 1000
cleaning up ...
# snf-mkimage --print-syspreps ubuntu.raw
snf-image-creator 0.7
===========================
Examining source media `ubuntu.raw' ... looks like an image file
Snapshotting media source ... done
Enabling recovery proc
Launching helper VM (may take a while) ... done
Inspecting Operating System ... ubuntu
Collecting image metadata ... done
Running OS inspection:
Checking if the media contains logical volumes (LVM)... no
Enabled system preparation operations:
cleanup-tmp:
Remove all files under /tmp and /var/tmp
remove-swap-entry:
Remove swap entry from /etc/fstab. If swap is the last partition
then the partition will be removed when shrinking is performed. If the
swap partition is not the last partition in the disk or if you are not
going to shrink the image you should probably disable this.
cleanup-cache:
Remove all regular files under /var/cache
cleanup-userdata:
Delete sensitive user data
cleanup-passwords:
Remove all passwords and lock all user accounts
cleanup-log:
Empty all files under /var/log
remove-persistent-net-rules:
Remove udev rules that will keep network interface names persistent
after hardware changes and reboots. Those rules will be created again
the next time the image runs.
use-persistent-block-device-names:
Scan fstab & grub configuration files and replace all non-persistent
device references with UUIDs.
fix-acpid:
Replace acpid powerdown action scripts to immediately shutdown the
system without checking if a GUI is running.
shrink:
Shrink the last file system and update the partition table
Disabled system preparation operations:
remove-user-accounts:
Remove all user accounts with id greater than 1000
cleanup-mail:
Remove all files under /var/mail and /var/spool/mail
cleaning up ...
If you want the image to have all normal user accounts and all mail files
removed, you should use *--enable-sysprep* option like this:
......@@ -152,6 +158,15 @@ removed, you should use *--enable-sysprep* option like this:
$ snf-mkimage --enable-sysprep cleanup-mail --enable-sysprep remove-user-accounts ...
Sysprep parameters are parameters used by some sysprep tasks. In most cases you
don't need to change their value. You can see the available sysprep parameters
and the default values they have by using the *--print-sysprep-params* option.
You can update their values by using the *--sysprep-param* option.
If the media is a Windows image, you can install or update its VirtIO drivers
by using the *--install-virtio* option. With this option you can point to a
directory that hosts a set of extracted Windows VirtIO drivers.
Dialog-based version
====================
......@@ -159,7 +174,7 @@ Dialog-based version
.. code-block:: console
$ snf-image-creator --help
# snf-image-creator --help
Usage: snf-image-creator [options] [<input_media>]
Options:
......@@ -184,13 +199,14 @@ be given the choice to run *snf-image-creator* in *wizard* or *expert* mode.
Wizard mode
-----------
When *snf-image-creator* runs in *wizard* mode, the user is just asked to provide the
following basic information:
When *snf-image-creator* runs in *wizard* mode, the user is just asked to
provide the following basic information:
* Cloud: The cloud account to use to upload and register the resulting image
* Name: A short name for the image (ex. "Slackware")
* Description: An one-line description for the image
(ex. "Slackware Linux 14.0 with KDE")
* VirtIO: A directory that hosts VirtIO drivers (for Windows images only)
* Registration Type: Private or Public
After confirming, the image will be extracted, uploaded to the storage service
......@@ -209,8 +225,8 @@ process. The main menu can be seen in the picture below:
In the *Customize* sub-menu the user can control:
* The installation of VirtIO drivers (only for Windows images)
* The system preparation operations that will be applied on the media
* Whether the image will be shrunk or not
* The properties associated with the image
* The configuration tasks that will run during image deployment
......@@ -229,12 +245,12 @@ Host bundling operation
=======================
As a new feature in *v0.2*, snf-image-creator can create images out of the host
system that runs the program. This is done either by specifying / as input
media or by using the *Bundle Host* button in the media selection dialog of
snf-mkimage. During this operation, the files of the disk are copied into a
temporary image file, which means that the file system that will host the
temporary image needs to have a lot of free space (see `large temporary files`_
for more information).
system that runs the program. This is done either by specifying */* as input
media or by using the *Bundle Host* button in the media selection dialog.
During this operation, the files of the disk are copied into a temporary image
file, which means that the file system that will host the temporary image needs
to have a lot of free space (see `large temporary files`_ for more
information).
Creating a new image
====================
......@@ -272,19 +288,19 @@ And install the Ubuntu system on this file:
use LVM partitions. They are not supported by snf-image-creator.
You will be able to boot your installed OS and make any changes you want
(e.g. install openssh-server) using the following command:
(e.g. install OpenSSH Server) using the following command:
.. code-block:: console
$ sudo kvm -m 1G -boot c -drive file=ubuntu.raw,format=raw,cache=none,if=virtio
After you're done, you may use *snf-image-creator* as root to create and upload the
image:
After you're done, you may use *snf-image-creator* as root to create and upload
the image:
.. code-block:: console
$ sudo -s
$ snf-image-creator ubuntu.raw
# snf-image-creator ubuntu.raw
In the first screen you will be asked to choose if you want to run the program
in *Wizard* or *Expert* mode. Choose *Wizard*.
......@@ -299,6 +315,21 @@ confirm the provided data.
Choosing *YES* will create and upload the image to your cloud account.
Working with different image formats
====================================
*snf-image-creator* is able to work with the most popular disk image formats.
It has been successfully tested with:
* Raw disk images
* VMDK (VMware)
* VHD (Microsoft Hyper-V)
* VDI (VirtualBox)
* qcow2 (QEMU)
It can support any image format QEMU supports as long as it represents a
bootable hard drive.
Limitations
===========
......@@ -315,12 +346,12 @@ Logical Volumes
---------------
The program cannot work on input media that contain LVM partitions inside
[#f1]_. The input media may only contain primary or logical partitions.
[#f2]_. The input media may only contain primary or logical partitions.
Para-virtualized drivers
------------------------
Most synnefo deployments uses the *VirtIO* framework. The disk I/O controller
Most Synnefo deployments uses the *VirtIO* framework. The disk I/O controller
and the Ethernet cards on the VM instances are para-virtualized and need
special *VirtIO* drivers. Those drivers are included in the Linux Kernel
mainline since version 2.6.25 and are shipped with all the popular Linux
......@@ -331,18 +362,15 @@ ramdisk, otherwise the VM won't be able to boot.
Many popular Linux distributions, like Ubuntu and Debian, will automatically
create a generic initial ramdisk file that contains many different modules,
including the VirtIO drivers. Others that target more experienced users, like
Slackware, won't do that [#f2]_. *snf-image-creator* cannot resolve this kind
Slackware, won't do that [#f3]_. *snf-image-creator* cannot resolve this kind
of problems and it's left to the user to do so. Please refer to your
distribution's documentation for more information on this. You can always check
if a system can boot with para-virtualized disk controller by launching it with
kvm using the *if=virtio* option (see the kvm command in the
`Creating a new image`_ section).
For Windows and FreeBSD systems, the needed drivers need to be manually
downloaded and installed on the media before the image creation process takes
place. For *FreeBSD* the virtio drivers can be found
`here <http://people.freebsd.org/~kuriyama/virtio/>`_. For Windows the drivers
are hosted by the
For Windows the program supports installing VirtIO drivers. You may download
the latest drivers from the
`Fedora Project <http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/>`_.
Some caveats on image creation
......@@ -372,19 +400,20 @@ Large temporary files
* During image shrinking, the input media snapshot file may reach the size of
the original media.
* When bundling the host system, the temporary image file may became as large
as the rest of the disk files altogether.
* When bundling the host system, the temporary image file may became 10%
larger than rest of the disk files altogether.
*/tmp* directory is not a good place for hosting large files. In many systems
the contents of */tmp* are stored in volatile memory and the size they may occupy
is limited. By default, *snf-image-creator* will use a heuristic approach to
determine where to store large temporary files. It will examine the free space
under */var/tmp*, the user's home directory and */mnt* and will pick the one
with the most available space. The user may overwrite this behaviour and
the contents of */tmp* are stored in volatile memory and the size they may
occupy is limited. By default, *snf-image-creator* will use a heuristic
approach to determine where to store large temporary files. It will examine the
free space under */var/tmp*, the user's home directory and */mnt* and will pick
the one with the most available space. The user may overwrite this behavior and
indicate a different directory using the *tmpdir* option. This option is
supported by both *snf-image-creator* and *snf-mkimage*.
.. rubric:: Footnotes
.. [#f1] http://sourceware.org/lvm2/
.. [#f2] http://mirrors.slackware.com/slackware/slackware-14.0/README.initrd
.. [#f1] http://technet.microsoft.com/en-us/library/bb676673.aspx
.. [#f2] http://sourceware.org/lvm2/
.. [#f3] http://mirrors.slackware.com/slackware/slackware-14.0/README.initrd
image_creator/__init__.py
View file @ ce3a618a
# -*- coding: utf-8 -*-
#
# Copyright 2012 GRNET S.A. All rights reserved.
# Copyright (C) 2011-2014 GRNET S.A.
#
# Redistribution and use in source and binary forms, with or
# without modification, are permitted provided that the following
# conditions are met:
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# 1. Redistributions of source code must retain the above
# copyright notice, this list of conditions and the following
# disclaimer.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# 2. Redistributions in binary form must reproduce the above
# copyright notice, this list of conditions and the following
# disclaimer in the documentation and/or other materials
# provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
#
# The views and conclusions contained in the software and
# documentation are those of the authors and should not be