Skip to content

GitLab

Commit d1bb5754 authored by Nikos Skalkotos's avatar Nikos Skalkotos
Browse files

Merge branch 'master' into debian-precise

parents 86aad523 072ab80f
Hide whitespace changes
Inline Side-by-side
ChangeLog
View file @ d1bb5754
2014-02-06, v0.13
* Add support for extending OpenBSD partitions
* Add a new HELPER_MEMORY option in /etc/default/snf-image. This can be
used to specify the Virtual RAM size of the helper VM
2014-01-24, v0.12.1
* Fix a bug affecting import and export scripts
2013-11-04, v0.12
* Add support for OpenBSD and NetBSD images (partition resizing is not
supported yet)
......
docs/advanced.rst
View file @ d1bb5754
......@@ -31,7 +31,7 @@ Known Issues
------------
* For Linux systems, having grub installed in the partition is fragile and
things can go wrong when resizing the partitions, especially when shrinking.
things can go wrong if you shrink the partition.
* More complicated partition schemes are not supported.
diskdump image format (recommended)
......@@ -44,18 +44,24 @@ This design decision has the following benefits:
* Swap partitions are supported
* The system may use multiple partitions:
* dedicated partitions for /boot, /home etc in Linux
* system and boot partition in Windows
* Dedicated partitions for /boot, /home etc. in Linux
* Separate system and boot partition in Windows
* There are no restrictions on starting sectors of partitions
Although diskdump is a lot more flexible than the older formats, there are
still some rules to follow:
* All devices in fstab should be specified by persistent names (UUID or LABEL)
* LVMs are not supported
* For Linux disks only ext{2,3,4} file systems are supported
* For FreeBSD disks only UFS file systems are supported
* For FreeBSD only GUID Partition Tables (GPT) are supported
* For Linux:
* All block devices in */etc/fstab* should be specified using persistent
names (UUID or LABEL)
* LVM partitions are not supported
* Only ext{2,3,4} file systems are supported
* For FreeBSD:
* GUID Partition Tables (GPT) should be used
* Only UFS2 file systems are supported
* Labels should be omitted in */etc/fstab* entries
* For {Open,Net}BSD:
* Only FFS file systems should be used
Progress Monitoring Interface
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
......@@ -67,7 +73,7 @@ the *PROGRESS_MONITOR* variable under ``/etc/default/snf-image`` and
program. In this section we will describe the format and the fields of the
progress messages.
The progress messages are json strings with standardized fields. All messages
The progress messages are JSON strings with standardized fields. All messages
have a **type** field whose value is a string and a **timestamp** field whose
value is a floating point number referring to a time encoded as the number of
seconds elapsed since the epoch. The rest of the field depend on the specific
......@@ -138,7 +144,7 @@ warning
This messages are produced to display a warning. The actual warning message
itself is present in the *messages* field:
``{"subtype": "warning", "type": "image-helper", "messages": [" No swap partition defined"], "timestamp": 1379075807.71704}``
``{"subtype": "warning", "type": "image-helper", "messages": ["No swap partition defined"], "timestamp": 1379075807.71704}``
error
-----
......
docs/conf.py
View file @ d1bb5754
......@@ -45,9 +45,9 @@ copyright = u'2011, 2012, 2013 GRNET S.A. All rights reserved'
# built documents.
#
# The short X.Y version.
version = '0.12'
version = '0.13'
# The full version, including alpha/beta/rc tags.
release = '0.12'
release = '0.13'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
......
docs/index.rst
View file @ d1bb5754
......@@ -19,16 +19,15 @@ snf-image supports `KVM <http://www.linux-kvm.org/page/Main_Page>`_ and
snf-image also supports Image customization via hooks. Hooks allow for:
* changing the password of root or arbitrary users
* injecting files at arbitrary locations inside the filesystem, e.g., SSH keys
* Changing the password of root or arbitrary users
* Injecting files into the file system, e.g., SSH keys
* setting a custom hostname
* re-creating SSH host keys to ensure the image uses unique keys
snf-image is being used in large scale production environments with Ganeti to
successfully deploy many major Linux distributions (Debian, Ubuntu/Kubuntu,
CentOS, Fedora, OpenSUSE), Windows 2008 R2 & Windows Server 2012, as well as
FreeBSD. Support for OpenBSD and NetBSD is also included with exception to
extending partitions.
CentOS, Fedora, OpenSUSE, Slackware, Arch Linux), Windows Server flavors
(2008 R2, 2012, 20012 R2), as well as BSD systems (FreeBSD, OpenBSD, NetBSD).
The snf-image Ganeti OS Definition is released under
`GPLv2 <http://www.gnu.org/licenses/gpl-2.0.html>`_.
......
docs/interface.rst
View file @ d1bb5754
......@@ -75,7 +75,7 @@ to be used. If no prefix is used, it defaults to the local back-end:
| For example, if we want to deploy using a full Pithos URI:
| ``img_id=pithos://<user-uuid>/<container>/<slackware-image>``
| or if we already know the map:
| ``img_id=pithosmap://<slackware-image-map-name>``
| ``img_id=pithosmap://<slackware-image-map-name>/<size>``
* **Null back-end**:
To select the Null back-end and skip the fetching and extraction step, we set
......
docs/usage.rst
View file @ d1bb5754
......@@ -27,10 +27,10 @@ that have been tested with snf-image and provided here for testing purposes:
[`diskdump <http://cdn.synnefo.org/centos-6.0-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/centos-6.0-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/centos-6.0-x86_64.diskdump.meta>`_]
* Fedora Desktop 18
[`diskdump <http://cdn.synnefo.org/fedora-18-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/fedora-18-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/fedora-18-x86_64.diskdump.meta>`_]
* Fedora Desktop 19
[`diskdump <http://cdn.synnefo.org/fedora-19-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/fedora-19-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/fedora-19-x86_64.diskdump.meta>`_]
* Ubuntu Desktop LTS 12.04
[`diskdump <http://cdn.synnefo.org/ubuntu_desktop-12.04-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/ubuntu_desktop-12.04-x86_64.diskdump.md5sum>`_]
......@@ -39,26 +39,34 @@ that have been tested with snf-image and provided here for testing purposes:
[`diskdump <http://cdn.synnefo.org/kubuntu_desktop-12.04-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/kubuntu_desktop-12.04-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/kubuntu_desktop-12.04-x86_64.diskdump.meta>`_]
* Ubuntu Desktop 13.04
[`diskdump <http://cdn.synnefo.org/ubuntu_desktop-13.04-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/ubuntu_desktop-13.04-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/ubuntu_desktop-13.04-x86_64.diskdump.meta>`_]
* Kubuntu 13.04
[`diskdump <http://cdn.synnefo.org/kubuntu_desktop-13.04-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/kubuntu_desktop-13.04-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/kubuntu_desktop-13.04-x86_64.diskdump.meta>`_]
* Ubuntu Desktop 13.10
[`diskdump <http://cdn.synnefo.org/ubuntu_desktop-13.10-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/ubuntu_desktop-13.10-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/ubuntu_desktop-13.10-x86_64.diskdump.meta>`_]
* Kubuntu 13.10
[`diskdump <http://cdn.synnefo.org/kubuntu_desktop-13.10-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/kubuntu_desktop-13.10-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/kubuntu_desktop-13.10-x86_64.diskdump.meta>`_]
* Ubuntu Server 12.04
[`diskdump <http://cdn.synnefo.org/ubuntu_server-12.04-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/ubuntu_server-12.04-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/ubuntu_server-12.04-x86_64.diskdump.meta>`_]
* OpenSUSE Desktop 12.3
[`diskdump <http://cdn.synnefo.org/opensuse_desktop-12.3-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/opensuse_desktop-12.3-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/opensuse_desktop-12.3-x86_64.diskdump.meta>`_]
* OpenSUSE Desktop 13.1
[`diskdump <http://cdn.synnefo.org/opensuse_desktop-13.1-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/opensuse_desktop-13.1-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/opensuse_desktop-13.1-x86_64.diskdump.meta>`_]
* FreeBSD 9.2
[`diskdump <http://cdn.synnefo.org/freebsd-9.2-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/freebsd-9.2-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/freebsd-9.2-x86_64.diskdump.meta>`_]
* OpenBSD 5.4
[`diskdump <http://cdn.synnefo.org/openbsd-5.4-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/openbsd-5.4-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/openbsd-5.4-x86_64.diskdump.meta>`_]
* NetBSD 6.1
[`diskdump <http://cdn.synnefo.org/netbsd-6.1-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/netbsd-6.1-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/netbsd-6.1-x86_64.diskdump.meta>`_]
Sample Usage
^^^^^^^^^^^^
......
snf-image-helper/Makefile.am
View file @ d1bb5754
......@@ -7,7 +7,7 @@ SUBDIRS = tasks
dist_doc_DATA = COPYING AUTHORS
dist_bin_SCRIPTS = snf-image-helper
dist_scripts_SCRIPTS= hashpwd.py inject-files.py decode-properties.py
dist_scripts_SCRIPTS= hashpwd.py inject-files.py decode-properties.py disklabel.py
dist_common_DATA = common.sh unattend.xml
edit = sed \
......@@ -24,4 +24,4 @@ edit = sed \
$(edit) $${srcdir}$@.in >$@.tmp
mv $@.tmp $@
CLEANFILES = snf-image-helper
CLEANFILES = snf-image-helper common.sh
snf-image-helper/common.sh → snf-image-helper/common.sh.in
View file @ d1bb5754
......@@ -34,6 +34,8 @@ CHNTPW=chntpw
SGDISK=sgdisk
GROWFS_UFS=growfs.ufs
DUMPFS_UFS=dumpfs.ufs
GROWFS_OPENBSD=growfs.openbsd
DUMPFS_OPENBSD=dumpfs.openbsd
DATE="date -u" # Time in UTC
EATMYDATA=eatmydata
MOUNT="mount -n"
......@@ -194,13 +196,13 @@ get_base_distro() {
elif [ -e "$root_dir/etc/freebsd-update.conf" ]; then
echo "freebsd"
elif [ -e "$root_dir/etc/release" ]; then
if grep -in netbsd "$root_dir/etc/release" &> /dev/null; then
if grep -i netbsd "$root_dir/etc/release" &> /dev/null; then
echo "netbsd"
else
warn "Unknown Unix flavor."
fi
elif [ -e "$root_dir/etc/magic" ]; then
if grep -in openbsd "$root_dir/etc/magic" &> /dev/null; then
elif [ -e "$root_dir/etc/motd" ]; then
if grep -i ^openbsd <(head -1 "$root_dir/etc/motd") &> /dev/null; then
echo "openbsd"
else
warn "Unknown Unix flavor"
......@@ -245,8 +247,8 @@ get_distro() {
else
warn "Unknown Unix flavor"
fi
elif [ -e "$root_dir/etc/magic" ]; then
if grep -in openbsd "$root_dir/etc/magic" &> /dev/null; then
elif [ -e "$root_dir/etc/motd" ]; then
if grep -i ^openbsd <(head -1 "$root_dir/etc/motd") &> /dev/null; then
echo "openbsd"
else
warn "Unknown Unix flavor"
......@@ -485,6 +487,110 @@ get_unattend() {
echo "$exists"
}
disklabel2linux() {
local partition i p
partition="$1"
i=4
# Partition 'c' traditionally used to describe the entire disk is not
# mapped to /dev/sda7 by the kernel
for p in a b {d..p}; do
let i++
if [ "$p" = "$partition" ]; then
echo "$i"
return 0
fi
done
log_error "Invalid BSD partition label: \`$partition'"
}
mount_all() {
local osfamily target fs device fstab entry duid opts types num
osfamily="$1"
device="$2"
target="$3"
case "$osfamily" in
linux)
fs="ext[234]|msdos|vfat|ntfs"
;;
freebsd)
fs="ufs|msdosfs|ntfs"
;;
openbsd)
fs="ffs|msdos|ntfs|ext2fs"
;;
netbsd)
fs="ffs|ufs|msdos|ext2fs|ntfs"
;;
*)
log_error "Unsupported osfamily: \`$osfamily'"
;;
esac
fstab="$(grep -v ^\# "${target}/etc/fstab" | awk "{ if (match(\$3, \"$fs\")) { print \$2,\$1,\$3 } }" | sort -bd)"
# <mpoint> <device> <fs>
while read -ra entry; do
# Skip root. It is already mounted
if [ "${entry[0]}" = "/" ]; then
continue
fi
opts="rw"
types="auto"
if [ "$osfamily" = linux ]; then
# Linux persistent block device naming
if [[ ${entry[1]} =~ ^(LABEL|UUID)= ]]; then
entry[1]=$(findfs "${entry[1]}")
else
log_error "fstab contains non-persistent block device names"
fi
else
if [[ "$osfamily" =~ ^(open|net)bsd$ ]]; then
# OpenBSD DUIDs device naming
if [[ "${entry[1]}" =~ ^[a-f0-9]{16}\.[a-p]$ ]]; then
duid="$(@scriptsdir@/disklabel.py --get-duid "$device")"
if [[ ! "${entry[1]}" =~ ^$duid ]]; then
warn "fstab refers to unknown DUID: \`$duid'"
continue
fi
fi
num="$(disklabel2linux "${entry[1]: -1}")"
if [ "${entry[2]}" = ffs -o "$entry[2]" = ufs ]; then
types="ufs"
opts="ufstype=44bsd,rw"
fi
else # FreeBSD
# We do not support FreeBSD labels for now
if [[ "${entry[1]}" =~ ^/dev/(ufs|label)/ ]]; then
log_error "fstab contains FreeBSD labels. We currently don't support them"
fi
num="${entry[1]: -1}"
if [ "${entry[2]}" = ufs ]; then
types="ufs"
opts="ufstype=ufs2,rw"
fi
fi
entry[1]="${device}${num}"
fi
$MOUNT -t "$types" -o "$opts" "${entry[1]}" "${target}${entry[0]}"
# In many cases when you try to mount a UFS file system read-write, the
# mount command returns SUCCESS and a message like this gets printed:
#
# mount: warning: <target> seems to be mounted read-only.
#
# remounting does the trick
if [ "$types" = ufs ]; then
$MOUNT -o remount,rw "${entry[1]}"
fi
done <<< "$fstab"
}
umount_all() {
local target mpoints
target="$1"
......
snf-image-helper/disklabel.py 0 → 100755
View file @ d1bb5754
#!/usr/bin/env python
#
# -*- coding: utf-8 -*-
#
# Copyright (C) 2013 GRNET S.A.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301, USA.
"""This module provides the code for handling OpenBSD disklabels"""
import struct
import sys
import cStringIO
import optparse
from collections import namedtuple
BLOCKSIZE = 512
LABELSECTOR = 1
LABELOFFSET = 0
BBSIZE = 8192 # size of boot area with label
SBSIZE = 8192 # max size of fs superblock
DISKMAGIC = 0x82564557
class MBR(object):
"""Represents a Master Boot Record."""
class Partition(object):
"""Represents a partition entry in MBR"""
format = "<B3sB3sLL"
def __init__(self, raw_part):
"""Create a Partition instance"""
(
self.status,
self.start,
self.type,
self.end,
self.first_sector,
self.sector_count
) = struct.unpack(self.format, raw_part)
def pack(self):
"""Pack the partition values into a binary string"""
return struct.pack(self.format,
self.status,
self.start,
self.type,
self.end,
self.first_sector,
self.sector_count)
@staticmethod
def size():
"""Returns the size of an MBR partition entry"""
return struct.calcsize(MBR.Partition.format)
def __str__(self):
start = self.unpack_chs(self.start)
end = self.unpack_chs(self.end)
return "%d %s %d %s %d %d" % (self.status, start, self.type, end,
self.first_sector, self.sector_count)
@staticmethod
def unpack_chs(chs):
"""Unpacks a CHS address string to a tuple."""
assert len(chs) == 3
head = struct.unpack('<B', chs[0])[0]
sector = struct.unpack('<B', chs[1])[0] & 0x3f
cylinder = (struct.unpack('<B', chs[1])[0] & 0xC0) << 2 | \
struct.unpack('<B', chs[2])[0]
return (cylinder, head, sector)
@staticmethod
def pack_chs(cylinder, head, sector):
"""Packs a CHS tuple to an address string."""
assert 1 <= sector <= 63
assert 0 <= head <= 255
assert 0 <= cylinder
# If the cylinders overflow then put the value (1023, 254, 63) to
# the tuple. At least this is what OpenBSD does.
if cylinder > 1023:
cylinder = 1023
head = 254
sector = 63
byte0 = head
byte1 = (cylinder >> 2) & 0xC0 | sector
byte2 = cylinder & 0xff
return struct.pack('<BBB', byte0, byte1, byte2)
format = "<444s2x16s16s16s16s2s"
"""
Offset Length Contents
0 440(max. 446) code area
440 2(optional) disk signature
444 2 Usually nulls
446 16 Partition 0
462 16 Partition 1
478 16 Partition 2
494 16 Partition 3
510 2 MBR signature
"""
def __init__(self, block):
"""Create an MBR instance"""
raw_part = {}
(self.code_area,
raw_part[0],
raw_part[1],
raw_part[2],
raw_part[3],
self.signature) = struct.unpack(self.format, block)
self.part = {}
for i in range(4):
self.part[i] = self.Partition(raw_part[i])
@staticmethod
def size():
"""Return the size of a Master Boot Record."""
return struct.calcsize(MBR.format)
def pack(self):
"""Pack an MBR to a binary string."""
return struct.pack(self.format,
self.code_area,
self.part[0].pack(),
self.part[1].pack(),
self.part[2].pack(),
self.part[3].pack(),
self.signature)
def __str__(self):
ret = ""
for i in range(4):
ret += "Partition %d: %s\n" % (i, self.part[i])
ret += "Signature: %s %s\n" % (hex(ord(self.signature[0])),
hex(ord(self.signature[1])))
return ret
class Disklabel:
"""Represents an OpenBSD Disklabel"""
format = "<IHH16s16sIIIIII8sIHHIII20sHH16sIHHII364s"
"""
Offset Length Contents
0 4 Magic
4 2 Drive Type
6 2 Subtype
8 16 Type Name
24 16 Pack Identifier
32 4 Bytes per sector
36 4 Data sectors per track
40 4 Tracks per cilinder
44 4 Data cylinders per unit
48 4 Data sectors per cylynder
52 4 Data sectors per unit
56 8 Unique label identifier
64 4 Alt cylinders per unit
68 2 Start of useable region (high part)
70 2 Size of usable region (high part)
72 4 Start of useable region
76 4 End of usable region
80 4 Generic Flags
84 5*4 Drive-type specific information
104 2 Number of data sectors (high part)
106 2 Version
108 4*4 Reserved for future use
124 4 Magic number
128 2 Xor of data Inclu. partitions
130 2 Number of partitions in following
132 4 size of boot area at sn0, bytes
136 4 Max size of fs superblock, bytes
140 16*16 Partition Table
"""
class PartitionTable:
"""Reprepsents an OpenBSD Partition Table"""
format = "<IIHHBBH"
"""
Partition Entry:
Offset Length Contents
0 4 Number of sectors in the partition
4 4 Starting sector
8 2 Starting sector (high part)
10 2 Number of sectors (high part)
12 1 Filesystem type
13 1 Filesystem Fragment per block
14 2 FS cylinders per group
"""
Partition = namedtuple(
'Partition', 'size, offset, offseth, sizeh, fstype, frag, cpg')
def __init__(self, ptable, pnumber):
"""Create a Partition Table instance"""
self.part = []
size = struct.calcsize(self.format)
raw = cStringIO.StringIO(ptable)
try:
for i in range(pnumber):
p = self.Partition(
*struct.unpack(self.format, raw.read(size)))
self.part.append(p)
finally:
raw.close()
def __str__(self):
"""Print the Partition table"""
val = ""
for i in range(len(self.part)):
val += "%c: %s\n" % (chr(ord('a') + i), str(self.part[i]))
return val
def pack(self):
"""Packs the partition table into a binary string."""
ret = ""
for i in range(len(self.part)):
ret += struct.pack(self.format,
self.part[i].size,
self.part[i].offset,
self.part[i].offseth,
self.part[i].sizeh,
self.part[i].fstype,
self.part[i].frag,
self.part[i].cpg)
return ret
def setpsize(self, i, size):
"""Set size for partition i"""
tmp = self.part[i]
self.part[i] = self.Partition(size & 0xffffffff, tmp.offset,
tmp.offseth, size >> 32, tmp.fstype,
tmp.frag, tmp.cpg)
def getpsize(self, i):
return (self.part[i].sizeh << 32) + self.part[i].size
def setpoffset(self, i, offset):
"""Set offset for partition i"""
tmp = self.part[i]
self.part[i] = self.Partition(tmp.size, offset & 0xffffffff,
offset >> 32, tmp.sizeh, tmp.frag,
tmp.cpg)
def getpoffset(self, i):
return (self.part[i].offseth << 32) + self.part[i].offset
def __init__(self, disk):
"""Create a DiskLabel instance"""
self.disk = disk
self.part_num = None
with open(disk, "rb") as d:
sector0 = d.read(BLOCKSIZE)
self.mbr = MBR(sector0)
for i in range(4):
if self.mbr.part[i].type == 0xa6: # OpenBSD type
self.part_num = i
break
assert self.part_num is not None, "No OpenBSD partition found"
d.seek(BLOCKSIZE * self.mbr.part[self.part_num].first_sector)
part_sector0 = d.read(BLOCKSIZE)
# The offset of the disklabel from the begining of the