Commit 2d76b580 authored by Michael Hanselmann's avatar Michael Hanselmann

Add daemon for instance import and export

This backend daemon for instance import and export will be used to
transfer instance data to other machines. It is implemented in a generic
way to support different ways of data input and output. The third-party
program “socat”, which is already used by the KVM hypervisor abstraction,
is used to connect to remote machines using SSL/TLS. After starting the
child processes in a separate process group, the import/export daemon
monitors their output and updates a status file regularily. This status
file can then be read by ganeti-noded (not in this patch).

Three I/O methods are supported: Raw disk, file and script. Each of these
can be used for import and export.

Similar to daemon-util, an incomplete set of tests written in Bash is
included.

Two future enhancements are planned:
- Run parts of the command chain as a dedicated user (privilege
  separation).
- Currently users of this daemon have to poll the status file while data
  is transferred. This is inefficient and creates unnecessary delays. By
  adding “dd” into the chain and sending it SIGUSR1 regularily, we can get
  some statistics, optimize the polling frequenc and even provide the user
  with an ETA (which isn't available with all current methods to
  import/export instance data).
Signed-off-by: default avatarMichael Hanselmann <hansmi@google.com>
Reviewed-by: default avatarIustin Pop <iustin@google.com>
parent 409e8ec2
......@@ -241,8 +241,12 @@ dist_tools_SCRIPTS = \
tools/lvmstrap \
tools/sanitize-config
pkglib_python_scripts = \
daemons/import-export
pkglib_SCRIPTS = \
daemons/daemon-util
daemons/daemon-util \
$(pkglib_python_scripts)
EXTRA_DIST = \
NEWS \
......@@ -256,6 +260,7 @@ EXTRA_DIST = \
$(RUN_IN_TEMPDIR) \
daemons/daemon-util.in \
daemons/ganeti-cleaner.in \
$(pkglib_python_scripts) \
devel/upload.in \
$(docdot) \
$(docpng) \
......@@ -322,7 +327,8 @@ TEST_FILES = \
test/data/cert1.pem \
test/data/proc_drbd8.txt \
test/data/proc_drbd80-emptyline.txt \
test/data/proc_drbd83.txt
test/data/proc_drbd83.txt \
test/import-export_unittest-helper
python_tests = \
test/ganeti.backend_unittest.py \
......@@ -351,6 +357,7 @@ python_tests = \
dist_TESTS = \
test/daemon-util_unittest.bash \
test/import-export_unittest.bash \
$(python_tests)
nodist_TESTS =
......@@ -368,6 +375,7 @@ TESTS_ENVIRONMENT = \
all_python_code = \
$(dist_sbin_SCRIPTS) \
$(dist_tools_SCRIPTS) \
$(pkglib_python_scripts) \
$(python_tests) \
$(pkgpython_PYTHON) \
$(hypervisor_PYTHON) \
......@@ -379,6 +387,7 @@ all_python_code = \
srclink_files = \
man/footer.sgml \
test/daemon-util_unittest.bash \
test/import-export_unittest.bash \
$(all_python_code)
check_python_code = \
......@@ -389,6 +398,7 @@ lint_python_code = \
ganeti \
$(dist_sbin_SCRIPTS) \
$(dist_tools_SCRIPTS) \
$(pkglib_python_scripts) \
$(BUILD_BASH_COMPLETION)
test/daemon-util_unittest.bash: daemons/daemon-util
......
This diff is collapsed.
......@@ -8,7 +8,7 @@ output: html
# note: the wildcards means the directories should be cleaned up after each
# run, otherwise there will be stale '*c' (compiled) files that will not be
# parsable and will break the epydoc run
modules: ganeti, scripts/gnt-*, daemons/ganeti-confd, daemons/ganeti-masterd, daemons/ganeti-noded, daemons/ganeti-rapi, daemons/ganeti-watcher, tools/*
modules: ganeti, scripts/gnt-*, daemons/ganeti-confd, daemons/ganeti-masterd, daemons/ganeti-noded, daemons/ganeti-rapi, daemons/ganeti-watcher, daemons/import-export, tools/*
graph: all
......
......@@ -193,6 +193,12 @@ X509_CERT_SIGN_DIGEST = "SHA1"
X509_CERT_SIGNATURE_HEADER = "X-Ganeti-Signature"
IMPORT_EXPORT_DAEMON = _autoconf.PKGLIBDIR + "/import-export"
# Import/export daemon mode
IEM_IMPORT = "import"
IEM_EXPORT = "export"
VALUE_DEFAULT = "default"
VALUE_AUTO = "auto"
VALUE_GENERATE = "generate"
......
......@@ -1007,6 +1007,17 @@ class BlockDevStatus(ConfigObject):
]
class ImportExportStatus(ConfigObject):
"""Config object representing the status of an import or export."""
__slots__ = [
"recent_output",
"listen_port",
"connected",
"exit_status",
"error_message",
] + _TIMESTAMPS
class ConfdRequest(ConfigObject):
"""Object holding a confd request.
......
#!/usr/bin/python
#
# Copyright (C) 2010 Google Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301, USA.
"""Helpers for testing import-export daemon"""
import os
import sys
import errno
from ganeti import constants
from ganeti import utils
from ganeti import objects
from ganeti import serializer
RETRY_INTERVAL = 0.1
TIMEOUT = int(os.getenv("TIMEOUT", 10))
def _GetImportExportData(filename):
try:
data = utils.ReadFile(filename)
except EnvironmentError, err:
if err.errno != errno.ENOENT:
raise
raise utils.RetryAgain()
return objects.ImportExportStatus.FromDict(serializer.LoadJson(data))
def _CheckConnected(filename):
if not _GetImportExportData(filename).connected:
raise utils.RetryAgain()
def WaitForListenPort(filename):
return utils.Retry(lambda: _GetImportExportData(filename).listen_port,
RETRY_INTERVAL, TIMEOUT)
def WaitForConnected(filename):
utils.Retry(_CheckConnected, RETRY_INTERVAL, TIMEOUT, args=(filename, ))
def main():
(filename, what) = sys.argv[1:]
if what == "listen-port":
print WaitForListenPort(filename)
elif what == "connected":
WaitForConnected(filename)
elif what == "gencert":
utils.GenerateSelfSignedSslCert(filename, validity=1)
else:
raise Exception("Unknown command '%s'" % what)
if __name__ == "__main__":
main()
#!/bin/bash
#
# Copyright (C) 2010 Google Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301, USA.
set -e
set -o pipefail
export PYTHON=${PYTHON:=python}
impexpd="$PYTHON daemons/import-export"
# Add "-d" for debugging
#impexpd+=' -d'
err() {
echo "$@"
echo 'Aborting'
exit 1
}
checkpids() {
local result=0
# Unlike combining the "wait" commands using || or &&, this ensures we
# actually wait for all PIDs.
for pid in "$@"; do
if ! wait $pid; then
result=1
fi
done
return $result
}
get_testpath() {
echo "${TOP_SRCDIR:-.}/test"
}
get_testfile() {
echo "$(get_testpath)/data/$1"
}
statusdir=$(mktemp -d)
trap "rm -rf $statusdir" EXIT
src_statusfile=$statusdir/src.status
src_x509=$statusdir/src.pem
dst_statusfile=$statusdir/dst.status
dst_x509=$statusdir/dst.pem
dst_portfile=$statusdir/dst.port
other_x509=$statusdir/other.pem
testdata=$statusdir/data1
cmd_prefix=
cmd_suffix=
$impexpd >/dev/null 2>&1 &&
err "daemon-util succeeded without parameters"
$impexpd foo bar baz moo boo >/dev/null 2>&1 &&
err "daemon-util succeeded with wrong parameters"
$impexpd $src_statusfile >/dev/null 2>&1 &&
err "daemon-util succeeded with insufficient parameters"
$impexpd $src_statusfile invalidmode >/dev/null 2>&1 &&
err "daemon-util succeeded with invalid mode"
cat $(get_testfile proc_drbd8.txt) $(get_testfile cert1.pem) > $testdata
impexpd_helper() {
$PYTHON $(get_testpath)/import-export_unittest-helper "$@"
}
reset_status() {
rm -f $src_statusfile $dst_statusfile $dst_portfile
}
write_data() {
# Wait for connection to be established
impexpd_helper $dst_statusfile connected
cat $testdata
}
do_export() {
# Wait for listening port
impexpd_helper $dst_statusfile listen-port > $dst_portfile
local port=$(< $dst_portfile)
test -n "$port" || err 'Empty port file'
do_export_to_port $port
}
do_export_to_port() {
local port=$1
$impexpd $src_statusfile export --bind=127.0.0.1 \
--host=127.0.0.1 --port=$port \
--key=$src_x509 --cert=$src_x509 --ca=$dst_x509 \
--cmd-prefix="$cmd_prefix" --cmd-suffix="$cmd_suffix"
}
do_import() {
$impexpd $dst_statusfile import --bind=127.0.0.1 \
--host=127.0.0.1 \
--key=$dst_x509 --cert=$dst_x509 --ca=$src_x509 \
--cmd-prefix="$cmd_prefix" --cmd-suffix="$cmd_suffix"
}
# Generate X509 certificates and keys
impexpd_helper $src_x509 gencert
impexpd_helper $dst_x509 gencert
impexpd_helper $other_x509 gencert
# Normal case
reset_status
do_import > $statusdir/recv1 & imppid=$!
write_data | do_export & exppid=$!
checkpids $exppid $imppid || err 'An error occurred'
cmp $testdata $statusdir/recv1 || err 'Received data does not match input'
# Export using wrong CA
reset_status
do_import > /dev/null 2>&1 & imppid=$!
: | dst_x509=$other_x509 do_export 2>/dev/null & exppid=$!
checkpids $exppid $imppid && err 'Export did not fail when using wrong CA'
# Import using wrong CA
reset_status
src_x509=$other_x509 do_import > /dev/null 2>&1 & imppid=$!
: | do_export 2> /dev/null & exppid=$!
checkpids $exppid $imppid && err 'Import did not fail when using wrong CA'
# Suffix command on import
reset_status
cmd_suffix="| cksum > $statusdir/recv2" do_import & imppid=$!
write_data | do_export & exppid=$!
checkpids $exppid $imppid || err 'Testing additional commands failed'
cmp $statusdir/recv2 <(cksum < $testdata) || \
err 'Checksum of received data does not match'
# Prefix command on export
reset_status
do_import > $statusdir/recv3 & imppid=$!
write_data | cmd_prefix="cksum |" do_export & exppid=$!
checkpids $exppid $imppid || err 'Testing additional commands failed'
cmp $statusdir/recv3 <(cksum < $testdata) || \
err 'Received checksum does not match'
# Failing prefix command on export
reset_status
: | cmd_prefix='exit 1;' do_export_to_port 0 & exppid=$!
checkpids $exppid && err 'Prefix command on export did not fail when it should'
# Failing suffix command on export
reset_status
do_import > /dev/null & imppid=$!
: | cmd_suffix='| exit 1' do_export & exppid=$!
checkpids $imppid $exppid && \
err 'Suffix command on export did not fail when it should'
# Failing prefix command on import
reset_status
cmd_prefix='exit 1;' do_import > /dev/null & imppid=$!
checkpids $imppid && err 'Prefix command on import did not fail when it should'
# Failing suffix command on import
reset_status
cmd_suffix='| exit 1' do_import > /dev/null & imppid=$!
: | do_export & exppid=$!
checkpids $imppid $exppid && \
err 'Suffix command on import did not fail when it should'
exit 0
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment