Commit d1fb3a36 authored by Nikos Skalkotos's avatar Nikos Skalkotos

Merge branch 'release-0.15'

parents 71842ea0 d3d77c06
2014-16-05, v0.15rc3
* Fix bugs in pithcat introduced in rc2
2014-15-05, v0.15rc2
* Add pithos backend version check in pithcat.
2014-13-05, v0.15rc1
* Add support for selectable pithos storage backend. Possible storage
backends for pithos are `nfs' or `rados'
* Fix bugs and update the documentation
2014-04-11, v0.14
* Add support for extending NetBSD partitions
* Change the default Virtual RAM size of the helper VM to 512M
......@@ -151,9 +162,9 @@
* Use Coordinated Universal Time when logging.
* Fix a bug in the way snf-image-update-helper reads the links of the
kernel and the initrd in the helper VM.
* Use environmental variables instead of input options to pass
sensitive data to pithcat. This is needed because when pithcat is
running, every user can see the program's command line by using ps.
* Use environment variables instead of input options to pass sensitive
data to pithcat. This is needed because when pithcat is running,
every user can see the program's command line by using ps.
2012-09-15, v0.6
* Fix import and export scripts
......
[ packages ]
[[ devflow ]]
version_file = "version.m4"
version_template = "version_template"
version_file = "version.m4","docs/version.py"
version_template = "version_template","docs/version_template"
......@@ -106,8 +106,12 @@ newly created block device. The following back-ends are supported:
* **Pithos backend**:
*snf-image* contains a special command-line tool (*pithcat*) for retrieving
images residing on a Pithos installation. To set up snf-image's Pithos
backend the user needs to setup the ``PITHOS_DATA`` and ``PITHOS_DB``
variables inside ``/etc/default/snf-image`` accordingly.
backend the user needs to setup the ``PITHOS_BACKEND_STORAGE`` variable
inside ``/etc/default/snf-image``.
Possible values are ``nfs`` and ``rados``. If ``nfs`` is used the user needs
to setup ``PITHOS_DATA`` variable, and when ``rados`` is used the user needs
to setup ``PITHOS_RADOS_POOL_MAPS`` and ``PITHOS_RADOS_POOL_BLOCKS``
accordingly.
* **Null backend**:
If the null backend is selected, no image copying is performed. This
......@@ -127,20 +131,20 @@ below for a description of each one of them:
**FixPartitionTable**: Enlarges the last partition in the partition table of
the instance, to consume all the available space and optionally adds a swap
partition in the end. The task will fail if the environmental variable
partition in the end. The task will fail if the environment variable
*SNF_IMAGE_DEV*, which specifies the device file of the instance's hard disk,
is missing.
**FilesystemResizeUnmounted**: Extends the file system of the last partition to
cover up the whole partition. This only works for ext{2,3,4}, FFS and UFS2 file
systems. Any other file system type is ignored and a warning is triggered. The
task will fail if *SNF_IMAGE_DEV* environmental variable is missing.
task will fail if *SNF_IMAGE_DEV* environment variable is missing.
**MountImage**: Mounts the root partition of the instance, specified by the
*SNF_IMAGE_PROPERTY_ROOT_PARTITION* variable. On Linux systems after the root
fs is mounted, the instance's ``/etc/fstab`` file is examined and the rest of
the disk file systems are mounted too, in a correct order. The script will fail
if any of the environmental variables *SNF_IMAGE_DEV*,
if any of the environment variables *SNF_IMAGE_DEV*,
*SNF_IMAGE_PROPERTY_ROOT_PARTITION* or *SNF_IMAGE_TARGET* is unset or has a
non-sane value.
......@@ -167,7 +171,7 @@ is needed by Windows in order to perform an unattended setup. The
**SELinuxAutorelabel**: Creates *.autorelabel* file in Red Hat images. This is
needed if SELinux is enabled to enforce an automatic file system relabeling
during the first boot. The only environmental variable required by this task is
during the first boot. The only environment variable required by this task is
*SNF_IMAGE_TARGET*.
**AssignHostname**: Assigns or changes the hostname of the instance. The task
......@@ -200,7 +204,7 @@ missing a warning is produced. Only *SNF_IMAGE_TARGET* is required for this
task to run.
**UmountImage**: Umounts the file systems previously mounted by MountImage. The
only environmental variable required is *SNF_IMAGE_TARGET*.
only environment variable required is *SNF_IMAGE_TARGET*.
+-------------------------------+---+--------------------------------------------+--------------------------------------------------+
......@@ -241,4 +245,4 @@ only environmental variable required is *SNF_IMAGE_TARGET*.
|UmountImage |80 |MountImage | |TARGET | |
+-------------------------------+---+------------------+-------------------------+-------------------------+------------------------+
.. [#] all environmental variables are prefixed with *SNF_IMAGE_*
.. [#] all environment variables are prefixed with *SNF_IMAGE_*
......@@ -12,7 +12,8 @@
# serve to show the default.
import sys, os
sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
from version import __version__
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
......@@ -45,9 +46,9 @@ copyright = u'2011, 2012, 2013 GRNET S.A. All rights reserved'
# built documents.
#
# The short X.Y version.
version = '0.14'
version = __version__
# The full version, including alpha/beta/rc tags.
release = '0.14'
release = __version__
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
......
......@@ -64,9 +64,25 @@ some external programs in ``/etc/default/snf-image``:
# PITHOS_DB: Pithos database in SQLAlchemy format
# PITHOS_DB="sqlite://///var/lib/pithos/backend.db"
# PITHOS_BACKEND_STORAGE: Select Pithos backend storage. Possible values are
# 'nfs' and 'rados'. According to the value you select, you need to set the
# corresponding variables that follow.
# If you select 'nfs' that's 'PITHOS_DATA'. If you select 'rados' then you
# need to set all the "*_RADOS_*" ones.
# PITHOS_BACKEND_STORAGE="nfs"
# PITHOS_DATA: Directory where Pithos data are hosted
# PITHOS_DATA="//var/lib/pithos/data"
# PITHOS_RADOS_CEPH_CONF: RADOS configuration file
# PITHOS_RADOS_CEPH_CONF="@sysconfdir@/ceph/ceph.conf"
# PITHOS_RADOS_POOL_MAPS: RADOS pool for storing Pithos maps
# PITHOS_RADOS_POOL_MAPS="maps"
# PITHOS_RADOS_POOL_BLOCKS: RADOS pool for storing Pithos blocks
# PITHOS_RADOS_POOL_BLOCKS="blocks"
# PROGRESS_MONITOR: External program that monitors the progress of image
# deployment. Monitoring messages will be redirected to the standard input of
# this program.
......
......@@ -8,17 +8,39 @@ identical for all nodes and should be repeated manually or automatically, e.g.,
with puppet.
Installing snf-image using packages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-----------------------------------
For Debian GNU/Linux and Ubuntu we provide packages in our APT repository.
Debian GNU/Linux
^^^^^^^^^^^^^^^^
For Debian Wheezy add the following lines to ``/etc/apt/sources.list`` file:
For Debian 7.x (Wheezy) we provide packages in our APT repository. To use
our repository add the following lines to file ``/etc/apt/sources.list``:
``deb http://apt.dev.grnet.gr wheezy/``
``deb-src http://apt.dev.grnet.gr wheezy/``
For Ubuntu 12.04 LTS add the following lines:
After you update ``/etc/apt/sources.list`` import the repo's GPG key:
.. code-block:: console
# curl https://dev.grnet.gr/files/apt-grnetdev.pub | apt-key add -
To install the package use the following commands:
.. code-block:: console
# apt-get update
# apt-get install snf-image
The last command will also download and install the *snf-image-helper* image in
the post install phase of the package installation.
Ubuntu
^^^^^^
For Ubuntu 12.04 LTS we provide packages in our APT repository. To use our
repository add the following lines to file ``/etc/apt/sources.list``:
``deb http://apt.dev.grnet.gr precise/``
......@@ -28,20 +50,43 @@ After you update ``/etc/apt/sources.list`` import the repo's GPG key:
.. code-block:: console
$ curl https://dev.grnet.gr/files/apt-grnetdev.pub | apt-key add -
# curl https://dev.grnet.gr/files/apt-grnetdev.pub | apt-key add -
To install the package use the following commands:
.. code-block:: console
$ apt-get update
$ apt-get install snf-image
# apt-get update
# apt-get install snf-image
The last command will also download and install the *snf-image-helper* image in
the post install phase of the package installation.
CentOS
^^^^^^
For CentOS 6.5 we provide packages in our Yum repository.
To add the GRNET repository in your system, run:
.. code-block:: console
# yum localinstall https://dev.grnet.gr/files/grnet-repo.rpm
You can verify the authenticity of the package using our public key found
`here <https://dev.grnet.gr/files/apt-grnetdev.pub>`_.
To install snf-image run:
.. code-block:: console
# yum install snf-image
The last command will also download and install the *snf-image-helper* image in
the post install phase of the package installation.
Installing snf-image from source
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
--------------------------------
To install snf-image from source, download the provided source package:
......@@ -63,13 +108,13 @@ Install snf-image:
.. code-block:: console
$ make install
$ install -Dm600 defaults /etc/default/snf-image
$ mkdir -p /var/lib/snf-image/helper
# make install
# install -Dm600 defaults /etc/default/snf-image
# mkdir -p /var/lib/snf-image/helper
Finally, install the helper image by executing:
.. code-block:: console
$ snf-image-update-helper
# snf-image-update-helper
......@@ -66,10 +66,12 @@ to be used. If no prefix is used, it defaults to the local back-end:
* **Pithos backend**:
If the **img_id** is prefixed with ``pithos://`` or ``pithosmap://`` the
image is considered to reside on a Pithos deployment. For ``pithosmap://``
images, the user needs to have set a valid value for the
``PITHOS_DATA`` variable in snf-image's configuration file
(``/etc/default/snf-image`` by default). For ``pithos://`` images, in
addition to ``PITHOS_DATA``, the user needs to have set a valid value for the
images, the user needs to have set a valid value for the ``PITHOS_DATA``
variable in snf-image's configuration file (``/etc/default/snf-image`` by
default) if the storage backend is ``nfs`` or ``PITHOS_RADOS_POOL_MAPS`` and
``PITHOS_RADOS_POOL_BLOCKS`` if the storage backend is ``rados``.
For ``pithos://`` images, in addition to ``PITHOS_DATA`` or
``PITHOS_RADOS_POOL_*``, the user needs to have set a valid value for the
``PITHOS_DB`` variable, too.
| For example, if we want to deploy using a full Pithos URI:
......
......@@ -11,10 +11,6 @@ following images are basic installations of some popular Linux distributions,
that have been tested with snf-image and provided here for testing purposes:
* Debian Squeeze Base System
[`diskdump <http://cdn.synnefo.org/debian_base-6.0-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/debian_base-6.0-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/debian_base-6.0-x86_64.diskdump.meta>`_]
* Debian Wheezy Base System
[`diskdump <http://cdn.synnefo.org/debian_base-7.0-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/debian_base-7.0-x86_64.diskdump.md5sum>`_]
......@@ -31,38 +27,34 @@ that have been tested with snf-image and provided here for testing purposes:
[`diskdump <http://cdn.synnefo.org/fedora-20-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/fedora-20-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/fedora-20-x86_64.diskdump.meta>`_]
* Ubuntu Desktop LTS 12.04
[`diskdump <http://cdn.synnefo.org/ubuntu_desktop-12.04-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/ubuntu_desktop-12.04-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/ubuntu_desktop-12.04-x86_64.diskdump.meta>`_]
* Kubuntu LTS 12.04
[`diskdump <http://cdn.synnefo.org/kubuntu_desktop-12.04-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/kubuntu_desktop-12.04-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/kubuntu_desktop-12.04-x86_64.diskdump.meta>`_]
* Ubuntu Desktop 13.10
[`diskdump <http://cdn.synnefo.org/ubuntu_desktop-13.10-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/ubuntu_desktop-13.10-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/ubuntu_desktop-13.10-x86_64.diskdump.meta>`_]
* Kubuntu 13.10
[`diskdump <http://cdn.synnefo.org/kubuntu_desktop-13.10-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/kubuntu_desktop-13.10-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/kubuntu_desktop-13.10-x86_64.diskdump.meta>`_]
* Ubuntu Server 12.04
[`diskdump <http://cdn.synnefo.org/ubuntu_server-12.04-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/ubuntu_server-12.04-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/ubuntu_server-12.04-x86_64.diskdump.meta>`_]
* Ubuntu Desktop LTS 14.04
[`diskdump <http://cdn.synnefo.org/ubuntu_desktop-14.04-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/ubuntu_desktop-14.04-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/ubuntu_desktop-14.04-x86_64.diskdump.meta>`_]
* Kubuntu LTS 14.04
[`diskdump <http://cdn.synnefo.org/kubuntu-14.10-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/kubuntu-14.10-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/kubuntu-14.10-x86_64.diskdump.meta>`_]
* Ubuntu Server LTS 14.04
[`diskdump <http://cdn.synnefo.org/ubuntu_server-14.04-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/ubuntu_server-14.04-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/ubuntu_server-14.04-x86_64.diskdump.meta>`_]
* OpenSUSE Desktop 13.1
[`diskdump <http://cdn.synnefo.org/opensuse_desktop-13.1-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/opensuse_desktop-13.1-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/opensuse_desktop-13.1-x86_64.diskdump.meta>`_]
* Oracle Linux 6.5
[`diskdump <http://cdn.synnefo.org/oraclelinux-6.5-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/oraclelinux-6.5-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/oraclelinux-6.5-x86_64.diskdump.meta>`_]
* FreeBSD 9.2
[`diskdump <http://cdn.synnefo.org/freebsd-9.2-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/freebsd-9.2-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/freebsd-9.2-x86_64.diskdump.meta>`_]
* OpenBSD 5.4
[`diskdump <http://cdn.synnefo.org/openbsd-5.4-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/openbsd-5.4-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/openbsd-5.4-x86_64.diskdump.meta>`_]
* OpenBSD 5.5
[`diskdump <http://cdn.synnefo.org/openbsd-5.5-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/openbsd-5.5-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/openbsd-5.5-x86_64.diskdump.meta>`_]
* NetBSD 6.1
[`diskdump <http://cdn.synnefo.org/netbsd-6.1-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/netbsd-6.1-x86_64.diskdump.md5sum>`_]
......
__version__ = "0.15rc3"
__version__ = "%(DEVFLOW_VERSION)s"
......@@ -413,7 +413,7 @@ create_partition() {
# For GPT
start=${start:0:${#start}-1} # remove the s at the end
end=${end:0:${#end}-1} # remove the s at the end
$SGDISK -n "$id":"$start":"$end" -t "$id":"$ptype" "$device"
$SGDISK -a 1 -n "$id":"$start":"$end" -t "$id":"$ptype" "$device"
fi
}
......
......@@ -4,3 +4,4 @@ Constantinos Venetsanopoulos <cven@grnet.gr>
Vangelis Koukis <vkoukis@grnet.gr>
Giorgos Verigakis <verigak@grnet.gr>
Christos Stavrakakis <cstavr@grnet.gr>
Chrysostomos Nanakos <cnanakos@grnet.gr>
......@@ -454,6 +454,10 @@ fi
: ${HELPER_MEMORY:="512"}
: ${PITHOS_DB:="sqlite:////@localstatedir@/lib/pithos/backend.db"}
: ${PITHOS_DATA:="@localstatedir@/lib/pithos/data/"}
: ${PITHOS_BACKEND_STORAGE:="nfs"}
: ${PITHOS_RADOS_CEPH_CONF:="@sysconfdir@/ceph/ceph.conf"}
: ${PITHOS_RADOS_POOL_MAPS:="maps"}
: ${PITHOS_RADOS_POOL_BLOCKS:="blocks"}
: ${PROGRESS_MONITOR:="@PROGRESS_MONITOR@"}
: ${UNATTEND:="@UNATTEND@"}
: ${XEN_SCRIPTS_DIR="@sysconfdir@/xen/scripts"}
......
......@@ -107,10 +107,14 @@ case $BACKEND_TYPE in
image_size=$($CURL -sI "$IMAGE_NAME" | grep ^Content-Length: | cut -d" " -f2)
;;
pithos)
# For security reasons pass the database url to pithcat as an
# environmental variable.
# For security reasons pass the various options to pithcat as
# environment variables.
export PITHCAT_INPUT_DB="$PITHOS_DB"
export PITHCAT_INPUT_DATA="$PITHOS_DATA"
export PITHCAT_BACKEND_STORAGE="$PITHOS_BACKEND_STORAGE"
export PITHCAT_RADOS_CEPH_CONF="$PITHOS_RADOS_CEPH_CONF"
export PITHCAT_RADOS_POOL_MAPS="$PITHOS_RADOS_POOL_MAPS"
export PITHCAT_RADOS_POOL_BLOCKS="$PITHOS_RADOS_POOL_BLOCKS"
cmd_args="$(printf "%q" "${IMAGE_NAME}")"
image_cmd="./pithcat $cmd_args"
image_size=$(./pithcat -s $cmd_args)
......
......@@ -50,13 +50,29 @@
# MULTISTRAP_APTPREFDIR="@MULTISTRAP_APTPREFDIR@"
# XEN_SCRIPTS_DIR: Directory where the Xen scripts are stored
# XEN_SCRIPTS_DIR=="@sysconfdir@/xen/scripts"
# XEN_SCRIPTS_DIR="@sysconfdir@/xen/scripts"
# PITHOS_DB: Pithos database in SQLAlchemy format
# PITHOS_DB="sqlite:////@localstatedir@/lib/pithos/backend.db"
# PITHOS_BACKEND_STORAGE: Select Pithos backend storage. Possible values are
# 'nfs' and 'rados'. According to the value you select, you need to set the
# corresponding variables that follow.
# If you select 'nfs' that's 'PITHOS_DATA'. If you select 'rados' then you need
# to set all the "*_RADOS_*" ones.
# PITHOS_BACKEND_STORAGE="nfs"
# PITHOS_DATA: Directory where Pithos data are hosted
# PITHOS_DATA="/@localstatedir@/lib/pithos/data"
#
# PITHOS_RADOS_CEPH_CONF: RADOS configuration file
# PITHOS_RADOS_CEPH_CONF="@sysconfdir@/ceph/ceph.conf"
# PITHOS_RADOS_POOL_MAPS: RADOS pool for storing Pithos maps
# PITHOS_RADOS_POOL_MAPS="maps"
# PITHOS_RADOS_POOL_BLOCKS: RADOS pool for storing Pithos blocks
# PITHOS_RADOS_POOL_BLOCKS="blocks"
# PROGRESS_MONITOR: External program that monitors the progress of the image
# deployment. The snf-image monitor messages will be redirected to the standard
......
#!/usr/bin/env python
# Copyright (C) 2011-2013 GRNET S.A.
# Copyright (C) 2011-2014 GRNET S.A.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
......@@ -30,29 +30,60 @@ from sys import exit, stdout, stderr
from os import environ
from binascii import hexlify, unhexlify
from collections import namedtuple
from pkg_resources import parse_version
try:
from pithos.backends.modular import ModularBackend
from pithos.backends.version import __version__ as pithos_backend_version
except ImportError:
stderr.write("Pithos backend was not found.\n")
exit(2)
SELECTABLE_BE_VER = "0.15.1"
note = """
NOTE: You can pass all arguments through environment variables instead of
the command line: Setting the environment variable PITHCAT_INPUX_XXX to
VALUE is equivalent to passing a '--xxx VALUE' argument.
Using the --db argument directly is dangerous, because it may
expose sensitive information in the output of 'ps'. Consider passing
the DB URI through the environment variable PITHOS_INPUT_DB instead.\n"""
OptionParser.format_epilog = lambda self, formattxt: self.epilog
parser = OptionParser(usage='%prog [options] <URL>', epilog=note)
if parse_version(pithos_backend_version) >= parse_version(SELECTABLE_BE_VER):
backend_group = OptionGroup(
parser, "Backend-specific Options",
"The backend-specific options depend on the specific "
"version of Pithos backend (package snf-pithos-backend) installed.\n"
"(Currently-installed version: Pithos backend '%s')" %
pithos_backend_version)
backend_group.add_option('--backend', dest='backend', metavar='BACKEND',
help='Pithos backend storage type', default=None)
backend_group.add_option('--rados-conf', dest='rconf', metavar='RCONF',
help='RADOS configuration file to use',
default=None)
backend_group.add_option('--rados-maps', dest='rmaps', metavar='RMAPS',
help='RADOS pool which Pithos maps reside',
default=None)
backend_group.add_option('--rados-blocks', dest='rblocks',
metavar='RBLOCKS',
help='RADOS pool which Pithos blocks reside',
default=None)
backend_group.add_option('--data', dest='data', metavar='DIR',
help='path to the directory where data are stored'
)
parser.add_option_group(backend_group)
else:
parser.add_option('--data', dest='data', metavar='DIR',
help='path to the directory where data are stored')
parser = OptionParser(usage='%prog [options] <URL>')
parser.add_option('--data', dest='data', metavar='DIR',
help='path to the directory where data are stored')
parser.add_option('-s', action='store_true', dest='size', default=False,
help='print file size and exit')
group = OptionGroup(
parser, "Dangerous Options",
"Caution: If the <URL> is a LocationURL (pithos://...), then you'll also "
"need to define a database URI. You can use the `--db' option to do so, "
"but this raises security concerns. For database URI's and pithos data "
"paths, the recommended way to define them is to use the PITHCAT_INPUT_DB "
"and PITHCAT_INPUT_DATA environmental variables respectfully.")
group.add_option('--db', dest='db', metavar='URI',
help='SQLAlchemy URI of the database', default=None)
parser.add_option_group(group)
parser.add_option('--db', dest='db', metavar='URI',
help='SQLAlchemy URI of the database [DANGEROUS: Do not use,'
'see NOTE below]', default=None)
LocationURL = namedtuple('LocationURL', ['account', 'container', 'object'])
HashmapURL = namedtuple('HashmapURL', ['hash', 'size'])
......@@ -114,30 +145,118 @@ def main():
url = parse_url(args[0])
if not options.data and 'PITHCAT_INPUT_DATA' not in environ:
stderr.write(
"Pithos data directory path is missing.\n"
"Use the PITHCAT_INPUT_DATA environmental variable (recommended) "
"or the --data command line option to define it.\n")
exit(1)
data_path = None
if parse_version(pithos_backend_version) >= \
parse_version(SELECTABLE_BE_VER):
data_path = environ['PITHCAT_INPUT_DATA'] if not options.data else \
options.data
if not options.backend and 'PITHCAT_BACKEND_STORAGE' not in environ:
stderr.write(
"Pithos backend storage is missing.\n"
"Use the PITHCAT_BACKEND_STORAGE environment variable "
"(recommended) "
"or the --backend command line option to define it.\n")
exit(1)
if options.backend == 'nfs' or \
environ.get('PITHCAT_BACKEND_STORAGE') == 'nfs':
if not options.data and 'PITHCAT_INPUT_DATA' not in environ:
stderr.write(
"Pithos data directory path is missing.\n"
"Use the PITHCAT_INPUT_DATA environment variable "
"(recommended) "
"or the --data command line option to define it.\n")
exit(1)
data_path = environ['PITHCAT_INPUT_DATA'] if not options.data else \
options.data
else:
if not options.data and 'PITHCAT_INPUT_DATA' not in environ:
stderr.write(
"Pithos data directory path is missing.\n"
"Use the PITHCAT_INPUT_DATA environment variable "
"(recommended) "
"or the --data command line option to define it.\n")
exit(1)
data_path = environ['PITHCAT_INPUT_DATA'] if not options.data else \
options.data
if options.db is None and 'PITHCAT_INPUT_DB' not in environ and \
type(url) is LocationURL:
stderr.write(
"Pithos database URI is missing.\n"
"Use the PITHCAT_INPUT_DB environmental variable (recommended) "
"Use the PITHCAT_INPUT_DB environment variable (recommended) "
"or the --db command line option to define it.\n")
exit(1)
db_uri = environ['PITHCAT_INPUT_DB'] if not options.db else options.db
if type(url) is HashmapURL:
db_uri = None
else:
db_uri = environ['PITHCAT_INPUT_DB'] if not options.db else options.db
backend = ModularBackend(None,
db_uri if type(url) is LocationURL else None,
None,
data_path)
if parse_version(pithos_backend_version) >= \
parse_version(SELECTABLE_BE_VER):
block_params = {'mappool': None, 'blockpool': None}
rados_ceph_conf = None
backend_storage = environ['PITHCAT_BACKEND_STORAGE'] if not \
options.backend else options.backend
if backend_storage == 'rados' and options.rconf is None and \
'PITHCAT_RADOS_CEPH_CONF' not in environ:
stderr.write(
"RADOS storage backend is selected but "
"RADOS conf file is missing.\n"
"Use the PITHCAT_RADOS_CEPH_CONF environment variable or the "
"--rados-conf command line option to define it.\n")
exit(1)
if backend_storage == 'rados':
rados_ceph_conf = environ['PITHCAT_RADOS_CEPH_CONF'] if not \
options.rconf else options.rconf
if options.rmaps is None and 'PITHCAT_RADOS_POOL_MAPS' not in \
environ:
stderr.write(
"RADOS storage backend is selected but "
"RADOS pool for Pithos maps is missing.\n"
"Use the PITHCAT_RADOS_POOL_MAPS environment variable or "
"the --rados-maps command line option to define it.\n")
exit(1)
rados_maps = environ['PITHCAT_RADOS_POOL_MAPS'] if not \
options.rmaps else options.rmaps
if options.rblocks is None and 'PITHCAT_RADOS_POOL_BLOCKS' not in \
environ:
stderr.write(
"RADOS storage backend is selected but "
"RADOS pool for Pithos blocks is missing.\n"
"Use the PITHCAT_RADOS_POOL_BLOCKS environment variable "
"or the --rados-blocks command line option to define it.\n"
)
exit(1)
rados_blocks = environ['PITHCAT_RADOS_POOL_BLOCKS'] if not \
options.rblocks else options.rblocks
block_params['blockpool'] = rados_blocks
block_params['mappool'] = rados_maps
backend = ModularBackend(None,
db_uri if type(url) is LocationURL else None,
None,
data_path, block_params=block_params,
backend_storage=backend_storage,
rados_ceph_conf=rados_ceph_conf)
else:
backend = ModularBackend(None,
db_uri if type(url) is LocationURL else None,
None,