Commit df284363 authored by Dimitris Aragiorgis's avatar Dimitris Aragiorgis
Browse files

Huuuuge snf-deploy refactor



Introduce the concept of Roles and Components.

Roles are ns, mq, db, nfs, astakos, cyclades, pithos, cms, stats,
ganeti, master, and client.

Each role consists of various SynnefoComponents which  should define:

* commands to check installation prerequisites (check)
* packages to install (REQUIRED_PACKAGES)
* commands to prepare installation (prepare)
* configuration templates (configure)
* reload commands (restart)
* initialization commands (initialize)
* test commands (test)

SynnefoComponents are unaware of fabric environment. They
get initialized with a Host object and a Env object.

Host includes all the necessary info for the installation node
(ip, hostname, alias, fqdn..) and Env includes all the configuration info
(created after parsing config files)

After defining components, fabfile should just do the following:

@roles("somerole")
def setup_somerole_role():
  SetupSynnefoRole("SomeRole")

In case you want to run a component's specific method just run

RunComponentMethod(SomeComponent, "method_name", *args, **kwargs)

During a role setup you might have to retrieve info from other
components already installed (user token, backend id, etc.) Use
'execute' fabric method and fill env object with required info.

Make deployment re-entrant.

Check if specific component is already installed and if yes then skip it.

Currently this is done on node/component level. The component's
status on a target node is either ok or nothing.

Introduce conflicting components.

Two components might not be able/or should not coexist in the same
node, e.g. NFS and Mount. These conflicts are defined in CONFLICTS
dict in roles.py.

Sync ci and deploy conf files.

ci: Change node's password in nodes.conf

Always enable CSRF. Install CMS only if it resides on different
node than pithos, cyclades and astakos.

Add copyright headers.
Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
parent 66fdbb73
[ganeti1]
cluster_nodes = node1
cluster_nodes =
master_node = node1
cluster_netdev = eth0
......
......@@ -21,8 +21,13 @@ node1 = 192.168.0.1
node1 = 52:54:00:00:00:01
# node2 = 52:54:00:00:00:02
[passwords]
node1 = 12345
# node2 = 67890
[info]
# Here we define which nodes from the predefined ones to use
# comma separated node names e.g. node1,node2
nodes = node1
# login credentials for the nodes
......
[debian]
rabbitmq-server = squeeze-backports
gunicorn = squeeze-backports
qemu-kvm = squeeze-backports
qemu = squeeze-backports
python-gevent = squeeze-backports
apache2 =
postgresql =
python-psycopg2 =
python-argparse =
nfs-kernel-server = squeeze-backports
nfs-common = squeeze-backports
bind9 =
vlan =
vlan =
lvm2 =
curl =
memcached =
python-memcache =
bridge-utils =
python-progress =
ganeti-instance-debootstrap =
python-django-south = squeeze-backports
drbd8-utils =
[synnefo]
snf-astakos-app = squeeze
snf-common = squeeze
snf-cyclades-app = squeeze
snf-cyclades-gtools = squeeze
snf-django-lib = squeeze
python-astakosclient = squeeze
snf-branding = squeeze
snf-webproject = squeeze
snf-pithos-app = squeeze
snf-pithos-backend = squeeze
snf-tools = squeeze
[ganeti]
snf-ganeti = 2.6.2+ippool11+hotplug5+extstorage3+rbdfix1+kvmfix2+nolvm+netxen-1~squeeze
ganeti-htools = 2.6.2+ippool11+hotplug5+extstorage3+rbdfix1+kvmfix2+nolvm+netxen-1~squeeze
[other]
snf-cloudcms = squeeze
snf-vncauthproxy = squeeze
snf-pithos-webclient = squeeze
snf-image = squeeze
snf-network = squeeze
python-objpool = squeeze
nfdhcpd = squeeze
kamaki = squeeze
python-bitarray = squeeze-backports
nfqueue-bindings-python = 0.3+physindev-1
......@@ -23,6 +23,7 @@ ns = node1
client = node1
router = node1
stats = node1
nfs = node1
[synnefo]
......
......@@ -24,6 +24,7 @@ python-django-south =
python-django =
drbd8-utils =
collectd =
dnsutils =
[synnefo]
......@@ -45,11 +46,12 @@ snf-stats-app = wheezy
snf-ganeti = wheezy
ganeti-htools = wheezy
ganeti-haskell = wheezy
ganeti2 = wheezy
[other]
snf-cloudcms = wheezy
snf-vncauthproxy = unstable
snf-vncauthproxy = wheezy
snf-pithos-webclient = wheezy
snf-image = wheezy
snf-network = wheezy
......
......@@ -893,6 +893,7 @@ class SynnefoCI(object):
self.logger.debug("Change password in nodes.conf file")
cmd = """
sed -i 's/^password =.*/password = {0}/' /etc/snf-deploy/nodes.conf
sed -i 's/12345/{0}/' /etc/snf-deploy/nodes.conf
""".format(fabric.env.password)
_run(cmd, False)
......
Copyright (C) 2010, 2011, 2012, 2013 GRNET S.A. All rights reserved.
Redistribution and use in source and binary forms, with or
without modification, are permitted provided that the following
conditions are met:
1. Redistributions of source code must retain the above
copyright notice, this list of conditions and the following
disclaimer.
2. Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials
provided with the distribution.
THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A. OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
The views and conclusions contained in the software and
documentation are those of the authors and should not be
interpreted as representing official policies, either expressed
or implied, of GRNET S.A.
[ganeti1]
cluster_nodes = node1
cluster_nodes =
master_node = node1
cluster_netdev = eth0
......@@ -8,8 +8,8 @@ cluster_ip = 192.168.0.13
vg = autovg
synnefo_public_network_subnet = 10.0.1.0/24
synnefo_public_network_gateway = 10.0.1.1
synnefo_public_network_subnet = 10.2.1.0/24
synnefo_public_network_gateway = 10.2.1.1
synnefo_public_network_type = CUSTOM
image_dir = /srv/okeanos
......
......@@ -4,7 +4,7 @@
domain = synnefo.live
[os]
node1 = squeeze
node1 = wheezy
# node2 = wheezy
[hostnames]
......@@ -21,8 +21,13 @@ node1 = 192.168.0.1
node1 = 52:54:00:00:00:01
# node2 = 52:54:00:00:00:02
[passwords]
node1 = 12345
# node2 = 67890
[info]
# Here we define which nodes from the predefined ones to use
# comma separated node names e.g. node1,node2
nodes = node1
# login credentials for the nodes
......
[debian]
rabbitmq-server = squeeze-backports
gunicorn = squeeze-backports
qemu-kvm = squeeze-backports
qemu = squeeze-backports
python-gevent = squeeze-backports
apache2 =
postgresql =
python-psycopg2 =
python-argparse =
nfs-kernel-server = squeeze-backports
nfs-common = squeeze-backports
bind9 =
vlan =
vlan =
lvm2 =
curl =
memcached =
python-memcache =
bridge-utils =
python-progress =
ganeti-instance-debootstrap =
python-django-south = squeeze-backports
python-django = squeeze-backports
drbd8-utils =
[synnefo]
snf-astakos-app = squeeze
snf-common = squeeze
snf-cyclades-app = squeeze
snf-cyclades-gtools = squeeze
snf-django-lib = squeeze
python-astakosclient = squeeze
snf-branding = squeeze
snf-webproject = squeeze
snf-pithos-app = squeeze
snf-pithos-backend = squeeze
snf-tools = squeeze
[ganeti]
snf-ganeti = 2.6.2+ippool11+hotplug5+extstorage3+rbdfix1+kvmfix2+nolvm+netxen-1~squeeze
ganeti-htools = 2.6.2+ippool11+hotplug5+extstorage3+rbdfix1+kvmfix2+nolvm+netxen-1~squeeze
[other]
snf-cloudcms = squeeze
snf-vncauthproxy = squeeze
snf-pithos-webclient = squeeze
snf-image = squeeze
snf-network = squeeze
python-objpool = squeeze
nfdhcpd = squeeze
kamaki = squeeze
python-bitarray = squeeze-backports
nfqueue-bindings-python = 0.3+physindev-1
......@@ -23,6 +23,7 @@ ns = node1
client = node1
router = node1
stats = node1
nfs = node1
[synnefo]
......
......@@ -24,6 +24,7 @@ python-django-south =
python-django =
drbd8-utils =
collectd =
dnsutils =
[synnefo]
......@@ -45,6 +46,8 @@ snf-stats-app = wheezy
snf-ganeti = wheezy
ganeti-htools = wheezy
ganeti-haskell = wheezy
ganeti2 = wheezy
[other]
snf-cloudcms = wheezy
......
......@@ -6,12 +6,37 @@
// organization
//include "/etc/bind/zones.rfc1918";
include "/etc/bind/ddns.key";
// all synnefo components share the same domain/zone
zone "%DOMAIN%" in {
type master;
notify no;
file "/etc/bind/zones/%DOMAIN%";
allow-update { key DDNS_UPDATE; };
};
# domain/zone for the VMs
zone "vm.%DOMAIN%" in {
type master;
notify no;
file "/etc/bind/zones/vm.%DOMAIN%";
allow-update { key DDNS_UPDATE; };
};
// reverse dns zone for all IPs
zone "in-addr.arpa" in {
type master;
notify no;
file "/etc/bind/rev/synnefo.in-addr.arpa.zone";
allow-update { key DDNS_UPDATE; };
};
// v6 reverse dns zone for all IPs
zone "ip6.arpa" in {
type master;
notify no;
file "/etc/bind/rev/synnefo.ip6.arpa.zone";
allow-update { key DDNS_UPDATE; };
};
$TTL 86400
$ORIGIN ip6.arpa.
@ IN SOA ns.%DOMAIN%. admin.%DOMAIN%. (
2012070900; the Serial Number
172800; the Refresh Rate
7200; the Retry Time
604800; the Expiration Time
3600) ; the Minimum Time
@ IN NS ns.%DOMAIN%.
$ORIGIN .
$TTL 86400 ; 1 day
ip6.arpa IN SOA ns.vm.qa.live. admin.vm.qa.live. (
2012071070 ; serial
172800 ; refresh (2 days)
7200 ; retry (2 hours)
604800 ; expire (1 week)
3600 ; minimum (1 hour)
)
NS ns.vm.qa.live.
$TTL 14400
$origin vm.%DOMAIN%.
@ IN SOA ns.vm.%DOMAIN%. admin.vm.%DOMAIN%. (
2012111903; the Serial Number
172800; the Refresh Rate
7200; the Retry Time
604800; the Expiration Time
3600; the Minimum Time
)
@ IN NS ns.vm.%DOMAIN%.
@ IN A %NS_NODE_IP%
ns IN A %NS_NODE_IP%
......@@ -8,7 +8,6 @@
# IMAGE_DIR: directory location for disk images
# IMAGE_DIR="/var/lib/snf-image"
IMAGE_DIR=%IMAGE_DIR%
# IMAGE_DEBUG: turn on debugging output for the scripts
# IMAGE_DEBUG=no
......@@ -43,7 +42,6 @@ IMAGE_DIR=%IMAGE_DIR%
# for days.
# HELPER_SOFT_TIMEOUT="20"
# HELPER_HARD_TIMEOUT="5"
HELPER_SOFT_TIMEOUT=100
# HELPER_USER: For security reasons, it is recommended that the helper VM
# runs as an unprivileged user. KVM drops root privileges and runs as
......@@ -60,17 +58,14 @@ HELPER_SOFT_TIMEOUT=100
# PITHOS_DB: Pithos database in SQLAlchemy format
# PITHOS_DB="sqlite:////var/lib/pithos/backend.db"
PITHOS_DB=postgresql://%SYNNEFO_USER%:%SYNNEFO_DB_PASSWD%@%DB_NODE%:5432/snf_pithos
# PITHOS_DATA: Directory where pithos data are hosted
# PITHOS_DATA="/var/lib/pithos/data"
PITHOS_DATA=%PITHOS_DIR%/data
# PROGRESS_MONITOR: External program that monitors the progress of the image
# deployment. The snf-image monitor messages will be redirected to the standard
# input of this program.
# PROGRESS_MONITOR=""
PROGRESS_MONITOR=snf-progress-monitor
# UNATTEND: This variables overwrites the unattend.xml file used when deploying
# a windows image. snf-image-helper will use its own unattend.xml file if this
......@@ -87,4 +82,9 @@ PROGRESS_MONITOR=snf-progress-monitor
# INSTALL_MBR="install-mbr"
# TIMELIMIT="timelimit"
# CURL="curl"
IMAGE_DIR=%IMAGE_DIR%
HELPER_SOFT_TIMEOUT=100
PITHOS_DB=postgresql://%SYNNEFO_USER%:%SYNNEFO_DB_PASSWD%@%DB_NODE%:5432/snf_pithos
PITHOS_DATA=%PITHOS_DIR%/data
PROGRESS_MONITOR=snf-progress-monitor
CURL="curl -k"
MAC_MASK=ff:ff:f0:00:00:00
TAP_CONSTANT_MAC=cc:47:52:4e:45:54 # GRNET in hex :-)
MAC2EUI64=/usr/bin/mac2eui64
NFDHCPD_STATE_DIR=/var/lib/nfdhcpd
GANETI_NIC_DIR=/var/run/ganeti/xen-hypervisor/nic
MAC_FILTERED_TAG=private-filtered
NFDHCPD_TAG=nfdhcpd
IP_LESS_ROUTED_TAG=ip-less-routed
MASQ_TAG=masq
PUBLIC_TAG=public
DNS_TAG=public
# Default options for runlocked helper script (uncomment to modify)
#RUNLOCKED_OPTS="--id 10001 --retry-sec 0.5"
# NS options needed by nsupdate
# A proper bind configuration is a prerequisite
# Please see: https://wiki.debian.org/DDNS
# If one of the following vars are not set dnshook wont do a thing
# Name server IP/FQDN
SERVER=%SERVER%
# zone for the vms
FZONE=vm.%DOMAIN%
# keyfile path to pass to nsupdate with -k option
# see man page for more info
KEYFILE=%KEYFILE%
/srv/ganeti/file-storage
/srv/ganeti/shared-file-storage
......@@ -8,7 +8,7 @@ CONFIG = {
'group': 'www-data',
'args': (
'--bind=127.0.0.1:8080',
'--workers=8',
'--workers=6',
'--worker-class=gevent',
# '--worker-class=sync',
'--log-level=debug',
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment