Commit 3c49605b authored by Stratos Psomadakis's avatar Stratos Psomadakis

Merge branch 'hotfix-0.16.1'

parents 012ead18 1569fe9f
......@@ -6,6 +6,47 @@ Unified Changelog file for Synnefo versions >= 0.13
Since v0.13 most of the Synnefo components have been merged into a single
repository and have aligned versions.
.. _Changelog-0.16.1:
v0.16.1
=======
Released: Fri Feb 6 12:05:44 EET 2015
Astakos
-------
* Include additional validation checks in project modification form to ensure
members limit is always set lower or equal than total limit.
* Prevent members limit from automatically being set to unlimited in project
modification form.
* Exclude terminated projects from responses of project list and user quota
api calls.
* Improve UI handling of inactive, terminated and empty projects.
* Change default pagination in projects list view.
Cyclades UI
-----------
* Improve handling of terminated projects in reassign view.
* Remove the COPYRECT encoding from the preferred noVNC client encodings, and
prefer TIGHT_PNG by default.
* Fix handling of SSH keys and password customization steps to images that
do not support them, e.g. snapshots.
* Add OS icons for Scientific Linux and CoreOS.
* Change project reassign button style for VMs and boot disks.
Stats
-----
* Aggregate the traffic of every VM interface in the net stats graphs.
Branding
--------
* Add CLOUDBAR_HOME_URL setting to allow Cloudbar home icon customization.
.. _Changelog-0.16:
v0.16
......
......@@ -5,6 +5,22 @@ Unified NEWS file for Synnefo versions >= 0.13
Since v0.13 all Synnefo components have been merged into a single repository.
.. _NEWS-0.16.1:
v0.16.1
=======
Released: Fri Feb 6 12:06:25 EET 2015
The Synnefo 0.16.1 release is mostly a bug fix version. The most notable
changes are:
* Improved handling of projects (both API- and UI-wise).
* Updates and fixes for snf-deploy.
* Various bug fixes and improvements across Synnefo. Please see the Synnefo
:ref:`Changelog <Changelog-0.16.1>` for a complete list.
.. _NEWS-0.16:
v0.16
......
......@@ -43,6 +43,9 @@ x2go_plugin_file = /tmp/x2go.html
[Deployment]
# Choose project id to use for created resources. Use
# `kamaki project list` to find the project id.
project =
# Choose the 'cloud' to use from .kamakirc
kamaki_cloud =
# Server name to use for our machine
......
......@@ -8,20 +8,5 @@ synnefo = True
[ganeti]
name = ganeti
domain = synnefo.live
ip = 10.1.2.101
netdev = eth0
[ganeti-qa]
name = ganeti
domain = qa.synnefo.live
ip = 10.1.2.101
netdev = eth0
synnefo =
[ganeti-vc]
name = ganeti
domain = vcluster.synnefo.live
ip = 10.1.2.101
ip = 10.0.0.101
netdev = eth0
......@@ -27,9 +27,9 @@ password =
# The interface with internet access
public_iface = eth0
# The interface for the instances' public traffic
vm_public_iface = eth1
vm_public_iface =
# The interface for the instances' private traffic
vm_private_iface = eth2
vm_private_iface =
# The extra disk for the Ganeti VG needed for plain and drbd disk templates
extra_disk = /dev/vdb
......@@ -42,63 +42,3 @@ extra_disk = /dev/vdb
name = node
ip = 192.0.2.1
extra_disk =
############
# qa nodes #
############
[dev]
name = qa2
ip = 10.1.2.10
public_iface = eth1
domain = qa.synnefo.live
[qa1]
name = qa1
ip = 10.1.2.11
public_iface = eth1
domain = qa.synnefo.live
[qa2]
name = qa2
ip = 10.1.2.12
public_iface = eth1
domain = qa.synnefo.live
############
# vc nodes #
############
[vc1]
mac = 52:54:00:00:00:01
name = vc1
ip = 10.1.2.1
public_iface = eth0
domain = vcluster.synnefo.live
[vc2]
mac = 52:54:00:00:00:02
name = vc2
ip = 10.1.2.2
public_iface = eth0
domain = vcluster.synnefo.live
[vc3]
mac = 52:54:00:00:00:03
name = vc3
ip = 10.1.2.3
public_iface = eth0
domain = vcluster.synnefo.live
[vc4]
mac = 52:54:00:00:00:04
name = vc4
ip = 10.1.2.4
public_iface = eth0
domain = vcluster.synnefo.live
[dummy]
name = dummy
ip = 1.2.3.4
public_iface = eth0
domain = synnefo.live
[debian]
python-nfqueue = 0.4+physindev-1~wheezy
python-scapy = 2.2.0+rfc6355-1
snf-ganeti = unstable
ganeti2 = unstable
snf-ganeti = wheezy
ganeti2 = wheezy
python-django-eztables = 0.3.3-1~snf~0.2
qemu-kvm = wheezy-backports
......
......@@ -28,52 +28,3 @@ clusters =
master = node
vmc =
node
###################################
# snf-deploy ganeti-qa --setup qa #
###################################
[qa]
ns = dev
client = dev
router = qa1
nfs = dev
dev = dev
clusters =
ganeti-qa
[ganeti-qa]
master = qa1
vmc =
qa1
qa2
##################################
# snf-deploy vcluster --setup vc #
##################################
[vc]
ns = vc1
client = vc4
router = vc1
nfs = vc1
db = vc2
mq = vc3
astakos = vc1
cyclades = vc2
pithos = vc3
cms = vc4
stats = vc1
dev = vc1
clusters =
ganeti-vc
[ganeti-vc]
master = vc1
vmc =
vc1
vc2
vc3
vc4
......@@ -36,4 +36,9 @@ vm_private_bridge = prv0
common_bridge = br0
synnefo_public_network_subnet = 10.2.1.0/24
synnefo_public_network_gateway = 10.2.1.1
synnefo_public_network_netmask = 255.255.255.0
synnefo_public_network_type = CUSTOM
# Ganeti git repo
# git://git.ganeti.org/ganeti.git
ganeti_url = https://code.grnet.gr/git/snf-ganeti
......@@ -266,6 +266,51 @@ class SynnefoCI(object):
self.compute_client = ComputeClient(compute_url, token)
self.compute_client.CONNECTION_RETRY_LIMIT = 2
__quota_cache = None
def _get_available_project(self, skip_config=False, **resources):
self.project_uuid = None
if self.config.has_option("Deployment", "project"):
self.project_uuid = self.config.get("Deployment",
"project").strip() or None
# user requested explicit project
if self.project_uuid and not skip_config:
return self.project_uuid
def _filter_projects(_project):
uuid, project_quota = _project
can_fit = False
for resource, required in resources.iteritems():
# transform dots in order to permit direct keyword
# arguments to be used.
# (cyclades__disk=1) -> 'cyclades.disk': 1
resource = resource.replace("__", ".")
project_resource = project_quota.get(resource)
if not project_resource:
raise Exception("Requested resource does not exist %s" \
% resource)
plimit, ppending, pusage, musage, mlimit, mpending = \
project_resource.values()
pavailable = plimit - ppending - pusage
mavailable = mlimit - mpending - musage
can_fit = (pavailable - required) >= 0 and \
(mavailable - required) >= 0
if not can_fit:
return None
return uuid
self.__quota_cache = quota = self.__quota_cache or \
self.astakos_client.get_quotas()
projects = filter(bool, map(_filter_projects, quota.iteritems()))
if not len(projects):
raise Exception("No project available for %r" % resources)
return projects[0]
def _wait_transition(self, server_id, current_status, new_status):
"""Wait for server to go from current_status to new_status"""
self.logger.debug("Waiting for server to become %s" % new_status)
......@@ -313,13 +358,15 @@ class SynnefoCI(object):
def _create_floating_ip(self):
"""Create a new floating ip"""
project_id = self._get_available_project(cyclades__floating_ip=1)
networks = self.network_client.list_networks(detail=True)
pub_nets = [n for n in networks
if n['SNF:floating_ip_pool'] and n['public']]
for pub_net in pub_nets:
# Try until we find a public network that is not full
try:
fip = self.network_client.create_floatingip(pub_net['id'])
fip = self.network_client.create_floatingip(
pub_net['id'], project_id=project_id)
except ClientError as err:
self.logger.warning("%s", str(err.message).strip())
continue
......@@ -353,6 +400,16 @@ class SynnefoCI(object):
# Find a flavor to use
flavor_id = self._find_flavor(flavor)
# get available project
flavor = self.cyclades_client.get_flavor_details(flavor_id)
quota = {
'cyclades.disk': flavor['disk'] * 1024 ** 3,
'cyclades.ram': flavor['ram'] * 1024 ** 2,
'cyclades.cpu': flavor['vcpus'],
'cyclades.vm': 1
}
project_id = self._get_available_project(**quota)
# Create Server
networks = []
if self.config.get("Deployment", "allocate_floating_ip") == "True":
......@@ -367,7 +424,8 @@ class SynnefoCI(object):
server_name = self.config.get("Deployment", "server_name")
server_name = "%s(BID: %s)" % (server_name, self.build_id)
server = self.cyclades_client.create_server(
server_name, flavor_id, image_id, networks=networks)
server_name, flavor_id, image_id, networks=networks,
project_id=project_id)
server_id = server['id']
self.write_temp_config('server_id', server_id)
self.logger.debug("Server got id %s" % _green(server_id))
......@@ -404,6 +462,9 @@ class SynnefoCI(object):
self.logger.debug("Setup apt")
cmd = """
echo 'APT::Install-Suggests "false";' >> /etc/apt/apt.conf
echo 'Package: python-gevent' >> /etc/apt/preferences.d/00-gevent
echo 'Pin: release o=Debian' >> /etc/apt/preferences.d/00-gevent
echo 'Pin-Priority: 990' >> /etc/apt/preferences.d/00-gevent
echo 'precedence ::ffff:0:0/96 100' >> /etc/gai.conf
apt-get update
apt-get install -q=2 curl --yes --force-yes
......@@ -1118,7 +1179,8 @@ class SynnefoCI(object):
if not os.path.exists(dest):
os.makedirs(dest)
self.fetch_compressed("synnefo_build-area", dest)
self.fetch_compressed("webclient_build-area", dest)
if self.config.get("Global", "build_pithos_webclient") == "True":
self.fetch_compressed("webclient_build-area", dest)
self.logger.info("Downloaded debian packages to %s" %
_green(dest))
......
......@@ -1851,7 +1851,7 @@ settings of their nodes. As a result, when installing Admin in a node, the
Astakos and Cyclades packages will also be installed.
In order to disable the Astakos/Cyclades API in the Admin node, the
administrator can add the following line in ``99-locals.conf`` (you can create
administrator can add the following line in ``99-local.conf`` (you can create
it if doesn't exist):
.. code-block:: console
......@@ -1890,9 +1890,16 @@ example setup is the following:
You may notice that there are three databases instead of two. That's because
Django requires that every ``DATABASES`` setting has a *default* database. In
our case, we suggest that you use as default the Cyclades database. Finally,
you must not forget to add the ``DATABASE_ROUTERS`` setting in the above
example that must always be used in multi-db setups.
our case, we suggest that you use as default the Cyclades database.
You should also make sure not to enable database (psycopg) connection pooling
(as described in the `installation guide <install-guide-debian.html#enable-pooling>`_)
by omitting (deleting or commenting out) all the relevant pooling options from
the DB configuration (i.e. the ``synnefo_poolsize`` option).
Finally, you must not forget to add the ``DATABASE_ROUTERS`` setting in the
above example that must always be used in multi-db setups.
Disabling Admin
---------------
......@@ -2303,6 +2310,7 @@ Image Name/extension convention Usage
=============== ============================ =========
Favicon favicon.ico Favicon for all services
Dashboard logo dashboard_logo.png Visible in all Astakos UI pages
Cloudbar icon cloudbar_home.png Homepage icon for admin & cloudbar
Compute logo compute_logo.png Visible in all Cyclades UI pages
Console logo console_logo.png Visible in the Cyclades Console Window
Storage logo storage_logo.png Visible in all Pithos UI pages
......@@ -2933,6 +2941,8 @@ Ganeti level or by the administrator.
In the future Synnefo will also support moving VMs across different Regions.
.. _upgrade-notes:
Upgrade Notes
=============
......@@ -2950,10 +2960,13 @@ Upgrade Notes
v0.15 -> v0.16 <upgrade/upgrade-0.16>
.. _changelog-news:
Changelog, NEWS
===============
* v0.16.1 :ref:`Changelog <Changelog-0.16.1>`, :ref:`NEWS <NEWS-0.16.1>`
* v0.16 :ref:`Changelog <Changelog-0.16>`, :ref:`NEWS <NEWS-0.16>`
* v0.15.2 :ref:`Changelog <Changelog-0.15.1>`, :ref:`NEWS <NEWS-0.15.2>`
* v0.15.1 :ref:`Changelog <Changelog-0.15.1>`, :ref:`NEWS <NEWS-0.15.1>`
......
......@@ -111,6 +111,8 @@ external world. Also documents all Synnefo external REST APIs.
Developer's Guide <dev-guide>
API Guide <api-guide>
* :ref:`Upgrade notes <upgrade-notes>`
* :ref:`Changelog and NEWS <changelog-news>`
Synnefo tools
=============
......
......@@ -1871,7 +1871,7 @@ Then, provide connectivity mode and link to the network:
.. code-block:: console
# gnt-network connect test-net-public bridged br1
# gnt-network connect --nic-parameters mode=bridged,link=br1 test-net-public
Now, it is time to test that the backend infrastracture is correctly setup for
the Public Network. We will add a new VM, almost the same way we did it on the
......@@ -1932,7 +1932,7 @@ means that the instances will have a second NIC connected to the ``br2``.
.. code-block:: console
# gnt-network add --network=192.168.1.0/24 --mac-prefix=aa:00:55 --tags=nfdhcpd,private-filtered test-net-prv-mac
# gnt-network connect test-net-prv-mac bridged br2
# gnt-network connect --nic-parameters mode=bridged,link=br2 test-net-prv-mac
# gnt-instance add -o snf-image+default --os-parameters \
img_passwd=my_vm_example_passw0rd,img_format=diskdump,img_id=debian_base-6.0-x86_64,img_properties='{"OSFAMILY":"linux"\,"ROOT_PARTITION":"1"}' \
......@@ -2046,7 +2046,7 @@ package by running on node1:
.. code-block:: console
# yum install snf-cyclades-app memcached python-memcached
# yum install snf-cyclades-app memcached python-memcached snf-vncauthproxy
If all packages install successfully, then Cyclades are installed and we
proceed with their configuration.
......
......@@ -1949,7 +1949,7 @@ Then, provide connectivity mode and link to the network:
.. code-block:: console
# gnt-network connect test-net-public bridged br1
# gnt-network connect --nic-parameters mode=bridged,link=br1 test-net-public
Now, it is time to test that the backend infrastracture is correctly setup for
the Public Network. We will add a new VM, almost the same way we did it on the
......@@ -2010,7 +2010,7 @@ means that the instances will have a second NIC connected to the ``br2``.
.. code-block:: console
# gnt-network add --network=192.168.1.0/24 --mac-prefix=aa:00:55 --tags=nfdhcpd,private-filtered test-net-prv-mac
# gnt-network connect test-net-prv-mac bridged br2
# gnt-network connect --nic-parameters mode=bridged,link=br2 test-net-prv-mac
# gnt-instance add -o snf-image+default --os-parameters \
img_passwd=my_vm_example_passw0rd,img_format=diskdump,img_id=debian_base-6.0-x86_64,img_properties='{"OSFAMILY":"linux"\,"ROOT_PARTITION":"1"}' \
......@@ -2124,7 +2124,7 @@ package by running on node1:
.. code-block:: console
# apt-get install snf-cyclades-app memcached python-memcache snf-pithos-backend
# apt-get install snf-cyclades-app memcached python-memcache snf-pithos-backend snf-vncauthproxy
If all packages install successfully, then Cyclades are installed and we
proceed with their configuration.
......@@ -2702,6 +2702,12 @@ settings file. Uncomment it and assign the following URL to it:
``https://node1.example.com/admin``
Restart gunicorn on node1:
.. code-block:: console
# /etc/init.d/gunicorn restart
Now, we can proceed with testing Admin.
Testing of Admin
......
......@@ -122,14 +122,14 @@ Existing network flavors are the following:
Flavor Name Mode Link MAC prefix Tags
============== ======= =============================== ====================== ==================
IP_LESS_ROUTED routed ``snf-link-$network_id`` ``DEFAULT_MAC_PREFIX`` 'ip-less-routed'
MAC_FILTERED bridged ``DEFAULT_MAC_FILTERED_BRIDGE`` 'pool' 'private'filtered'
PHYSICAL_VLAN bridged 'pool' ``DEFAULT_MAC_PREFIX`` 'physical-vlan'
MAC_FILTERED bridged ``DEFAULT_MAC_FILTERED_BRIDGE`` ``pool`` 'private'filtered'
PHYSICAL_VLAN bridged ``pool`` ``DEFAULT_MAC_PREFIX`` 'physical-vlan'
CUSTOM bridged ``DEFAULT_BRIDGE`` ``DEFAULT_MAC_PREFIX``
============== ======= =============================== ====================== ==================
``DEFAULT_MAC_PREFIX``, ``DEFAULT_BRIDGE``,
``DEFAULT_MAC_FILTERED_BRIDGE`` are all configurable settings in
``/etc/synnefo/20-snf-cyclades-app-api.conf``. 'pool' is used to denote that a
``/etc/synnefo/20-snf-cyclades-app-api.conf``. ``pool`` is used to denote that a
link or MAC prefix will be allocated from the corresponding Pool. Finally,
most of these attributes, may be overridden when creating networks with
`snf-manage network-create command`.
......@@ -258,7 +258,7 @@ issuing:
# gnt-network add --network=5.6.7.0/27 --gateway=5.6.7.1 --network6=2001:648:2FFC:1322::/64 --gateway6=2001:648:2FFC:1322::1 --network-type=public --tags=nfdhcpd snf-net-1
# gnt-network connect snf-net-1 default bridged br100
# gnt-network connect --nic-parameters mode=bridged,link=br100 snf-net-1
# gnt-network list snf-net-1
Network Subnet Gateway NetworkType MacPrefix GroupList Tags
snf-net-1 5.6.7.0/27 5.6.7.1 public None default(bridged, br100) nfdhcpd
......@@ -313,7 +313,7 @@ issuing:
# gnt-network add --network=5.6.7.0/27 --gateway=5.6.7.1 --network6=2001:648:2FFC:1322::/64 --gateway6=2001:648:2FFC:1322::1 --network-type=public --tags=nfdhcpd,ip-less-routed snf-net-2
# gnt-network connect snf-net-2 default bridged br100
# gnt-network connect --nic-parameters mode=bridged,link=br100 snf-net-2
# gnt-network list snf-net-2
Network Subnet Gateway NetworkType MacPrefix GroupList Tags
dimara-net-1 62.217.123.128/27 62.217.123.129 public None default(routed, snf_public) nfdhcpd,ip-less-routed
......@@ -370,7 +370,7 @@ issuing:
# gnt-network add --network=192.168.1.0/24 --gateway=192.168.1.1 --network-type=private --tags=nfdhcpd,private-filtered snf-net-3
# gnt-network connect snf-net-3 default bridged prv0
# gnt-network connect --nic-parameters mode=bridged,link=prv0 snf-net-3
# gnt-network list snf-net-3
Network Subnet Gateway NetworkType MacPrefix GroupList Tags
snf-net-3 192.168.1.0/24 192.168.1.1 private aa:00:01 default(bridged, prv0) nfdhcpd,private-filtered
......@@ -432,7 +432,7 @@ issuing:
# gnt-network add --network=192.168.1.0/24 --gateway=192.168.1.1 --network-type=private --tags=nfdhcpd,physica-vlan snf-net-4
# gnt-network connect snf-net-4 default bridged prv1
# gnt-network connect --nic-parameters mode=bridged,link=prv1 snf-net-4
# gnt-network list snf-net-4
Network Subnet Gateway NetworkType MacPrefix GroupList Tags
snf-net-4 192.168.1.0/24 192.168.1.1 private None default(bridged, prv1) nfdhcpd,physical-vlan
......@@ -466,7 +466,7 @@ issuing:
# gnt-network add --network=192.168.1.0/24 --gateway=192.168.1.1 --network-type=private --tags=nfdhcpd snf-net-5
# gnt-network connect snf-net-5 default bridged br200
# gnt-network connect --nic-parameters mode=bridged,link=br200 snf-net-5
# gnt-network list snf-net-5
Network Subnet Gateway NetworkType MacPrefix GroupList Tags
snf-net-5 192.168.1.0/24 192.168.1.1 private bb:00:55 default(bridged, br200) nfdhcpd,private-filtered
......
......@@ -377,25 +377,6 @@ set True, run:
snf-deploy setup --setup synnefo --cluster ganeti2 -vvv
snf-deploy for Ganeti
=====================
`snf-deploy` can be used to deploy a Ganeti cluster on pre-existing nodes
by issuing:
.. code-block:: console
snf-deploy ganeti --setup ganeti -vvv
It will install a nameserver, nfs server and a Ganeti cluster. To
install a development node along with a Ganeti cluster ready for QA, run:
.. code-block:: console
snf-deploy ganeti-qa --setup qa -vvv
snf-deploy as a DevTool
=======================
......@@ -407,8 +388,7 @@ option to ``True``. Then run:
.. code-block:: console
snf-deploy setup --setup SETUP --node nodeX \
--role ROLE --cluster CLUSTER
snf-deploy setup --setup SETUP --node nodeX --component COMPONENT
to setup a specific role on a target node of a specific cluster and setup.
......@@ -416,10 +396,36 @@ For instance, to add another node to an existing ganeti backend run:
.. code-block:: console
snf-deploy setup --node node5 --role vmc --cluster ganeti3 --setup synnefo
snf-deploy setup --node node5 --component VMC --cluster ganeti3 --setup synnefo
`snf-deploy` keeps track of installed components per node in
``/var/lib/snf-deploy/snf_deploy_status``. If a deployment command
fails, the developer can make the required fix and then re-run the same
command; `snf-deploy` will not re-install components that have been