Commit 462aec17 authored by Zenon Mousmoulas's avatar Zenon Mousmoulas

Merge branch '1.0_docs' into master

parents 7fec7431 5871ae32
......@@ -118,15 +118,15 @@ NRO_COUNTRY_NAME = _('My Country')
NRO_COUNTRY_CODE = 'tld'
# main domain url used in right top icon, eg. http://www.grnet.gr
NRO_DOMAIN_MAIN_URL = "http://www.example.com"
# developer info for footer
NRO_PROV_BY_DICT = {"name": "GRNET NOC", "url": "//noc.grnet.gr"}
#provider social media contact (Use: // to preserve https)
# "provided by" info for footer
NRO_PROV_BY_DICT = {"name": "EXAMPLE NRO TEAM", "url": "http://noc.example.com"}
# social media contact (Use: // to preserve https)
NRO_PROV_SOCIAL_MEDIA_CONTACT = [
{"url": "//facebook.com/noc.grnet.gr", "icon":"/static/img/facebook_img.png", "name":"Facebook"},
{"url": "//twitter.com/grnetnoc", "icon":"/static/img/twitter_img.png", "name":"Twitter"},
{"url": "//facebook.com/example.com", "icon":"/static/img/facebook_img.png", "name":"Facebook"},
{"url": "//twitter.com/example_com", "icon":"/static/img/twitter_img.png", "name":"Twitter"},
]
#Helpdesk, used in base.html:
# Helpdesk, used in base.html:
NRO_DOMAIN_HELPDESK_DICT = {"name": _("Domain Helpdesk"), 'email':'helpdesk@example.com', 'phone': '12324567890', 'uri': 'helpdesk.example.com'}
#Countries for Realm model:
......@@ -134,7 +134,7 @@ REALM_COUNTRIES = (
('country_2letters', 'Country' ),
)
#Shibboleth attribute map
# Shibboleth attribute map
SHIB_USERNAME = ['HTTP_EPPN']
SHIB_MAIL = ['mail', 'HTTP_MAIL', 'HTTP_SHIB_INETORGPERSON_MAIL']
SHIB_FIRSTNAME = ['HTTP_SHIB_INETORGPERSON_GIVENNAME']
......@@ -169,12 +169,12 @@ SOCIAL_AUTH_GOOGLE_OAUTH2_SCOPE = []
# "CAT_API_KEY": "<provided API key>",
# "CAT_API_URL": "https://cat.eduroam.org/admin/API.php",
# "CAT_PROFILES_URL": "https://cat.eduroam.org/",
# "CAT_IDPMGMT_URL": "https://cat.eduroam.org/admin/overview_federation.php"
# "CAT_IDPMGMT_URL": "https://cat.eduroam.org/admin/overview_idp.php"
# },
# 'testing': {
# "CAT_API_KEY": "<provided API key>",
# "CAT_API_URL": "https://cat-test.eduroam.org/test/admin/API.php",
# "CAT_PROFILES_URL": "https://cat-test.eduroam.org/test",
# "CAT_IDPMGMT_URL": "https://cat-test.eduroam.org/test/admin/overview_federation.php"
# "CAT_IDPMGMT_URL": "https://cat-test.eduroam.org/test/admin/overview_idp.php"
# },
# }
......@@ -4,40 +4,36 @@ DjNRO = Django + NRO (Django National Roaming Operator or how to manage your edu
In the [eduroam](http://www.eduroam.org) world, NRO stands for National Roaming Operator.
Maintaining and managing a local eduroam database is quite an important responsibility of an eduroam NRO.
eduroam.org periodically polls and gathers information from all participating domains.
Information is provided upstream, in a structured way (XML format) and consists of participating institutions' data, location data along with monitoring data - though provisioning of monitoring data has been superseeded by the f-Ticks mechanism.
Information is provided upstream, in a structured way (XML format) and consists of participating institutions' data, location data along with statistical data - though collection of statistical data has been superseeded by the F-Ticks mechanism.
The source of information should be the local eduroam database. So, changes to the database should be reflected to the XML files.
New eduroam locations, changes in contacts and information about each location should be up-to-date so as to ease the eduroam usage and assist eduroam users whenever they need support.
The source of information should be the local eduroam database. Changes in the database should be reflected in the XML files.
New eduroam locations, changes in contacts and information about each location should be up-to-date so as to ease discovery of information and assist eduroam users whenever they need support.
DjNRO is a Django platform that eases the management process of a National Roaming Operator. DjNRO complies with the [eduroam database](http://monitor.eduroam.org/database.php) and the eduroam XSDs.
Thus, apart from domain management, it can generate the necessary xml files for eduroam.org monitoring.
DjNRO is a Django platform that eases the management process for a National Roaming Operator. DjNRO complies with the [eduroam database](http://monitor.eduroam.org/database.php) and the eduroam XML schema.
Thus, apart from domain management, it can generate the necessary XML files for harvesting by eduroam.org.
DjNRO is more than keeping eduroam.org updated with data.
In essence it is a distributed management application. It is distributed in the sense that information about each institution locations and services is kept up-to-date by each local eduroam administrator. Keeping in pace with eduroam's federated nature, our implementation uses federated authentication/authorisation mechanisms, namely Shibboleth.
In case Shibboleth is not an option for an institution, a social media auth mechanism comes in handy. The local institution eduroam administrators can become DjNRO admins. Local eduroam administrators register to the platform via Shibboleth or social media auth. The NRO's responsibility is to activate their accounts.
In essence it is a distributed management application. It is distributed in the sense that information about each institution locations and services is kept up-to-date by each local eduroam administrator. Keeping in pace with eduroam's federated nature, our implementation uses federated authentication/authorisation mechanisms, namely SAML Web SSO.
In case SAML is not an option for an institution, social authentication mechanisms come in handy. The local institution eduroam administrators can become DjNRO admins. Local eduroam administrators register in the platform through SAML or social auth. The NRO's responsibility is to activate their accounts.
From then on they can manage their eduroam locations, contact points and institution information. The administrative interface especially the locations management part, is heavily implemented with Google Maps. This makes editing easier, faster and accurate.
From then on they can manage their eduroam locations, contact points and institution information. The administrative interface and specifically service locations management makes heavy use of Google Maps. This makes editing easier, faster and more accurate.
Installation and customization is fairly easy and is described in the following sections.
Currently the source code is availiable at code.grnet.gr and github and can be cloned via git::
The source code is available on and can be downloaded/fetched from GitHub::
git clone https://code.grnet.gr/git/djnro
git clone https://github.com/grnet/djnro.git
The Greek eduroam webpage is a living example of DjNRO: [eduroam|gr](http://www.eduroam.gr)
The eduroam web site of GRNET (the eduroam NRO in Greece) is a living example of DjNRO: [eduroam|gr](http://www.eduroam.gr)
## Features
* Allow your local eduroam admins to edit their local eduroam data (AP locations, server params, etc)
* Allow your local eduroam admins to edit their institution eduroam data (AP locations, server params, etc)
* Visualize the information via Google Maps
* Eduroam world maps overview via daily update on eduroam.org KML file
* Find your closest eduroam in the world
* **New** Allow for eduroam CAT institution enrollments
* **New** Extract contact info for mailing list creation
* **New** Server monitoring data
* **New** Pebble watch app with closest eduroam walking instrunctions
Bootstrap 3 CSS framework with responsive design makes it work on every device
* eduroam world maps overview via daily update on eduroam.org KML file
* Find your closest eduroam location around the world
* Allow for eduroam CAT institution enrollments
* Extract contact info for mailing list creation
* Server monitoring data
* Pebble watch app with closest eduroam walking instructions
This diff is collapsed.
# Use case: migration from 0.8 to 1.0
GRNET had been running the production DjNRO v0.8 instance on Django 1.2 (on Debian Squeeze). The MySQL database had been created with MyISAM tables. GRNET wanted to migrate this installation (both the application and the database) to a different host (Debian Wheezy), upgrade to latest code and Django 1.4 and at the same time migrate the database to the InnoDB engine. Achieving this by means of dumping and restoring the database was deemed too complicated and risky, as the SQL dump would have to be edited in order to properly recreate the database with InnoDB. Therefore it was decided to use Django dumpdata/loaddata to transfer the data to the new installation. The following sections document the steps taken in this process (most actions performed on the new host).
### Installing required software
We wanted to install both the old (v0.8) and current versions of DjNRO on the new host. The old version would serve as a snapshot of the production instance, so as to test actions in preparation of the migration without touching production.
We also prefer installing Debian-provided python packages for essential software, so as to have security updates, whereas installation through PyPI would provide latest versions of the software, which are however not required in our installation, but would also require compilers and development tools that we would rather not install in this environment. Therefore, with regards to python requirements, the installation would use a mix of Debian packages and packages installed with `pip` in a virtual environment.
1. Install required software with Debian packages:
- apache2-mpm-worker
- memcached
- mysql-server
- libapache2-mod-wsgi
- libapache2-mod-shib2 (for SAML federated login)
- python (= python 2.7)
- python-pip (this will also install python 2.6)
- python-virtualenv (this also requires python-pip)
- git
1. Install optional Debian packages:
- mysql-client (for manage.py dbshell)
- gettext (for translation development)
- ipython (for a better python shell)
- winpdb (for python debugging)
- patch
1. Install Debian python packages (rather than using pip to install them as requirements in a virtualenv):
- python-mysqldb
- python-memcache
- python-lxml
- python-yaml
1. Likewise install these packages instead of *security* requirements for `python-requests` (which will be installed in a virtualenv):
- python-openssl
- python-ndg-httpsclient
- python-pyasn1
1. Install these packages to fulfill some dependencies for Python Social Auth:
- python-httplib2
- python-oauth2
- python-openid
1. Install Debian packages for Django and some addons:
- python-django
- python-django-auth-ldap (for LDAP authentication, this will also install `python-django`)
**Note:** Installing `python-django-auth-ldap` would also install Django 1.4 outside a virtual environment. That would not be a problem as we were still targetting to use Debian-provided Django 1.4. However one could also install the `python-ldap` Debian package and install `django-ldap` and `Django` inside the virtualenv. In any case we don't want to install `python-ldap` with pip for the reasons stated earlier.
### Creating virtual environments
For a number of reasons we want to use *relocatable* virtual environments, for example so that we could move a virtualenv to a different path and apps installed therein would still work after that.
We thus create a virtualenv like this:
virtualenv --python=/usr/bin/python2.7 --system-site-packages /path/to/virtualenv
We then need to patch `/path/to/virtualenv/bin/activate` so that it will work with a *relocated* virtualenv:
```
patch -p1 /path/to/virtualenv/bin/activate <<'EOF'
--- a/activate 2015-09-28 03:12:24.000000000 +0300
+++ b/activate 2015-09-28 03:12:57.000000000 +0300
@@ -37,7 +37,16 @@
# unset irrelavent variables
deactivate nondestructive
-VIRTUAL_ENV="/path/to/virtualenv"
+if [ -n "$BASH" ]; then
+ VIRTUAL_ENV=$(readlink -f "${BASH_SOURCE}" | sed 's#/bin/activate[^/]*$##')
+elif [ -n "$ZSH_VERSION" ]; then
+ VIRTUAL_ENV=$(readlink -f "${(%):-%N}" | sed 's#/bin/activate[^/]*$##')
+else
+ return 1
+fi
+if [ -z "$VIRTUAL_ENV" ]; then
+ return 1
+fi
export VIRTUAL_ENV
_OLD_VIRTUAL_PATH="$PATH"
EOF
```
A similar treatment is needed for scripts targetting different shells (`/path/to/virtualenv/bin/activate.{csh,fish}`), otherwise the `VIRTUAL_ENV` variable set therein should be updated once the virtualenv is *relocated*.
We then make the virtualenv relocatable. This command should be executed whenever virtualenv-installed packages install files in `/path/to/virtualenv/bin`.
virtualenv --relocatable /path/to/virtualenv
Using this process we shall create two virtual environments:
- one for running the old version of DjNRO on Django 1.2 and
- one for setting up the target version of DjNRO on Django 1.4
In each of these we shall install a number of packages using `pip`:
#### *Old* virtualenv
- `Django==1.2.3`
- `South==0.7.5` (the particular version is required due to [an issue affecting `edumanage/migrations/0028...`](https://lists.grnet.gr/wws/arc/djnro/2013-02/msg00001.html))
- `django-extensions==0.5`
- `django-registration==0.7`
- `django-tinymce==1.5` (**with patch**, see below)
- `django-fixture-magic==0.0.7` (we will need to reorder fixtures with forward references)
- `ipaddr==2.1.11`
- `oauth==1.0.1`
**Note:** There is a minor issue with django-tinymce 1.5 that must be fixed:
```
patch -p1 /path/to/virtualenv/lib/python2.7/site-packages/tinymce/widgets.py <<'EOF'
--- a/widgets.py
+++ b/widgets.py
@@ -11,7 +11,7 @@
from django.contrib.admin import widgets as admin_widgets
from django.core.urlresolvers import reverse
from django.forms.widgets import flatatt
-from django.forms.util import smart_unicode
+from django.utils.encoding import smart_unicode
from django.utils.html import escape
from django.utils import simplejson
from django.utils.datastructures import SortedDict
EOF
```
Even though we have previously installed Debian-provided python-django 1.4 outside the virtualenv and we set it up with `--system-site-packages`, `pip` will happily install Django 1.2 inside:
```
(old_djnro)# pip install Django==1.2.3
[...]
Installing collected packages: Django
Found existing installation: Django 1.4.5
Not uninstalling Django at /usr/lib/python2.7/dist-packages, outside environment /srv/venv/old_djnro
[...]
```
#### *New* virtualenv
- `South==1.0`
- `django-registration==1.0`
- `django-tinymce==1.5.3`
- `ipaddr==2.1.11`
- `oauth==1.0.1`
- `longerusername==0.4`
- `python-social-auth==0.2.10`
- `requests==2.7.0` (also required by PSA)
### Installing DjNRO and migrating data
After creating the virtual environments and installing the required packages in each one, these are the steps to install both the old and new versions of DjNRO and migrate the data as necessary:
* Clone the DjNRO repo and checkout the `old-stable` branch:
`git clone --branch old-stable https://github.com:grnet/djnro.git`
* Modify `manage.py` so that it will use the virtualenv python interpreter: change the first line from `#!/usr/bin/python` to `#!/usr/bin/env python`
* Copy `settings.py`, `urls.py` and `django.wsgi` over from the old host, adjusting any paths therein as necessary for matching the installation of DjNRO on the new host.
* Create the MySQL database (and grant rights to the DjNRO user as on the old host):
`create database eduroam character set = 'utf8' collate = 'utf8_unicode_ci';`
* Activate the *old* virtualenv:
`source /path/to/old-virtualenv/bin/activate`
* Create the DjNRO schema and run South migrations:
`./manage.py syncdb --setings='settings'`
`./manage.py migrate --settings='settings'`
* Dump DjNRO data on the **old** host:
`./manage.py dumpdata --indent=4 --natural --exclude=contenttypes --exclude=auth.Permission > /tmp/eduroam_production.json`
**Note:** The `contenttypes` app and `auth.Permission` model were excluded because they are not necessary (they are recreated by `syncdb`) and they could not be properly imported on the new host. If there are custom permissions assignments, however, they would be lost.
* Copy over the JSON data dump to the **new** host and reorder the *fixtures*, so that e.g. a model having a foreign key to another model will be re-created after the latter upon import:
```
./manage.py reorder_fixtures --settings='settings' /tmp/eduroam_production.json \
edumanage.name_i18n \
edumanage.url_i18n \
edumanage.contact \
edumanage.realm \
edumanage.realmdata \
edumanage.institution \
edumanage.institutiondetails \
edumanage.institutioncontactpool \
edumanage.serviceloc \
edumanage.instserver \
edumanage.instrealm \
edumanage.instrealmmon \
edumanage.monlocalauthnparam \
edumanage.monproxybackclient \
> /tmp/eduroam_production.ordered.json
```
* Import the reordered data:
`./manage.py loaddata --settings='settings' /tmp/eduroam_production.ordered.json`
* Copy over the apache vhost and adjust as necessary.
* Verify that the old version of DjNRO is running and data has been properly imported.
* Before switching to the new version of DjNRO, undo the modification to `manage.py`:
`git checkout -- manage.py`
* Checkout the target version:
`git checkout master`
* Create `djnro/local_settings.py` by copying `djnro/local_settings.py.dist` and adjusting it according to `settings.py`.
* Activate the *new* virtualenv:
`source /path/to/new-virtualenv/bin/activate`
* *Fake* the initial migration for PSA because we are migrating from Django Social Auth:
`./manage.py migrate default 0001_initial --fake`
* Run the rest of the migrations:
`./manage.py migrate`
* Modify/update the apache vhost configuration as necessary.
* Verify that the new version of DjNRO is running and data has been properly migrated.
......@@ -22,7 +22,7 @@ DjNRO heavily depends on the following:
* python-django-auth-ldap: if ldap authentication backend will be used.
## Django Social Auth
User authentication via social media is carried out by the [python-social-auth](http://http://django-social-auth.readthedocs.org/en/latest/index.html) python-social-auth package.
User authentication via social media is carried out by the [python-social-auth](http://http://django-social-auth.readthedocs.org/en/latest/index.html) package.
## Pip requirements.txt file
......
# Upgrading DjNRO from 0.8 to 1.0 or later
DjNRO 0.8 was developed with django 1.2. Version 1.0 was developed with django 1.4.2.
Dump data with the help of south (always keep a backup):
./manage.py dumpdata --indent=4 --natural --exclude=contenttypes --exclude=auth.Permission > /tmp/eduroam_0.8.json
## Install DjNro
Install DjNRO >= v1.0 by following the instructions.
## Patch widgets.py
Patch widgets.py (there is a bug in tinymce)
--- widgets.py
+++ widgets.py
@@ -11,7 +11,7 @@
from django.contrib.admin import widgets as admin_widgets
from django.core.urlresolvers import reverse
from django.forms.widgets import flatatt
-from django.forms.util import smart_unicode
+from django.utils.encoding import smart_unicode
from django.utils.html import escape
from django.utils import simplejson
from django.utils.datastructures import SortedDict
## Migrate
We have to introduce south the the models of social auth:
./manage.py migrate default 0001_initial --fake
And then run the real migration:
./manage.py migrate
# Loading old data to a new instance
In case you want to load data to a new database one has to follow these extra
steps.
In the old installation of DjNRO:
- install and add 'fixture_magic' to INSTALLED_APPS in settings.py
- run:
./manage.py reorder_fixtures --settings='settings' /tmp/eduroam_0.8.json edumanage.name_i18n edumanage.url_i18n edumanage.contact edumanage.realm edumanage.realmdata edumanage.institution edumanage.institutiondetails edumanage.institutioncontactpool edumanage.serviceloc edumanage.instserver edumanage.instrealm edumanage.instrealmmon edumanage.monlocalauthnparam edumanage.monproxybackclient > /tmp/eduroam_0.8.ordered.json
Now in the new installation:
- run `./manage.py loaddata --settings='settings' /tmp/eduroam_0.8.ordered.json`
- Continue with the `Migrate` step from above.
site_name: DjNRO
repo_url: https://github.com/grnet/djnro/
docs_dir: docs
site_author: Stavros Kroustouris
site_author: DjNRO
theme: readthedocs
pages:
- 'Introduction': 'index.md'
- 'Installation':
- 'Requirements': 'installation/requirements.md'
- 'Installing DjNRO': 'installation/install.md'
- 'Upgrading from 0.8': 'installation/upgrading-from-0.8.md'
- 'Migrating from 0.8 to 1.0': 'installation/migrating-from-0.8-to-1.0.md'
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment