Commit d8d456de authored by Christodoulos Psaltis's avatar Christodoulos Psaltis
Browse files

Merge branch 'master' into ui-0.2

parents 82ebed37 83dcbbe1
Continuous integration with Jenkins
-----------------------------------
===================================
Preparing a GIT mirror
----------------------
Jenkins cannot currently work with Git over encrypted HTTP. To solve
this problem we currently mirror the central Git repository locally
on the jenkins installation machine. To setup such a mirror do the
following:
-edit .netrc
machine code.grnet.gr
login accountname
password accountpasswd
-Create the mirror
git clone --mirror https://code.grnet.gr/git/synnefo synnefo
-Setup cron to pull from the mirror periodically. Ideally, Git mirror updates
should run just before Jenkins jobs check the mirror for changes.
4,14,24,34,44,54 * * * * cd /path/to/mirror && git fetch && git remote prune origin
Jenkins setup
-------------
The following instructions will setup Jenkins to run synnefo tests with
the SQLite database. To run the tests on MySQL and/or Postgres, step 5
must be replicated. Also, the correct configuration file must be copied
(line 5 of the build script).
(line 6 of the build script).
1. Install and start Jenkins. On Debian Squeeze:
......
......@@ -12,7 +12,9 @@ Synnefo is written in Python 2.6 and depends on the following Python modules
- pyzmq-static [pyzmq==2.0.10.1]
- pycurl [pycurl==7.19.0]
- python-dateutil [python-dateutil==1.4.1]
WARNING: version python-dateutil==2.0 downloaded by pip known *not* to work with Python 2.6
WARNING: version python-dateutil==2.0 downloaded by pip known *not* to work
with Python 2.6
- south [south==0.7.1]
also, depending on the database engine of choice, on one of the following:
- MySQL-python [MySQL-python==1.2.3]
......@@ -27,8 +29,8 @@ The easiest method is to setup a working environment through virtualenv.
Alternatively, you can use your system's package manager to install
the dependencies (e.g. Macports has them all).
*On Snow Leopard, you have to set the following environment variable for
pip to compile the dependencies correctly.
*On Snow Leopard and linux (64-bit), you have to set the following environment
variable for pip to compile the dependencies correctly.
$export ARCHFLAGS="-arch x86_64"
......@@ -135,7 +137,7 @@ and then copy/edit according to the database used:
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'synnefo',
'USER': 'USERNAME'
'USER': 'USERNAME',
'PASSWORD': 'PASSWORD',
'HOST': 'HOST',
'PORT': 'PORT',
......@@ -184,6 +186,124 @@ username
9. (Hopefully) Done
South Database Migrations
------------------------
*Initial Migration
First, remember to add the south app to settings.py (it is already included in the
settings.py.dist).
To initialise south migrations in your database the following commands must be executed:
$ ./bin/python manage.py syncdb # Create / update the database with the south tables
$ ./bin/python manage.py migrate db # Perform migration in the database
Note that syncdb will create the latest models that exist in the db app, so some migrations may fail.
If you are sure a migration has already taken place you must use the "--fake" option, to apply it.
For example:
$ ./bin/python manage.py migrate db 0001 --fake
To be sure that all migrations are applied type:
$ ./bin/python manage.py migrate db --list
All starred migrations are applied.
Remember, the migration is performed mainly for the data, not for the database schema. If you do not want to migrate the
data, a syncdb and fake migrations for all the migration versions will suffice.
*Schema migrations:
Do not use the syncdb management command. It can only be used
the first time and/or if you drop the database and must recreate it from scratch.
See "Initial Migration" section.
Each time you make changes to the database and data migration is not required (WARNING: always
perform this with extreme care):
$ ./bin/python schemamigration db --auto
The above will create the migration script. Now this must be applied to the live database.
$ ./bin/python migrate db
Consider this example (adding a field to the SynnefoUser model):
bkarak@nefarian:~/devel/synnefo$ ./bin/python manage.py schemamigration db --auto
+ Added field new_south_test_field on db.SynnefoUser
Created 0002_auto__add_field_synnefouser_new_south_test_field.py.
You can now apply this migration with: ./manage.py migrate db
$ ./manage.py migrate db
Running migrations for db:
- Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field.
> db:0002_auto__add_field_synnefouser_new_south_test_field
- Loading initial data for db.
Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'.
Installed 1 object(s) from 1 fixture(s)
South needs some extra definitions to the model to preserve and migrate the existing data, for example, if we add a field
in a model, we should declare its default value. If not, South will propably fail, after indicating the error.
$ ./bin/python manage.py schemamigration db --auto
? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL.
? Since you are adding or removing this field, you MUST specify a default
? value to use for existing rows. Would you like to:
? 1. Quit now, and add a default to the field in models.py
? 2. Specify a one-off value to use for existing columns now
? Please select a choice: 1
*Data migrations:
If we need to do data migration as well, for example rename a field, we use tha 'datamigration' management command.
In contrast with schemamigration, to perform complex data migration, we must write the script manually. The process is
the following:
1. Introduce the changes in the code and fixtures (initial data).
2. Execute:
$ ./bin/python manage.py datamigration <migration_name_here>
For example:
$ ./bin/python manage.py datamigration db rename_credit_wallet
Created 0003_rename_credit_wallet.py.
3. We edit the generated script. It contains two methods: forwards and backwards.
For database operations (column additions, alter tables etc) we use the South database API
(http://south.aeracode.org/docs/databaseapi.html).
To access the data, we use the database reference (orm) provided as parameter in forwards, backwards method declarations
in the migration script. For example:
class Migration(DataMigration):
def forwards(self, orm):
orm.SynnefoUser.objects.all()
4. To migrate the database to the latest version, we execute:
./manage.py migrate db
To see which migrations are applied:
$ ./bin/python manage.py migrate db --list
db
(*) 0001_initial
(*) 0002_auto__add_field_synnefouser_new_south_test_field
(*) 0003_rename_credit_wallet
More information and more thorough examples can be found in the South web site.
http://south.aeracode.org/
UI Testing
----------
The functional ui tests require the Selenium server and the synnefo app to
......
......@@ -6,4 +6,5 @@ python-dateutil==1.4.1
django-hudson
MySQL-python==1.2.3
psycopg2==2.4
south==0.7.1
......@@ -13,4 +13,5 @@ DATABASES = {
}
}
BACKEND_PREFIX_ID = "jenkins-"
INSTALLED_APPS += ('django_hudson',)
......@@ -9,4 +9,5 @@ DATABASES = {
}
}
BACKEND_PREFIX_ID = "jenkins-"
INSTALLED_APPS += ('django_hudson',)
......@@ -6,4 +6,5 @@ DATABASES = {
}
}
BACKEND_PREFIX_ID = "jenkins-"
INSTALLED_APPS += ('django_hudson',)
# encoding: utf-8
import datetime
from south.db import db
from south.v2 import SchemaMigration
from django.db import models
class Migration(SchemaMigration):
def forwards(self, orm):
# Adding model 'SynnefoUser'
db.create_table('db_synnefouser', (
('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('name', self.gf('django.db.models.fields.CharField')(max_length=255)),
('credit', self.gf('django.db.models.fields.IntegerField')()),
('created', self.gf('django.db.models.fields.DateTimeField')(auto_now_add=True, blank=True)),
('updated', self.gf('django.db.models.fields.DateTimeField')(auto_now=True, blank=True)),
))
db.send_create_signal('db', ['SynnefoUser'])
# Adding model 'Image'
db.create_table('db_image', (
('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('name', self.gf('django.db.models.fields.CharField')(max_length=255)),
('state', self.gf('django.db.models.fields.CharField')(max_length=30)),
('owner', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.SynnefoUser'], null=True, blank=True)),
('created', self.gf('django.db.models.fields.DateTimeField')(auto_now_add=True, blank=True)),
('updated', self.gf('django.db.models.fields.DateTimeField')(auto_now=True, blank=True)),
('sourcevm', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.VirtualMachine'], null=True)),
))
db.send_create_signal('db', ['Image'])
# Adding model 'ImageMetadata'
db.create_table('db_imagemetadata', (
('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('meta_key', self.gf('django.db.models.fields.CharField')(max_length=50)),
('meta_value', self.gf('django.db.models.fields.CharField')(max_length=500)),
('image', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.Image'])),
))
db.send_create_signal('db', ['ImageMetadata'])
# Adding model 'Limit'
db.create_table('db_limit', (
('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('user', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.SynnefoUser'])),
('name', self.gf('django.db.models.fields.CharField')(max_length=30)),
('value', self.gf('django.db.models.fields.IntegerField')()),
))
db.send_create_signal('db', ['Limit'])
# Adding model 'Flavor'
db.create_table('db_flavor', (
('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('cpu', self.gf('django.db.models.fields.IntegerField')(default=0)),
('ram', self.gf('django.db.models.fields.IntegerField')(default=0)),
('disk', self.gf('django.db.models.fields.IntegerField')(default=0)),
))
db.send_create_signal('db', ['Flavor'])
# Adding unique constraint on 'Flavor', fields ['cpu', 'ram', 'disk']
db.create_unique('db_flavor', ['cpu', 'ram', 'disk'])
# Adding model 'FlavorCost'
db.create_table('db_flavorcost', (
('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('cost_active', self.gf('django.db.models.fields.PositiveIntegerField')()),
('cost_inactive', self.gf('django.db.models.fields.PositiveIntegerField')()),
('effective_from', self.gf('django.db.models.fields.DateTimeField')()),
('flavor', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.Flavor'])),
))
db.send_create_signal('db', ['FlavorCost'])
# Adding model 'VirtualMachine'
db.create_table('db_virtualmachine', (
('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('name', self.gf('django.db.models.fields.CharField')(max_length=255)),
('owner', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.SynnefoUser'])),
('created', self.gf('django.db.models.fields.DateTimeField')(auto_now_add=True, blank=True)),
('updated', self.gf('django.db.models.fields.DateTimeField')(auto_now=True, blank=True)),
('charged', self.gf('django.db.models.fields.DateTimeField')(default=datetime.datetime(2011, 4, 12, 13, 36, 55, 200332))),
('sourceimage', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.Image'])),
('hostid', self.gf('django.db.models.fields.CharField')(max_length=100)),
('ipfour', self.gf('django.db.models.fields.IPAddressField')(max_length=15)),
('ipsix', self.gf('django.db.models.fields.CharField')(max_length=100)),
('flavor', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.Flavor'])),
('deleted', self.gf('django.db.models.fields.BooleanField')(default=False, blank=True)),
('suspended', self.gf('django.db.models.fields.BooleanField')(default=False, blank=True)),
('action', self.gf('django.db.models.fields.CharField')(max_length=30, null=True)),
('operstate', self.gf('django.db.models.fields.CharField')(max_length=30, null=True)),
('backendjobid', self.gf('django.db.models.fields.PositiveIntegerField')(null=True)),
('backendopcode', self.gf('django.db.models.fields.CharField')(max_length=30, null=True)),
('backendjobstatus', self.gf('django.db.models.fields.CharField')(max_length=30, null=True)),
('backendlogmsg', self.gf('django.db.models.fields.TextField')(null=True)),
))
db.send_create_signal('db', ['VirtualMachine'])
# Adding model 'VirtualMachineGroup'
db.create_table('db_virtualmachinegroup', (
('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('name', self.gf('django.db.models.fields.CharField')(max_length=255)),
('created', self.gf('django.db.models.fields.DateTimeField')(auto_now_add=True, blank=True)),
('updated', self.gf('django.db.models.fields.DateTimeField')(auto_now=True, blank=True)),
('owner', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.SynnefoUser'])),
))
db.send_create_signal('db', ['VirtualMachineGroup'])
# Adding M2M table for field machines on 'VirtualMachineGroup'
db.create_table('db_virtualmachinegroup_machines', (
('id', models.AutoField(verbose_name='ID', primary_key=True, auto_created=True)),
('virtualmachinegroup', models.ForeignKey(orm['db.virtualmachinegroup'], null=False)),
('virtualmachine', models.ForeignKey(orm['db.virtualmachine'], null=False))
))
db.create_unique('db_virtualmachinegroup_machines', ['virtualmachinegroup_id', 'virtualmachine_id'])
# Adding model 'VirtualMachineMetadata'
db.create_table('db_virtualmachinemetadata', (
('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('meta_key', self.gf('django.db.models.fields.CharField')(max_length=50)),
('meta_value', self.gf('django.db.models.fields.CharField')(max_length=500)),
('vm', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.VirtualMachine'])),
))
db.send_create_signal('db', ['VirtualMachineMetadata'])
# Adding model 'Debit'
db.create_table('db_debit', (
('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('when', self.gf('django.db.models.fields.DateTimeField')()),
('user', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.SynnefoUser'])),
('vm', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.VirtualMachine'])),
('description', self.gf('django.db.models.fields.TextField')()),
))
db.send_create_signal('db', ['Debit'])
# Adding model 'Disk'
db.create_table('db_disk', (
('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('name', self.gf('django.db.models.fields.CharField')(max_length=255)),
('created', self.gf('django.db.models.fields.DateTimeField')(auto_now_add=True, blank=True)),
('updated', self.gf('django.db.models.fields.DateTimeField')(auto_now=True, blank=True)),
('size', self.gf('django.db.models.fields.PositiveIntegerField')()),
('vm', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.VirtualMachine'], null=True, blank=True)),
('owner', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['db.SynnefoUser'], null=True, blank=True)),
))
db.send_create_signal('db', ['Disk'])
def backwards(self, orm):
# Deleting model 'SynnefoUser'
db.delete_table('db_synnefouser')
# Deleting model 'Image'
db.delete_table('db_image')
# Deleting model 'ImageMetadata'
db.delete_table('db_imagemetadata')
# Deleting model 'Limit'
db.delete_table('db_limit')
# Deleting model 'Flavor'
db.delete_table('db_flavor')
# Removing unique constraint on 'Flavor', fields ['cpu', 'ram', 'disk']
db.delete_unique('db_flavor', ['cpu', 'ram', 'disk'])
# Deleting model 'FlavorCost'
db.delete_table('db_flavorcost')
# Deleting model 'VirtualMachine'
db.delete_table('db_virtualmachine')
# Deleting model 'VirtualMachineGroup'
db.delete_table('db_virtualmachinegroup')
# Removing M2M table for field machines on 'VirtualMachineGroup'
db.delete_table('db_virtualmachinegroup_machines')
# Deleting model 'VirtualMachineMetadata'
db.delete_table('db_virtualmachinemetadata')
# Deleting model 'Debit'
db.delete_table('db_debit')
# Deleting model 'Disk'
db.delete_table('db_disk')
models = {
'db.debit': {
'Meta': {'object_name': 'Debit'},
'description': ('django.db.models.fields.TextField', [], {}),
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'user': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.SynnefoUser']"}),
'vm': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.VirtualMachine']"}),
'when': ('django.db.models.fields.DateTimeField', [], {})
},
'db.disk': {
'Meta': {'object_name': 'Disk'},
'created': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'name': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
'owner': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.SynnefoUser']", 'null': 'True', 'blank': 'True'}),
'size': ('django.db.models.fields.PositiveIntegerField', [], {}),
'updated': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'}),
'vm': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.VirtualMachine']", 'null': 'True', 'blank': 'True'})
},
'db.flavor': {
'Meta': {'unique_together': "(('cpu', 'ram', 'disk'),)", 'object_name': 'Flavor'},
'cpu': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
'disk': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'ram': ('django.db.models.fields.IntegerField', [], {'default': '0'})
},
'db.flavorcost': {
'Meta': {'object_name': 'FlavorCost'},
'cost_active': ('django.db.models.fields.PositiveIntegerField', [], {}),
'cost_inactive': ('django.db.models.fields.PositiveIntegerField', [], {}),
'effective_from': ('django.db.models.fields.DateTimeField', [], {}),
'flavor': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.Flavor']"}),
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'})
},
'db.image': {
'Meta': {'object_name': 'Image'},
'created': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'name': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
'owner': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.SynnefoUser']", 'null': 'True', 'blank': 'True'}),
'sourcevm': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.VirtualMachine']", 'null': 'True'}),
'state': ('django.db.models.fields.CharField', [], {'max_length': '30'}),
'updated': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'})
},
'db.imagemetadata': {
'Meta': {'object_name': 'ImageMetadata'},
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'image': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.Image']"}),
'meta_key': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
'meta_value': ('django.db.models.fields.CharField', [], {'max_length': '500'})
},
'db.limit': {
'Meta': {'object_name': 'Limit'},
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'name': ('django.db.models.fields.CharField', [], {'max_length': '30'}),
'user': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.SynnefoUser']"}),
'value': ('django.db.models.fields.IntegerField', [], {})
},
'db.synnefouser': {
'Meta': {'object_name': 'SynnefoUser'},
'created': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
'credit': ('django.db.models.fields.IntegerField', [], {}),
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'name': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
'updated': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'})
},
'db.virtualmachine': {
'Meta': {'object_name': 'VirtualMachine'},
'action': ('django.db.models.fields.CharField', [], {'max_length': '30', 'null': 'True'}),
'backendjobid': ('django.db.models.fields.PositiveIntegerField', [], {'null': 'True'}),
'backendjobstatus': ('django.db.models.fields.CharField', [], {'max_length': '30', 'null': 'True'}),
'backendlogmsg': ('django.db.models.fields.TextField', [], {'null': 'True'}),
'backendopcode': ('django.db.models.fields.CharField', [], {'max_length': '30', 'null': 'True'}),
'charged': ('django.db.models.fields.DateTimeField', [], {'default': 'datetime.datetime(2011, 4, 12, 13, 36, 55, 200332)'}),
'created': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
'deleted': ('django.db.models.fields.BooleanField', [], {'default': 'False', 'blank': 'True'}),
'flavor': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.Flavor']"}),
'hostid': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'ipfour': ('django.db.models.fields.IPAddressField', [], {'max_length': '15'}),
'ipsix': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
'name': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
'operstate': ('django.db.models.fields.CharField', [], {'max_length': '30', 'null': 'True'}),
'owner': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.SynnefoUser']"}),
'sourceimage': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.Image']"}),
'suspended': ('django.db.models.fields.BooleanField', [], {'default': 'False', 'blank': 'True'}),
'updated': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'})
},
'db.virtualmachinegroup': {
'Meta': {'object_name': 'VirtualMachineGroup'},
'created': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'machines': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['db.VirtualMachine']", 'symmetrical': 'False'}),
'name': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
'owner': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.SynnefoUser']"}),
'updated': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'})
},
'db.virtualmachinemetadata': {
'Meta': {'object_name': 'VirtualMachineMetadata'},
'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'meta_key': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
'meta_value': ('django.db.models.fields.CharField', [], {'max_length': '500'}),
'vm': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['db.VirtualMachine']"})
}
}
complete_apps = ['db']
......@@ -133,6 +133,7 @@ INSTALLED_APPS = (
'synnefo.db',
'synnefo.ganeti',
'synnefo.logic',
'south'
)
# The RAPI endpoint and associated credentials to use
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment