GitLab

Fix south migration files. In data migrations, access the Models from
the orm, instead importing Models from synnefo.db. This is necessary as
the orm provides access to the version of the models that existed when
the migration file was created.

Do not perform reconciliation for VMs that are hosted in Backends
that are currently offline, since it is impossible to get the
state of these VMs from the backend. Before this patch, the
reconciliation mechanism would report these VMs as stale (and with
--fix-* would also mark them as deleted in the DB).

Add new management command 'reconcile-pools' which checks the consistency of the
pools and the uniqueness of the associated resources. Currently this
commands detects the inconcistencies, without trying to resolve them.

In order to guarantee uniqueness of MAC address accross all backends,
synnefo assignes a unique mac prefix to each network in each backend.

A mac-prefix of 20 bits, which is picked from the MacPrefixPool, is
assigned to each Network. The mac-prefix of the network in each
backend results from combining the mac-prefix of the Network with the
unique index of the Backend, resulting in a 3-byte mac-prefix (up to 16
Backends).

Settings PRIVATE_MAC_FILTERED_BASE_MAC_PREFIX and
PRIVATE_MAC_FILTERED_MAX_PREFIX_NUMBER have been converted to
MAC_POOL_BASE and MAC_POOL_LIMIT, since mac-prefixes are now assigned to
all networks.

Assert that this is an 'real' deletion of a Network before releasing
it's resources. Without this patch, a resource used by an other Network
could be released, if someone called the Network.update_state function
explicitly.

Do not find the public network by the primary key, as the initial_data.json
is obsolete. Instead, ise the public attribute of the Network Model.
Currently only one public Network is supported, and the used can not
create or delete the public Network. But this can easily change in the
future.

Fix wrong reporting messages when adding a new backend.

Add PRIVATE_PHYSICAL_VLAN_TAGS setting

Check that nic has an ipv4 before trying to release it.

Introduce new setting 'PUBLIC_ROUTED_USE_POOL', to indicate whether
synnefo will be responsible for the IP management of the public network,
or this responsibility will be delegated to Ganeti.

Specially for create_instance and create_network methods, since a
VirtualMachine, Network object is created before sending the request
to the backend.

Implement ForUpdateManager that uses SELECT .. FOR UPDATE statement in
order to acquire row-level exclusive locks to DB. Essentially this
manager backports the select_for_update() method that is implemented in
Django >= 1.4.

Non-blocking reads are not implemented, and each query including a row
that is locked by another transaction will block until the lock is
released. Also care must be taken in order to avoid deadlocks or retry
transactions that abort due to deadlocks.

Also, update Network object to use this manager, in order to guarantee
consistency of the IP pool of each Network.

Reconcile the contents of each networks IP pool by comparing it
with combination of IP pools of all ganeti backends.

Add extra option for detecting conflicting IPs.

Limit the size of networks that users can create, since big IP
pools can have negative impact on performance. Allowed network
size is controled by 'MAX_CIDR_BLOCK' setting.

Delegate IP management to synnefo, in order to be able to allocate IP
addresses accross multiple Ganeti backends. This is achieved by creating a
Pool of IPs for each Network, in the form of a bitarray, which is stored
in the DB as a base64 string.

Reservation of IPs is done beforing creating the NIC in Ganeti, unlike
releasing which is only done after the notification from Ganeti(hook) is
received.

Reservation and release of address must be performed inside a
transaction, and with the use of select_for_update() statement in order
to guarantee atomic and isolated access to the pool.

Create a management command for creating networks. Also remove
the initial_data.json, since it is currently not necessary.

- Extend the Network model with fields for holding the IPv6 subnet and
  gateway.
- Update network API methods.
- Modify ganeti hooks to return IPv6 address for all network interfaces.
  IPv6 is now derived from the IPv6 subnet of the network in Ganeti,
  instead of the old setting (PUBLIC_IPV6_PREFIX)
- Remove obsolete setting (`PUBLIC_IPV6_PREFIX`)
- Update management commands for IPv6

Use cipher to encrypt the passwords of the Ganeti backends in the DB.
Encryption/decryption is based on a configurable key, defined by the
ENCRYPTION_KEY setting. After modifying this setting, synnefo will be
unable to decrypt already stored passwords, and passwords must be
re-entered, using 'snf-manage backend-modify' command and restaring the
server. A migration file is responsible for initially encrypting already
stored passwords.

Modify list_networks and list_servers to filter deleted entries when
if-modified-since HTTP header is used, because deleted entries can be
updated after deletion through the reconciliation mechanism.

Extend NetworkInterface model with a dirty bit, indicating whether
the index of the NIC is consistent with the one in Ganeti. This flag is
set when deleting a NIC, and is switched off after processing the
notification from Ganeti. Removing a NIC with the dirty bit on is not
allowed.

Raise Overlimit exception when either bridge or mac-prefix pool
is exhausted. Also prohibit public network creation.