GitLab

Make Cyclades and Pithos require PithosBackend and Archip.
Make VMC require also ExtStorage and ArchipGaneti.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Up until now components were simple objects decoupled from fabric
execution. Their methods were just returning a list of bash commands
which utils module was executing in the correct context.

This patch changes this rational. Specifically:

A Component() gets initialized with an execution context that is a
configuration snapshot of the target setup, cluster and node. A
component implements the following helper methods: check, install,
prepare, configure, restart, initialize, and test. All those methods
will be executed on the target node with this order during setup.

Additionally each Component class implements admin_pre, and
admin_post methods which invoke actions on different components on
the same execution context before and after installation. For
example before a backend gets installed, its FQDN must resolve to
the master floating IP, so we have to run some actions on the ns
node and after installation we must add it to cyclades (snf-manage
backend-add on the cyclades node).

Component() inherits ComponentRunner() which practically exports the
setup() method. This will first check if the required components are
installed, will install them if not and update the status of target
node.

ComponentRunner() inherits FabricRunner() which practically wraps
basic fabric commands (put, get, run) with the correct execution
environment.

The fabfile practically gets the target nodes for each role from the
configuration and uses the roles module which knows the
role-component mapping to get the target component with the proper
execution context. Then it just invokes the setup() method.

Each component gets initialized with an execution context and uses
the config module for accessing global wide options. The context
provides node, cluster, and setup related info.

We introduce some helper decorators that wrap methods of Component
class and update its execution context (self, self.ctx):

- parse_* executes the wrapped method and parses the output.
  The desired vars (user_id, backend_id, user_uuid, etc.) are
  stored in the execution context and made available to other
  methods.

- update_admin initializes the admin roles (NS, Astakos,
  Cyclades, etc.) of the current context and make them available
  under self.NS, self.ASTAKOS, etc. These have the same execution
  context of the current components besides the target node
  which gets derived from the corresponding config.

- update_cluster_admin initializes the cluster's admin role
  and make it available under self.MASTER.

- run_cmds runs the list of returned commands in the target node

The update_ns() method of NS component gets the
self.ctx.admin_fqdn and invokes nsupdate accordingly.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

This is used to create a development environment for Ganeti.
It clones the official repo, installs all packages needed,
and adds a qa-sample.json file that reflects to the
Ganeti cluster found in the configuration.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

To install a minimal Ganeti cluster for QA run:

 snf-deploy ganeti-qa

This will create a ganeti cluster. The test instances will resolve on
the local nameserver. On the client node it clones the official repo
and installs all required packages required for ganeti development.
A sample qa json file is uploaded under /tmp.

To run the qa go to the client node and run:

 # cd ganeti
 # python ganeti-qa.py /tmp/qa-sample.json --yes-do-it

Note that the installed version in the qa cluster should be the same
used in the qa source.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Ganeti needs a VG in order to support plain/drbd disk templates.
In case an extra disk exists, deploy will create a PV and a VG
on this disk.

Use options `extra_disk` in nodes.conf and `vg` in ganeti.conf for
this purpose.

VMs spawned currently during ci do not have extra disks. Update the
corresponding schema.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

The components PithosBackend, Image, GTools, and GanetiCollectd
are related to Ganeti but used only by Synnefo.

Define new role, named 'snf-ganeti' to include those components and use
it with the `snf-deploy ganeti` command.

Install ganeti-instance-debootstrap so that we have a working os
provider.

snf-image takes a lot of time to be configured (curl,
snf-image-update-helper, wget debian-base image) and is not needed
for a simple Ganeti cluster (i.e. QA cluster).

Move wget of debian base image to Image component and not nfs.
In order not to download many times check if it already exists
in the proper path.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

..to actually mount the dirs exported by nfs node and not pithos.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

According to the decision of the GRNET Board of Directors,
switch license to GPLv3.

This commit will be propagated to the release
and master branches based on git flow, and the next
release will be licensed as GPLv3.

..and use 2G for ci.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Since archipelago-ganeti installs the provider under
/usr/share/ganeti/extstorage we can/should not clone our repo in the
same dir. Thus we clone it under /usr/local/lib/ganeti/ which is
also in ExtStorage search path.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Clone extstorage repo to /usr/share/ganeti/extstorage.
and add ext_vlmc flavor_storage.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Currently cyclades and pithos require a specific gunicorn setup
with GeventArchipelagoWorker. Plus they require a config
file that is included in snf-pithos-app; pithos.conf.py

This file must be installed both in pithos and cyclades. Therefore
we have it as a template and install it under /etc/archipelago.

Add archipelago.conf template in order to customize SEGMENT_SIZE.

For the above responsible is Archip component.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

..since adding it to setup.py results to:

pkg_resources.DistributionNotFound: svipc

Note that `pip search svipc` does not return any python module.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Extend snf-deploy to support installation and configuration of
Archipelago.

Archipelago related packages are found in apt.dev unstable an in
ceph's repo, which are added to the source.list.

A new nfs mount point is created /srv/archip and two new components,
Archip and ArchipGaneti are added to be included in pithos, cyclades,
ganeti and master roles.

Since we are inserting ssh keys, we first backup existing
authorized_keys and append them later to our uploaded file.

If authorized_keys did not exist, neither will authorized_keys.bak .
So `test -e` in the initialize method of SSH component will
fail and fabric will abort. Here we fix this case.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Update Copyright for files merged into develop in 2014 concerning the new
projects mechanism.

Add support in resource-modify for changing the project_default.

Remove interactive operations, rename default-quota to base-default.

Add a management command that directly modifies an initialized project.

Enable eventd during prepare() so that after restarting it, init
script will not fail.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

CYCLADES_NODE_IP is needed for customizing cyclades.conf
template for UI_MEDIA_URL.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

This is necessary so that future migrations know where to start upgrading.

/var/lib/vncauthproxy dir is created by package itself.
Thus mkdir -p to avoid error.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Add users files.

TODO: use vncauthproxy-passwd with correct conf settings
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

..which broke after applying pep8 fixes.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Those fixes concern errors that are introduced in latest
pep8 version (and not the wheezy's one).
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Introduce the concept of Roles and Components.

Roles are ns, mq, db, nfs, astakos, cyclades, pithos, cms, stats,
ganeti, master, and client.

Each role consists of various SynnefoComponents which  should define:

* commands to check installation prerequisites (check)
* packages to install (REQUIRED_PACKAGES)
* commands to prepare installation (prepare)
* configuration templates (configure)
* reload commands (restart)
* initialization commands (initialize)
* test commands (test)

SynnefoComponents are unaware of fabric environment. They
get initialized with a Host object and a Env object.

Host includes all the necessary info for the installation node
(ip, hostname, alias, fqdn..) and Env includes all the configuration info
(created after parsing config files)

After defining components, fabfile should just do the following:

@roles("somerole")
def setup_somerole_role():
  SetupSynnefoRole("SomeRole")

In case you want to run a component's specific method just run

RunComponentMethod(SomeComponent, "method_name", *args, **kwargs)

During a role setup you might have to retrieve info from other
components already installed (user token, backend id, etc.) Use
'execute' fabric method and fill env object with required info.

Make deployment re-entrant.

Check if specific component is already installed and if yes then skip it.

Currently this is done on node/component level. The component's
status on a target node is either ok or nothing.

Introduce conflicting components.

Two components might not be able/or should not coexist in the same
node, e.g. NFS and Mount. These conflicts are defined in CONFLICTS
dict in roles.py.

Sync ci and deploy conf files.

ci: Change node's password in nodes.conf

Always enable CSRF. Install CMS only if it resides on different
node than pithos, cyclades and astakos.

Add copyright headers.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>