GitLab

Make Cyclades and Pithos require PithosBackend and Archip.
Make VMC require also ExtStorage and ArchipGaneti.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

The fabfile is now responsible to find the target nodes for each
execution context. It practically gets the corresponding component
for each role and invokes its setup() method.

To get the execution context for each action we use fabric's execute()
and update the context based on env.host.

Some things can be done concurrently (setup of each VMC node).
To do so, we use the fabric's @parallel decorator.

Additionally we support setup of a single component/role on a target
node or even the execution of a single method of a component.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Up until now components were simple objects decoupled from fabric
execution. Their methods were just returning a list of bash commands
which utils module was executing in the correct context.

This patch changes this rational. Specifically:

A Component() gets initialized with an execution context that is a
configuration snapshot of the target setup, cluster and node. A
component implements the following helper methods: check, install,
prepare, configure, restart, initialize, and test. All those methods
will be executed on the target node with this order during setup.

Additionally each Component class implements admin_pre, and
admin_post methods which invoke actions on different components on
the same execution context before and after installation. For
example before a backend gets installed, its FQDN must resolve to
the master floating IP, so we have to run some actions on the ns
node and after installation we must add it to cyclades (snf-manage
backend-add on the cyclades node).

Component() inherits ComponentRunner() which practically exports the
setup() method. This will first check if the required components are
installed, will install them if not and update the status of target
node.

ComponentRunner() inherits FabricRunner() which practically wraps
basic fabric commands (put, get, run) with the correct execution
environment.

The fabfile practically gets the target nodes for each role from the
configuration and uses the roles module which knows the
role-component mapping to get the target component with the proper
execution context. Then it just invokes the setup() method.

Each component gets initialized with an execution context and uses
the config module for accessing global wide options. The context
provides node, cluster, and setup related info.

We introduce some helper decorators that wrap methods of Component
class and update its execution context (self, self.ctx):

- parse_* executes the wrapped method and parses the output.
  The desired vars (user_id, backend_id, user_uuid, etc.) are
  stored in the execution context and made available to other
  methods.

- update_admin initializes the admin roles (NS, Astakos,
  Cyclades, etc.) of the current context and make them available
  under self.NS, self.ASTAKOS, etc. These have the same execution
  context of the current components besides the target node
  which gets derived from the corresponding config.

- update_cluster_admin initializes the cluster's admin role
  and make it available under self.MASTER.

- run_cmds runs the list of returned commands in the target node

The update_ns() method of NS component gets the
self.ctx.admin_fqdn and invokes nsupdate accordingly.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Use config, context, status and vcluster modules.

The main method of snf-deploy script will initialize them with
the args passed. Additionally make some minor changes related
to the command line:

 - snf-deploy all -> snf-deploy synnefo.
 - add --setup option
 - add --cluster option
 - use --<name>-dir for template, state, package dirs
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Remove unused Env, Status, Conf classes. Host and Alias classes have
been substituted by FQDN. The latter will be used for Node and
Cluster info.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

..with the relevant helper methods for creating a local
virtual cluster with KVM. snf-deploy base script will import
this module and use it if vcluster command is requested.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

This is used to get the target component for each node.
It has a role-component mapping and should be used like:

NS = roles.get(constants.NS, ctx)

where the ctx is a Context() with proper node, role, cluster, setup.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

This module will be used as a common execution context
just like fabric's env. A context is basically the node, role,
cluster, and setup that a components gets installed. The new
Context class is a configuration snapshot for the given context.
It exports helper methods to get the node's and cluster's info, etc.
This class is used to initialize the target component class.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

This defines the base classes 'FabricRunner', 'ComponentRunner' and
'Component'. These classes will be used by the classes defined in
the components module.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

This is used to keep track of the deployment status. The status file
is kept under the 'state' dir. The check() method is used to find
conflicting components or the ones that have already been installed.
The update() method adds an entry in the node's section. If in dry-run
this file is not written back to permanent storage (kept in memory).
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

This module reads the configuration and includes
several methods for extracting info. It gets
initialized with the command line arguments.

It reads the DEFAULT section of synnefo, vcluster, and deploy conf
files and evaluates their options in a sense that once it gets
imported, one has easy access to generic configuration variables,
e.g. config.synnefo_user.

On the other hand the setups, nodes, ganeti and packages conf files
are proper ConfigParser files with sections and default values.

nodes: All info needed for each node.
ganeti: All info needed for each cluster.
setups: All role-nodes mappings for each setup and each cluster.
packages: All special package versions for each OS.

The get_info() helper method gets a node or cluster identifier
(e.g. node2, ganeti1) and returns a FQDN object with all related
options as attributes.

The get_package() helper method is used to get the special version
(if any) of a package.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Use constants to refer to roles defined in conf files.
Add constants related to snf-deploy state.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

This is used to create a development environment for Ganeti.
It clones the official repo, installs all packages needed,
and adds a qa-sample.json file that reflects to the
Ganeti cluster found in the configuration.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

To install a minimal Ganeti cluster for QA run:

 snf-deploy ganeti-qa

This will create a ganeti cluster. The test instances will resolve on
the local nameserver. On the client node it clones the official repo
and installs all required packages required for ganeti development.
A sample qa json file is uploaded under /tmp.

To run the qa go to the client node and run:

 # cd ganeti
 # python ganeti-qa.py /tmp/qa-sample.json --yes-do-it

Note that the installed version in the qa cluster should be the same
used in the qa source.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Ganeti needs a VG in order to support plain/drbd disk templates.
In case an extra disk exists, deploy will create a PV and a VG
on this disk.

Use options `extra_disk` in nodes.conf and `vg` in ganeti.conf for
this purpose.

VMs spawned currently during ci do not have extra disks. Update the
corresponding schema.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

The components PithosBackend, Image, GTools, and GanetiCollectd
are related to Ganeti but used only by Synnefo.

Define new role, named 'snf-ganeti' to include those components and use
it with the `snf-deploy ganeti` command.

Install ganeti-instance-debootstrap so that we have a working os
provider.

snf-image takes a lot of time to be configured (curl,
snf-image-update-helper, wget debian-base image) and is not needed
for a simple Ganeti cluster (i.e. QA cluster).

Move wget of debian base image to Image component and not nfs.
In order not to download many times check if it already exists
in the proper path.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

..from choices of command positional argument and thus
support: `snf-deploy ganeti`.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

..to actually mount the dirs exported by nfs node and not pithos.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

..to include constants introduced in 0.16.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Use nosetests to test astakosclient. If coverage is present in the
system, use coverage in cojunction with nosetests.

Also, fix reports generated by coverage as well as other coverage
issues.

Update the url for the Synnefo repo and add a flavor with more RAM as
default.

Choose from the user list of an Image the first user as the admin user.

Allow to test only an app of a component, if the input has the following
form:

    component.app

e.g. astakos.im

According to the decision of the GRNET Board of Directors,
switch license to GPLv3.

This commit will be propagated to the release
and master branches based on git flow, and the next
release will be licensed as GPLv3.

Extend the 'volume-create' management command with the '--source' option
which can be used to create a volume which is initialized with data from
an image, snapshot or volume. Currently, this option is useful only
for volumes that have been created with the 'ext_archipelago' disk template.

Closes: #45

Format arguments were applied to the last part of the string which contained
no % format operators. Concat strings before applying format.

This patch-set fixes a number of issues relative with handling unicode strings.
These commits try to avoid mixing bytestrings and unicode objects, by decoding
bytestrings as early as possible (Input) and encoding unicodes as late as
possible (Ouput). In the API the encoding that is used is always 'UTF-8'. In
the CLI the encoding that is used is the user's preferred encoding.

Besides unicode issues, this patch-set fixes handling of image metadata
(Plankton) which are views as HTTP headers. Since, image metadata must be valid
HTTP headers, metadata keys and values which contain user defined values must
be properly quoted and unquoted where needed.

When writing Pithos+ mapfiles with direct I/O enabled Archipelago takes
care to use properly aligned and sized I/O based on the basic block unit,
which is 512 bytes. The result is to pad the file with zeros and finally
produce the wrong mapfile.

Use the 'get_json_body' function, which takes care of encoding errors.
Also, check that the POST body is a dictionary, where needed.

Define a custom CommandError, which applies smart_locale_str() in its
__str__ method. Use this custom CommandError in Astakos management
commands.

Use recipe from http://docs.python.org/2/library/csv.html to write unicode
in csv format.