1. 02 May, 2014 17 commits
    • Dimitris Aragiorgis's avatar
      deploy: Refactor fabfile · ac8113ce
      Dimitris Aragiorgis authored
      The fabfile is now responsible to find the target nodes for each
      execution context. It practically gets the corresponding component
      for each role and invokes its setup() method.
      To get the execution context for each action we use fabric's execute()
      and update the context based on env.host.
      Some things can be done concurrently (setup of each VMC node).
      To do so, we use the fabric's @parallel decorator.
      Additionally we support setup of a single component/role on a target
      node or even the execution of a single method of a component.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy: Refactor components module · 3699d1fe
      Dimitris Aragiorgis authored
      Up until now components were simple objects decoupled from fabric
      execution. Their methods were just returning a list of bash commands
      which utils module was executing in the correct context.
      This patch changes this rational. Specifically:
      A Component() gets initialized with an execution context that is a
      configuration snapshot of the target setup, cluster and node. A
      component implements the following helper methods: check, install,
      prepare, configure, restart, initialize, and test. All those methods
      will be executed on the target node with this order during setup.
      Additionally each Component class implements admin_pre, and
      admin_post methods which invoke actions on different components on
      the same execution context before and after installation. For
      example before a backend gets installed, its FQDN must resolve to
      the master floating IP, so we have to run some actions on the ns
      node and after installation we must add it to cyclades (snf-manage
      backend-add on the cyclades node).
      Component() inherits ComponentRunner() which practically exports the
      setup() method. This will first check if the required components are
      installed, will install them if not and update the status of target
      ComponentRunner() inherits FabricRunner() which practically wraps
      basic fabric commands (put, get, run) with the correct execution
      The fabfile practically gets the target nodes for each role from the
      configuration and uses the roles module which knows the
      role-component mapping to get the target component with the proper
      execution context. Then it just invokes the setup() method.
      Each component gets initialized with an execution context and uses
      the config module for accessing global wide options. The context
      provides node, cluster, and setup related info.
      We introduce some helper decorators that wrap methods of Component
      class and update its execution context (self, self.ctx):
      - parse_* executes the wrapped method and parses the output.
        The desired vars (user_id, backend_id, user_uuid, etc.) are
        stored in the execution context and made available to other
      - update_admin initializes the admin roles (NS, Astakos,
        Cyclades, etc.) of the current context and make them available
        under self.NS, self.ASTAKOS, etc. These have the same execution
        context of the current components besides the target node
        which gets derived from the corresponding config.
      - update_cluster_admin initializes the cluster's admin role
        and make it available under self.MASTER.
      - run_cmds runs the list of returned commands in the target node
      The update_ns() method of NS component gets the
      self.ctx.admin_fqdn and invokes nsupdate accordingly.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
    • Dimitris Aragiorgis's avatar
      deploy: Make snf-deploy script use new modules · f4f1dfa9
      Dimitris Aragiorgis authored
      Use config, context, status and vcluster modules.
      The main method of snf-deploy script will initialize them with
      the args passed. Additionally make some minor changes related
      to the command line:
       - snf-deploy all -> snf-deploy synnefo.
       - add --setup option
       - add --cluster option
       - use --<name>-dir for template, state, package dirs
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy: Cleanup lib module · 2ce89484
      Dimitris Aragiorgis authored
      Remove unused Env, Status, Conf classes. Host and Alias classes have
      been substituted by FQDN. The latter will be used for Node and
      Cluster info.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy: Add vcluster module · d783fd6e
      Dimitris Aragiorgis authored
      ..with the relevant helper methods for creating a local
      virtual cluster with KVM. snf-deploy base script will import
      this module and use it if vcluster command is requested.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy: Add roles module · 4ba6e61d
      Dimitris Aragiorgis authored
      This is used to get the target component for each node.
      It has a role-component mapping and should be used like:
      NS = roles.get(constants.NS, ctx)
      where the ctx is a Context() with proper node, role, cluster, setup.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy: Add context module · 2d0a73fe
      Dimitris Aragiorgis authored
      This module will be used as a common execution context
      just like fabric's env. A context is basically the node, role,
      cluster, and setup that a components gets installed. The new
      Context class is a configuration snapshot for the given context.
      It exports helper methods to get the node's and cluster's info, etc.
      This class is used to initialize the target component class.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy: Add new base module · 41376567
      Dimitris Aragiorgis authored
      This defines the base classes 'FabricRunner', 'ComponentRunner' and
      'Component'. These classes will be used by the classes defined in
      the components module.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy: Add status module · 9689b718
      Dimitris Aragiorgis authored
      This is used to keep track of the deployment status. The status file
      is kept under the 'state' dir. The check() method is used to find
      conflicting components or the ones that have already been installed.
      The update() method adds an entry in the node's section. If in dry-run
      this file is not written back to permanent storage (kept in memory).
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy: Add config module · f6532cfc
      Dimitris Aragiorgis authored
      This module reads the configuration and includes
      several methods for extracting info. It gets
      initialized with the command line arguments.
      It reads the DEFAULT section of synnefo, vcluster, and deploy conf
      files and evaluates their options in a sense that once it gets
      imported, one has easy access to generic configuration variables,
      e.g. config.synnefo_user.
      On the other hand the setups, nodes, ganeti and packages conf files
      are proper ConfigParser files with sections and default values.
      nodes: All info needed for each node.
      ganeti: All info needed for each cluster.
      setups: All role-nodes mappings for each setup and each cluster.
      packages: All special package versions for each OS.
      The get_info() helper method gets a node or cluster identifier
      (e.g. node2, ganeti1) and returns a FQDN object with all related
      options as attributes.
      The get_package() helper method is used to get the special version
      (if any) of a package.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy: Add constants module · 5cfb352a
      Dimitris Aragiorgis authored
      Use constants to refer to roles defined in conf files.
      Add constants related to snf-deploy state.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
    • Dimitris Aragiorgis's avatar
      deploy: Add new component GanetiDev · e330e80a
      Dimitris Aragiorgis authored
      This is used to create a development environment for Ganeti.
      It clones the official repo, installs all packages needed,
      and adds a qa-sample.json file that reflects to the
      Ganeti cluster found in the configuration.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy: Add ganeti-qa command · 9093eb07
      Dimitris Aragiorgis authored
      To install a minimal Ganeti cluster for QA run:
       snf-deploy ganeti-qa
      This will create a ganeti cluster. The test instances will resolve on
      the local nameserver. On the client node it clones the official repo
      and installs all required packages required for ganeti development.
      A sample qa json file is uploaded under /tmp.
      To run the qa go to the client node and run:
       # cd ganeti
       # python ganeti-qa.py /tmp/qa-sample.json --yes-do-it
      Note that the installed version in the qa cluster should be the same
      used in the qa source.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy/ci: Setup lvm in case extra disk exists · 79707d0e
      Dimitris Aragiorgis authored
      Ganeti needs a VG in order to support plain/drbd disk templates.
      In case an extra disk exists, deploy will create a PV and a VG
      on this disk.
      Use options `extra_disk` in nodes.conf and `vg` in ganeti.conf for
      this purpose.
      VMs spawned currently during ci do not have extra disks. Update the
      corresponding schema.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
    • Dimitris Aragiorgis's avatar
      deploy: Extract synnefo's components from ganeti · de81d118
      Dimitris Aragiorgis authored
      The components PithosBackend, Image, GTools, and GanetiCollectd
      are related to Ganeti but used only by Synnefo.
      Define new role, named 'snf-ganeti' to include those components and use
      it with the `snf-deploy ganeti` command.
      Install ganeti-instance-debootstrap so that we have a working os
      snf-image takes a lot of time to be configured (curl,
      snf-image-update-helper, wget debian-base image) and is not needed
      for a simple Ganeti cluster (i.e. QA cluster).
      Move wget of debian base image to Image component and not nfs.
      In order not to download many times check if it already exists
      in the proper path.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
  2. 28 Apr, 2014 4 commits
  3. 17 Apr, 2014 5 commits
  4. 14 Apr, 2014 3 commits
    • Vangelis Koukis's avatar
      Switch license to GPLv3 · 02071b96
      Vangelis Koukis authored
      According to the decision of the GRNET Board of Directors,
      switch license to GPLv3.
      This commit will be propagated to the release
      and master branches based on git flow, and the next
      release will be licensed as GPLv3.
    • Kostas Papadimitriou's avatar
      cyclades: Fix typo · ff4c4658
      Kostas Papadimitriou authored
    • Christos Stavrakakis's avatar
      cyclades: Create volumes initialized from a source · b140a83e
      Christos Stavrakakis authored
      Extend the 'volume-create' management command with the '--source' option
      which can be used to create a volume which is initialized with data from
      an image, snapshot or volume. Currently, this option is useful only
      for volumes that have been created with the 'ext_archipelago' disk template.
      Closes: #45
  5. 07 Apr, 2014 2 commits
    • Giorgos Korfiatis's avatar
      snf_django: Fix string formatting · 654ef946
      Giorgos Korfiatis authored
      Format arguments were applied to the last part of the string which contained
      no % format operators. Concat strings before applying format.
    • Ilias Tsitsimpis's avatar
      Merge pull request #28 from cstavr/feature-unicode-handling · 8d477a6d
      Ilias Tsitsimpis authored
      This patch-set fixes a number of issues relative with handling unicode strings.
      These commits try to avoid mixing bytestrings and unicode objects, by decoding
      bytestrings as early as possible (Input) and encoding unicodes as late as
      possible (Ouput). In the API the encoding that is used is always 'UTF-8'. In
      the CLI the encoding that is used is the user's preferred encoding.
      Besides unicode issues, this patch-set fixes handling of image metadata
      (Plankton) which are views as HTTP headers. Since, image metadata must be valid
      HTTP headers, metadata keys and values which contain user defined values must
      be properly quoted and unquoted where needed.
  6. 03 Apr, 2014 1 commit
    • Chrysostomos Nanakos's avatar
      snf-deploy: Disable direct I/O in Archipelago · 98838c30
      Chrysostomos Nanakos authored
      When writing Pithos+ mapfiles with direct I/O enabled Archipelago takes
      care to use properly aligned and sized I/O based on the basic block unit,
      which is 512 bytes. The result is to pad the file with zeros and finally
      produce the wrong mapfile.
  7. 31 Mar, 2014 8 commits