1. 02 May, 2014 19 commits
    • Dimitris Aragiorgis's avatar
      deploy: Install blktap in Achip component · 62bdc889
      Dimitris Aragiorgis authored
      
      
      Currently latest version of archipelago uses blktap instead of
      xseg. Install official blktap-dkms and our custom blktap-utils
      for this purpose.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      62bdc889
    • Dimitris Aragiorgis's avatar
      deploy: Use cname instead of fqdn · 3a9f2ac6
      Dimitris Aragiorgis authored
      
      
      ..for astakos, cyclades, pithos, cms, db, mq, nfs.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      3a9f2ac6
    • Dimitris Aragiorgis's avatar
      deploy: Use archipelago related components · 523cb739
      Dimitris Aragiorgis authored
      
      
      Make Cyclades and Pithos require PithosBackend and Archip.
      Make VMC require also ExtStorage and ArchipGaneti.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      523cb739
    • Dimitris Aragiorgis's avatar
      deploy: Refactor fabfile · ac8113ce
      Dimitris Aragiorgis authored
      
      
      The fabfile is now responsible to find the target nodes for each
      execution context. It practically gets the corresponding component
      for each role and invokes its setup() method.
      
      To get the execution context for each action we use fabric's execute()
      and update the context based on env.host.
      
      Some things can be done concurrently (setup of each VMC node).
      To do so, we use the fabric's @parallel decorator.
      
      Additionally we support setup of a single component/role on a target
      node or even the execution of a single method of a component.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      ac8113ce
    • Dimitris Aragiorgis's avatar
      deploy: Refactor components module · 3699d1fe
      Dimitris Aragiorgis authored
      
      
      Up until now components were simple objects decoupled from fabric
      execution. Their methods were just returning a list of bash commands
      which utils module was executing in the correct context.
      
      This patch changes this rational. Specifically:
      
      A Component() gets initialized with an execution context that is a
      configuration snapshot of the target setup, cluster and node. A
      component implements the following helper methods: check, install,
      prepare, configure, restart, initialize, and test. All those methods
      will be executed on the target node with this order during setup.
      
      Additionally each Component class implements admin_pre, and
      admin_post methods which invoke actions on different components on
      the same execution context before and after installation. For
      example before a backend gets installed, its FQDN must resolve to
      the master floating IP, so we have to run some actions on the ns
      node and after installation we must add it to cyclades (snf-manage
      backend-add on the cyclades node).
      
      Component() inherits ComponentRunner() which practically exports the
      setup() method. This will first check if the required components are
      installed, will install them if not and update the status of target
      node.
      
      ComponentRunner() inherits FabricRunner() which practically wraps
      basic fabric commands (put, get, run) with the correct execution
      environment.
      
      The fabfile practically gets the target nodes for each role from the
      configuration and uses the roles module which knows the
      role-component mapping to get the target component with the proper
      execution context. Then it just invokes the setup() method.
      
      Each component gets initialized with an execution context and uses
      the config module for accessing global wide options. The context
      provides node, cluster, and setup related info.
      
      We introduce some helper decorators that wrap methods of Component
      class and update its execution context (self, self.ctx):
      
      - parse_* executes the wrapped method and parses the output.
        The desired vars (user_id, backend_id, user_uuid, etc.) are
        stored in the execution context and made available to other
        methods.
      
      - update_admin initializes the admin roles (NS, Astakos,
        Cyclades, etc.) of the current context and make them available
        under self.NS, self.ASTAKOS, etc. These have the same execution
        context of the current components besides the target node
        which gets derived from the corresponding config.
      
      - update_cluster_admin initializes the cluster's admin role
        and make it available under self.MASTER.
      
      - run_cmds runs the list of returned commands in the target node
      
      The update_ns() method of NS component gets the
      self.ctx.admin_fqdn and invokes nsupdate accordingly.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      3699d1fe
    • Dimitris Aragiorgis's avatar
      a43cc462
    • Dimitris Aragiorgis's avatar
      deploy: Make snf-deploy script use new modules · f4f1dfa9
      Dimitris Aragiorgis authored
      
      
      Use config, context, status and vcluster modules.
      
      The main method of snf-deploy script will initialize them with
      the args passed. Additionally make some minor changes related
      to the command line:
      
       - snf-deploy all -> snf-deploy synnefo.
       - add --setup option
       - add --cluster option
       - use --<name>-dir for template, state, package dirs
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      f4f1dfa9
    • Dimitris Aragiorgis's avatar
      deploy: Cleanup lib module · 2ce89484
      Dimitris Aragiorgis authored
      
      
      Remove unused Env, Status, Conf classes. Host and Alias classes have
      been substituted by FQDN. The latter will be used for Node and
      Cluster info.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      2ce89484
    • Dimitris Aragiorgis's avatar
      deploy: Add vcluster module · d783fd6e
      Dimitris Aragiorgis authored
      
      
      ..with the relevant helper methods for creating a local
      virtual cluster with KVM. snf-deploy base script will import
      this module and use it if vcluster command is requested.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      d783fd6e
    • Dimitris Aragiorgis's avatar
      deploy: Add roles module · 4ba6e61d
      Dimitris Aragiorgis authored
      
      
      This is used to get the target component for each node.
      It has a role-component mapping and should be used like:
      
      NS = roles.get(constants.NS, ctx)
      
      where the ctx is a Context() with proper node, role, cluster, setup.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      4ba6e61d
    • Dimitris Aragiorgis's avatar
      deploy: Add context module · 2d0a73fe
      Dimitris Aragiorgis authored
      
      
      This module will be used as a common execution context
      just like fabric's env. A context is basically the node, role,
      cluster, and setup that a components gets installed. The new
      Context class is a configuration snapshot for the given context.
      It exports helper methods to get the node's and cluster's info, etc.
      This class is used to initialize the target component class.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      2d0a73fe
    • Dimitris Aragiorgis's avatar
      deploy: Add new base module · 41376567
      Dimitris Aragiorgis authored
      
      
      This defines the base classes 'FabricRunner', 'ComponentRunner' and
      'Component'. These classes will be used by the classes defined in
      the components module.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      41376567
    • Dimitris Aragiorgis's avatar
      deploy: Add status module · 9689b718
      Dimitris Aragiorgis authored
      
      
      This is used to keep track of the deployment status. The status file
      is kept under the 'state' dir. The check() method is used to find
      conflicting components or the ones that have already been installed.
      The update() method adds an entry in the node's section. If in dry-run
      this file is not written back to permanent storage (kept in memory).
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      9689b718
    • Dimitris Aragiorgis's avatar
      deploy: Add config module · f6532cfc
      Dimitris Aragiorgis authored
      
      
      This module reads the configuration and includes
      several methods for extracting info. It gets
      initialized with the command line arguments.
      
      It reads the DEFAULT section of synnefo, vcluster, and deploy conf
      files and evaluates their options in a sense that once it gets
      imported, one has easy access to generic configuration variables,
      e.g. config.synnefo_user.
      
      On the other hand the setups, nodes, ganeti and packages conf files
      are proper ConfigParser files with sections and default values.
      
      nodes: All info needed for each node.
      ganeti: All info needed for each cluster.
      setups: All role-nodes mappings for each setup and each cluster.
      packages: All special package versions for each OS.
      
      The get_info() helper method gets a node or cluster identifier
      (e.g. node2, ganeti1) and returns a FQDN object with all related
      options as attributes.
      
      The get_package() helper method is used to get the special version
      (if any) of a package.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      f6532cfc
    • Dimitris Aragiorgis's avatar
      deploy: Add constants module · 5cfb352a
      Dimitris Aragiorgis authored
      
      
      Use constants to refer to roles defined in conf files.
      Add constants related to snf-deploy state.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      5cfb352a
    • Dimitris Aragiorgis's avatar
      deploy: Add new component GanetiDev · e330e80a
      Dimitris Aragiorgis authored
      
      
      This is used to create a development environment for Ganeti.
      It clones the official repo, installs all packages needed,
      and adds a qa-sample.json file that reflects to the
      Ganeti cluster found in the configuration.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      e330e80a
    • Dimitris Aragiorgis's avatar
      deploy: Add ganeti-qa command · 9093eb07
      Dimitris Aragiorgis authored
      
      
      To install a minimal Ganeti cluster for QA run:
      
       snf-deploy ganeti-qa
      
      This will create a ganeti cluster. The test instances will resolve on
      the local nameserver. On the client node it clones the official repo
      and installs all required packages required for ganeti development.
      A sample qa json file is uploaded under /tmp.
      
      To run the qa go to the client node and run:
      
       # cd ganeti
       # python ganeti-qa.py /tmp/qa-sample.json --yes-do-it
      
      Note that the installed version in the qa cluster should be the same
      used in the qa source.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      9093eb07
    • Dimitris Aragiorgis's avatar
      deploy/ci: Setup lvm in case extra disk exists · 79707d0e
      Dimitris Aragiorgis authored
      
      
      Ganeti needs a VG in order to support plain/drbd disk templates.
      In case an extra disk exists, deploy will create a PV and a VG
      on this disk.
      
      Use options `extra_disk` in nodes.conf and `vg` in ganeti.conf for
      this purpose.
      
      VMs spawned currently during ci do not have extra disks. Update the
      corresponding schema.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      79707d0e
    • Dimitris Aragiorgis's avatar
      deploy: Extract synnefo's components from ganeti · de81d118
      Dimitris Aragiorgis authored
      
      
      The components PithosBackend, Image, GTools, and GanetiCollectd
      are related to Ganeti but used only by Synnefo.
      
      Define new role, named 'snf-ganeti' to include those components and use
      it with the `snf-deploy ganeti` command.
      
      Install ganeti-instance-debootstrap so that we have a working os
      provider.
      
      snf-image takes a lot of time to be configured (curl,
      snf-image-update-helper, wget debian-base image) and is not needed
      for a simple Ganeti cluster (i.e. QA cluster).
      
      Move wget of debian base image to Image component and not nfs.
      In order not to download many times check if it already exists
      in the proper path.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      de81d118
  2. 28 Apr, 2014 3 commits
  3. 14 Apr, 2014 1 commit
    • Vangelis Koukis's avatar
      Switch license to GPLv3 · 02071b96
      Vangelis Koukis authored
      According to the decision of the GRNET Board of Directors,
      switch license to GPLv3.
      
      This commit will be propagated to the release
      and master branches based on git flow, and the next
      release will be licensed as GPLv3.
      02071b96
  4. 31 Mar, 2014 7 commits
  5. 12 Mar, 2014 4 commits
    • Dimitris Aragiorgis's avatar
      deploy: Remove old fabfile · 6f9c1490
      Dimitris Aragiorgis authored
      
      
      ..and replace it with the `fabfile2.py`.
      
      Update Copyright dates.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      6f9c1490
    • Dimitris Aragiorgis's avatar
      deploy/ci: Small refactor regarding ssh/ddns keys · f1efbb09
      Dimitris Aragiorgis authored
      
      
      In case `keygen` founds the keys it does nothing. If `--force` is
      passed then it re-creates the keys.
      
      Currently ci installs the deploy package during `build` command.
      Move keygen action from ci's `deploy_synnefo` phase to the
      `build_synnefo` phase.
      
      Do not use the `--force` flag so that deploy can be re-entrant on
      any level.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      f1efbb09
    • Dimitris Aragiorgis's avatar
      deploy: Add state dir and use it for status · 19fa4790
      Dimitris Aragiorgis authored
      
      
      Introduce new setting in [dirs] section of deploy.conf: `state`.
      Override this setting with `--state-dir` option (useful when running
      deploy from source).
      
      This dir is currently used to store snf-deploy's `snf_deploy_status`
      file, that shows which components on which nodes have been
      installed. Do not update this file if `--dry-run` is used.
      
      Additionally introduce `--templates-dir` option that overrides
      `template` setting in [dirs] section of `deploy.conf`.
      
      Note that override options do not modify the configuration files.
      Changes are performed in the execution context of each snf-deploy
      command.
      
      To run snf-deploy from source use:
      
      $ python setup.py develop
      $ snf-deploy keygen -c conf -t files -s /tmp
      $ snf-deploy all --autoconf -c conf -t files -s /tmp
      
      Add the above setting to snf-ci's schema files.
      
      Update Copyright dates.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      19fa4790
    • Dimitris Aragiorgis's avatar
      deploy: Create the ddns and .ssh dirs · e329c2be
      Dimitris Aragiorgis authored
      
      
      These dirs are needed to store the ddns and .ssh files created by
      snf-deploy keygen. All these files should reside in the template dir
      since are going to be moved to target nodes.
      
      Add those dirs in .gitignore since we don't want to track any temp
      files created if we run snf-deploy from source.
      
      Update Copyright dates.
      Signed-off-by: default avatarDimitris Aragiorgis <dimara@grnet.gr>
      e329c2be
  6. 28 Feb, 2014 2 commits
  7. 13 Feb, 2014 3 commits
  8. 03 Feb, 2014 1 commit