1. 20 Dec, 2012 2 commits
    • Michele Tartara's avatar
      Add Confd client to the Haskell code base · 04063ba7
      Michele Tartara authored
      The client queries all the master candidates in parallel, until the minimum
      number of replies, defined in the constant file, is received.
      A timeout prevents the waiting from being of indefinite length.
      The reply to be returned to the function that made the query is decided
      according to the Confd design document.
      Signed-off-by: default avatarMichele Tartara <mtartara@google.com>
      Reviewed-by: default avatarIustin Pop <iustin@google.com>
    • Constantinos Venetsanopoulos's avatar
      Implement the External Storage Interface · 376631d1
      Constantinos Venetsanopoulos authored
      With this commit we introduce the External Storage Interface
      to Ganeti, abbreviated: ExtStorage Interface.
      The ExtStorage Interface provides Ganeti with the ability to interact
      with externally connected shared storage pools, visible by all
      VM-capable nodes. This means that Ganeti is able to handle VM disks
      that reside inside a NAS/SAN or any distributed block storage provider.
      The ExtStorage Interface provides a clear API, heavily inspired by the
      gnt-os-interface API, that can be used by storage vendors or sysadmins
      to write simple ExtStorage Providers (correlated to gnt-os-interface's
      OS Definitions). Those Providers will glue externally attached shared
      storage with Ganeti, without the need of preprovisioned block devices
      on Ganeti VM-capable nodes as confined be the current `blockdev' disk
      To do so, we implement a new disk template called `ext' (of type
      DTS_EXT_MIRROR) that passes control to externally provided scripts
      (the ExtStorage Provider) for the template's basic functions:
       create / attach / detach / remove / grow
      The scripts reside under ES_SEARCH_PATH (correlated to OS_SEARCH_PATH)
      and only one ExtStorage Provider is supported called `ext'.
      The disk's logical id is the tuple ('ext', UUID.ext.diskX), where UUID
      is generated as in disk template `plain' and X is the disk's index.
      Signed-off-by: default avatarConstantinos Venetsanopoulos <cven@grnet.gr>
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      [iustin@google.com: small simplification in bdev code, pylint fixes]
      Reviewed-by: default avatarIustin Pop <iustin@google.com>
  2. 19 Dec, 2012 8 commits
    • Guido Trotter's avatar
      Add hroller htools personality · 3504d6c8
      Guido Trotter authored
      This is a new personality that for the moment doesn't do anything.
      Signed-off-by: default avatarGuido Trotter <ultrotter@google.com>
      Reviewed-by: default avatarIustin Pop <iustin@google.com>
    • Iustin Pop's avatar
      Add support for job queries in hconfd · a7e484c4
      Iustin Pop authored
      This adds support for job queries, including (basic) unit-tests.
      I've tested this for memory and cpu usage as follows:
      - 3600 jobs (live queue):
        - via masterd, default: ~1.1s (masterd: ~60MB ram)
        - via confd,   default: ~1.1s (hconfd:  ~25MB ram)
        - via masterd, id only: ~1.0s (masterd: ~57MB ram)
        - via confd,   id only: ~0.2s (hconfd:  ~15MB ram)
      - all jobs (128K in total, around 570MB size on disk):
        - via masterd, default: 1m22s (masterd cpu: 48s), masterd: 1.4G ram
        - via confd,   default: 1m16s (hconfd  cpu: 51s), hconfd:  570MB ram peak
          (peak only right before starting luxi send, hconfd decreases in RSS as
          results are streamed over luxi, back to ~18MB after the send)
        - via masterd, id only:  ~49s (masterd cpu: 45s), masterd: 1.3G ram
        - via confd,   id only:  ~39s (hconfd cpu: 35s),  hconfd:  110MB ram peak
          (right before luxi send, decreasing as results are sent, back to ~14MB
          after the send)
      Given this, and that in production it's not likely to have hundreds of
      thousand of job files, I believe the implementation is safe from this
      point of view.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarHelga Velroyen <helgav@google.com>
    • Iustin Pop's avatar
      Add a read-only job queue module · aa79e62e
      Iustin Pop authored
      This patch adds implementation for a read-only job queue module,
      together with "full" test (as full as can be in a lazy language…).
      One note about the behaviour of the job queue is the handling of
      opcodes that fail validation: the 'input' opcode actually is a
      meta-type, which can hold either a real opcode or a plain JSValue, so
      that we can still load jobs with invalid opcodes for querying. The
      only downside of this is that, as opposed to Python code, we can't
      show the correct summary for such an opcode - we try to parse the
      OP_ID but not the extended OP_DSC_FIELD-equivalent.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarHelga Velroyen <helgav@google.com>
    • Iustin Pop's avatar
      Remove two hlint overrides · 86aa9ba3
      Iustin Pop authored
      "Use comparing" was not needed, and "Use on" can be solved by actually
      using on in the single case we have (the patch also reorders imports
      in that file).
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
    • Iustin Pop's avatar
      Add more mon-collector shell tests · 570274e4
      Iustin Pop authored
      These test the newly-changed behaviour, and fix an inconsistency in
      the hs-check target (versus hs-coverage).
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichele Tartara <mtartara@google.com>
    • Iustin Pop's avatar
      Enable bash completion for mon-collector · 988dc5ca
      Iustin Pop authored
      This just ties in all the pieces so far and enables bash completion
      for it.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
    • Iustin Pop's avatar
      Add a custom CLI module for DataCollectors · 55abd2c7
      Iustin Pop authored
      Currently, the mon-collector binary uses the HTools/CLI module, which
      is OK but mean it links in lots of htools code. By copying that module
      to DataCollectors/CLI and removing the unneeded code, we reduce the
      number of modules it depends on fro 20 to 12, meaning both a shorter
      compiler time (24s to 9s) and a smaller binary (~9.6MB to ~7MB).
      Also fixes a typo in the original HTools/CLI module, thanks Michele!
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichele Tartara <mtartara@google.com>
    • Iustin Pop's avatar
      Fix profiling targets · 675f65b7
      Iustin Pop authored
      As noted by Guido, there are problems when using the hs-prof and
      hs-prof-quick targets in the default configuration (compiling all
      programs). The errors manifest in the form of wrong symbols during
      I knew that the hs-prof targets, which compiled multiple objects with
      the '.o' suffix, could be problematic; but the objects that are
      actually needed in the Template Haskell phase are very standard and
      don't differ between the binaries (Constants, BasicTypes, JSON,
      THH). What I didn't realise (although it's obvious) is that also the
      hs_prof_quick targets (the final binaries) are also compiled with a
      single suffix ('.prof_o'), which means that the object files are
      actually compiled for the last binary.
      This means that targets later in the HS_ALL_PROGS list would work
      correct, but early targets, especially htools/htools, would fail.
      So the obvious, and the single simple solution is to make these two
      rules only work on a single binary at a time. This should be fine,
      since one is looking at a specific problem usually, and it has the
      advantage that the hs-prof step is much faster (since it wasn't
      buildable in parallel anyway).
      Thanks to Guido for finding and making the initial diagnose on this!
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
  3. 17 Dec, 2012 2 commits
  4. 13 Dec, 2012 2 commits
  5. 12 Dec, 2012 3 commits
  6. 11 Dec, 2012 1 commit
  7. 10 Dec, 2012 1 commit
  8. 07 Dec, 2012 1 commit
  9. 06 Dec, 2012 3 commits
  10. 05 Dec, 2012 1 commit
    • Michael Hanselmann's avatar
      Makefile: Reset environment for epydoc · de645b5b
      Michael Hanselmann authored
      epydoc allows overriding configuration values via environment variables.
      While this might be useful in certain use cases, but as no prefix
      whatsoever is used, conflicts are easily created. Some people have the
      environment variable “NAME” set, effectively overriding the project name
      set in epydoc.conf. A bug in epydoc causes an error if non-ASCII
      characters, such as German umlauts, are used in NAME.
      $ NAME=Täscht make py-apidoc
      'ascii' codec can't decode byte 0xc3 in position 73: ordinal not in range(128)
      $ parse=false make py-apidoc
      epydoc: error: Invalid option combination: --parse-only and --introspect-only.
      This patch changes the call in Makefile to reset the environment given
      to epydoc save for PATH and PYTHONPATH.
      Signed-off-by: default avatarMichael Hanselmann <hansmi@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
  11. 04 Dec, 2012 4 commits
  12. 03 Dec, 2012 3 commits
  13. 30 Nov, 2012 1 commit
  14. 29 Nov, 2012 1 commit
  15. 28 Nov, 2012 1 commit
  16. 21 Nov, 2012 1 commit
  17. 20 Nov, 2012 5 commits