1. 20 Dec, 2010 2 commits
    • Iustin Pop's avatar
      Change the Node.group attribute · 10ef6b4e
      Iustin Pop authored
      
      
      Currently, the Node.group attribute is the UUID of the group, as until
      recently Ganeti didn't export the node group properties. Since it does
      so now, we make the following changes (again apologies for a big
      patch):
      
      - we change the group attribute to be an index, similar to the way an
        Instance.pnode and snode attributes point to the parent node(s)
      - on load, we read the group.uuid attribute and we use that to lookup
        the actual group index, from previously-loaded groups info
      - this means that we now first read groups, then read nodes using the
        group info, and then read instances using the node info
      
      This patch leaves a few functions showing the group index (ugly since
      it's htools internal), will be converted later.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarBalazs Lecz <leczb@google.com>
      10ef6b4e
    • Iustin Pop's avatar
      Rework the data loader pipelines to read groups · a679e9dc
      Iustin Pop authored
      
      
      This (invasive) patch changes all the loader pipelines to read the node
      groups data from the cluster, via the various backends. It is invasive
      as it needs coordinated changes across all the loaders.
      
      Note that the new group data is not used, just returned.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarBalazs Lecz <leczb@google.com>
      a679e9dc
  2. 01 Dec, 2010 2 commits
    • Iustin Pop's avatar
      Rework the types used during data loading · 99b63608
      Iustin Pop authored
      
      
      This improves on the previous change. Currently, the node and instance
      lists shipped around during data loading are (again) association lists.
      For instances it's not a big issue, but the node list is rewritten
      continuously while we assign instances to nodes, and that is very slow.
      The code was originally written for small (10-20 node) clusters, and
      today with multinodes… :)
      
      Rewriting to use Node.List/Instance.List makes a bit of a messy patch,
      but it allows to remove some custom functions for assoc. list
      processing, and also some custom unittests.
      
      At the end, the load time is almost halved, and we spend time now just
      in the string parsing code (which is, as we know, slow…).
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarBalazs Lecz <leczb@google.com>
      99b63608
    • Iustin Pop's avatar
      Convert some leftovers to NameAssoc · 6ff78049
      Iustin Pop authored
      
      
      The type alias NameAssoc has been introduced a long time ago, but there
      are some few not-yet-converted cases. In preparation for changes to that
      type, let's make sure we use it consistently.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarBalazs Lecz <leczb@google.com>
      6ff78049
  3. 24 Nov, 2010 2 commits
  4. 19 Nov, 2010 1 commit
  5. 27 May, 2010 2 commits
    • Iustin Pop's avatar
      Replace fromJResult with annotateJResult · c96d44df
      Iustin Pop authored
      This patch removes all old uses of fromJResult with the annotated
      version, and removes the non-annotated version. All JSON parsing points
      should now have annotated errors.
      c96d44df
    • Iustin Pop's avatar
      Add annotations to loadJSArray · c8b662f1
      Iustin Pop authored
      This allows, for example, the RAPI backend to detail which information
      (instance or node data) fails to parse.
      c8b662f1
  6. 25 Feb, 2010 2 commits
  7. 11 Dec, 2009 1 commit
    • Iustin Pop's avatar
      Use the oper_ram field if available · 6402a260
      Iustin Pop authored
      For the RAPI and LUXI backends, we can get the actual memory usage (if
      instances are running) via the oper_ram, whereas backend/memory only
      tell what the instance will use at the next boot.
      
      Not using oper_ram means that the node model is flawed and we consider
      wrong values for the instance's memory (resulting sometimes in hilarious
      values such as x_mem = -700 MB).
      6402a260
  8. 09 Dec, 2009 1 commit
    • Iustin Pop's avatar
      rapi, luxi: treat drained nodes as offline · b45222ce
      Iustin Pop authored
      Commit e97f211e changed the iallocator backend to handle drained nodes as
      offline. This commit completes that change by making the rapi and luxi
      backend do the same (the text backend ignores any '?' values which are
      returned by ganeti when nodes have problems, so it doesn't need this
      change).
      b45222ce
  9. 01 Dec, 2009 1 commit
  10. 27 Nov, 2009 1 commit
  11. 11 Nov, 2009 1 commit
    • Iustin Pop's avatar
      Add a tags attribute to instances · 17e7af2b
      Iustin Pop authored
      … and read it in all the loaders. hscan is modified to save it to the
      files it generates.
      
      The attribute is not yet used in any place.
      17e7af2b
  12. 06 Nov, 2009 2 commits
    • Iustin Pop's avatar
      Enhance the error reporting for Rapi and Luxi · 117dc2d8
      Iustin Pop authored
      Currently the JSON conversion in Rapi and Luxi are giving something
      like:
        Error: failed to load data. Details:
        Unable to read Double
      
      This doesn't tell one where the error is (in a node specification? and
      which node? etc.). This patch annotates such messages with the owner
      node:
        Error: failed to load data. Details:
        Node 'node1' key 'mtotal': Unable to read Double
      
      For errors during parsing of the node/instance name (unlikely, but
      still), the output is:
        Error: failed to load data. Details:
        Parsing new node key 'name': Unable to read String
      117dc2d8
    • Iustin Pop's avatar
      Change the Utils.fromObj signature · 262f3e6c
      Iustin Pop authored
      Currently the fromObj function takes a JSON object which is then
      converted into a list of (String, JSValue) in which we make a lookup.
      However, most of the callers of this function call it repeatedly on the
      same object, which means we do the object→list conversion repeatedly.
      
      This patch converts it to take directly the list, and converts its
      callers to do the conversion themselves (and only once).
      
      While this is not in the hot-path today, it would be if we ever were to
      process much data over Luxi (or RAPI), and is a good cleanup in any
      case.
      262f3e6c
  13. 14 Jul, 2009 1 commit
    • Iustin Pop's avatar
      Introduce timeout in RAPI queries · 135a6c6a
      Iustin Pop authored
      The patch adds two constants in Types.hs for connect and query timeout,
      then modifies Rapi.hs to use them as the connect and general curl
      timeout.
      
      Rapi could be improved more, as currently we wait double the total
      timeout due to not aborting early in case the node queries failed.
      135a6c6a
  14. 06 Jul, 2009 1 commit
    • Iustin Pop's avatar
      Fix hlint-generated warnings · 9f6dcdea
      Iustin Pop authored
      This big patch cleans up the code per hlint indications. Many removals
      of extra parentheses, replacements of concat . map with concabtMap,
      extra dollar signs, eta reductions, etc. were performed.
      
      The code still compiles and passes a couple of manual tests on sample
      files. The individual changes are also small enough as to be visually
      easy to confirm.
      9f6dcdea
  15. 10 Jun, 2009 2 commits
    • Iustin Pop's avatar
      Add cpu-count-related attributes to nodes · 1a82215d
      Iustin Pop authored
      This patch adds cpu-count related attributes to nodes:
        - total cpus
        - cpus in use
        - ratio of virtual:physical cpus
      
      We also set correctly the cpu values at load time, but we don't do
      anything yet while moving instances around. The cpu ratio is shown in
      the cluster list.
      1a82215d
    • Iustin Pop's avatar
      Add a new vcpus attribute to instances · d752eb39
      Iustin Pop authored
      This patch adds reading of vcpu count for instances, in preparation for
      using the vcpu ratio in cluster scoring.
      d752eb39
  16. 01 Jun, 2009 2 commits
    • Iustin Pop's avatar
      Add copyright/license information · e2fa2baf
      Iustin Pop authored
      This doc-patch adds copyright and license information to (hopefully) all
      needed files.
      e2fa2baf
    • Iustin Pop's avatar
      Lots of documentation updates · 9188aeef
      Iustin Pop authored
      This patch does only doc build changes, doc changes and function move
      around (for more logical documentation). It should have no impact at all
      on the code.
      9188aeef
  17. 27 May, 2009 1 commit
    • Iustin Pop's avatar
      Add type synonyms for the node/instance indices · 608efcce
      Iustin Pop authored
      This is a first step towards full datatype renaming. That requires more
      changes, so at first we only want to document clearly what is a node
      index, what is an instance index, and what is a plain Int.
      608efcce
  18. 25 May, 2009 1 commit
  19. 23 May, 2009 2 commits
    • Iustin Pop's avatar
      Remove ktn/kti from first half of loader · e3a684c5
      Iustin Pop authored
      This patch removes the ktn/kti lists from most parts of the first half
      of the loading sequence. Some remain as the [(String, Int)] is the
      nicest way to lookup names and get indices back.
      e3a684c5
    • Iustin Pop's avatar
      Add a small class for Nodes and Instances · 497e30a1
      Iustin Pop authored
      Since both nodes and instances support some common functionality (names
      and indices), we add a class so that we can access these attributes in a
      generic way.
      497e30a1
  20. 22 May, 2009 3 commits
    • Iustin Pop's avatar
      Add back names to nodes/instances · 2727257a
      Iustin Pop authored
      In order to simplify the data structures, we add back the name on the
      node and instance objects. We still keep the index for, well, indexing,
      but we will use the name directly from the object, in order to get rid
      of the ktn/kti arguments which are passed around everywhere.
      2727257a
    • Iustin Pop's avatar
      More code reorganizations · e4c5beaf
      Iustin Pop authored
      This new big patch does a couple of more cleanups in the loading of data
      chapter:
        - introduce a Types module that holds most types (except the base
          Node/Instance/etc.) so that multiple other modules can use these
          (instead of only Cluster and its users)
        - bring IAlloc.hs in line with the recent changes of providing data
          types and not strings
        - removal of obsolete Utils.hs json-related functions
      e4c5beaf
    • Iustin Pop's avatar
      Rework the loader model · 040afc35
      Iustin Pop authored
      This big patch changes the loader model from “string data as common
      format” to actual object structures as common format.
      
      The text loading function move from Cluster.hs to a new Text.hs module,
      some common functions are moved to a new Loader.hs module, and the
      return values from both Rapi.hs and Text.hs are uniformized.
      040afc35
  21. 20 May, 2009 5 commits
    • Iustin Pop's avatar
      Small syntax improvement · 1297ce13
      Iustin Pop authored
      D'oh, one can extract from a wrapped variable, not only from functions.
      1297ce13
    • Iustin Pop's avatar
      Allow overriding the RAPI port/scheme · e015b554
      Iustin Pop authored
      This patch adds a very dumb way to override the port and scheme for
      RAPI: if the master string doesn't contain a colon, we consider it a
      simple hostname specification, and thus prepend ‘https://’ and append
      ‘:5080’, (the default RAPI port); otherwise, we consider it a fully
      specified URL, and don't do any mangling to it.
      
      This allows to use saved RAPI responses in testing, with the “file://”
      scheme, or to use another port or non https mode with an actual RAPI
      instance.
      e015b554
    • Iustin Pop's avatar
      Remove some 1.2 specific code · ba00ad4d
      Iustin Pop authored
      Now that we dropped 1.2 compatibility, we can removed the special casing
      for RAPI differences.
      ba00ad4d
    • Iustin Pop's avatar
      Generalize some Result function into monad ones · 5aa48dbe
      Iustin Pop authored
      We don't really needed, but is more clean like this.
      5aa48dbe
    • Iustin Pop's avatar
      Switch from hand-written monads to a real one · 942403e6
      Iustin Pop authored
      This big patch converts from our home-grown monad-like constructs
      (the Either stuff) to a real, Either-like-but-another-name monad.
      
      We introduce a “Result a” monad, and this allows dropping many of the
      extra constructs. Hopefully the code is also more clear.
      
      Many of the functions could now be written in a generic-monad style,
      instead of Result specifically, but that will come in future patches.
      
      IAlloc.hs also has some unrelated patches.
      942403e6
  22. 19 May, 2009 2 commits
    • Iustin Pop's avatar
      Drop RAPI v1 compatiblity · aff363a4
      Iustin Pop authored
      Since we only support Ganeti 2.0, we drop the 1.2 compatibility.
      aff363a4
    • Iustin Pop's avatar
      Add support for 'offline' nodes · 00b15752
      Iustin Pop authored
      This patch drops compatiblity with Ganeti 1.2 and adds support for
      offline nodes in the cluster. When reading from RAPI, the drained nodes
      are considered offline so that we don't allocate on them too.
      00b15752
  23. 18 May, 2009 1 commit
  24. 22 Mar, 2009 1 commit
    • Iustin Pop's avatar
      Include DRBD overhead in sda/sdb size · 190ce47c
      Iustin Pop authored
      For Ganeti 1.2 which doesn't have the ‘disk_usage’ instance query field,
      we need to manually include the DRBD overhead (per disk). This patch
      modifies the RAPI collection to do this, but loading from disk does not
      as it's unknown if the query came from hscan or RAPI 1.2 or RAPI 2.0...
      190ce47c