1. 18 Nov, 2013 1 commit
  2. 04 Oct, 2013 1 commit
    • Klaus Aehlig's avatar
      Provide means of submitting jobs to a drained queue · 346c3037
      Klaus Aehlig authored
      During an upgrade, the job queue needs to be drained in order to avoid
      new jobs coming to the cluster.  Nevertheless, the upgrade process
      needs to carry out some maintenance, like redistributing the new
      configuration, therefore, this patch provides a means of submitting
      jobs to a drained queue.
      
      Of course, once the more fine-grained job queue control will be implemented,
      this functionality can be removed.
      Signed-off-by: default avatarKlaus Aehlig <aehlig@google.com>
      Reviewed-by: default avatarJose Lopes <jabolopes@google.com>
      346c3037
  3. 18 Sep, 2013 1 commit
    • Jose A. Lopes's avatar
      Flip dependency between Haskell types and constants · 1c31b263
      Jose A. Lopes authored
      Before this patch, Haskell types, such as, 'GanetiDaemon' and
      'GanetiGroup', and related functions were taking their values from
      Haskell constants.  However, given that the role of Haskell to Python
      constants is to leverage Haskell and its typesystem, it makes sense to
      first define the Haskell types and then have the constants depend on
      these types.  In other words, this patch series inverts the dependency
      between (some) Haskell types and constants.
      Signed-off-by: default avatarJose A. Lopes <jabolopes@google.com>
      Reviewed-by: default avatarMichele Tartara <mtartara@google.com>
      1c31b263
  4. 07 Aug, 2013 1 commit
  5. 22 Jul, 2013 1 commit
  6. 18 Jul, 2013 1 commit
  7. 12 Jul, 2013 1 commit
  8. 24 Dec, 2012 1 commit
  9. 17 Dec, 2012 2 commits
    • Iustin Pop's avatar
      Switch Luxi sendMsg from strict to lazy ByteStrings · 62d5242b
      Iustin Pop authored
      Commit e821050d (“Switch the Luxi interface from Strings to
      ByteStrings”) was designed to optimise the receive interface, but has
      an unfortunate side-effect: when sending non-trivial messages, it
      means that both the entire String and the ByteString versions must be
      in memory at the same time, leading to much increased memory usage.
      
      By changing the "hPut" from strict to lazy ByteStrings, it means that
      both the String and the ByteString values can be evaluated lazily,
      with significant effects: for a test query answer, instead of having
      a peak from ~600MB to 1.4G during the entire Luxi send operation,
      memory consumption actually decreased during the send operation, as
      the ByteString chunks are released individually and even the backing
      storage of the items that create the JSON string serialisation is
      released lazily as well. So instead of slow growth 10→550MB then quick
      peak to 1.4GB during Luxi send, we now have an even slower growth to
      ~580MB and then decrease of memory as the Luxi send progresses.
      
      The only downside is of a small increase in CPU time of a few percents
      for the above case; for our use cases, I think this is much desirable.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarHelga Velroyen <helgav@google.com>
      62d5242b
    • Iustin Pop's avatar
      Support 'null' in Luxi QueryJobs names field · d2970809
      Iustin Pop authored
      Python code sometimes sends this, so let's support it even though it's
      non-standard.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarHelga Velroyen <helgav@google.com>
      d2970809
  10. 05 Dec, 2012 1 commit
  11. 04 Dec, 2012 3 commits
  12. 30 Nov, 2012 1 commit
    • Iustin Pop's avatar
      Remove read instances from our Haskell code · 139c0683
      Iustin Pop authored
      It turns out that optimising 'read' derived instances (via -O) for
      complex data types (like OpCode, or the various objects) can be slow
      to very slow. Disabling such instances results in (time make
      $all_our_haskell_binaries) large compile-time savings and also smaller
      (unstripped) binaries (by a significant amount):
      
      ghc 6.12:        time  htools sz  hconfd sz
        with read:    4m50s 12,244,694 14,927,928
        no read:      3m30s 10,234,305 12,536,745
      ghc 7.6:
        with read:   14m45s 13,694,761 15,741,755
        no read:      3m40s 11,631,373 13,245,134
      
      So let's remove these instances, since we never use read in production
      for our custom types, and even when debugging in GHCI, we can simply
      use the 'show' representation to create the types, without needing to
      actually parse from strings.
      
      Note: for the very slow ghc 7.6 compilation time, I filled a ticket
      (ghc #7450), since it is surprising(ly slow).
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichele Tartara <mtartara@google.com>
      139c0683
  13. 13 Nov, 2012 1 commit
  14. 12 Nov, 2012 2 commits
    • Iustin Pop's avatar
      Convert tag objects to a safer type · d8e7c45e
      Iustin Pop authored
      Currently, we keep information about the "target" of a tag operation
      in a data type similar to (TagKind, Maybe String). This is unsafe, as
      nothing (at the type level) prevents us from accidentally having
      (TagCluster, Just "instance1.example.com"), or (TagInstance, Nothing).
      
      To fix this problem, we rename the current TagObject type to TagType
      (an internal utility type), and create TagObject as a better/safer
      data type (see the definition), which doesn't allow such possibilities
      in the future.
      
      The downside is that, since at encoding level (both opcode and luxi)
      this is done in an ugly way (type elements spread at the same level as
      level as other value), we have to add custom encoders/decoders. The
      encoder is shared between the OpCode and Luxi usage, the decoder is
      different however as Luxi uses custom decoding.
      
      This also fixes the recent breakage in confd w.r.t. QueryTags.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      d8e7c45e
    • Iustin Pop's avatar
      Switch Luxi TH code from simple to custom fields · 88609f00
      Iustin Pop authored
      This is needed so that we have more flexibility in generating Luxi
      serialisation code (deserialisation is still custom). Also, only
      exceptions are now using the 'simple' field types, so we might be able
      later to convert and remove that TH code as well.
      
      Since we will use custom serialisation fields in the future, we change
      the order of serialisation for custom-save fields; Luxi uses
      positional as opposed to name-based ordering, so we need to keep this
      stable.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarHelga Velroyen <helgav@google.com>
      88609f00
  15. 08 Nov, 2012 1 commit
  16. 07 Nov, 2012 1 commit
  17. 06 Nov, 2012 1 commit
  18. 26 Oct, 2012 2 commits
    • Iustin Pop's avatar
      Convert Luxi results to Ganeti errors · 7adb7dff
      Iustin Pop authored
      This a bit too complex patch converts the result of Luxi calls
      (submitJob, query*, etc.) from Result to ErrorResult. It then
      immediately revers this in the HTools/Backend/Luxi module, where we
      don't need necessarily the full error type (just a nice error
      message), and does the same in Hbal's job execution functions.
      
      While at first sight this doesn't seem to do much, what we get is
      actual error messages from Ganeti, plus improvements to the result
      parsing: instead of "can't parse char", we now get properly (note,
      wrapped manually):
      
        Executing jobset for instances instance1, …
        Job submission error: Failure: the job queue is marked for drain and
          doesn't accept new requests
      
      Or:
      
        Job submission error: Unhandled exception: LuxiError "parsing job
          id: cannot parse string 'a956101'"
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      7adb7dff
    • Iustin Pop's avatar
      Fix a few issues found by newer hlint · 66ad857a
      Iustin Pop authored
      Testing with a newer hlint found a few minor issues; but all are real,
      valid recommendations:
      
      - don't use "if cond then f x else f y", but "f (if cond then x else y)"
      - "if a then b else True" is equivalent to the simpler "not a || b"
      - and as usual, one more ignore to our "testing basic properties"
        module
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      66ad857a
  19. 11 Oct, 2012 1 commit
    • Iustin Pop's avatar
      Cleanup network timeouts and htools imports · 4cd79ca8
      Iustin Pop authored
      This patch removes the last HTools module imports from non-htools code
      (the HTools.Types module), but it requires an associated cleanup:
      using luxi-specific constants for luxi timeouts (the only effect is
      that one timeout decreases from 15 to 10, the default value in the
      python code), and moving of the (now) RAPI specific constants to
      RAPI.hs (which allows simplifying their type/usage).
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      4cd79ca8
  20. 08 Oct, 2012 1 commit
  21. 05 Sep, 2012 1 commit
    • Iustin Pop's avatar
      Further hlint fixes · 5b11f8db
      Iustin Pop authored
      Commit 2cdaf225, “Re-enable standard hlint warnings”, got it almost
      right. The only problem is that (confusingly) the default set of hints
      is not in HLint.Default, but in HLint.HLint (it includes Default and
      some built-ins).
      
      After changing the lint file to correctly include the defaults, we had
      another 128 suggestions:
      
        - Error: Eta reduce (2)
        - Error: Redundant bracket (4)
        - Error: Redundant do (17)
        - Error: Redundant lambda (7)
        - Error: Redundant return (1)
        - Warning: Avoid lambda (2)
        - Warning: Redundant $ (42)
        - Warning: Redundant bracket (35)
        - Warning: Use : (1)
        - Warning: Use String (4)
        - Warning: Use camelCase (10)
        - Warning: Use section (3)
      
      which are fixed by the current patch. Note that the 10 "Use camelCase"
      were all due to hlint not “knowing” the idiom of ‘case_’ (it does for
      ‘prop_’), for which I filled
      http://code.google.com/p/ndmitchell/issues/detail?id=558.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarRené Nussbaumer <rn@google.com>
      5b11f8db
  22. 04 Sep, 2012 3 commits
  23. 03 Sep, 2012 1 commit
  24. 28 Aug, 2012 8 commits
  25. 13 Aug, 2012 2 commits
    • Iustin Pop's avatar
      Add a server-side Luxi implementation · 13f2321c
      Iustin Pop authored
      This is a trivial code change, but it allows us to finally test the
      send-receive code on both client and server sides via a simple
      in-process server.
      
      The unittest works, but it won't handle timeouts very nicely; it will
      wait until the actual Luxi timeout expires, instead of using much
      shorter timeouts as we could in the same process.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
      13f2321c
    • Iustin Pop's avatar
      Switch the Luxi interface from Strings to ByteStrings · e821050d
      Iustin Pop authored
      I'm doing this change for future performance optimisations. Currently
      we use the Luxi interface just as a client, so not in the hot path,
      but when we'll use this as a server interface, we're interested to
      both reduce the space and time consumption of the interface.
      
      We have to simultaneous changes here:
      
      - switch from using socket-related function (sendto, recv, etc.) to
        handle-based functions, since the standard network library doesn't
        work with sockets
      - switch from using Strings for the internal buffer to strict
        ByteStrings; the only downside is that we now have the issue of
        decoding/encoding from binary to UTF-8 strings, a fact which brings
        its own issues into the mix (we have to check for failed decodings,
        etc.); but this is similar to what we'll have to handle on the
        Python side when moving to Python 3.x
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
      e821050d