1. 04 Dec, 2012 2 commits
    • Iustin Pop's avatar
      Add a 'real' type for JobIds · c48711d5
      Iustin Pop authored
      
      
      Currently, the job ID is a simple type alias. This is suboptimal, as
      it means we can't use a custom JSON (or Arbitrary) instance for it.
      
      The patch changes it into a newtype, and then a) simplifies some
      deserialisation code and b) changes some more fields to this new type
      (rather than plain 'Int').
      
      We also move the JobId to types, since it will be needed in opcodes as
      well.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
      c48711d5
    • Iustin Pop's avatar
      Rework custom fields handling · fa10983e
      Iustin Pop authored
      
      
      This patch changes a bit the handling of custom fields. Since in
      general we use custom fields to aggregate multiple entries in the JSON
      object into a safer data-type, we should also have a way to declare
      which extra entries this field covers (so that in the future we can
      say what are all the JSON keys for an object).
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
      fa10983e
  2. 30 Nov, 2012 1 commit
    • Iustin Pop's avatar
      Remove read instances from our Haskell code · 139c0683
      Iustin Pop authored
      
      
      It turns out that optimising 'read' derived instances (via -O) for
      complex data types (like OpCode, or the various objects) can be slow
      to very slow. Disabling such instances results in (time make
      $all_our_haskell_binaries) large compile-time savings and also smaller
      (unstripped) binaries (by a significant amount):
      
      ghc 6.12:        time  htools sz  hconfd sz
        with read:    4m50s 12,244,694 14,927,928
        no read:      3m30s 10,234,305 12,536,745
      ghc 7.6:
        with read:   14m45s 13,694,761 15,741,755
        no read:      3m40s 11,631,373 13,245,134
      
      So let's remove these instances, since we never use read in production
      for our custom types, and even when debugging in GHCI, we can simply
      use the 'show' representation to create the types, without needing to
      actually parse from strings.
      
      Note: for the very slow ghc 7.6 compilation time, I filled a ticket
      (ghc #7450), since it is surprising(ly slow).
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichele Tartara <mtartara@google.com>
      139c0683
  3. 13 Nov, 2012 1 commit
  4. 12 Nov, 2012 2 commits
    • Iustin Pop's avatar
      Convert tag objects to a safer type · d8e7c45e
      Iustin Pop authored
      
      
      Currently, we keep information about the "target" of a tag operation
      in a data type similar to (TagKind, Maybe String). This is unsafe, as
      nothing (at the type level) prevents us from accidentally having
      (TagCluster, Just "instance1.example.com"), or (TagInstance, Nothing).
      
      To fix this problem, we rename the current TagObject type to TagType
      (an internal utility type), and create TagObject as a better/safer
      data type (see the definition), which doesn't allow such possibilities
      in the future.
      
      The downside is that, since at encoding level (both opcode and luxi)
      this is done in an ugly way (type elements spread at the same level as
      level as other value), we have to add custom encoders/decoders. The
      encoder is shared between the OpCode and Luxi usage, the decoder is
      different however as Luxi uses custom decoding.
      
      This also fixes the recent breakage in confd w.r.t. QueryTags.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      d8e7c45e
    • Iustin Pop's avatar
      Switch Luxi TH code from simple to custom fields · 88609f00
      Iustin Pop authored
      
      
      This is needed so that we have more flexibility in generating Luxi
      serialisation code (deserialisation is still custom). Also, only
      exceptions are now using the 'simple' field types, so we might be able
      later to convert and remove that TH code as well.
      
      Since we will use custom serialisation fields in the future, we change
      the order of serialisation for custom-save fields; Luxi uses
      positional as opposed to name-based ordering, so we need to keep this
      stable.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarHelga Velroyen <helgav@google.com>
      88609f00
  5. 08 Nov, 2012 1 commit
  6. 07 Nov, 2012 1 commit
  7. 06 Nov, 2012 1 commit
  8. 26 Oct, 2012 2 commits
    • Iustin Pop's avatar
      Convert Luxi results to Ganeti errors · 7adb7dff
      Iustin Pop authored
      
      
      This a bit too complex patch converts the result of Luxi calls
      (submitJob, query*, etc.) from Result to ErrorResult. It then
      immediately revers this in the HTools/Backend/Luxi module, where we
      don't need necessarily the full error type (just a nice error
      message), and does the same in Hbal's job execution functions.
      
      While at first sight this doesn't seem to do much, what we get is
      actual error messages from Ganeti, plus improvements to the result
      parsing: instead of "can't parse char", we now get properly (note,
      wrapped manually):
      
        Executing jobset for instances instance1, …
        Job submission error: Failure: the job queue is marked for drain and
          doesn't accept new requests
      
      Or:
      
        Job submission error: Unhandled exception: LuxiError "parsing job
          id: cannot parse string 'a956101'"
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      7adb7dff
    • Iustin Pop's avatar
      Fix a few issues found by newer hlint · 66ad857a
      Iustin Pop authored
      
      
      Testing with a newer hlint found a few minor issues; but all are real,
      valid recommendations:
      
      - don't use "if cond then f x else f y", but "f (if cond then x else y)"
      - "if a then b else True" is equivalent to the simpler "not a || b"
      - and as usual, one more ignore to our "testing basic properties"
        module
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      66ad857a
  9. 11 Oct, 2012 1 commit
    • Iustin Pop's avatar
      Cleanup network timeouts and htools imports · 4cd79ca8
      Iustin Pop authored
      
      
      This patch removes the last HTools module imports from non-htools code
      (the HTools.Types module), but it requires an associated cleanup:
      using luxi-specific constants for luxi timeouts (the only effect is
      that one timeout decreases from 15 to 10, the default value in the
      python code), and moving of the (now) RAPI specific constants to
      RAPI.hs (which allows simplifying their type/usage).
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      4cd79ca8
  10. 08 Oct, 2012 1 commit
  11. 05 Sep, 2012 1 commit
    • Iustin Pop's avatar
      Further hlint fixes · 5b11f8db
      Iustin Pop authored
      Commit 2cdaf225, “Re-enable standard hlint warnings”, got it almost
      right. The only problem is that (confusingly) the default set of hints
      is not in HLint.Default, but in HLint.HLint (it includes Default and
      some built-ins).
      
      After changing the lint file to correctly include the defaults, we had
      another 128 suggestions:
      
        - Error: Eta reduce (2)
        - Error: Redundant bracket (4)
        - Error: Redundant do (17)
        - Error: Redundant lambda (7)
        - Error: Redundant return (1)
        - Warning: Avoid lambda (2)
        - Warning: Redundant $ (42)
        - Warning: Redundant bracket (35)
        - Warning: Use : (1)
        - Warning: Use String (4)
        - Warning: Use camelCase (10)
        - Warning: Use section (3)
      
      which are fixed by the current patch. Note that the 10 "Use camelCase"
      were all due to hlint not “knowing” the idiom of ‘case_’ (it does for
      ‘prop_’), for which I filled
      http://code.google.com/p/ndmitchell/issues/detail?id=558
      
      .
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarRené Nussbaumer <rn@google.com>
      5b11f8db
  12. 04 Sep, 2012 3 commits
  13. 03 Sep, 2012 1 commit
  14. 28 Aug, 2012 8 commits
  15. 13 Aug, 2012 2 commits
    • Iustin Pop's avatar
      Add a server-side Luxi implementation · 13f2321c
      Iustin Pop authored
      
      
      This is a trivial code change, but it allows us to finally test the
      send-receive code on both client and server sides via a simple
      in-process server.
      
      The unittest works, but it won't handle timeouts very nicely; it will
      wait until the actual Luxi timeout expires, instead of using much
      shorter timeouts as we could in the same process.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
      13f2321c
    • Iustin Pop's avatar
      Switch the Luxi interface from Strings to ByteStrings · e821050d
      Iustin Pop authored
      
      
      I'm doing this change for future performance optimisations. Currently
      we use the Luxi interface just as a client, so not in the hot path,
      but when we'll use this as a server interface, we're interested to
      both reduce the space and time consumption of the interface.
      
      We have to simultaneous changes here:
      
      - switch from using socket-related function (sendto, recv, etc.) to
        handle-based functions, since the standard network library doesn't
        work with sockets
      - switch from using Strings for the internal buffer to strict
        ByteStrings; the only downside is that we now have the issue of
        decoding/encoding from binary to UTF-8 strings, a fact which brings
        its own issues into the mix (we have to check for failed decodings,
        etc.); but this is similar to what we'll have to handle on the
        Python side when moving to Python 3.x
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
      e821050d
  16. 07 Aug, 2012 1 commit
    • Iustin Pop's avatar
      Switch job IDs to numeric · 76b62028
      Iustin Pop authored
      
      
      This has been a long-standing cleanup item, which we've always
      refrained from doing due to the high estimated effort needed.
      
      In reality, it turned out that after some infrastructure improvements
      (the previous patches), the actual job queue-related changes are quite
      small.
      
      We will need to update the NEWS file later, but so far the RAPI
      documentation doesn't mention that the job ID is a string (it only
      says it is "a number"), so it doesn't look like it needs update.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarRené Nussbaumer <rn@google.com>
      76b62028
  17. 31 Jul, 2012 2 commits
  18. 08 May, 2012 3 commits
  19. 13 Jan, 2012 1 commit
  20. 08 Dec, 2011 1 commit
    • Iustin Pop's avatar
      Cleanup hlint errors · 3603605a
      Iustin Pop authored
      
      
      First, we update the recommended hlint version to what I used to get a
      clean output (1.8.15). Most of the changes are:
      
      - remove unneeded parentheses
      - some simplifications (intercalate " " → unwords, maybe … id →
        fromMaybe, etc.)
      - removal of some duplicate code (in previous patches)
      
      There are still some warnings which I didn't clean out but plain
      ignored:
      
      - 'Eta reduce' in some specific files, because the type inference
        specialises the function on the first call, and annotating the type
        properly would be too verbose
      - use of 'first', 'comparing', and 'on', since these don't seem to be
        widely or consistently used (outside ganeti/htools, I mean)
      - use of Control.Exception.catch, as we only care about I/O errors; at
        one point yes, we will need to transition to this new API
      - 'Reduce duplication', since hlint warns even for 3 duplicate lines,
        and abstracting that away seems overkill to me
      
      After this patch, make hlint is clean and doesn't exit with an error
      anymore; we could enable it automatically on 'make lint' if hlint is
      detected (future patch).
      
      Note that we explicitly skip the THH.hs file from checking because it
      seems that hlint doesn't parse correctly for now the splice notation.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarAgata Murawska <agatamurawska@google.com>
      3603605a
  21. 17 Nov, 2011 1 commit
  22. 26 Oct, 2011 2 commits
  23. 14 Oct, 2011 1 commit
    • Iustin Pop's avatar
      Adjust htools code to new Luxi argument format · b20cbf06
      Iustin Pop authored
      This partially undoes commit 92678b3c
      
      , more specifically it removes the
      Store data type and the associated code, since all Luxi arguments are
      now lists.
      
      Furthermore, since the qfilter field on Query is complex (it's
      actually a tree structure), and we don't support it, turn it into a
      plain () type, which always gets encoded as JSNull ('null'), so that
      we can remove the optional field handling from Luxi (all fields are
      always required).
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      b20cbf06