1. 11 Apr, 2013 2 commits
  2. 21 Mar, 2013 1 commit
  3. 20 Mar, 2013 1 commit
  4. 04 Mar, 2013 1 commit
  5. 07 Feb, 2013 1 commit
  6. 24 Dec, 2012 1 commit
  7. 19 Dec, 2012 2 commits
    • Iustin Pop's avatar
      Add support for job queries in hconfd · a7e484c4
      Iustin Pop authored
      
      
      This adds support for job queries, including (basic) unit-tests.
      
      I've tested this for memory and cpu usage as follows:
      
      - 3600 jobs (live queue):
        - via masterd, default: ~1.1s (masterd: ~60MB ram)
        - via confd,   default: ~1.1s (hconfd:  ~25MB ram)
        - via masterd, id only: ~1.0s (masterd: ~57MB ram)
        - via confd,   id only: ~0.2s (hconfd:  ~15MB ram)
      
      - all jobs (128K in total, around 570MB size on disk):
        - via masterd, default: 1m22s (masterd cpu: 48s), masterd: 1.4G ram
        - via confd,   default: 1m16s (hconfd  cpu: 51s), hconfd:  570MB ram peak
          (peak only right before starting luxi send, hconfd decreases in RSS as
          results are streamed over luxi, back to ~18MB after the send)
        - via masterd, id only:  ~49s (masterd cpu: 45s), masterd: 1.3G ram
        - via confd,   id only:  ~39s (hconfd cpu: 35s),  hconfd:  110MB ram peak
          (right before luxi send, decreasing as results are sent, back to ~14MB
          after the send)
      
      Given this, and that in production it's not likely to have hundreds of
      thousand of job files, I believe the implementation is safe from this
      point of view.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarHelga Velroyen <helgav@google.com>
      a7e484c4
    • Iustin Pop's avatar
      Change makeSimpleFilter accept NumericValues as well · 037762a9
      Iustin Pop authored
      
      
      This is required for QueryJobs implementation; we change
      makeSimpleFilter to support both string and integers as names.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarHelga Velroyen <helgav@google.com>
      037762a9
  8. 13 Dec, 2012 1 commit
    • Iustin Pop's avatar
      Log only partial response in Luxi when in debug mode · f74b88fa
      Iustin Pop authored
      
      
      Currently, we log the entire response (at debug level) in the Luxi
      replies. This is not a good idea; the logging library operates on
      strings, and as such it will use huge amounts of memory: without debug
      logging, a certain gnt-job list invocation uses 295MB (+RTS -s) and
      2m35s time, when in debug mode, it's 1525MB and 48m!
      
      So we make two changes:
      
      - first, we switch from "show (pp_value a)" to "encode a", which
        generates a non-formatted string rather than a indented one
      - second we log only the first 2000 characters; this should be enough
        to understand the first part of the response
      
      We could go for higher, or for logging in batches (that would be
      faster, as well).
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarHelga Velroyen <helgav@google.com>
      f74b88fa
  9. 21 Nov, 2012 1 commit
    • Iustin Pop's avatar
      Split 'Query.Language.ItemType' in two sub-types · 1283cc38
      Iustin Pop authored
      
      
      The QR_VIA_OP/QR_VIA_LUXI types in Python are using yet another
      validation mode: QR_VIA_OP is the base type, and QR_VIA_LUXI extends
      it (when doing luxi queries). But on the wire they have the same
      representation.
      
      To accommodate this properly, we split the ItemType in two: a
      QueryTypeOp and a QueryTypeLuxi, joining them back together in
      ItemType itself. This requires custom serialisation/deserialisation,
      but allows us to express correctly that at opcode level, we only
      accept a QueryTypeOp, but at Luxi query level, we accept either of
      them.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
      1283cc38
  10. 12 Nov, 2012 4 commits
  11. 08 Nov, 2012 1 commit
  12. 25 Oct, 2012 1 commit
    • Iustin Pop's avatar
      Convert query path from string errors to GanetiException · 5183e8be
      Iustin Pop authored
      
      
      This patch converts all the call paths from 'Result' (which contains
      just string errors) to 'ErrorResult', which holds
      GanetiException-encoded errors. We can now return proper
      OpPrereq/OpExec errors to the clients of the luxi/query socket.
      
      The patch touches many files as we had to convert the entire call
      chains in a single round. But it should be pretty straightforward
      otherwise:
      
      - change 'Result' into 'ErrorResult'
      - add error annotations: change "Bad msg" into "Bad (XXXEror msg)"
      - add a helper function for confd, where we don't send to client
        formatted exceptions, to convert back from ErrorResult into Result
      - change tests similarly, where needed
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      5183e8be
  13. 11 Oct, 2012 1 commit
  14. 26 Sep, 2012 2 commits
  15. 18 Sep, 2012 1 commit
  16. 05 Sep, 2012 1 commit
    • Iustin Pop's avatar
      Further hlint fixes · 5b11f8db
      Iustin Pop authored
      Commit 2cdaf225, “Re-enable standard hlint warnings”, got it almost
      right. The only problem is that (confusingly) the default set of hints
      is not in HLint.Default, but in HLint.HLint (it includes Default and
      some built-ins).
      
      After changing the lint file to correctly include the defaults, we had
      another 128 suggestions:
      
        - Error: Eta reduce (2)
        - Error: Redundant bracket (4)
        - Error: Redundant do (17)
        - Error: Redundant lambda (7)
        - Error: Redundant return (1)
        - Warning: Avoid lambda (2)
        - Warning: Redundant $ (42)
        - Warning: Redundant bracket (35)
        - Warning: Use : (1)
        - Warning: Use String (4)
        - Warning: Use camelCase (10)
        - Warning: Use section (3)
      
      which are fixed by the current patch. Note that the 10 "Use camelCase"
      were all due to hlint not “knowing” the idiom of ‘case_’ (it does for
      ‘prop_’), for which I filled
      http://code.google.com/p/ndmitchell/issues/detail?id=558
      
      .
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarRené Nussbaumer <rn@google.com>
      5b11f8db
  17. 04 Sep, 2012 2 commits
  18. 03 Sep, 2012 4 commits
  19. 28 Aug, 2012 2 commits