• Iustin Pop's avatar
    Add support for job queries in hconfd · a7e484c4
    Iustin Pop authored
    This adds support for job queries, including (basic) unit-tests.
    
    I've tested this for memory and cpu usage as follows:
    
    - 3600 jobs (live queue):
      - via masterd, default: ~1.1s (masterd: ~60MB ram)
      - via confd,   default: ~1.1s (hconfd:  ~25MB ram)
      - via masterd, id only: ~1.0s (masterd: ~57MB ram)
      - via confd,   id only: ~0.2s (hconfd:  ~15MB ram)
    
    - all jobs (128K in total, around 570MB size on disk):
      - via masterd, default: 1m22s (masterd cpu: 48s), masterd: 1.4G ram
      - via confd,   default: 1m16s (hconfd  cpu: 51s), hconfd:  570MB ram peak
        (peak only right before starting luxi send, hconfd decreases in RSS as
        results are streamed over luxi, back to ~18MB after the send)
      - via masterd, id only:  ~49s (masterd cpu: 45s), masterd: 1.3G ram
      - via confd,   id only:  ~39s (hconfd cpu: 35s),  hconfd:  110MB ram peak
        (right before luxi send, decreasing as results are sent, back to ~14MB
        after the send)
    
    Given this, and that in production it's not likely to have hundreds of
    thousand of job files, I believe the implementation is safe from this
    point of view.
    Signed-off-by: default avatarIustin Pop <iustin@google.com>
    Reviewed-by: default avatarHelga Velroyen <helgav@google.com>
    a7e484c4
Query.hs 12 KB