Skip to content
  • Iustin Pop's avatar
    Add a luxi call for multi-job submit · 2971c913
    Iustin Pop authored
    
    
    As a workaround for the job submit timeouts that we have, this patch
    adds a new luxi call for multi-job submit; the advantage is that all the
    jobs are added in the queue and only after the workers can start
    processing them.
    
    This is definitely faster than per-job submit, where the submission of
    new jobs competes with the workers processing jobs.
    
    On a pure no-op OpDelay opcode (not on master, not on nodes), we have:
      - 100 jobs:
        - individual: submit time ~21s, processing time ~21s
        - multiple:   submit time 7-9s, processing time ~22s
      - 250 jobs:
        - individual: submit time ~56s, processing time ~57s
                      run 2:      ~54s                  ~55s
        - multiple:   submit time ~20s, processing time ~51s
                      run 2:      ~17s                  ~52s
    
    which shows that we indeed gain on the client side, and maybe even on
    the total processing time for a high number of jobs. For just 10 or so I
    expect the difference to be just noise.
    
    This will probably require increasing the timeout a little when
    submitting too many jobs - 250 jobs at ~20 seconds is close to the
    current rw timeout of 60s.
    
    Signed-off-by: default avatarIustin Pop <iustin@google.com>
    Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
    2971c913