1. 28 Dec, 2009 1 commit
  2. 25 Sep, 2009 1 commit
  3. 07 Sep, 2009 1 commit
    • Iustin Pop's avatar
      Optimise multi-job submit · 009e73d0
      Iustin Pop authored
      Currently, on multi-job submits we simply iterate over the
      single-job-submit function. This means we grab a new serial, write and
      replicate (and wait for the remote nodes to ack) the serial file, and
      only then create the job file; this is repeated N times, once for each
      Since job identifiers are ‘cheap’, it's simpler to simply grab at the
      start a block of new IDs, write and replicate the serial count file a
      single time, and then proceed with the jobs as before. This is a cheap
      change that reduces I/O and reduces slightly the CPU consumption of the
      master daemon: submit time seems to be cut in half for big batches of
      jobs and the masterd cpu time by (I can't get consistent numbers)
      between 15%-50%.
      Note that this doesn't change anything for single-job submits and most
      probably for < 5 job submits either.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
  4. 03 Aug, 2009 2 commits
  5. 19 Jul, 2009 4 commits
    • Iustin Pop's avatar
      job queue: fix loss of finalized opcode result · 34327f51
      Iustin Pop authored
      Currently, unclean master daemon shutdown overwrites all of a job's
      opcode status and result with error/None. This is incorrect, since the
      any already finished opcode(s) should have their status and result
      preserved, and only not-yet-processed opcodes should be marked as
      ‘error’. Cancelling jobs between opcodes does the same (but this is not
      allowed currently by the code, so it's not as important as unclean
      This patch adds a new _QueuedJob function that only overwrites the
      status and result of finalized opcodes, which is then used in job queue
      init and in the cancel job functions. The patch also adds some comments
      and a new set constants in constants.py highlighting the finalized vs.
      non-finalized opcode statuses.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
    • Iustin Pop's avatar
      Add a luxi call for multi-job submit · 56d8ff91
      Iustin Pop authored
      As a workaround for the job submit timeouts that we have, this patch
      adds a new luxi call for multi-job submit; the advantage is that all the
      jobs are added in the queue and only after the workers can start
      processing them.
      This is definitely faster than per-job submit, where the submission of
      new jobs competes with the workers processing jobs.
      On a pure no-op OpDelay opcode (not on master, not on nodes), we have:
        - 100 jobs:
          - individual: submit time ~21s, processing time ~21s
          - multiple:   submit time 7-9s, processing time ~22s
        - 250 jobs:
          - individual: submit time ~56s, processing time ~57s
                        run 2:      ~54s                  ~55s
          - multiple:   submit time ~20s, processing time ~51s
                        run 2:      ~17s                  ~52s
      which shows that we indeed gain on the client side, and maybe even on
      the total processing time for a high number of jobs. For just 10 or so I
      expect the difference to be just noise.
      This will probably require increasing the timeout a little when
      submitting too many jobs - 250 jobs at ~20 seconds is close to the
      current rw timeout of 60s.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
      (cherry picked from commit 2971c913)
    • Iustin Pop's avatar
      job queue: fix interrupted job processing · f6424741
      Iustin Pop authored
      If a job with more than one opcodes is being processed, and the master
      daemon crashes between two opcodes, we have the first N opcodes marked
      successful, and the rest marked as queued. This means that the overall
      jbo status is queued, and thus on master daemon restart it will be
      resent for completion.
      However, the RunTask() function in jqueue.py doesn't deal with
      partially-completed jobs. This patch makes it simply skip such opcodes.
      An alternative option would be to not mark partially-completed jobs as
      QUEUED but instead RUNNING, which would result in aborting of the job at
      restart time.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
    • Iustin Pop's avatar
      Fix an error path in job queue worker's RunTask · ed21712b
      Iustin Pop authored
      In case the job fails, we try to set the job's run_op_idx to -1.
      However, this is a wrong variable, which wasn't detected until the
      __slots__ addition. The correct variable is run_op_index.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
  6. 17 Jul, 2009 1 commit
  7. 07 Jul, 2009 1 commit
  8. 12 Feb, 2009 1 commit
    • Iustin Pop's avatar
      job queue: log the opcode error too · 0f6be82a
      Iustin Pop authored
      Currently we only log "Error in opcode ...", but we don't log the error itself.
      This is not good for debugging.
      Reviewed-by: ultrotter
  9. 28 Jan, 2009 1 commit
    • Iustin Pop's avatar
      Fix some issues related to job cancelling · df0fb067
      Iustin Pop authored
      This patch fixes two issues with the cancel mechanism:
        - cancelled jobs show as such, and not in error state (we mark them as
        - queued jobs which are cancelled don't raise errors in the master (we
          treat OP_STATUS_CANCELED now)
      Reviewed-by: imsnah
  10. 27 Jan, 2009 1 commit
  11. 20 Jan, 2009 1 commit
    • Iustin Pop's avatar
      Update the logging output of job processing · d21d09d6
      Iustin Pop authored
      (this is related to the master daemon log)
      Currently it's not possible to follow (in the non-debug runs) the
      logical execution thread of jobs. This is due to the fact that we don't
      log the thread name (so we lose the association of log messages to jobs)
      and we don't log the start/stop of job and opcode execution.
      This patch adds a new parameter to utils.SetupLogging that enables
      thread name logging, and promotes some log entries from debug to info.
      With this applied, it's easier to understand which log messages relate
      to which jobs/opcodes.
      The patch also moves the "INFO client closed connection" entry to debug
      level, since it's not a very informative log entry.
      Reviewed-by: ultrotter
  12. 15 Jan, 2009 1 commit
    • Iustin Pop's avatar
      Some docstring updates · 25e7b43f
      Iustin Pop authored
      This patch rewraps some comments to shorter lengths, changes
      double-quotes to single-quotes inside triple-quoted docstrings for
      better editor handling.
      It also fixes some epydoc errors, namely invalid crossreferences (after
      method rename), documentation for inexistent (removed) parameters, etc.
      Reviewed-by: ultrotter
  13. 18 Dec, 2008 5 commits
  14. 17 Dec, 2008 1 commit
    • Michael Hanselmann's avatar
      Add job queue size limit · f87b405e
      Michael Hanselmann authored
      A job queue with too many jobs can increase memory usage and/or make
      the master daemon slow. The current limit is just an arbitrary number.
      A "soft" limit for automatic job archival is prepared.
      Reviewed-by: iustinp
  15. 14 Dec, 2008 1 commit
  16. 11 Dec, 2008 1 commit
    • Iustin Pop's avatar
      Fix epydoc format warnings · c41eea6e
      Iustin Pop authored
      This patch should fix all outstanding epydoc parsing errors; as such, we
      switch epydoc into verbose mode so that any new errors will be visible.
      Reviewed-by: imsnah
  17. 02 Dec, 2008 1 commit
    • Iustin Pop's avatar
      Restrict job propagation to master candidates only · 59303563
      Iustin Pop authored
      This patch restricts the job propagation to master candidates only, by
      not registering non-candidates in the job queue node lists.
      Note that we do intentionally purge the job queue if a node is toggled
      to non-master status.
      Reviewed-by: imsnah
  18. 28 Nov, 2008 2 commits
  19. 27 Nov, 2008 1 commit
  20. 26 Nov, 2008 2 commits
  21. 12 Nov, 2008 1 commit
  22. 27 Oct, 2008 2 commits
  23. 20 Oct, 2008 1 commit
    • Iustin Pop's avatar
      Convert the job queue rpcs to address-based · 99aabbed
      Iustin Pop authored
      The two main multi-node job queue RPC calls (jobqueue_update,
      jobqueue_rename) are converted to address-based calls, in order to speed
      up queue changes. For this, we need to change the _nodes attribute on
      the jobqueue to be a dict {name: ip}, instead of a set.
      Reviewed-by: imsnah
  24. 16 Oct, 2008 2 commits
    • Iustin Pop's avatar
      Fix job queue behaviour when loading jobs · 94ed59a5
      Iustin Pop authored
      Currently, if loading a job fails, the job queue code raises an
      exception and prevents the proper processing of the jobs in the queue.
      We change this so that unparseable jobs are instead archived (if not
      Reviewed-by: imsnah
    • Iustin Pop's avatar
      Add an interface for the drain flag changes/query · 3ccafd0e
      Iustin Pop authored
      This adds the set/reset in the jqueue and luxi modules, and a way to
      query it in OpQueryConfigValues, and also the comand line interface for
      $ gnt-cluster queue info
      The drain flag is unset
      $ gnt-cluster queue drain
      $ gnt-cluster queue info
      The drain flag is set
      $ gnt-cluster queue undrain
      $ gnt-cluster queue info
      The drain flag is unset
      The choice of making the setting via luxi and not an opcode is that
      opcodes can't be executed when drained, but we don't query via luxi
      since in the future it might become a cluster property as opposed to a
      node one.
      Reviewed-by: imsnah
  25. 15 Oct, 2008 1 commit
    • Iustin Pop's avatar
      Implement the job queue drain flag · 686d7433
      Iustin Pop authored
      We add a (per-node) queue drain flag that blocks new job submission.
      There is not yet an interface to add/remove the flag (will come in next
      Reviewed-by: imsnah
  26. 10 Oct, 2008 1 commit
    • Iustin Pop's avatar
      Convert rpc module to RpcRunner · 72737a7f
      Iustin Pop authored
      This big patch changes the call model used in internode-rpc from
      standalong function calls in the rpc module to via a RpcRunner class,
      that holds all the methods. This can be used in the future to enable
      smarter processing in the RPC layer itself (some quick examples are not
      setting the DiskID from cmdlib code, but only once in each rpc call,
      There are a few RPC calls that are made outside of the LU code, and
      these calls are left as staticmethods, so they can be used without a
      class instance (which requires a ConfigWriter instance).
      Reviewed-by: imsnah
  27. 07 Oct, 2008 1 commit
    • Iustin Pop's avatar
      Implement job 'waiting' status · e92376d7
      Iustin Pop authored
      Background: when we have multiple jobs in the queue (more than just a
      few), many of the jobs (up to the number of threads) will be in state
      'running', although many of them could be actually blocked, waiting for
      some locks. This is not good, as one cannot easily see what is
      The patch extends the opcode/job possible statuses with another one,
      waiting, which shows that the LU is in the acquire locks phase. The
      mechanism for doing so is simple, we initialize (in the job queue) the
      opcode with OP_STATUS_WAITLOCK, and when the processor is ready to give
      control to the LU's Exec, it will call a notifier back into the
      _JobQueueWorker that sets the opcode status to OP_STATUS_RUNNING (with
      the proper queue locking). Because this mechanism does not save the job,
      all opcodes on disk will be in status WAITLOCK and not RUNNING anymore,
      so we also change the load sequence to consider WAITLOCK as RUNNING.
      With the patch applied, creating in parallel (via burnin) five instances
      on a five node cluster shows that only two are executing, while three
      are waiting for locks.
      Reviewed-by: imsnah
  28. 06 Oct, 2008 1 commit
    • Iustin Pop's avatar
      Implement job auto-archiving · 07cd723a
      Iustin Pop authored
      This patch adds a new luxi call that implements auto-archiving of jobs
      older than a certain age (or -1 for all completed jobs), and the gnt-job
      command that makes use of this (with 'all' for -1).
      Reviewed-by: imsnah