1. 29 Dec, 2010 1 commit
  2. 15 Dec, 2010 2 commits
    • Michael Hanselmann's avatar
      jqueue: Keep jobs in “waitlock” while returning to queue · 5fd6b694
      Michael Hanselmann authored
      
      
      Iustin Pop reported that a job's file is updated many times while it
      waits for locks held by other thread(s). After an investigation it was
      concluded that the reason was a design decision for job priorities to
      return jobs to the “queued” status if they couldn't acquire all locks.
      Changing a jobs' status or priority requires an update to permanent
      storage.
      
      In a high-level view this is what happens:
      1. Mark as waitlock
      2. Write to disk as permanent storage (jobs left in this state by a
         crashing master daemon are resumed on restart)
      3. Wait for lock (assume lock is held by another thread)
      4. Mark as queued
      5. Write to disk again
      6. Return to workerpool
      
      Another option originally discussed was to leave the job in the
      “waitlock” status. Ignoring priority changes, this is what would happen:
      1. If not in waitlock
      1.1. Assert state == queued
      1.2. Mark as waitlock
      1.3. Set start_timestamp
      1.4. Write to disk as permanent storage
      3. Wait for locks (assume lock is held by another thread)
      4. Leave in waitlock
      5. Return to workerpool
      
      Now let's assume the lock is released by the other thread:
      […]
      3. Wait for locks and get them
      4. Assert state == waitlock
      5. Set state to running
      6. Set exec_timestamp
      7. Write to disk
      
      As this change reduces the number of writes from two per lock acquire
      attempt to two per opcode and one per priority increase (as happens
      after 24 acquire attempts (see mcpu._CalculateLockAttemptTimeouts) until
      the highest priority is reached), here's the patch to implement it.
      Unittests are updated.
      Signed-off-by: default avatarMichael Hanselmann <hansmi@google.com>
      Reviewed-by: default avatarIustin Pop <iustin@google.com>
      5fd6b694
    • Michael Hanselmann's avatar
      Improve jqueue unittests · ebb2a2a3
      Michael Hanselmann authored
      
      
      - Verify job file updates
      - Ensure queue lock is released while executing opcode
      Signed-off-by: default avatarMichael Hanselmann <hansmi@google.com>
      Reviewed-by: default avatarIustin Pop <iustin@google.com>
      ebb2a2a3
  3. 11 Nov, 2010 1 commit
  4. 28 Oct, 2010 3 commits
  5. 22 Oct, 2010 2 commits
  6. 20 Oct, 2010 1 commit
  7. 13 Oct, 2010 2 commits
  8. 12 Oct, 2010 1 commit
  9. 07 Oct, 2010 1 commit
  10. 05 Oct, 2010 2 commits
  11. 30 Sep, 2010 1 commit
    • Iustin Pop's avatar
      Abstract OS name/variant functions · 870dc44c
      Iustin Pop authored
      
      
      Currently, the computation of the 'pure' name or the variant is
      hardcoded and spread around the functions that need it. This is not
      nice, and in the future we'd spread it even more with more usage of
      variants/pure os names.
      
      This patch abstracts these functions into the OS class, and then
      replaces the hardcoded uses with the new functions.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
      870dc44c
  12. 29 Sep, 2010 1 commit
  13. 24 Sep, 2010 2 commits
  14. 23 Sep, 2010 2 commits
  15. 22 Sep, 2010 1 commit
  16. 20 Sep, 2010 1 commit
    • Michael Hanselmann's avatar
      jqueue: Change model from per-job to per-opcode processing · be760ba8
      Michael Hanselmann authored
      
      
      In order to support priorities, the processing of jobs needs to be
      changed. Instead of processing jobs as a whole, the code is changed to
      process one opcode at a time and then return to the queue. See the
      Ganeti 2.3 design document for details.
      
      This patch does not yet use priorities for acquiring locks.
      
      The enclosed unittests increase the test coverage of jqueue.py from
      about 34% to 58%. Please note that they also test some parts not added
      by this patch, but testing them became only possible with some
      infrastructure added by this patch. For the first time, many
      implications and assumptions for the job queue are codified in these
      unittests.
      Signed-off-by: default avatarMichael Hanselmann <hansmi@google.com>
      Reviewed-by: default avatarIustin Pop <iustin@google.com>
      be760ba8
  17. 16 Sep, 2010 2 commits
  18. 13 Sep, 2010 3 commits
  19. 07 Sep, 2010 3 commits
  20. 06 Sep, 2010 1 commit
  21. 27 Aug, 2010 1 commit
  22. 25 Aug, 2010 1 commit
  23. 24 Aug, 2010 4 commits
  24. 23 Aug, 2010 1 commit