diff --git a/doc/design-2.0-job-queue.rst b/doc/design-2.0-job-queue.rst new file mode 100644 index 0000000000000000000000000000000000000000..c711a3234a5fd65d0975589e022afe92a01433c8 --- /dev/null +++ b/doc/design-2.0-job-queue.rst @@ -0,0 +1,135 @@ +Job Queue +========= + +.. contents:: + +Overview +-------- + +In Ganeti 1.2, operations in a cluster have to be done in a serialized way. +Virtually any operation locks the whole cluster by grabbing the global lock. +Other commands can't return before all work has been done. + +By implementing a job queue and granular locking, we can lower the latency of +command execution inside a Ganeti cluster. + + +Detailed Design +--------------- + +Job executionββLife of a Ganeti jobβ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +#. Job gets submitted by the client. A new job identifier is generated and + assigned to the job. The job is then automatically replicated to all nodes + in the cluster. The identifier is returned to the client. +#. A pool of worker threads waits for new jobs. If all are busy, the job has + to wait and the first worker finishing its work will grab it. Otherwise any + of the waiting threads will pick up the new job. +#. Client waits for job status updates by calling a waiting RPC function. + Log message may be shown to the user. Until the job is started, it can also + be cancelled. +#. As soon as the job is finished, its final result and status can be retrieved + from the server. +#. If the client archives the job, it gets moved to a history directory. + This could also be done regularily using a cron script. + + +Queue structure +~~~~~~~~~~~~~~~ + +All file operations have to be done atomically by writing to a temporary file +and subsequent renaming. Except for log messages, every change in a job is +stored and replicated to other nodes. + +:: + + /var/lib/ganeti/queue/ + job-1 (JSON encoded job description and status) + [β¦] + job-37 + job-38 + job-39 + lock (Queue managing process opens this file in exclusive mode) + serial (Last job ID used) + version (Queue format version) + + +Locking +~~~~~~~ + +Locking in the job queue is a complicated topic. It is called from more than +one thread and must be thread-safe. For simplicity, a single lock is used for +the whole job queue. + +A more detailed description can be found in doc/locking.txt. + + +Internal RPC +~~~~~~~~~~~~ + +RPC calls available between Ganeti master and node daemons: + +jobqueue_update(file_name, content) + Writes a file in the job queue directory. +jobqueue_purge() + Cleans the job queue directory completely, including archived job. +jobqueue_rename(old, new) + Renames a file in the job queue directory. + + +Client RPC +~~~~~~~~~~ + +RPC between Ganeti clients and the Ganeti master daemon supports the following +operations: + +SubmitJob(ops) + Submits a list of opcodes and returns the job identifier. The identifier is + guaranteed to be unique during the lifetime of a cluster. +WaitForJobChange(job_id, fields, [β¦], timeout) + This function waits until a job changes or a timeout expires. The condition + for when a job changed is defined by the fields passed and the last log + message received. +QueryJobs(job_ids, fields) + Returns field values for the job identifiers passed. +CancelJob(job_id) + Cancels the job specified by identifier. This operation may fail if the job + is already running, canceled or finished. +ArchiveJob(job_id) + Moves a job into the β¦/archive/ directory. This operation will fail if the + job has not been canceled or finished. + + +Job and opcode status +~~~~~~~~~~~~~~~~~~~~~ + +Each job and each opcode has, at any time, one of the following states: + +Queued + The job/opcode was submitted, but did not yet start. +Running + The job/opcode is running. +Canceled + The job/opcode was canceled before it started. +Success + The job/opcode ran and finished successfully. +Error + The job/opcode was aborted with an error. + +If the master is aborted while a job is running, the job will be set to the +Error status once the master started again. + + +History +~~~~~~~ + +Archived jobs are kept in a separate directory, /var/lib/ganeti/queue/archive/. +The idea is to speed up the queue handling. + + +Ganeti updates +~~~~~~~~~~~~~~ + +The queue has to be completely empty for Ganeti updates with changes in the job +queue structure.