GitLab

Jobs usually are archived a given time after they have finished.
For finalized jobs without end-time, the start-time is taken en
lieu. This function provides the pure predicate for this decision.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

As jobs are archived in groups of 10000, creating new subdirectories
of the archive might be necessary when archiving a job. Use a
function that takes care of this.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...that also creates the target directory, if needed.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

On sucess, jobqueue_rename returns a list containing one
null per change request.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

With luxid taking over the tasks of masterd, archiving
jobs also belongs to its responsibilities. As archiving
a job affects the global state of the job queue, synchronise
over the queue lock.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Archiving jobs is also replicated to all master candidates.
Therefore luxid needs to be aware of this RPC call.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

When replicating parts of the job queue, allow for virtual
paths in the RPC call. In this way, replication will also
work correctly in a vcluster setup. Note that makeVirtualPath
lives in IO, and hence cannot be part of the pure encoding
function of the RPC.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Virtual clusters are an efficient way to test how Ganeti behaves
on a large cluster without requiring a large number of machines.
Now that more tasks like job replication are done by luxid, provide
that functionality in Haskell as well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...as, in that way, they will also be available in Haskell,
where job replication happens as well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Now that the number of jobs maximally running in parallel is
a run-time option, this magic constant is not needed any more.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Use the run-time configuration to decide on the number of jobs
scheduled for execution instead of using a hard-coded constant.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

In this way, scheduling decisions can depend on the configuration
of the cluster. At the moment, this is only the maximal number
jobs to be run in parallel, but in the future this will also include
job filters.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

As we have introduced a new cluster parameter, it should
be also visible when querying about the cluster configuration.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This parameter of OpClusterSetParams  will allow to set the
maximal number of jobs to be run simultaneously.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This cluster-wide parameter will determine how many non-finalized jobs maximally
should be in a not queued state at the same time.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

As luxid handles the job queue, this daemon is the natural
place to handle job cancellation. Answering to CancelJob requests
is also necessary for luxid to be feature compliant with masterd,
even for command-line requests only.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

When a job gets canceled while still queued, dequeuing requires
luxid to mark it as cancelled. So provide the necessary pure
function to do so.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Even after jobs have been handed over for execution, it might
still be possible to cancel them. On such case would be the
job still waiting for a lock. Eventually, we will have to
communicate to the job directly, but as long as execution is
still managed by masterd, ask masterd to cancel the job.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This only removes queued jobs from the queue
and indicates whether the job was found in the queue.
For jobs that are already started from the queue's
point of view, it might still be possible to cancel
them, e.g., if they are still waiting for locks.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Add user shutdown parameter for KVM.  Based on this parameter, decide
what information to report for a KVM instance, for example,
distinguish between 'ADMIN_down' and 'USER_down'.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Add KVM daemon entry point, command-line options, backgrounding, etc
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Add KVM daemon logic, which contains monitors for Qmp sockets and
directory/file watching.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Refactor module 'Ganeti.UDSServer' so the KVM daemon can reuse code
declared in this module to handle Unix domain sockets.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* add user and group to 'configure.ac', 'Makefile.am' and
  'AutoConf.hs.in'
* extend 'Daemon' datatype with 'GanetiKvmd' and implement all related
  functions, such as, 'daemonName', etc.
* export KVM daemon name as constant
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Fix whitespace in several modules.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

If the query fields don't require live data, we use the shortcut
and don't request live data. However, we cannot take this shortcut
if the fields the filter depends on requires live data.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Now that all jobs are monitored with inotify, increase the polling interval.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

In order to obtain a higher throughput of jobs, schedule new jobs
as soon as a job was detected to have finished.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Add a function that can serve as an event handler for inotify
updating a job in the job queue if the corresponding job file
changes. Also attach it to all jobs selected to be run.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

When attaching inotifies to jobs, we need to preserve
it through potential requeuing actions. Also, this information
is needed for cleaning up.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

When cleaning up finished jobs, remove the inotify
attached to them, if any.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

This provides the infrastructure to monitor running jobs
by inotify, and hence update the queue promptly upon
job changes.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Make luxid also handle queries to drain the job queue.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

As luxid is also responsible for handling requests to drain the job queue,
we need the corresponding RPC in Haskell as well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The drain flag is set, if the queue is not open.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

When watching a file, reinstantiate the inotify if notified
of an event that removes the watch. Such events are likely
to happen, as our usual way to "modify" a file is to atomically
replace it by another one.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Also log, at debug level only, when a change of a watched
file was observed, but the change did not result in any
change of derived value.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>