GitLab

Now that mandatory opcode fields are mandatory also on disk,
document that, once we use soft drains during upgrades, the disk
representation of the job queue needs to be updated by cfgupgrade
as well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

In order to guarantee all mandatory slots to be present, add
a custom constructor to BaseOpCode adding those fields with
default value, instead of inheriting the constructor from
outils.ValidatedSlots.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

While it's generally better to beg for forgiveness than ask for
permission in Python, it's not a good idea to rely on exceptions to
model control flow, especially if the resulting code is simpler.
Signed-off-by: Santi Raffa <rsanti@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

DTS_FILEBASED is a constant that exists and this commit makes sure
that it is used whenever sensible, rather than resorting to hardcoding
the pair of templates in very many files.
Signed-off-by: Santi Raffa <rsanti@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

As the responsibility for writing the job queue changed
to luxid, make masterd call to luxid when it is necessary
to write a job.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

As the base parameters also includes the reason trail,
set it, if not set already, together with the other
base parameters in the case of an opcode expanding to
a list of opcodes.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Handle this request by writing the jobs to the
queue and inform masterd; masterd will then also
distribute the jobs to all master candidates.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

As luxid is to take over responsibility for the job queue,
handle this request by writing the job to the queue and then
informing masterd; masterd will also distribute the job to
all master candidates.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Adding jobs to the queue is only allowed if the queue
is not drained.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Since luxid is going to write to the job queue,
it needs to honor drains of the queue as well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

During the transition to the new daemon layout, from step 2
onwards, luxid will write to the queue but masterd will trigger
the execution. Therefore, add a new luxi request to tell masterd
to pick up a job that has already been written to the queue.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

The current restart procedure for masterd includes functionality
to pick up a job from the queue and restart it, if it hasn't been
started before. Move this functionality into a separate function
to be able to have the enqueuing be done by luxid.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Two avoid two processes simultaneously accessing the same
on-file structure, like the job queue, file locks are used.
Therefore, provide this functionality in Haskell as well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

To avoid several processes accessing the queue at the same time,
Ganeti locks the queue via a lock file on disk. Provide the path
to this file.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This function writes a (non-archived) job to disk. The file
name can be computed from the job id, which is part of the job.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Add utility functions to allocate new job ids by increasing
the value stored in the serial file. As this function is
used in a multi-threaded program, synchronize access over
an internal lock.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

With luxi daemon taking over part of the job queue management, it will
also be responsible for replicating the queue to all master candidates.
Therefore, add a function to extract the list of master candidates from
the configuration.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

To be able to replicate the job queue, in particular the serial,
luxid needs to be able to send the jobqueue_update RPC. So add
its definition.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This version of 'the' properly lives in the 'Result' monad,
as opposed to traditional one calling 'error'. The reason
why it is 'Bad' that not precisely one element is returned
is given as argument.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This methods allows reading the maximal job serial number from
disk.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This function handles the pure part of generating a job,
i.e., assuming the job id already assigned and not setting
time stamps.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

When queueing many jobs, the dependencies between them need to
be resolved with the knowledge of their respective job id.
Lift the computation of the absolute dependency to the level
of MetaOpCodes.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

SubmitManyJobs also accepts jobs with dependencies given as
relative ids. Together with the absolute id of the job, once
assigned, the dependency can be resolved. Add a function doing
this computation.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

When generating jobs from sequences of op-codes, it is necessary
to wrap op-codes into queued form.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This function allows to use 'IO a' objects in a safe
way, using the 'try' function; the outcome is reported
as a 'Result'. IOErrors are logged and the result is
'Bad', while in the case of no errors, a result-yielding
function is applied to the value.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

To keep our on-file data consistent at any moment, we change
file contents by atomically replacing the file with a new one.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

See issue 235. It turned out the feature is a bit more complex than expected,
hence this design doc.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The idea is to compile (on demand, that is, when necessary) each
source file to a normal object file, a coverage object file, a
profiling object file, and a test object file.  Also, a given Haskell
binary is linked with the proper object files.  This is achieved with
the following Makefile variables:

Compilation modes (profiles):
1. HPROFILE enables/disables profiling
2. HCOVERAGE enables/disables coverage
3. HTEST enables/disables 'TEST' preprocessor definition

A few words on testing: testing means the problem described in issue
535: https://code.google.com/p/ganeti/issues/detail?id=535.  With
HTEST enable, ghc will be instructed to define the preprocessor
definition TEST, for modules that use '#ifdef TEST'.

Haskell binary targets fetch the proper dependencies.  They are also
'.PHONY' targets so that 'make' does not check for the file timestamp
and instead it will always call 'ghc --make ...'.  This is not a
problem because 'ghc' with the '--make' flag only compiles the
necessary object files.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

In the configuration, the master node is now given
by its uuid. Therefore, compare the uuid and not the
name to find out if a given node is the master.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* stable-2.10
  Fix documentation
  Replace all constant definitions with re-exports
  Prepare constants for automatic reexport
  Hs2Py constants: 'hvsParameterTypes' and 'hvsParameters'
  Fix indentation that triggers PEP8 error
  Check if hotplug is supported in CheckPrereq

* stable-2.9
  Revision bump for 2.9.0
  Schedule release of 2.9.0

* stable-2.8
  Improve error message for replace-disks
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* stable-2.9
  Revision bump for 2.9.0
  Schedule release of 2.9.0

* stable-2.8
  Improve error message for replace-disks

Conflicts:
	NEWS: trivial
	configure.ac: drop suffix bump of stable-2.9
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

...and mention the last change pulled in from stable-2.8.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* stable-2.8
  Improve error message for replace-disks

Conflicts:
	lib/cmdlib/instance_storage.py
Resolved by manually applying the node name to uuid
transition on the version of stable-2.9.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

In some conditions, replace-disks will fail if the disks are not properly
activated. Improve the error message suggesting to run activate-disks before
executing replace-disks.

Fixes Issue 606.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Fix documentation in constants containing values in seconds.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Remove all constant definitions in 'lib/constants.py' and re-export
the module 'lib/_constants.py', which contains the generated
constants.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* separate constants and non-constants in 'lib/constants.py' to make
  it simple to replace their re-export definitions with a single
  re-export statement at the begining of the module

* fix access 'CONFD_REQFIELD_*' related constants in
  'tools/confd-client' to convert them to 'String', rather then doing
  it in 'lib/constants.py' in order to allow the replacement of the
  re-export definitions with a single re-export statement
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Add constants 'hvsParameters' and 'hvsParameterTypes' to the Haskell
to Python constant generation.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

 * Use different visual marks at the beginning of lines (such as "<<<<" for
    starting a test and ">>>>" for finishing it)

 * Show more specifically if a test fails/passes.

 * If stdout is a terminal, use colors to distinguish different messages
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>