GitLab

noded is designed to live on a normal node and hence
will not depend on any other daemon. In particular, it
is safe to start this daemon first. Starting it after
wconfd also allows wconfd to carry out a full voting.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. mark it as failed only if it's not finalized, and let the queue
management handle the cleanup.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The job is atomically moved from the list of running jobs into the list
of manipulated jobs, an action is performed on them, and then it's
returned.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Also add lenses for the fields.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Make the official (exported) getFQDN function honor the vcluster
setup. In this way, also Haskell daemons can obtain their name
correctly in a vcluster. In particular, voting will work.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Document that luxid does a voting before starting up. Also
document that this can be prevented by the --no-voting option,
confirmed by --yes-do-it.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

As luxid now starts jobs, make it verify that it
is running on the master node by carrying out the
Ganeti voting process.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

As the job handling is now done by luxid, that
daemon is also in charge of verifying that it runs
on the master node. So provide these options in
the Haskell world as well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

When voting for the master node, it might be that too many
nodes are temporarily not reachable, e.g., after a mass
reboot. In this case, we wait a given interval and retry.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...as it is only included in newer Versions of Data.Either
which are not available in all library versions Ganeti has
to work on.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...as it is also needed in the voting process.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

In Haskell, support the RPC call "master_node_name" by which a
node is asked who it believes is the master node. Luxid will need
this to carry out a voting.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Move the starting of the job scheduler to a later stage
in the startup. In particular, only start it after the
job-queue lock file is obtained.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

* The call to update the metadata is moved inside the
  'RunOsScriptsVirtualized' but before 'TemporaryDisks' because this
  call updates the instance configuration in the metadata daemon, and
  we want this configuration to be available when the helper VM
  starts, but we don't want it to contain information about the
  temporary disks.

* The other call to update the metadata is extracted from the 'else'
  branch to the top-level of the function because we want to update
  metadata independently of whether the instance has an OS.

* Disk paths must be overridden when running the instance inside a
  safe virtualized environment because, from the host perserspective,
  a disk path that looks like

    /srv/ganeti/file-storage/<instance>/e0105543-0979-4896-95c1-d2c477fafa9b.file.disk0

  should be presented to the instance as

    /dev/xvda (Xen)
    /dev/vda  (KVM)

  Therefore, we must pass the correct '/dev/*' disk paths to the safe
  virtualized environment when creating the environment file.  To
  achieve this, extend backend and RPC for export OS to allow disk
  paths to be overridden.

* The environment variable 'OS_SCRIPT' is introduce to tell the init
  script that runs inside the virtualized environment which OS script
  to run.  In this patch, the OS script to run is 'create_untrusted',
  but in the future it can be overridden for import, export, and
  rename.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

... which converts decimal numbers into '/dev/sda', '/dev/sdb', etc.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Extend export OS RPC and backend to allow the environment file that
goes in the OS archive to be overridden.  This will be necessary to
introduce new OS environment variables, as well as, override the disk
labels.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Making the temporary disk readonly has absolutely no advantages and it
has the disadvantage that it prevents the OS install helper from
saving temporary data to it, for example, when downloading the OS
install package which is necessary to run the OS scripts.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

...from the jobs code, as this functionality is now
carried out by wconfd. Also remove dead code resulting
from this removal.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Otherwise it was possible that a node was demoted (by calling noded),
its configuration copy removed, but before the configuration change took
place, it was distributed to the node again.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This ensures that the configuration is fully distributed and doesn't
change during the verification process.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

If a node is removed from the list of master candidates, we want to make
sure that after the operation the configuration will not be distributed
to the node any more. Without requiring full distribution, it's possible
that the node still receives configuration update from a previous call.

Also include the master node(s).
(While this shouldn't change without master failover, if it does, it's
definitely a good idea to flush the configuration.)
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. so that the unit tests work without the presence of WConfd.

Also fix mocked GenerateMAC, which used _temporary_ids by mistake in the
original ConfigWriter.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Also WConfd handles proper synchronization internally so there is no
need to explicitly lock the configuration in ConfigWriter.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Expose the generation method through WConfd's RPC interface.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This will be needed to properly generate and reserve new secrets.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. instead of just String. This is a minor step towards better
distinguishing distinct concepts with types.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

The same pattern that has been used for gathering DRBD minors will be
used for DRBD secrets etc.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. since it's a general, randomized function.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

.. and add a function for random secrets, just as we have in Python
code.
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Add flags '--install-image' and '--zeroing-image' to cluster init and
cluster modify man pages.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

... in 'gnt-cluster modify'
Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

To avoid further cluttering the already overly long Utils.hs
move the Statistics data type with its related functions into
a separate submodule. This also provides a better encapsulation
of this type where the Constructors are not exported.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

* stable-2.11
  (no changes)

* stable-2.10
  Use more efficient statistics for the standard deviation
  Use statistics updates when allocating on pairs
  Factor score computation through abstract statistics
  Verify the update of the standard deviation statistics
  Add data type for abstract statistics
  Relax test requirements
  Fix gnt-network client wrt instances report
  Fix QueryNetworks wrt instances

Conflicts:
	src/Ganeti/HTools/Cluster.hs: manually apply
	    patch b366d732 to stable-2.11 version
	src/Ganeti/Utils.hs: take all addition
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

* stable-2.10
  Use more efficient statistics for the standard deviation
  Use statistics updates when allocating on pairs
  Factor score computation through abstract statistics
  Verify the update of the standard deviation statistics
  Add data type for abstract statistics
  Relax test requirements
  Fix gnt-network client wrt instances report
  Fix QueryNetworks wrt instances

Conflicts:
	lib/cmdlib/network.py
Resolved by ignoring the change as the corresponding code
has been removed in 2.11.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Instead of using the full sample as statistics providing
enough information to compute the standard deviation, use
a slightly more elaborate one. It contains the standard
statistics count, sum, and sum of squares, which can also
very easily be updated. However, computing the standard
deviation from these values unfortunately is not precise
enough for our purpose (small difference of large values).
Therefore, we also carry the naively computed variance in
the statistics. The latter can also be updated using the
standard statistics and, moreover, this can be done in a
way that does not introduce too large inaccuracies for a
small number of updates.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

When considering the various ways of positioning an instance
on a pair of nodes, make use of the fact that the statistics
are extremely similar (only two nodes changed) and obtain the
new statistics by updating the old one, rather than by recomputing
it from scratch.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Logically separate the computation of the cluster score into
two steps: the computation of the abstract statistics and its
evaluation. In this way, we obtain an abstract value which we
can update instead of recomputing it when considering different
instance placements.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add a test that verifies that the error introduced by updating
a standard-deviation statistics of a sample with at least two
elements is not too large, as compared to the direct computation.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>