GitLab

No longer index nodes by their name but by their UUID in the cluster
config. This change changes large parts of the code, as the following
adjustments were necessary:
 * Change the index key to UUID in the configuration and the
   ConfigWriter, including all methods.
 * Change all cross-references to nodes to use UUID's.
 * External interfaces (command line interface, IAllocator interface,
   hook scripts, etc.) are kept stable.
 * RPC-calls can resolve UUID's as target node arguments, if the RPC
   runner is based on a ConfigWriter instance. The result dictionary is
   presented in the form the nodes are addressed: by UUID if UUID's were
   given, or by name if names were given.
 * Node UUID's are resolved in ExpandNames and then stored in the
   OpCode. This allows to check for node renames if the OpCode is
   reloaded after a cluster restart. This check is currently only done
   for single node parameters.
 * Variable names are renamed to follow the following pattern:
   - Suffix is 'node' or 'nodes': Variable holds Node objects
   - Suffix is 'name' or 'names': Variable holds node names
   - Suffix is 'uuid' or 'uuids': Variable holds node UUID's
 * Tests are adapted.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This fixes a 'FIXME' in the xen hypervisor. The instance
migration requires the cluster's name. This should come
from the RPC call, but so far was hackishly taken from
ssconf. Ssconf is intended to be used by scripts only and
not by the backend itself.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Currently, the VNC bind address can only be specified as an IP
address. Given that this is a cluster-wide parameter, there
aren't many sensible choices besides 127.0.0.1 and 0.0.0.0. This
patch allows to also specify it as the address of a specifed
interface.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Removes the option --with-xen-cmd from the configure
script.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch extend the RPC to powercycle a node by a
hvparams parameter. This parameter is used by the
xen hypervisor to determine which xen command to use.
The respective LU is adjusted as well.

Since this was the last operation depending on the
constants.XEN_CMD, it is removed eventually.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch removes the fallback to the xen command from the
auto config and adds an exception instead. Also, the
hvparams parameter of the functions GetCommand, _RunXen and
_GetInstanceList are made mandatory.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch extends the GetAllInstancesInfo method of the
hypervisors by a hvparams parameter. This change propagates
up to the RPC call all_instances_info, which also needed
to be extended. All callers of this call were adjusted
as well.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch extends the GetInstanceInfo function of the
hypervisors by an hvparams parameters. This leads to
an extension of the instance_info RPC call and the
adjustmend of all classes using this call.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This removes the fallback implementation of the backend's
GetNodeInfo function to the old implementation. By now,
all calling code is adjusted to the new hvparams parameter
and thus, the old implementation should not be necessary
anymore.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch contains all adjustments necessary to code that
uses the 'call_node_info' call. The adjustment is necessary
because the call's signature changed from receiving a list
of hypervisor names to a list of tuples
(hypervisor_name, hvparams).
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch adds the hvparams parameter to the RPC call
'node_info'. It also adjusts the related code in noded.py
and Query/Node.hs
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch extends the GetNodeInfo function of the backend
module by a hvparams parameter. For now, it contains a
fallback solution for when the parameter is None. This
fallback will be removed in a later patch of this series.
It was necessary to be abled to gradually adjust all
calling code without breaking everything at once.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch extends the GetNodeInfo function of the
hypervisors by a hvparams parameter. The parameter
is currently only used by the xen hypervisor to determine
which xen toolstack ('xm' or 'xl') to use.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Some methods still refered to 'xm', which is inaccurate now
that xen can also be run with 'xl'. No functional changes
otherwise.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch makes the watcher use the hvparams from ssconf
to retrieve the list of instances from the hypervisors.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

The watcher needs access to the cluster's hypervisor
parameters. This patch extends the ssconf infrastructure
by adding a hvparams file for each hypervisor. The format
of the hvparams file is a simple key=value format.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch makes use of hvparams on instance migration to
determine the list of instance before the migration. This way, in
xen, the xl/xm choice is respected.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch enhances node verification by:
- Factoring out the verification of the node info
  and the instance list to increase testability.
- Make both of them use the hvparams parameter.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch extends the node verification by:
- Adding a check for the xen toolstack when the hypervisor
  is verified.
- Factoring out the hypervisor verification in a
  subfunction to increase testability.
- Factoring out the hvparams verification in a
  subfunction for the same reason.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

The 'instance_list' call needs to be extended to accept
hvparams. This is one of the few RPCs where it is
necessary, because there is no instance given whose
hvparams could be reused.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

The backend module's function 'ListInstances' is called
within the module several times. In many cases, it is
possible to reuse the instance's hvparams for the
'ListInstances' call and thus avoiding having to extend
RPC calls.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

In same cases, the call 'GetInstanceList' is only done for
one hypervisor. In those cases, it is more convenient to
not provide the full collection of hypervisor parameters
for all hypervisors, but only the ones for this hypervisor.
This patch factors out this particular case.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Propagating the use of hvparams further up the calling
hierarchy, this patch makes the backend module use the
hvparams in 'GetInstanceList'. Unit tests provided.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Propagating the use of hvparams further up in the calling
hierarchy, the functions '_RunXen' and 'GetInstanceList'
use the given hvparams. Unit tests provided. Calling function
'StopInstance' adjusted in this patch as well.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Since one can retrieve xen's instance list with both, 'xm'
or 'xl', the function name 'RunXmList' is no longer
appropriate. Renaming it to 'RunInstanceList'. No
functional changes otherwise.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

Since one can retrieve xen's instance list via both 'xm'
or 'xl', it is no longer appropriate to call the function
'GetXmList'. This patch renames it to 'GetInstanceList'.
Other than that, there are no functional changes.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch adds a (so far optional) hvparams parameter
to the '_GetCommand' function. This dictionary is used
to retrieve the xen command (xm or xl). It is optional
for now to make the refactoring possible without breaking
everything at once.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch adds unit tests for the current state of the
'GetCommand' method. The tests are rather trivial, but
having tests before starting to refactor seems to be
a good idea.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This patch adds the xen command (xm, xl) as hypervisor
parameter to the constants and adds validation to
the xen hypervisors.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

The introductory comment of hv_xen_unittest.py referred to
the wrong hypervisor.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

As opposed to SubmitOrSend, the SubmitOpCode function is
called also by commands not supporting submit options. Hence
only inspect these options, if they are actually present.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

When submitting a job, the job id, prefixed by 'JobID: ', is output to
stderr. While this is easy to understand for humans, for scripting, it
is more convenient to have (additionally) the id on stdout (and only
the id). Add an option to provide this.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

This command tacitly waits for a job to finish. In that
way, job dependency can also be handled in shell scripts.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Even though we need the let-bound variable showMoves only
at type [(String, String)] -> IO (), it's most general type
would be (PrintfArg a, PrintfArg b) => [(a, b)] -> IO ().
This causes the monomorphism restriction apply to that binding,
which is a warning (and, thanks to -Werror, fatal) on some
ghc versions.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

If non-redundant instances are present in the cluster, hroller will
plan for them to move to other nodes while the group is rebooted.
This adds an option to also show this plan.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

hspace is run on clusters with exclusive storage enabled.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Statistics about spindles are tracked. In human-readable output, spindles
are printed only when used (i.e., exclusive storage is enabled). For
machine-oriented output, they are always there.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This makes tiered allocation in hspace work also with respect to spindles.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Spindles are now part of resource spec. Instances get created with spindles
specified (which are just ignored when exclusive storage is disabled).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

When exclusive storage is enabled, spindles are used instead of disk space
to compute the cluster score.

Comments and variable names in computePDsk has been changed to match the
actual code.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>