Design document for hsqueeze

This design document describes a new htool, hsqueeze, to
be used to move the externally-mirrored instances in such a
way that as few nodes as possible host instances at all.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Martin Zobel-Helas <zobel@debian.org>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Makefile.am

@@ -538,6 +538,7 @@ docinput = \
 	doc/design-hroller.rst \
 	doc/design-storagetypes.rst \
         doc/design-upgrade.rst \
+	doc/design-hsqueeze.rst \
 	doc/devnotes.rst \
 	doc/glossary.rst \
 	doc/hooks.rst \

doc/design-draft.rst

@@ -23,6 +23,7 @@ Design document drafts
    design-ceph-ganeti-support.rst
    design-upgrade.rst
    design-daemons.rst
+   design-hsqueeze.rst
 
 .. vim: set textwidth=72 :
 .. Local Variables:

doc/design-hsqueeze.rst

+=============
+HSqueeze tool
+=============
+
+.. contents:: :depth: 4
+
+This is a design document detailing the node-freeing scheduler, HSqueeze.
+
+
+Current state and shortcomings
+==============================
+
+Externally-mirrored instances can be moved between nodes at low
+cost. Therefore, it is attractive to free up nodes and power them down
+at times of low usage, even for small periods of time, like nights or
+weekends.
+
+Currently, the best way to find out a suitable set of nodes to shut down
+is to use the property of our balancedness metric to move instances
+away from drained nodes. So, one would manually drain more and more
+nodes and see, if `hbal` could find a solution freeing up all those
+drained nodes.
+
+
+Proposed changes
+================
+
+We propose the addition of a new htool command-line tool, called
+`hsqueeze`, that aims at keeping resource usage at a constant high
+level by evacuating and powering down nodes, or powering up nodes and
+rebalancing, as appropriate. By default, only externally-mirrored
+instances are moved, but options are provided to additionally take
+DRBD instances (which can be moved without downtimes), or even all
+instances into consideration.
+
+Tagging of standy nodes
+-----------------------
+
+Powering down nodes that are technically healthy effectively creates a
+new node state: nodes on standby. To avoid further state
+proliferation, and as this information is only used by `hsqueeze`,
+this information is recorded in node tags. `hsqueeze` will assume
+that offline nodes having a tag with prefix `htools:standby:` can
+easily be powered on at any time.
+
+Minimum available resources
+---------------------------
+
+To keep the squeezed cluster functional, a minimal amount of resources
+will be left available on every node. While the precise amount will
+be specifiable via command-line options, a sensible default is chosen,
+like enough resource to start an additional instance at standard
+allocation on each node. If the available resources fall below this
+limit, `hsqueeze` will, in fact, try to power on more nodes, till
+enough resources are available, or all standy nodes are online.
+
+To avoid flapping behavior, a second, higher, amount of reserve
+resources can be specified, and `hsqueeze` will only power down nodes,
+if after the power down this higher amount of reserve resources is
+still available.
+
+Computation of the set to free up
+---------------------------------
+
+To determine which nodes can be powered down, `hsqueeze` basically
+follows the same algorithm as the manual process. It greedily goes
+through all non-master nodes and tries if the algorithm used by `hbal`
+would find a solution (with the appropriate move restriction) that
+frees up the extended set of nodes to be drained, while keeping enough
+resources free. Being based on the algorithm used by `hbal`, all
+restrictions respected by `hbal`, in particular memory reservation
+for N+1 redundancy, are also respected by `hsqueeze`.
+The order in which the nodes are tried is choosen by a
+suitable heuristics, like trying the nodes in order of increasing
+number of instances; the hope is that this reduces the number of
+instances that actually have to be moved.
+
+If the amount of free resources has fallen below the lower limit,
+`hsqueeze` will determine the set of nodes to power up in a similar
+way; it will hypothetically add more and more of the standby
+nodes (in some suitable order) till the algorithm used by `hbal` will
+finally balance the cluster in a way that enough resources are available,
+or all standy nodes are online.
+
+
+Instance moves and execution
+----------------------------
+
+Once the final set of nodes to power down is determined, the instance
+moves are determined by the algorithm used by `hbal`. If
+requested by the `-X` option, the nodes freed up are drained, and the
+instance moves are executed in the same way as `hbal` does. Finally,
+those of the freed-up nodes that do not already have a
+`htools:standby:` tag are tagged as `htools:standby:auto`, all free-up
+nodes are marked as offline and powered down via the
+:doc:`design-oob`.
+
+Similarly, if it is determined that nodes need to be added, then first
+the nodes are powered up via the :doc:`design-oob`, then they're marked
+as online and finally,
+the cluster is balanced in the same way, as `hbal` would do. For the
+newly powered up nodes, the `htools:standby:auto` tag, if present, is
+removed, but no other tags are removed (including other
+`htools:standby:` tags).
+
+
+Design choices
+==============
+
+The proposed algorithm builds on top of the already present balancing
+algorithm, instead of greedily packing nodes as full as possible. The
+reason is, that in the end, a balanced cluster is needed anyway;
+therefore, basing on the balancing algorithm reduces the number of
+instance moves. Additionally, the final configuration will also
+benefit from all improvements to the balancing algorithm, like taking
+dynamic CPU data into account.
+
+We decided to have a separate program instead of adding an option to
+`hbal` to keep the interfaces, especially that of `hbal`, cleaner. It is
+not unlikely that, over time, additional `hsqueeze`-specific options
+might be added, specifying, e.g., which nodes to prefer for
+shutdown. With the approach of the `htools` of having a single binary
+showing different behaviors, having an additional program also does not
+introduce significant additional cost.
+
+We decided to have a whole prefix instead of a single tag reserved
+for marking standby nodes (we consider all tags starting with
+`htools:standby:` as serving only this purpose). This is not only in
+accordance with the tag
+reservations for other tools, but it also allows for further extension
+(like specifying priorities on which nodes to power up first) without
+changing name spaces.