During the sequence of moves while cluster balancing the situation on
cluster may change (e.g. because of adding new instance or because of
instance or node parameters change) and desired moves can become
unprofitable.

Partly prevent this effect by introducing new hbal option
*--avoid-disk-moves=FACTOR* which will admit only profitable enough
disk moves.
Signed-off-by: Oleg Ponomarev <onponomarev@gmail.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This daemon will take over cluster maintenance as per
our design document. As it will heavily depend on the monitoring
daemon, it will only be enabled (at configure time) if the
monitoring daemon is enabled as well. It will also run as
the same user and group.

In this commit only the plain daemon is added with the only supported
request being the question for the supported protocol versions.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Support restarting of failed repair events, by allowing unconditional
forgetting of a failed event.  Also, rename it to maintenance daemon
to emphasize that it does more than just coordinating repairs.
Signed-off-by: Andrew King <ahking@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

This patch updates the cluster-keys-replacement document
which assists user about how to replace the crypto keys
for their cluster. This now reflects the changes wrt
server/client certificates.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

* Actualize instance pinning documentation in the location design document
* Add instance pinning documentation to the hbal man page.
Signed-off-by: Oleg Ponomarev <onponomarev@gmail.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

While we cannot avoid data loss on node crashs if we
have plain instances, we can ensure that the cluster
has enough capacity to reinstall the instances on a
new node. Add a design describing how we enusre this.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Fixing issue 1094 unfortunately will result in a bigger
change. This change is big enough to be documented in
the node-security design doc.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

This renames the deprecated `printTestCase` to its replacement
`counterexample`, add provides a CPP-guarded fallback for QuickCheck < 2.7.
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Conflicts:
	test/hs/Test/Ganeti/JQScheduler.hs
          - removed file not present in 2.12
	test/hs/Test/Ganeti/SlotMap.hs
          - removed file not present in 2.12

Cherry-picked-from: 077c415aSigned-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

It is a frequent request to be able to specify desired locations
for particular instances. Describe a tag-based way of achieving
this without adding hard enforcements.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add a design describing a new Ganeti daemon to coordinate
Ganeti's task in automated repairs. While Ganeti itself will
not handle hardware swaps, it can still automatically prepare
everything, e.g., by evacuating a node.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

The metadata daemon was previously running as root due to its need to
open port 80 to provide information to instances. To allow the daemon
to run in a more secure way, this patch adds a separate metadata user,
and grants the metad executable the CAP_NET_BIND_SERVICE capability.
As a result, the metadata daemon can use the port 80 without having to
acquire the full set of root capabilities and drop it later.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

While we have never actually heard of anyone using it, Ganeti does
offer the possibility of using a different prefix for the groups and
users used in split-user mode. The groups.in file contained a mishmash
of users and groups, and this did not work well with different
prefixes. This patch fixes the situation.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Signed-off-by: Lisa Velden <velden@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

When ganeti is removed (not purged) `/etc/cron.d/ganeti` is not deleted,
thus after a reboot cron tries to execute gnt-cluster upgrade and fails.

The same pattern is used on all other cron entries.
Signed-off-by: Christos Trochalakis <christos@skroutz.gr>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Add a mention into the install file, and provide a link to the official
DRBD documentation.
Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...and also explain why the original approach was good in theory but
not able to cope with out-of-memory killers killing daemons in just
the wrong moment.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

As it turns out, simply iterating the steps that hail does
is fast enough for shared storage, so that we can have accurate
predictions with hspace.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

In the presence of global N+1 redundancy properties, the hack
of just marking non-locked nodes offline in opportunistic locking
does not work any more. Describe an extension to the allocation
protocol allowing for proper planning.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

And mention partial and complete designs in there.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

We always supported underscores in tags since \w in pythons
interpretation of regular expressions does include the underscore.
While this might have happened by accident, there is no reason to
change the implementation. The motivation for the restriction
was to avoid accidents with wrong (manual) shell escaping; the
underscore, however, has no special meaning to the shell. So
just make the documentation say what we implemented.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

When branching off 2.14, updating those got forgotten.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Bump various versions to 2.16 and remove downgrade code
from cfgupgrade. Also adapt cfgupgrade tests.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Signed-off-by: Gangbiao Liu <lgb.nwpu@gmail.com>
Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Add a design document describing how Ganeti will take care of N+1
redundancy in the presence of shared storage. As opposed to DRBD,
instances using shared storage can be started on any node. Therefore,
at no particular node we have to reserve memory---however, in total
enough memory for any node to fail has to be reserved nevertheless.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Signed-off-by: Lisa Velden <velden@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Fix typos in doc/design-file-based-storage.rst.
Signed-off-by: Gangbiao Liu <lgb.nwpu@gmail.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Signed-off-by: Lisa Velden <velden@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

This design describes an addition to the cluster
metrics that improves allocation efficiency. So
far, only the balancedness of the amount of reserved
memory was taken into account, but not its total
amount. The reserved memory, however, is not preserved
by instance moves; it can be lowered by spreading the
secondaries more widely over the cluster.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

It is a deliberate choice to have the full generated documentation
part of the distribution, as this reduces the dependencies for users
who just want to install and use Ganeti. It has, however, the disadvantage
that we ship files with configure-time paths substituted in. So be
careful with the wording to indicate that those paths depend on
configure-time choices.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Introduce a new environment variable named VOL_OPEN_EXCLUSIVE
and export it to the ExtStorage's open script to denote
whether this device will be opened in shared mode (before
migrations) or not (instance start, disk activation).

VOL_OPEN_EXCLUSIVE will be either True or False.

Update design doc and man pages accordingly.
Signed-off-by: Dimitris Aragiorgis <dimitris.aragiorgis@gmail.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Introduce open/close scripts to the ExtStorage interface in order
to start a unification among BlockDev classes.

Both scripts will be optional for backwards compatibility.
Signed-off-by: Dimitris Aragiorgis <dimitris.aragiorgis@gmail.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

In preparation of the 2.14 release cycle, update the
documentation about which design documents have been
implemented for 2.14.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

http://downloads.ganeti.org is now the primary location for
obtaining ganeti release tar balls.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

...thus moving the design document appropriately.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

Add a design document detailing how allocation is changed
on dedicated clusters in order to avoid not being able
to put a big instance on the cluster as all nodes are taken
up by small instances.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

...as it is not implemented yet, let alone in a
released version.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>

Currently the htools don't have the concept of differnt storage types
(and neither does LUXI), so this patch adds an initial design on how
to handle multiple different storage units on a node (and on the
cluster in general).
Signed-off-by: Aaron Karper <akarper@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

Add a design doc regarding the attachment/detachment of file-based disks. This
doc highlights the issue of disk ownership at the filesystem level and proposes
a way to implement it correctly.
Signed-off-by: Alex Pyrgiotis <apyrgio@grnet.gr>
Reviewed-by: Aaron Karper <akarper@google.com>

 create mode 100644 doc/design-file-based-disks-ownership.rst
Signed-off-by: Aaron Karper <akarper@google.com>