Commit 0aae4013 authored by Apollon Oikonomopoulos's avatar Apollon Oikonomopoulos Committed by Klaus Aehlig
Browse files

Add systemd integration design document



This patch adds a design document describing the necessary changes to
provide native systemd support by shipping the relevant unit files and
changing Ganeti to use systemd for process control if applicable.

Also, the possibility of adding socket activation support to various
daemons is briefly discussed.
Signed-off-by: default avatarApollon Oikonomopoulos <apoikos@gmail.com>
Signed-off-by: default avatarKlaus Aehlig <aehlig@google.com>
Reviewed-by: default avatarKlaus Aehlig <aehlig@google.com>
parent af11c51e
......@@ -20,6 +20,7 @@ Design document drafts
design-hsqueeze.rst
design-os.rst
design-node-security.rst
design-systemd.rst
.. vim: set textwidth=72 :
.. Local Variables:
......
===================
Systemd integration
===================
.. contents:: :depth: 4
This design document outlines the implementation of native systemd
support in Ganeti by providing systemd unit files. It also briefly
discusses the possibility of supporting socket activation.
Current state and shortcomings
==============================
Ganeti currently ships an example init script, compatible with Debian
(and derivatives) and RedHat/Fedora (and derivatives). The initscript
treats the whole Ganeti system as a single service wrt. starting and
stopping (but allows starting/stopping/restarting individual daemons).
The initscript is aided by ``daemon-util``, which takes care of correctly
ordering the startup/shutdown of daemons using an explicit order.
Finally, process supervision is achieved by (optionally) running
``ganeti-watcher`` via cron every 5 minutes. ``ganeti-watcher`` will - among
other things - try to start services that should be running but are not.
The example initscript currently shipped with Ganeti will work with
systemd's LSB compatibility wrappers out of the box, however there are
a number of areas where we can benefit from providing native systemd
unit files:
- systemd is the `de-facto choice`_ of almost all major Linux
distributions. Since it offers a stable API for service control,
providing our own systemd unit files means that Ganeti will run
out-of-the-box and in a predictable way in all distributions using
systemd.
- systemd performs constant process supervision with immediate
service restarts and configurable back-off. Ganeti currently offers
supervision only via ganeti-watcher, running via cron in 5-minute
intervals and unconditionally starting missing daemons even if they
have been manually stopped.
- systemd offers `socket activation`_ support, which may be of
interest for use at least with masterd, luxid and noded. Socket
activation offers two main advantages: no explicit service
dependencies or ordering needs to be defined as services will be
activated when needed; and seamless restarts / upgrades are possible
without rejecting new client connections.
- systemd offers a number of `security features`_, primarily using
the Linux kernel's namespace support, which may be of interest to
better restrict daemons running as root (noded and mond).
.. _de-facto choice: https://en.wikipedia.org/wiki/Systemd#Adoption
.. _socket activation: http://0pointer.de/blog/projects/socket-activation.html
.. _security features: http://0pointer.de/blog/projects/security.html
Proposed changes
================
We propose to extend Ganeti to natively support systemd, in addition to
shipping the init-script as is. This requires the addition of systemd
unit files, as well as some changes in daemon-util and ganeti-watcher to
use ``systemctl`` on systems where Ganeti is managed by systemd.
systemd unit files
------------------
Systemd uses unit files to store information about a service, device,
mount point, or other resource it controls. Each unit file contains
exactly one unit definition, consisting of a ``Unit`` an (optional)
``Install`` section and an (optional) type-specific section (e.g.
``Service``). Unit files are dropped in pre-determined locations in the
system, where systemd is configured to read them from. Systemd allows
complete or partial overrides of the unit files, using overlay
directories. For more information, see `systemd.unit(5)`_.
.. _systemd.unit(5): http://www.freedesktop.org/software/systemd/man/systemd.unit.html
We will create one systemd `service unit`_ per daemon (masterd, noded,
mond, luxid, confd, rapi) and an additional oneshot service for
ensure-dirs (``ganeti-common.service``). All services will ``Require``
``ganeti-common.service``, which will thus run exactly once per
transaction (regardless of starting one or all daemons).
.. _service unit: http://www.freedesktop.org/software/systemd/man/systemd.service.html
All daemons will run in the foreground (already implemented by the
``-f`` flag), directly supervised by systemd, using
``Restart=on-failure`` in the respective units. Master role units will
also treat ``EXIT_NOTMASTER`` as a successful exit and not trigger
restarts. Additionally, systemd's conditional directives will be used to
avoid starting daemons when they will certainly fail (e.g. because of
missing configuration).
Apart from the individual daemon units, we will also provide three
`target units`_ as synchronization points:
- ``ganeti-node.target``: Regular node/master candidate functionality,
including ``ganeti-noded.service``, ``ganeti-mond.service`` and
``ganeti-confd.service``.
- ``ganeti-master.target``: Master node functionality, including
``ganeti-masterd.service``, ``ganeti-luxid.service`` and
``ganeti-rapi.service``.
- ``ganeti.target``: A "meta-target" depending on
``ganeti-node.target`` and ``ganti-master.target``.
``ganeti.target`` itself will be ``WantedBy`` ``multi-user.target``,
so that Ganeti starts automatically on boot.
.. _target units: http://www.freedesktop.org/software/systemd/man/systemd.target.html
To allow starting/stopping/restarting the different roles, all units
will include a ``PartOf`` directive referencing their direct ancestor
target. In this way ``systemctl restart ganeti-node.target`` or ``systemctl
restart ganeti.target`` will work as expected, i.e. restart only the node
daemons or all daemons respectively.
The full dependency tree is as follows:
::
ganeti.target
├─ganeti-master.target
│ ├─ganeti-luxid.service
│ │ └─ganeti-common.service
│ ├─ganeti-masterd.service
│ │ └─ganeti-common.service
│ └─ganeti-rapi.service
│ └─ganeti-common.service
└─ganeti-node.target
├─ganeti-confd.service
│ └─ganeti-common.service
├─ganeti-mond.service
│ └─ganeti-common.service
└─ganeti-noded.service
└─ganeti-common.service
Installation
~~~~~~~~~~~~
The systemd unit files will be built from templates under
doc/examples/systemd, much like what is currently done for the
initscript. They will not be installed with ``make install``, but left
up to the distribution packagers to ship them at the appropriate
locations.
SysV compatibility
~~~~~~~~~~~~~~~~~~
Systemd automatically creates a service for each SysV initscript on the
system, appending ``.service`` to the initscript name, except if a
service with the given name already exists. In our case however, the
initscript's functionality is implemented by ``ganeti.target``.
Systemd provides the ability to *mask* a given service, rendering it
unusable, but in the case of SysV services this also results in
failure to use tools like ``invoke-rc.d`` or ``service``. Thus we have
to ship a ``ganeti.service`` (calling ``/bin/true``) of type
``oneshot``, that depends on ``ganeti.target`` for these tools to
continue working as expected. ``ganeti.target`` on the other hand will
be marked as ``PartOf = ganeti.service`` for stop and restart to be
propagated to the whole service.
The ``ganeti.service`` unit will not be marked to be enabled by systemd
(i.e. will not be started at boot), but will be available for manual
invocation and only be used for compatibility purposes.
Changes to daemon-util
----------------------
``daemon-util`` is used wherever daemon control is required:
- In the sample initscript, to start and stop all daemons.
- In ``ganeti.backend`` to start the master daemons on master failover and
to stop confd when leaving the cluster.
- In ``ganeti.bootstrap``, to start the daemons on cluster initialization.
- In ``ganeti.cli``, to control the daemon run state during certain
operations (e.g. renew-crypto).
Currently, ``daemon-util`` uses two auxiliary tools for managing daemons
``start-stop-daemon`` and ``daemon``, in this order of preference. In
order not to confuse systemd in its process supervision, ``daemon-util``
will have to be modified to start and stop the daemons via ``systemctl``
in preference to ``start-stop-daemon`` and ``daemon``. This
will require a basic check against run-time environment integrity:
- Make sure that ``systemd`` runs as PID 1, which is a `simple
check`_ against the existence of ``/run/systemd/system``.
- Make sure ``systemd`` knows how to handle Ganeti natively. This can
be a check against the ``LoadState`` of the ``ganeti.target`` unit.
Unless both of these checks pass, ``daemon-util`` will fall back to its
current behavior.
.. _simple check: http://www.freedesktop.org/software/systemd/man/sd_booted.html
Changes to ganeti-watcher
-------------------------
Since the daemon process supervision will be systemd's responsibility,
the watcher must detect systemd's presence and not attempt to start any
missing services. Again, systemd can be detected by the existence of
``/run/systemd/system``.
Future work
===========
Socket activation
-----------------
Systemd offers support for `socket activation`_. A daemon supporting
socket-based activation, can inherit its listening socket(s) by systemd.
This in turn means that the socket can be created and bound by systemd
during early boot and it can be used to provide implicit startup
ordering; as soon as a client connects to the listening socket, the
respective service (and all its dependencies) will be started and the
client will wait until its connection is accepted.
Also, because the socket remains bound even if the service is
restarting, new client connections will never be rejected, making
service restarts and upgrades seamless.
Socket activation support is trivial to implement (see
`sd_listen_fds(3)`_) and relies on information passed by systemd via
environment variables to the started processes.
.. _sd_listen_fds(3): http://www.freedesktop.org/software/systemd/man/sd_listen_fds.html
.. vim: set textwidth=72 :
.. Local Variables:
.. mode: rst
.. fill-column: 72
.. End:
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment