Commit 30114136 authored by Dimitris Aragiorgis's avatar Dimitris Aragiorgis Committed by Helga Velroyen
Browse files

Add a revised design document for networks

This design doc describes how to extend the existing network
management and make it more flexible and able to deal with more
generic use cases. It proposes support for:

 - Networks with multiple subnets
 - Subnets with multiple IP pools
 - NICs with multiple IPs from various subnets of a single network
Signed-off-by: default avatarDimitris Aragiorgis <>
Reviewed-by: default avatarHelga Velroyen <>
parent 06710954
......@@ -631,6 +631,7 @@ docinput = \
doc/design-multi-reloc.rst \
doc/design-multi-version-tests.rst \
doc/design-network.rst \
doc/design-network2.rst \
doc/design-node-add.rst \
doc/design-node-security.rst \
doc/design-oob.rst \
......@@ -25,6 +25,7 @@ Design document drafts
.. vim: set textwidth=72 :
.. Local Variables:
Network Management (revised)
.. contents:: :depth: 4
This is a design document detailing how to extend the existing network
management and make it more flexible and able to deal with more generic
use cases.
Current state and shortcomings
Currently in Ganeti, networks are tightly connected with IP pools,
since creation of a network implies the existence of one subnet
and the corresponding IP pool. This design does not allow common
scenarios like:
- L2 only networks
- IPv6 only networks
- Networks without an IP pool
- Networks with an IPv6 pool
- Networks with multiple IP pools (alternative to externally reserving
Additionally one cannot have multiple IP pools inside one network.
Finally, from the instance perspective, a NIC cannot get more than one
IPs (v4 and v6).
Proposed changes
In order to deal with the above shortcomings, we propose to extend
the existing networks in Ganeti and support:
a) Networks with multiple subnets
b) Subnets with multiple IP pools
c) NICs with multiple IPs from various subnets of a single network
These changes bring up some design and implementation issues that we
discuss in the following sections.
Quoting the initial network management design doc "an IP pool consists
of two bitarrays. Specifically the ``reservations`` bitarray which holds
all IP addresses reserved by Ganeti instances and the ``external
reservations`` bitarray with all IPs that are excluded from the IP pool
and cannot be assigned automatically by Ganeti to instances (via
Without violating those semantics, here, we clarify the following
**network**: A cluster level taggable configuration object with a
user-provider name, (e.g. network1, network2), UUID and MAC prefix.
**L2**: The `mode` and `link` with which we connect a network to a
nodegroup. A NIC attached to a network will inherit this info, just like
connecting an Ethernet cable to a physical NIC. In this sense we only
have one L2 info per NIC.
**L3**: A CIDR and a gateway related to the network. Since a NIC can
have multiple IPs on the same cable each network can have multiple L3
info with the restriction that they do not overlap with each other.
The gateway is optional (just like with current implementation). No
gateway can be used for private networks that do not have a default
**subnet**: A subnet is the above L3 info plus some additional information
(see below).
**ip**: A valid IP should reside in a network's subnet, and should not
be used by more than one instance. An IP can be either obtained dynamically
from a pool or requested explicitly from a subnet (or a pool).
**range**: Sequential IPs inside one subnet calculated either from the
first IP and a size (e.g. start=, size=10) or the first IP and
the last IP (e.g. start=, end= A single IP can
also be thought of as an IP range with size=1 (see configuration
**reservations**: All IPs that are used by instances in the cluster at
any time.
**external reservations**: All IPs that are supposed to be reserved
by the admin for either some external component or specific instances.
If one instance requests an external IP explicitly (ip=,
Ganeti will allow the operation only if ``--force`` is given. Still, the
admin can externally reserve an IP that is already in use by an
instance, as happens now. This helps to reserve an IP for future use and
at the same time prevent any possible race between the instance that
releases this IP and another that tries to retrieve it.
**pool**: A (range, reservations, name) tuple from which instances can
dynamically obtain an IP. Reservations is a bitarray with
length the size of the range, and is needed so that we know which IPs
are available at any time without querying all instances. The use of
name is explained below. A subnet can have multiple pools.
Split L2 from L3
Currently networks in Ganeti do not separate L2 from L3. This means
that one cannot use L2 only networks. The reason is because the CIDR
(passed currently with the ``--network`` option) and the derived IP pool
are mandatory. This design makes L3 info optional. This way we can have
an L2 only network just by connecting a Ganeti network to a nodegroup
with the desired `mode` and `link`. Then one could add one or more subnets
to the existing network.
Multiple Subnets per Network
Currently the IPv4 CIDR is mandatory for a network. Also a network can
obtain at most one IPv4 CIDR and one IPv6 CIDR. These restrictions will
be lifted.
This design doc introduces support for multiple subnets per network. The
L3 info will be moved inside the subnet. A subnet will have a `name` and
a `uuid` just like NIC and Disk config objects. Additionally it will contain
the `dhcp` flag which is explained below, and the `pools` and `external`
fields which are mentioned in the next section. Only the `cidr` will be
Any subnet related actions will be done via the new ``--subnet`` option.
Its syntax will be similar to ``--net``.
The network's subnets must not overlap with each other. Logic will
validate any operations related to reserving/releasing of IPs and check
whether a requested IP is included inside one of the network's subnets.
Just like currently, the L3 info will be exported to NIC configuration
hooks and scripts as environment variables. The example below adds
subnets to a network:
gnt-network modify --subnet add:cidr=,gateway=,dhcp=true net1
gnt-network modify --subnet add:cidr=2001::/64,gateway=2001::1,dhcp=true net1
To remove a subnet from a network one should use:
gnt-network modify --subnet some-ident:remove network1
where some-ident can be either a CIDR, a name or a UUID. Ganeti will
allow this operation only if no instances use IPs from this subnet.
Since DHCP is allowed only for a single CIDR on the same cable, the
subnet must have a `dhcp` flag. Logic must not allow more that one
subnets of the same version (4 or 6) in the same network to have DHCP enabled.
To modify a subnet's name or the dhcp flag one could use:
gnt-network modify --subnet some-ident:modify,dhcp=false,name=foo network1
This would search for a registered subnet that matches the identifier,
disable DHCP on it and change its name.
The ``dhcp`` parameter is used only for validation purposes and does not
make Ganeti starting a DHCP service. It will just be exported to
external scripts (ifup and hooks) and handled accordingly.
Changing the CIDR or the gateway of a subnet should also be supported.
gnt-network modify --subnet some-ident:modify,cidr= net1
gnt-network modify --subnet some-ident:modify,cidr= net1
gnt-network modify --subnet some-ident:modify,gateway= net1
Before expanding a subnet logic should should check for overlapping
subnets. Shrinking the subnet should be allowed only if the ranges
that are about to be trimmed are not included either in pool
reservations or external ranges.
Multiple IP pools per Subnet
Currently IP pools are automatically created during network creation and
include the whole subnet. Some IPs can be excluded from the pool by
passing them explicitly with ``--add-reserved-ips`` option.
Still for IPv6 subnets or even big IPv4 ones this might be insufficient.
It is impossible to have two bitarrays for a /64 prefix. Even for IPv4
networks a /20 subnet currently requires 8K long bitarrays. And the
second 4K is practically useless since the external reservations are way
less than the actual reservations.
This design extract IP pool management from the network logic, and pools
will become optional. Currently the pool is created based on the
network's CIDR. With multiple subnets per network, we should be able to
create and add IP pools to a network (and eventually to the
corresponding subnet). Each pool will have an optional user friendly
`name` so that the end user can refer to it (see instance related
The user will be able to obtain dynamically an IP only if we have
already defined a pool for a network's subnet. One would use ``ip=pool``
for the first available IP of the first available pool, or
``ip=some-pool-name`` for the first available IP of a specific pool.
Any pool related actions will be done via the new ``--pool`` option.
In order to add a pool a relevant subnet should pre-exist. Overlapping
pools won't be allowed. For example:
gnt-network modify --pool add:,name=pool1 net1
gnt-network modify --pool add: net1
gnt-network modify --pool add: net1
will first parse and find the ranges. Then for each range, Ganeti will
try to find a matching subnet meaning that a pool must be a subrange of
the subnet. If found, the range with empty reservations will be appended
to the list of the subnet's pools. Moreover, logic must be added to
reserve the IPs that are currently in use by instances of this network.
Adding a pool can be easier if we associate it directly with a subnet.
For example on could use the following shortcuts:
gnt-network modify --subnet add:cidr=,pool net1
gnt-network modify --pool add:subnet=some-ident
gnt-network modify --pool add: net1
During pool removal, logic should be added to split pools if ranges
given overlap existing ones. For example:
gnt-network modify --pool remove: net1
will split the pool previously added (10-100) into two new ones;
10-19 and 51-100. The corresponding bitarrays will be trimmed
accordingly. The name will be preserved.
The same things apply to external reservations. Just like now,
modifications will take place via the ``--add|remove-reserved-ips``
option. Logic must be added to support IP ranges.
gnt-network modify --add-reserved-ips net1
Based on the aforementioned we propose the following changes:
1) Change the IP pool representation in config data.
Existing `reservations` and `external_reservations` bitarrays will be
removed. Instead, for each subnet we will have:
* `pools`: List of (IP range, reservations bitarray) tuples.
* `external`: List of IP ranges
For external ranges the reservations bitarray is not needed
since this will be all 1's.
A configuration example could be::
net1 {
subnets [
uuid1 {
name: subnet1
pools: [
{range:Range(,, reservations: 00000, name:pool1}
reserved: []
uuid2 {
name: subnet2
pools: [
{range:, reservations: 00000000, name:pool3}
{range:, reservations: 000000, name:pool3}
reserved: [Range(,,]
Range(start, end) will be some json representation of an IPRange().
We decide not to store external reservations as pools (and in the
same list) since we get the following advantages:
- Keep the existing semantics for pools and external reservations.
- Each list has similar entries: one has pools the other has ranges.
The pool must have a bitarray, and has an optional name. It is
meaningless to add a name and a bitarray to external ranges.
- Each list must not have overlapping ranges. Still external
reservations can overlap with pools.
- The --pool option supports add|remove|modify command just like
`--net` and `--disk` and operate on single entities (a restriction that
is not needed for external reservations).
- Another thing, and probably the most important, is that in order to
get the first available IP, only the reserved list must be checked for
conflicts. The ipaddr.summarize_address_range(first, last) could be very
2) Change the network module logic.
The above changes should be done in the network module and be transparent
to the rest of the Ganeti code. If a random IP from the networks is
requested, Ganeti searches for an available IP from the first pool of
the first subnet. If it is full it gets to the next pool. Then to the
next subnet and so on. Of course the `external` IP ranges will be
excluded. If an IP is explicitly requested, Ganeti will try to find a
matching subnet. Its pools and external will be checked for
availability. All this logic will be extracted in a separate class
with helper methods for easier manipulation of IP ranges and
Bitarray processing can be optimized too. The usage of bitarrays will
be reduced since (a) we no longer have `external_reservations` and (b)
pools will have shorter bitarrays (i.e. will not have to cover the whole
subnet). Besides that, we could keep the bitarrays in memory, so that
in most cases (e.g. adding/removing reservations, querying), we don't
keep converting strings to bitarrays and vice versa. Also, the Haskell
code could as well keep this in memory as a bitarray, and validate it
on load.
3) Changes in config module.
We should not have instances with the same IP inside the same network.
We introduce _AllIPs() helper config method that will hold all existing
(IP, network) tuples. Config logic will check this list as well
before passing it to TemporaryReservationManager.
4) Change the query mechanism.
Since we have more that one subnets the new `subnets` field will
include a list of:
* cidr: IPv4 or IPv6 CIDR
* gateway: IPv4 or IPv6 address
* dhcp: True or False
* name: The user friendly name for the subnet
Since we want to support small pools inside big subnets, current query
results are not practical as far as the `map` field is concerned. It
should be replaced with the new `pools` field for each subnet, which will
* start: The first IP of the pool
* end: The last IP of the pool
* map: A string with 'X' for reserved IPs (either external or not) and
with '.' for all available ones inside the pool
Multiple IPs per NIC
Currently IP is a simple string inside the NIC object and there is a
one-to-one mapping between the `ip` and the `network` slots. The whole
logic behind this is that a NIC belongs to a network (cable) and
inherits its mode and link. This rational will not change.
Since this design adds support for multiple subnets per network, a NIC
must be able to obtain multiple IPs from various subnets of the same
network. Thus we change the `ip` slot into list.
We introduce a new `ipX` attribute. For backwards compatibility `ip`
will denote `ip0`.
During instance related operations one could use something like:
gnt-instance add --net 0:ip0=,ip1=pool,ip2=some-pool-name,network=network1 inst1
gnt-instance add --net 0:ip=pool,network1 inst1
This will be parsed, converted to a proper list (e.g. ip = [,
"pool", "some-pool-name"]) and finally passed to the corresponding opcode.
Based on the previous example, here the first IP will match subnet1, the
second IP will be retrieved from the first available pool of the first
available subnet, and the third from the pool with the some-pool name.
During instance modification, the `ip` option will refer to the first IP
of the NIC, whereas the `ipX` will refer to the X'th IP. As with NICs
we start counting from 0 so `ip1` will refer to the second IP. For example
one should pass:
--net 0:modify,ip1=
to change the second IP of the first NIC to,
--net -1:add,ip0=pool,ip1=,network=test
to add a new NIC with two IPs, and
--net 1:modify,ip1=none
to remove the second IP of the second NIC.
Configuration changes
IPRange config object:
Introduce new config object that will hold ranges needed by pools, and
reservations. It will be either a tuple of (start, size, end) or a
simple string. The `end` is redundant and can derive from start and
size in runtime, but will appear in the representation for readability
and debug reasons.
Pool config object:
Introduce new config object to represent a single subnet's pool. It
will have the `range`, `reservations`, `name` slots. The range slot
will be an IPRange config object, the reservations a bitarray and the
name a simple string.
Subnet config object:
Introduce new config object with slots: `name`, `uuid`, `cidr`,
`gateway`, `dhcp`, `pools`, `external`. Pools is a list of Pool config
objects. External is a list of IPRange config objects. All ranges must
reside inside the subnet's CIDR. Only `cidr` will be mandatory. The
`dhcp` attribute will be False by default.
Network config objects:
The L3 and the IP pool representation will change. Specifically all
slots besides `name`, `mac_prefix`, and `tag` will be removed. Instead
the slot `subnets` with a list of Subnet config objects will be added.
NIC config objects:
NIC's network slot will be removed and the `ip` slot will be modified
to a list of strings.
KVM runtime files:
Any change done in config data must be done also in KVM runtime files.
For this purpose the existing _UpgradeSerializedRuntime() can be used.
Exported variables
The exported variables during instance related operations will be just
like Linux uses aliases for interfaces. Specifically:
``IP:i`` for the ith IP.
``NETWORK_*:i`` for the ith subnet. * is SUBNET, GATEWAY, DHCP.
In case of hooks those variables will be prefixed with ``INSTANCE_NICn``
for the nth NIC.
Backwards Compatibility
The existing networks representation will be internally modified.
They will obtain one subnet, and one pool with range the whole subnet.
During `gnt-network add` if the deprecated ``--network`` option is passed
will still create a network with one subnet, and one IP pool with the
size of the subnet. Otherwise ``--subnet`` and ``--pool`` options
will be needed.
The query mechanism will also include the deprecated `map` field. For the
newly created network this will contain only the mapping of the first
pool. The deprecated `network`, `gateway`, `network6`, `gateway6` fields
will point to the first IPv4 and IPv6 subnet accordingly.
During instance related operation the `ip` argument of the ``--net``
option will refer to the first IP of the NIC.
Hooks and scripts will still have the same environment exported in case
of single IP per NIC.
This design allows more fine-grained configurations which in turn yields
more flexibility and a wider coverage of use cases. Still basic cases
(the ones that are currently available) should be easy to set up.
Documentation will be enriched with examples for both typical and
advanced use cases of gnt-network.
.. vim: set textwidth=72 :
.. Local Variables:
.. mode: rst
.. fill-column: 72
.. End:
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment