iallocator.rst

Ganeti automatic instance allocation

Documents Ganeti version 2.1

Contents

• Introduction
  • User-visible changes
  • Cluster configuration
  • Command line interface changes
• IAllocator API
  • Input message
  • Response message
• Examples
  • Input messages to scripts
  • Response messages
  • Command line messages

Introduction

Currently in Ganeti the admin has to specify the exact locations for an instance's node(s). This prevents a completely automatic node evacuation, and is in general a nuisance.

The iallocator framework will enable automatic placement via external scripts, which allows customization of the cluster layout per the site's requirements.

User-visible changes

There are two parts of the ganeti operation that are impacted by the auto-allocation: how the cluster knows what the allocator algorithms are and how the admin uses these in creating instances.

An allocation algorithm is just the filename of a program installed in a defined list of directories.

Cluster configuration

At configure time, the list of the directories can be selected via the --with-iallocator-search-path=LIST option, where LIST is a comma-separated list of directories. If not given, this defaults to $libdir/ganeti/iallocators, i.e. for an installation under /usr, this will be /usr/lib/ganeti/iallocators.

Ganeti will then search for allocator script in the configured list, using the first one whose filename matches the one given by the user.

Command line interface changes

The node selection options in instanece add and instance replace disks can be replace by the new --iallocator=NAME option (shortened to -I), which will cause the auto-assignement of nodes with the passed iallocator. The selected node(s) will be show as part of the command output.

IAllocator API

The protocol for communication between Ganeti and an allocator script will be the following:

1. ganeti launches the program with a single argument, a filename that contains a JSON-encoded structure (the input message)
2. if the script finishes with exit code different from zero, it is considered a general failure and the full output will be reported to the users; this can be the case when the allocator can't parse the input message
3. if the allocator finishes with exit code zero, it is expected to output (on its stdout) a JSON-encoded structure (the response)

Input message

The input message will be the JSON encoding of a dictionary containing the following:

version

the version of the protocol; this document specifies version 2

cluster_name

the cluster name

cluster_tags

the list of cluster tags

enabled_hypervisors

the list of enabled hypervisors

request

a dictionary containing the request data:

type

the request type; this can be either allocate, relocate or multi-evacuate; the allocate request is used when a new instance needs to be placed on the cluster, while the relocate request is used when an existing instance needs to be moved within the cluster; the multi-evacuate protocol requests that the script computes the optimal relocate solution for all secondary instances of the given nodes

The following keys are needed in allocate/relocate mode:

name

the name of the instance; if the request is a realocation, then this name will be found in the list of instances (see below), otherwise is the FQDN of the new instance

required_nodes

how many nodes should the algorithm return; while this information can be deduced from the instace's disk template, it's better if this computation is left to Ganeti as then allocator scripts are less sensitive to changes to the disk templates

disk_space_total

the total disk space that will be used by this instance on the (new) nodes; again, this information can be computed from the list of instance disks and its template type, but Ganeti is better suited to compute it

If the request is an allocation, then there are extra fields in the request dictionary:

disks

list of dictionaries holding the disk definitions for this instance (in the order they are exported to the hypervisor):

mode

either r or w denoting if the disk is read-only or writable

size

the size of this disk in mebibytes

nics

a list of dictionaries holding the network interfaces for this instance, containing:

ip

the IP address that Ganeti know for this instance, or null

mac

the MAC address for this interface

bridge

the bridge to which this interface will be connected

vcpus

the number of VCPUs for the instance

disk_template

the disk template for the instance

memory

the memory size for the instance

os

the OS type for the instance

tags

the list of the instance's tags

hypervisor

the hypervisor of this instance

If the request is of type relocate, then there is one more entry in the request dictionary, named relocate_from, and it contains a list of nodes to move the instance away from; note that with Ganeti 2.0, this list will always contain a single node, the current secondary of the instance.

The multi-evacuate mode has instead a single request argument:

nodes

the names of the nodes to be evacuated

instances

a dictionary with the data for the current existing instance on the cluster, indexed by instance name; the contents are similar to the instance definitions for the allocate mode, with the addition of:

admin_up

if this instance is set to run (but not the actual status of the instance)

nodes

list of nodes on which this instance is placed; the primary node of the instance is always the first one

nodes

dictionary with the data for the nodes in the cluster, indexed by the node name; the dict contains [*] :

total_disk

the total disk size of this node (mebibytes)

free_disk

the free disk space on the node

total_memory

the total memory size

free_memory

free memory on the node; note that currently this does not take into account the instances which are down on the node

total_cpus

the physical number of CPUs present on the machine; depending on the hypervisor, this might or might not be equal to how many CPUs the node operating system sees;

primary_ip

the primary IP address of the node

secondary_ip

the secondary IP address of the node (the one used for the DRBD replication); note that this can be the same as the primary one

tags

list with the tags of the node

master_candidate:

a boolean flag denoting whether this node is a master candidate

drained:

a boolean flag denoting whether this node is being drained

offline:

a boolean flag denoting whether this node is offline

i_pri_memory:

total memory required by primary instances

i_pri_up_memory:

total memory required by running primary instances

No allocations should be made on nodes having either the drained or offline flags set. More details about these of node status flags is available in the manpage :manpage:`ganeti(7)`.

[*]	Note that no run-time data is present for offline or drained nodes; this means the tags total_memory, reserved_memory, free_memory, total_disk, free_disk, total_cpus, i_pri_memory and i_pri_up memory will be absent

Response message

The response message is much more simple than the input one. It is also a dict having three keys:

success

a boolean value denoting if the allocation was successful or not

info

a string with information from the scripts; if the allocation fails, this will be shown to the user

result

the output of the algorithm; even if the algorithm failed (i.e. success is false), this must be returned as an empty list

for allocate/relocate, this is the list of node(s) for the instance; note that the length of this list must equal the requested_nodes entry in the input message, otherwise Ganeti will consider the result as failed

for multi-evacuation mode, this is a list of lists; each element of the list is a list of instance name and the new secondary node

Note

Current Ganeti version accepts either result or nodes as a backwards-compatibility measure (older versions only supported nodes)

Examples

Input messages to scripts

Input message, new instance allocation:

{
  "cluster_tags": [],
  "request": {
    "required_nodes": 2,
    "name": "instance3.example.com",
    "tags": [
      "type:test",
      "owner:foo"
    ],
    "type": "allocate",
    "disks": [
      {
        "mode": "w",
        "size": 1024
      },
      {
        "mode": "w",
        "size": 2048
      }
    ],
    "nics": [
      {
        "ip": null,
        "mac": "00:11:22:33:44:55",
        "bridge": null
      }
    ],
    "vcpus": 1,
    "disk_template": "drbd",
    "memory": 2048,
    "disk_space_total": 3328,
    "os": "etch-image"
  },
  "cluster_name": "cluster1.example.com",
  "instances": {
    "instance1.example.com": {
      "tags": [],
      "should_run": false,
      "disks": [
        {
          "mode": "w",
          "size": 64
        },
        {
          "mode": "w",
          "size": 512
        }
      ],
      "nics": [
        {
          "ip": null,
          "mac": "aa:00:00:00:60:bf",
          "bridge": "xen-br0"
        }
      ],
      "vcpus": 1,
      "disk_template": "plain",
      "memory": 128,
      "nodes": [
        "nodee1.com"
      ],
      "os": "etch-image"
    },
    "instance2.example.com": {
      "tags": [],
      "should_run": false,
      "disks": [
        {
          "mode": "w",
          "size": 512
        },
        {
          "mode": "w",
          "size": 256
        }
      ],
      "nics": [
        {
          "ip": null,
          "mac": "aa:00:00:55:f8:38",
          "bridge": "xen-br0"
        }
      ],
      "vcpus": 1,
      "disk_template": "drbd",
      "memory": 512,
      "nodes": [
        "node2.example.com",
        "node3.example.com"
      ],
      "os": "etch-image"
    }
  },
  "version": 1,
  "nodes": {
    "node1.example.com": {
      "total_disk": 858276,
      "primary_ip": "192.168.1.1",
      "secondary_ip": "192.168.2.1",
      "tags": [],
      "free_memory": 3505,
      "free_disk": 856740,
      "total_memory": 4095
    },
    "node2.example.com": {
      "total_disk": 858240,
      "primary_ip": "192.168.1.3",
      "secondary_ip": "192.168.2.3",
      "tags": ["test"],
      "free_memory": 3505,
      "free_disk": 848320,
      "total_memory": 4095
    },
    "node3.example.com.com": {
      "total_disk": 572184,
      "primary_ip": "192.168.1.3",
      "secondary_ip": "192.168.2.3",
      "tags": [],
      "free_memory": 3505,
      "free_disk": 570648,
      "total_memory": 4095
    }
  }
}

Input message, reallocation. Since only the request entry in the input message is changed, we show only this changed entry:

"request": {
  "relocate_from": [
    "node3.example.com"
  ],
  "required_nodes": 1,
  "type": "relocate",
  "name": "instance2.example.com",
  "disk_space_total": 832
},

Input message, node evacuation:

"request": {
  "evac_nodes": [
    "node2"
  ],
  "type": "multi-evacuate"
},

Response messages

Successful response message:

{
  "info": "Allocation successful",
  "result": [
    "node2.example.com",
    "node1.example.com"
  ],
  "success": true
}

Failed response message:

{
  "info": "Can't find a suitable node for position 2 (already selected: node2.example.com)",
  "result": [],
  "success": false
}

Successful node evacuation message:

{
  "info": "Request successful",
  "result": [
    [
      "instance1",
      "node3"
    ],
    [
      "instance2",
      "node1"
    ]
  ],
  "success": true
}

Command line messages

# gnt-instance add -t plain -m 2g --os-size 1g --swap-size 512m --iallocator dumb-allocator -o etch-image instance3
Selected nodes for the instance: node1.example.com
* creating instance disks...
[...]

# gnt-instance add -t plain -m 3400m --os-size 1g --swap-size 512m --iallocator dumb-allocator -o etch-image instance4
Failure: prerequisites not met for this operation:
Can't compute nodes using iallocator 'dumb-allocator': Can't find a suitable node for position 1 (already selected: )

# gnt-instance add -t drbd -m 1400m --os-size 1g --swap-size 512m --iallocator dumb-allocator -o etch-image instance5
Failure: prerequisites not met for this operation:
Can't compute nodes using iallocator 'dumb-allocator': Can't find a suitable node for position 2 (already selected: node1.example.com)