iallocator.rst 18 KB
Newer Older
1 2 3
Ganeti automatic instance allocation
====================================

4
Documents Ganeti version 2.16
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

.. contents::

Introduction
------------

Currently in Ganeti the admin has to specify the exact locations for
an instance's node(s). This prevents a completely automatic node
evacuation, and is in general a nuisance.

The *iallocator* framework will enable automatic placement via
external scripts, which allows customization of the cluster layout per
the site's requirements.

User-visible changes
~~~~~~~~~~~~~~~~~~~~

There are two parts of the ganeti operation that are impacted by the
auto-allocation: how the cluster knows what the allocator algorithms
are and how the admin uses these in creating instances.

An allocation algorithm is just the filename of a program installed in
a defined list of directories.

Cluster configuration
~~~~~~~~~~~~~~~~~~~~~

At configure time, the list of the directories can be selected via the
``--with-iallocator-search-path=LIST`` option, where *LIST* is a
comma-separated list of directories. If not given, this defaults to
``$libdir/ganeti/iallocators``, i.e. for an installation under
``/usr``, this will be ``/usr/lib/ganeti/iallocators``.

Ganeti will then search for allocator script in the configured list,
using the first one whose filename matches the one given by the user.

Command line interface changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

44
The node selection options in instance add and instance replace disks
45 46
can be replace by the new ``--iallocator=NAME`` option (shortened to
``-I``), which will cause the auto-assignement of nodes with the
Helga Velroyen's avatar
Helga Velroyen committed
47
passed iallocator. The selected node(s) will be shown as part of the
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70
command output.

IAllocator API
--------------

The protocol for communication between Ganeti and an allocator script
will be the following:

#. ganeti launches the program with a single argument, a filename that
   contains a JSON-encoded structure (the input message)

#. if the script finishes with exit code different from zero, it is
   considered a general failure and the full output will be reported to
   the users; this can be the case when the allocator can't parse the
   input message

#. if the allocator finishes with exit code zero, it is expected to
   output (on its stdout) a JSON-encoded structure (the response)

Input message
~~~~~~~~~~~~~

The input message will be the JSON encoding of a dictionary containing
71 72 73 74 75 76 77 78
all the required information to perform the operation. We explain the
contents of this dictionary in two parts: common information that every
type of operation requires, and operation-specific information.

Common information
++++++++++++++++++

All input dictionaries to the IAllocator must carry the following keys:
79 80 81

version
  the version of the protocol; this document
82
  specifies version 2
83 84 85 86 87 88 89

cluster_name
  the cluster name

cluster_tags
  the list of cluster tags

90 91 92
enabled_hypervisors
  the list of enabled hypervisors

93 94 95 96
ipolicy
  the cluster-wide instance policy (for information; the per-node group
  values take precedence and should be used instead)

97
request
98 99 100
  a dictionary containing the details of the request; the keys vary
  depending on the type of operation that's being requested, as
  explained in `Operation-specific input`_ below.
101

102 103 104 105 106 107 108
nodegroups
  a dictionary with the data for the cluster's node groups; it is keyed
  on the group UUID, and the values are a dictionary with the following
  keys:

  name
    the node group name
109
  alloc_policy
110 111
    the allocation policy of the node group (consult the semantics of
    this attribute in the :manpage:`gnt-group(8)` manpage)
112 113
  networks
    the list of network UUID's this node group is connected to
114 115
  ipolicy
    the instance policy of the node group
116 117
  tags
    the list of node group tags
118

119 120 121 122 123
instances
  a dictionary with the data for the current existing instance on the
  cluster, indexed by instance name; the contents are similar to the
  instance definitions for the allocate mode, with the addition of:

124
  admin_state
125 126 127 128 129 130 131 132 133
    if this instance is set to run (but not the actual status of the
    instance)

  nodes
    list of nodes on which this instance is placed; the primary node
    of the instance is always the first one

nodes
  dictionary with the data for the nodes in the cluster, indexed by
134
  the node name; the dict contains [*]_ :
135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163

  total_disk
    the total disk size of this node (mebibytes)

  free_disk
    the free disk space on the node

  total_memory
    the total memory size

  free_memory
    free memory on the node; note that currently this does not take
    into account the instances which are down on the node

  total_cpus
    the physical number of CPUs present on the machine; depending on
    the hypervisor, this might or might not be equal to how many CPUs
    the node operating system sees;

  primary_ip
    the primary IP address of the node

  secondary_ip
    the secondary IP address of the node (the one used for the DRBD
    replication); note that this can be the same as the primary one

  tags
    list with the tags of the node

164 165 166 167 168 169 170 171 172 173 174 175 176 177 178
  master_candidate:
    a boolean flag denoting whether this node is a master candidate

  drained:
    a boolean flag denoting whether this node is being drained

  offline:
    a boolean flag denoting whether this node is offline

  i_pri_memory:
    total memory required by primary instances

  i_pri_up_memory:
    total memory required by running primary instances

179 180 181
  group:
    the node group that this node belongs to

182 183
  No allocations should be made on nodes having either the ``drained``
  or ``offline`` flags set. More details about these of node status
Iustin Pop's avatar
Iustin Pop committed
184
  flags is available in the manpage :manpage:`ganeti(7)`.
185

186 187 188 189
.. [*] Note that no run-time data is present for offline, drained or
   non-vm_capable nodes; this means the tags total_memory,
   reserved_memory, free_memory, total_disk, free_disk, total_cpus,
   i_pri_memory and i_pri_up memory will be absent
190

191 192 193 194 195 196 197 198 199 200 201
Operation-specific input
++++++++++++++++++++++++

All input dictionaries to the IAllocator carry, in the ``request``
dictionary, detailed information about the operation that's being
requested. The required keys vary depending on the type of operation, as
follows.

In all cases, it includes:

  type
202
    the request type; this can be either ``allocate``, ``relocate``,
203
    ``change-group`` or ``node-evacuate``. The
204 205 206
    ``allocate`` request is used when a new instance needs to be placed
    on the cluster. The ``relocate`` request is used when an existing
    instance needs to be moved within its node group.
207 208 209

    The ``multi-evacuate`` protocol used to request that the script
    computes the optimal relocate solution for all secondary instances
210 211 212
    of the given nodes. It is now deprecated and needs only be
    implemented if backwards compatibility with Ganeti 2.4 and lower is
    needed.
213 214 215 216 217

    The ``change-group`` request is used to relocate multiple instances
    across multiple node groups. ``node-evacuate`` evacuates instances
    off their node(s). These are described in a separate :ref:`design
    document <multi-reloc-detailed-design>`.
218

219 220 221 222 223
    The ``multi-allocate`` request is used to allocate multiple
    instances on the cluster. The request is beside of that very
    similiar to the ``allocate`` one. For more details look at
    :doc:`Ganeti bulk create <design-bulk-create>`.

224 225 226 227 228 229
For both allocate and relocate mode, the following extra keys are needed
in the ``request`` dictionary:

  name
    the name of the instance; if the request is a realocation, then this
    name will be found in the list of instances (see below), otherwise
230
    is the FQDN of the new instance; type *string*
231 232 233 234 235

  required_nodes
    how many nodes should the algorithm return; while this information
    can be deduced from the instace's disk template, it's better if
    this computation is left to Ganeti as then allocator scripts are
236
    less sensitive to changes to the disk templates; type *integer*
237 238 239 240 241

  disk_space_total
    the total disk space that will be used by this instance on the
    (new) nodes; again, this information can be computed from the list
    of instance disks and its template type, but Ganeti is better
242
    suited to compute it; type *integer*
243

244 245 246 247 248
.. pyassert::

   constants.DISK_ACCESS_SET == set([constants.DISK_RDONLY,
     constants.DISK_RDWR])

249 250 251 252 253 254 255
Allocation needs, in addition:

  disks
    list of dictionaries holding the disk definitions for this
    instance (in the order they are exported to the hypervisor):

    mode
256 257
      either :pyeval:`constants.DISK_RDONLY` or
      :pyeval:`constants.DISK_RDWR` denoting if the disk is read-only or
258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296
      writable

    size
      the size of this disk in mebibytes

  nics
    a list of dictionaries holding the network interfaces for this
    instance, containing:

    ip
      the IP address that Ganeti know for this instance, or null

    mac
      the MAC address for this interface

    bridge
      the bridge to which this interface will be connected

  vcpus
    the number of VCPUs for the instance

  disk_template
    the disk template for the instance

  memory
   the memory size for the instance

  os
   the OS type for the instance

  tags
    the list of the instance's tags

  hypervisor
    the hypervisor of this instance

Relocation:

  relocate_from
297 298 299 300 301
     a list of nodes to move the instance away from; for DRBD-based
     instances, this will contain a single node, the current secondary
     of the instance, whereas for shared-storage instance, this will
     contain also a single node, the current primary of the instance;
     type *list of strings*
302

303
As for ``node-evacuate``, it needs the following request arguments:
304 305

  instances
306 307 308 309 310
    a list of instance names to evacuate; type *list of strings*

  evac_mode
    specify which instances to evacuate; one of ``primary-only``,
    ``secondary-only``, ``all``, type *string*
311

312 313 314 315 316
``change-group`` needs the following request arguments:

  instances
    a list of instance names whose group to change; type
    *list of strings*
317 318

  target_groups
319 320 321
    must either be the empty list, or contain a list of group UUIDs that
    should be considered for relocating instances to; type
    *list of strings*
322

323 324 325 326 327
``multi-allocate`` needs the following request arguments:

  instances
    a list of request dicts

328 329 330 331 332 333 334 335 336 337
MonD data
+++++++++

Additional information is available from mond. Mond's data collectors
provide information that can help an allocator script make better
decisions when allocating a new instance. Mond's information may also be
accessible from a mock file mainly for testing purposes. The file will
be in JSON format and will present an array of :ref:`report objects
<monitoring-agent-format-of-the-report>`.

338 339
Response message
~~~~~~~~~~~~~~~~
340 341 342 343 344

The response message is much more simple than the input one. It is
also a dict having three keys:

success
Michael Hanselmann's avatar
Michael Hanselmann committed
345
  a boolean value denoting if the allocation was successful or not
346 347 348 349 350

info
  a string with information from the scripts; if the allocation fails,
  this will be shown to the user

351 352 353 354 355 356 357 358 359
result
  the output of the algorithm; even if the algorithm failed
  (i.e. success is false), this must be returned as an empty list

  for allocate/relocate, this is the list of node(s) for the instance;
  note that the length of this list must equal the ``requested_nodes``
  entry in the input message, otherwise Ganeti will consider the result
  as failed

360 361 362 363
  for the ``node-evacuate`` and ``change-group`` modes, this is a
  dictionary containing, among other information, a list of lists of
  serialized opcodes; see the :ref:`design document
  <multi-reloc-result>` for a detailed description
364

365 366 367 368 369 370
  for the ``multi-allocate`` mode this is a tuple of 2 lists, the first
  being element of the tuple is a list of succeeded allocation, with the
  instance name as first element of each entry and the node placement in
  the second. The second element of the tuple is the instance list of
  failed allocations.

371 372 373
.. note:: Current Ganeti version accepts either ``result`` or ``nodes``
   as a backwards-compatibility measure (older versions only supported
   ``nodes``)
374 375 376 377 378 379 380

Examples
--------

Input messages to scripts
~~~~~~~~~~~~~~~~~~~~~~~~~

381 382
Input message, new instance allocation (common elements are listed this
time, but not included in further examples below)::
383 384

  {
385 386
    "version": 2,
    "cluster_name": "cluster1.example.com",
387
    "cluster_tags": [],
388 389 390 391 392 393
    "enabled_hypervisors": [
      "xen-pvm"
    ],
    "nodegroups": {
      "f4e06e0d-528a-4963-a5ad-10f3e114232d": {
        "name": "default",
394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429
        "alloc_policy": "preferred",
        "networks": ["net-uuid-1", "net-uuid-2"],
        "ipolicy": {
          "disk-templates": ["drbd", "plain"],
          "minmax": [
            {
              "max": {
                "cpu-count": 2,
                "disk-count": 8,
                "disk-size": 2048,
                "memory-size": 12800,
                "nic-count": 8,
                "spindle-use": 8
              },
              "min": {
                "cpu-count": 1,
                "disk-count": 1,
                "disk-size": 1024,
                "memory-size": 128,
                "nic-count": 1,
                "spindle-use": 1
              }
            }
          ],
          "spindle-ratio": 32.0,
          "std": {
            "cpu-count": 1,
            "disk-count": 1,
            "disk-size": 1024,
            "memory-size": 128,
            "nic-count": 1,
            "spindle-use": 1
          },
          "vcpu-ratio": 4.0
        },
        "tags": ["ng-tag-1", "ng-tag-2"]
430
      }
431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458
    },
    "instances": {
      "instance1.example.com": {
        "tags": [],
        "should_run": false,
        "disks": [
          {
            "mode": "w",
            "size": 64
          },
          {
            "mode": "w",
            "size": 512
          }
        ],
        "nics": [
          {
            "ip": null,
            "mac": "aa:00:00:00:60:bf",
            "bridge": "xen-br0"
          }
        ],
        "vcpus": 1,
        "disk_template": "plain",
        "memory": 128,
        "nodes": [
          "nodee1.com"
        ],
Guido Trotter's avatar
Guido Trotter committed
459
        "os": "debootstrap+default"
460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487
      },
      "instance2.example.com": {
        "tags": [],
        "should_run": false,
        "disks": [
          {
            "mode": "w",
            "size": 512
          },
          {
            "mode": "w",
            "size": 256
          }
        ],
        "nics": [
          {
            "ip": null,
            "mac": "aa:00:00:55:f8:38",
            "bridge": "xen-br0"
          }
        ],
        "vcpus": 1,
        "disk_template": "drbd",
        "memory": 512,
        "nodes": [
          "node2.example.com",
          "node3.example.com"
        ],
Guido Trotter's avatar
Guido Trotter committed
488
        "os": "debootstrap+default"
489 490 491 492 493
      }
    },
    "nodes": {
      "node1.example.com": {
        "total_disk": 858276,
494 495
        "primary_ip": "198.51.100.1",
        "secondary_ip": "192.0.2.1",
496
        "tags": [],
497
        "group": "f4e06e0d-528a-4963-a5ad-10f3e114232d",
498 499 500 501 502 503
        "free_memory": 3505,
        "free_disk": 856740,
        "total_memory": 4095
      },
      "node2.example.com": {
        "total_disk": 858240,
504 505
        "primary_ip": "198.51.100.2",
        "secondary_ip": "192.0.2.2",
506
        "tags": ["test"],
507
        "group": "f4e06e0d-528a-4963-a5ad-10f3e114232d",
508 509 510 511 512 513
        "free_memory": 3505,
        "free_disk": 848320,
        "total_memory": 4095
      },
      "node3.example.com.com": {
        "total_disk": 572184,
514 515
        "primary_ip": "198.51.100.3",
        "secondary_ip": "192.0.2.3",
516
        "tags": [],
517
        "group": "f4e06e0d-528a-4963-a5ad-10f3e114232d",
518 519 520 521
        "free_memory": 3505,
        "free_disk": 570648,
        "total_memory": 4095
      }
522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553
    },
    "request": {
      "type": "allocate",
      "name": "instance3.example.com",
      "required_nodes": 2,
      "disk_space_total": 3328,
      "disks": [
        {
          "mode": "w",
          "size": 1024
        },
        {
          "mode": "w",
          "size": 2048
        }
      ],
      "nics": [
        {
          "ip": null,
          "mac": "00:11:22:33:44:55",
          "bridge": null
        }
      ],
      "vcpus": 1,
      "disk_template": "drbd",
      "memory": 2048,
      "os": "debootstrap+default",
      "tags": [
        "type:test",
        "owner:foo"
      ],
      hypervisor: "xen-pvm"
554 555 556
    }
  }

557
Input message, reallocation::
558

559 560 561 562 563 564 565 566 567 568 569 570 571
  {
    "version": 2,
    ...
    "request": {
      "type": "relocate",
      "name": "instance2.example.com",
      "required_nodes": 1,
      "disk_space_total": 832,
      "relocate_from": [
        "node3.example.com"
      ]
    }
  }
572

573

574 575 576 577 578
Response messages
~~~~~~~~~~~~~~~~~
Successful response message::

  {
579
    "success": true,
580
    "info": "Allocation successful",
581
    "result": [
582 583
      "node2.example.com",
      "node1.example.com"
584
    ]
585 586 587 588 589
  }

Failed response message::

  {
590
    "success": false,
591
    "info": "Can't find a suitable node for position 2 (already selected: node2.example.com)",
592
    "result": []
593 594
  }

595 596 597
Successful node evacuation message::

  {
598
    "success": true,
599 600 601 602 603 604 605 606 607 608
    "info": "Request successful",
    "result": [
      [
        "instance1",
        "node3"
      ],
      [
        "instance2",
        "node1"
      ]
609
    ]
610 611 612
  }


613 614 615 616
Command line messages
~~~~~~~~~~~~~~~~~~~~~
::

Guido Trotter's avatar
Guido Trotter committed
617
  # gnt-instance add -t plain -m 2g --os-size 1g --swap-size 512m --iallocator hail -o debootstrap+default instance3
618 619 620 621
  Selected nodes for the instance: node1.example.com
  * creating instance disks...
  [...]

Guido Trotter's avatar
Guido Trotter committed
622
  # gnt-instance add -t plain -m 3400m --os-size 1g --swap-size 512m --iallocator hail -o debootstrap+default instance4
623
  Failure: prerequisites not met for this operation:
Guido Trotter's avatar
Guido Trotter committed
624
  Can't compute nodes using iallocator 'hail': Can't find a suitable node for position 1 (already selected: )
625

Guido Trotter's avatar
Guido Trotter committed
626
  # gnt-instance add -t drbd -m 1400m --os-size 1g --swap-size 512m --iallocator hail -o debootstrap+default instance5
627
  Failure: prerequisites not met for this operation:
Guido Trotter's avatar
Guido Trotter committed
628 629 630 631 632
  Can't compute nodes using iallocator 'hail': Can't find a suitable node for position 2 (already selected: node1.example.com)

Reference implementation
~~~~~~~~~~~~~~~~~~~~~~~~

633 634 635
Ganeti's default iallocator is "hail" which is available when "htools"
components have been enabled at build time (see :doc:`install-quick` for
more details).
636 637

.. vim: set textwidth=72 :
638 639 640 641
.. Local Variables:
.. mode: rst
.. fill-column: 72
.. End: