NEWS

News
====


Version 2.6.0 beta1
-------------------

*(unreleased)*

New features
~~~~~~~~~~~~

Instance run status
+++++++++++++++++++

The current ``admin_up`` field, which used to denote whether an instance
should be running or not, has been removed. Instead, ``admin_state`` is
introduced, with 3 possible values -- ``up``, ``down`` and ``offline``.

The rational behind this is that an instance being “down” can have
different meanings:

- it could be down during a reboot
- it could be temporarily be down for a reinstall
- or it could be down because it is deprecated and kept just for its
  disk

The previous Boolean state was making it difficult to do capacity
calculations: should Ganeti reserve memory for a down instance? Now, the
tri-state field makes it clear:

- in ``up`` and ``down`` state, all resources are reserved for the
  instance, and it can be at any time brought up if it is down
- in ``offline`` state, only disk space is reserved for it, but not
  memory or CPUs

The field can have an extra use: since the transition between ``up`` and
``down`` and vice-versus is done via ``gnt-instance start/stop``, but
transition between ``offline`` and ``down`` is done via ``gnt-instance
modify``, it is possible to given different rights to users. For
example, owners of an instance could be allowed to start/stop it, but
not transition it out of the offline state.

Instance policies and specs
+++++++++++++++++++++++++++

In previous Ganeti versions, an instance creation request was not
limited on the minimum size and on the maximum size just by the cluster
resources. As such, any policy could be implemented only in third-party
clients (RAPI clients, or shell wrappers over ``gnt-*``
tools). Furthermore, calculating cluster capacity via ``hspace`` again
required external input with regards to instance sizes.

In order to improve these workflows and to allow for example better
per-node group differentiation, we introduced instance specs, which
allow declaring:

- minimum instance disk size, disk count, memory size, cpu count
- maximum values for the above metrics
- and “standard” values (used in ``hspace`` to calculate the standard
  sized instances)

The minimum/maximum values can be also customised at node-group level,
for example allowing more powerful hardware to support bigger instance
memory sizes.

Beside the instance specs, there are a few other settings belonging to
the instance policy framework. It is possible now to customise, per
cluster and node-group:

- the list of allowed disk templates
- the maximum ratio of VCPUs per PCPUs (to control CPU oversubscription)
- the maximum ratio of instance to spindles (see below for more
  information) for local storage

All these together should allow all tools that talk to Ganeti to know
what are the ranges of allowed values for instances and the
over-subscription that is allowed.

For the VCPU/PCPU ratio, we already have the VCPU configuration from the
instance configuration, and the physical CPU configuration from the
node. For the spindle ratios however, we didn't track before these
values, so new parameters have been added:

- a new node parameter ``spindle_count``, defaults to 1, customisable at
  node group or node level
- at new backend parameter (for instances), ``spindle_use`` defaults to 1

Note that spindles in this context doesn't need to mean actual
mechanical hard-drives; it's just a relative number for both the node
I/O capacity and instance I/O consumption.

Instance migration behaviour
++++++++++++++++++++++++++++

While live-migration is in general desirable over failover, it is
possible that for some workloads it is actually worse, due to the
variable time of the “suspend” phase during live migration.

To allow the tools to work consistently over such instances (without
having to hard-code instance names), a new backend parameter
``always_failover`` has been added to control the migration/failover
behaviour. When set to True, all migration requests for an instance will
instead fall-back to failover.

Instance memory ballooning
++++++++++++++++++++++++++

Initial support for memory ballooning has been added. The memory for an
instance is no longer fixed (backend parameter ``memory``), but instead
can vary between minimum and maximum values (backend parameters
``minmem`` and ``maxmem``). Currently we only change an instance's
memory when:

- live migrating or failing over and instance and the target node
  doesn't have enough memory
- user requests changing the memory via ``gnt-instance modify
  --runtime-memory``

Instance CPU pinning
++++++++++++++++++++

In order to control the use of specific CPUs by instance, support for
controlling CPU pinning has been added for the Xen, HVM and LXC
hypervisors. This is controlled by a new hypervisor parameter
``cpu_mask``; details about possible values for this are in the
:manpage:`gnt-instance(8)`. Note that use of the most specific (precise
VCPU-to-CPU mapping) form will work well only when all nodes in your
cluster have the same amount of CPUs.

Disk parameters
+++++++++++++++

Another area in which Ganeti was not customisable were the parameters
used for storage configuration, e.g. how many stripes to use for LVM,
DRBD resync configuration, etc.

To improve this area, we've added disks parameters, which are
customisable at cluster and node group level, and which allow to
specify various parameters for disks (DRBD has the most parameters
currently), for example:

- DRBD resync algorithm and parameters (e.g. speed)
- the default VG for meta-data volumes for DRBD
- number of stripes for LVM (plain disk template)
- the RBD pool

These parameters can be modified via ``gnt-cluster modify -D …`` and
``gnt-group modify -D …``, and are used at either instance creation (in
case of LVM stripes, for example) or at disk “activation” time
(e.g. resync speed).

Rados block device support
++++++++++++++++++++++++++

A Rados (http://ceph.com/wiki/Rbd) storage backend has been added,
denoted by the ``rbd`` disk template type. This is considered
experimental, feedback is welcome. For details on configuring it, see
the :doc:`install` document and the :manpage:`gnt-cluster(8)` man page.

Master IP setup
+++++++++++++++

The existing master IP functionality works well only in simple setups (a
single network shared by all nodes); however, if nodes belong to
different networks, then the ``/32`` setup and lack of routing
information is not enough.

To allow the master IP to function well in more complex cases, the
system was reworked as follows:

- a master IP netmask setting has been added
- the master IP activation/turn-down code was moved from the node daemon
  to a separate script
- whether to run the Ganeti-supplied master IP script or a user-supplied
  on is a ``gnt-cluster init`` setting

Details about the location of the standard and custom setup scripts are
in the man page :manpage:`gnt-cluster(8)`; for information about the
setup script protocol, look at the Ganeti-supplied script.

SPICE support
+++++++++++++

The `SPICE <http://www.linux-kvm.org/page/SPICE>`_ support has been
improved.

It is now possible to use TLS-protected connections, and when renewing
or changing the cluster certificates (via ``gnt-cluster renew-crypto``,
it is now possible to specify spice or spice CA certificates. Also, it
is possible to configure a password for SPICE sessions via the
hypervisor parameter ``spice_password_file``.

There are also new parameters to control the compression and streaming
options (e.g. ``spice_image_compression``, ``spice_streaming_video``,
etc.). For details, see the man page :manpage:`gnt-instance(8)` and look
for the spice parameters.

Lastly, it is now possible to see the SPICE connection information via
``gnt-instance console``.

OVF converter
+++++++++++++

A new tool (``tools/ovfconverter``) has been added that supports
conversion between Ganeti and the `Open Virtualization Format
<http://en.wikipedia.org/wiki/Open_Virtualization_Format>`_ (both to and
from).

This relies on the ``qemu-img`` tool to convert the disk formats, so the
actual compatibility with other virtualization solutions depends on it.

Confd daemon changes
++++++++++++++++++++

The configuration query daemon (``ganeti-confd``) is now optional, and
has been rewritten in Haskell; whether to use the daemon at all, use the
Python (default) or the Haskell version is selectable at configure time
via the ``--enable-confd`` parameter, which can take one of the
``haskell``, ``python`` or ``no`` values. If not used, disabling the
daemon will result in a smaller footprint; for larger systems, we
welcome feedback on the Haskell version which might become the default
in future versions.


User interface changes
~~~~~~~~~~~~~~~~~~~~~~

We have replaced the ``--disks`` option of ``gnt-instance
replace-disks`` with a more flexible ``--disk`` option, which allows
adding and removing disks at arbitrary indices (Issue 188). Furthermore,
disk size and mode can be changed upon recreation (via ``gnt-instance
recreate-disks``, which accepts the same ``--disk`` option).

As many people are used to a ``show`` command, we have added that as an
alias to ``info`` on all ``gnt-*`` commands.

The ``gnt-instance grow-disk`` command has a new mode in which it can
accept the target size of the disk, instead of the delta; this can be
more safe since two runs in absolute mode will be idempotent, and
sometimes it's also easier to specify the desired size directly.

API changes
~~~~~~~~~~~

RAPI coverage has improved, with (for example) new resources for
recreate-disks, node power-cycle, etc.

Compatibility
~~~~~~~~~~~~~

There is partial support for ``xl`` in the Xen hypervisor; feedback is
welcome.

Python 2.7 is better supported, and after Ganeti 2.6 we will investigate
whether to still support Python 2.4 or move to Python 2.6 as minimum
required version.

Internal changes
~~~~~~~~~~~~~~~~

The deprecated ``QueryLocks`` LUXI request has been removed. Use
``Query(what=QR_LOCK, ...)`` instead.

The LUXI requests :pyeval:`luxi.REQ_QUERY_JOBS`,
:pyeval:`luxi.REQ_QUERY_INSTANCES`, :pyeval:`luxi.REQ_QUERY_NODES`,
:pyeval:`luxi.REQ_QUERY_GROUPS`, :pyeval:`luxi.REQ_QUERY_EXPORTS` and
:pyeval:`luxi.REQ_QUERY_TAGS` are deprecated and will be removed in a
future version. :pyeval:`luxi.REQ_QUERY` should be used instead.

RAPI client: ``CertificateError`` now derives from
``GanetiApiError``. This should make it more easy to handle Ganeti
errors.

Deprecation warnings due to PyCrypto/paramiko import in
``tools/setup-ssh`` have been silenced, as usually they are safe; please
make sure to run an up-to-date paramiko version, if you use this tool.

The QA scripts now depend on Python 2.5 or above (the main code base
still works with Python 2.4).

The configuration file (``config.data``) is now written without
indentation for performance reasons; if you want to edit it, it can be
re-formatted via ``tools/fmtjson``.

A number of bugs has been fixed in the cluster merge tool.

``x509`` certification verification (used in import-export) has been
changed to allow the same clock skew as permitted by the cluster
verification. This will remove some rare but hard to diagnose errors in
import-export.



Version 2.5.1
-------------

*(Released Fri, 11 May 2012)*

A small bugfix release.

The main issues solved are on the topic of compatibility with newer LVM
releases:

- fixed parsing of ``lv_attr`` field
- adapted to new ``vgreduce --removemissing`` behaviour where sometimes
  the ``--force`` flag is needed

Also on the topic of compatibility, ``tools/lvmstrap`` has been changed
to accept kernel 3.x too (was hardcoded to 2.6.*).

A regression present in 2.5.0 that broke handling (in the gnt-* scripts)
of hook results and that also made display of other errors suboptimal
was fixed; the code behaves now like 2.4 and earlier.

Another change in 2.5, the cleanup of the OS scripts environment, is too
aggressive: it removed even the ``PATH`` variable, which requires the OS
scripts to *always* need to export it. Since this is a bit too strict,
we now export a minimal PATH, the same that we export for hooks.

The fix for issue 201 (Preserve bridge MTU in KVM ifup script) was
integrated into this release.

Finally, a few other miscellaneous changes were done (no new features,
just small improvements):

- Fix ``gnt-group --help`` display
- Fix hardcoded Xen kernel path
- Fix grow-disk handling of invalid units
- Update synopsis for ``gnt-cluster repair-disk-sizes``
- Accept both PUT and POST in noded (makes future upgrade to 2.6 easier)


Version 2.5.0
-------------

*(Released Thu, 12 Apr 2012)*

Incompatible/important changes and bugfixes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- The default of the ``/2/instances/[instance_name]/rename`` RAPI
  resource's ``ip_check`` parameter changed from ``True`` to ``False``
  to match the underlying LUXI interface.
- The ``/2/nodes/[node_name]/evacuate`` RAPI resource was changed to use
  body parameters, see :doc:`RAPI documentation <rapi>`. The server does
  not maintain backwards-compatibility as the underlying operation
  changed in an incompatible way. The RAPI client can talk to old
  servers, but it needs to be told so as the return value changed.
- When creating file-based instances via RAPI, the ``file_driver``
  parameter no longer defaults to ``loop`` and must be specified.
- The deprecated ``bridge`` NIC parameter is no longer supported. Use
  ``link`` instead.
- Support for the undocumented and deprecated RAPI instance creation
  request format version 0 has been dropped. Use version 1, supported
  since Ganeti 2.1.3 and :doc:`documented <rapi>`, instead.
- Pyparsing 1.4.6 or above is required, see :doc:`installation
  documentation <install>`.
- The "cluster-verify" hooks are now executed per group by the
  ``OP_CLUSTER_VERIFY_GROUP`` opcode. This maintains the same behavior
  if you just run ``gnt-cluster verify``, which generates one opcode per
  group.
- The environment as passed to the OS scripts is cleared, and thus no
  environment variables defined in the node daemon's environment will be
  inherited by the scripts.
- The :doc:`iallocator <iallocator>` mode ``multi-evacuate`` has been
  deprecated.
- :doc:`New iallocator modes <design-multi-reloc>` have been added to
  support operations involving multiple node groups.
- Offline nodes are ignored when failing over an instance.
- Support for KVM version 1.0, which changed the version reporting format
  from 3 to 2 digits.
- TCP/IP ports used by DRBD disks are returned to a pool upon instance
  removal.
- ``Makefile`` is now compatible with Automake 1.11.2
- Includes all bugfixes made in the 2.4 series

New features
~~~~~~~~~~~~

- The ganeti-htools project has been merged into the ganeti-core source
  tree and will be built as part of Ganeti (see :doc:`install-quick`).
- Implemented support for :doc:`shared storage <design-shared-storage>`.
- Add support for disks larger than 2 TB in ``lvmstrap`` by supporting
  GPT-style partition tables (requires `parted
  <http://www.gnu.org/s/parted/>`_).
- Added support for floppy drive and 2nd CD-ROM drive in KVM hypervisor.
- Allowed adding tags on instance creation.
- Export instance tags to hooks (``INSTANCE_TAGS``, see :doc:`hooks`)
- Allow instances to be started in a paused state, enabling the user to
  see the complete console output on boot using the console.
- Added new hypervisor flag to control default reboot behaviour
  (``reboot_behavior``).
- Added support for KVM keymaps (hypervisor parameter ``keymap``).
- Improved out-of-band management support:

  - Added ``gnt-node health`` command reporting the health status of
    nodes.
  - Added ``gnt-node power`` command to manage power status of nodes.
  - Added command for emergency power-off (EPO), ``gnt-cluster epo``.

- Instance migration can fall back to failover if instance is not
  running.
- Filters can be used when listing nodes, instances, groups and locks;
  see *ganeti(7)* manpage.
- Added post-execution status as variables to :doc:`hooks <hooks>`
  environment.
- Instance tags are exported/imported together with the instance.
- When given an explicit job ID, ``gnt-job info`` will work for archived
  jobs.
- Jobs can define dependencies on other jobs (not yet supported via
  RAPI or command line, but used by internal commands and usable via
  LUXI).

  - Lock monitor (``gnt-debug locks``) shows jobs waiting for
    dependencies.

- Instance failover is now available as a RAPI resource
  (``/2/instances/[instance_name]/failover``).
- ``gnt-instance info`` defaults to static information if primary node
  is offline.
- Opcodes have a new ``comment`` attribute.
- Added basic SPICE support to KVM hypervisor.
- ``tools/ganeti-listrunner`` allows passing of arguments to executable.

Node group improvements
~~~~~~~~~~~~~~~~~~~~~~~

- ``gnt-cluster verify`` has been modified to check groups separately,
  thereby improving performance.
- Node group support has been added to ``gnt-cluster verify-disks``,
  which now operates per node group.
- Watcher has been changed to work better with node groups.

  - One process and state file per node group.
  - Slow watcher in one group doesn't block other group's watcher.

- Added new command, ``gnt-group evacuate``, to move all instances in a
  node group to other groups.
- Added ``gnt-instance change-group`` to move an instance to another
  node group.
- ``gnt-cluster command`` and ``gnt-cluster copyfile`` now support
  per-group operations.
- Node groups can be tagged.
- Some operations switch from an exclusive to a shared lock as soon as
  possible.
- Instance's primary and secondary nodes' groups are now available as
  query fields (``pnode.group``, ``pnode.group.uuid``, ``snodes.group``
  and ``snodes.group.uuid``).

Misc
~~~~

- Numerous updates to documentation and manpages.

  - :doc:`RAPI <rapi>` documentation now has detailed parameter
    descriptions.
  - Some opcode/job results are now also documented, see :doc:`RAPI
    <rapi>`.

- A lockset's internal lock is now also visible in lock monitor.
- Log messages from job queue workers now contain information about the
  opcode they're processing.
- ``gnt-instance console`` no longer requires the instance lock.
- A short delay when waiting for job changes reduces the number of LUXI
  requests significantly.
- DRBD metadata volumes are overwritten with zeros during disk creation.
- Out-of-band commands no longer acquire the cluster lock in exclusive
  mode.
- ``devel/upload`` now uses correct permissions for directories.


Version 2.5.0 rc6
-----------------

*(Released Fri, 23 Mar 2012)*

This was the sixth release candidate of the 2.5 series.


Version 2.5.0 rc5
-----------------

*(Released Mon, 9 Jan 2012)*

This was the fifth release candidate of the 2.5 series.


Version 2.5.0 rc4
-----------------

*(Released Thu, 27 Oct 2011)*

This was the fourth release candidate of the 2.5 series.


Version 2.5.0 rc3
-----------------

*(Released Wed, 26 Oct 2011)*

This was the third release candidate of the 2.5 series.


Version 2.5.0 rc2
-----------------

*(Released Tue, 18 Oct 2011)*

This was the second release candidate of the 2.5 series.


Version 2.5.0 rc1
-----------------

*(Released Tue, 4 Oct 2011)*

This was the first release candidate of the 2.5 series.


Version 2.5.0 beta3
-------------------

*(Released Wed, 31 Aug 2011)*

This was the third beta release of the 2.5 series.


Version 2.5.0 beta2
-------------------

*(Released Mon, 22 Aug 2011)*

This was the second beta release of the 2.5 series.


Version 2.5.0 beta1
-------------------

*(Released Fri, 12 Aug 2011)*

This was the first beta release of the 2.5 series.


Version 2.4.5
-------------

*(Released Thu, 27 Oct 2011)*

- Fixed bug when parsing command line parameter values ending in
  backslash
- Fixed assertion error after unclean master shutdown
- Disable HTTP client pool for RPC, significantly reducing memory usage
  of master daemon
- Fixed queue archive creation with wrong permissions


Version 2.4.4
-------------

*(Released Tue, 23 Aug 2011)*

Small bug-fixes:

- Fixed documentation for importing with ``--src-dir`` option
- Fixed a bug in ``ensure-dirs`` with queue/archive permissions
- Fixed a parsing issue with DRBD 8.3.11 in the Linux kernel


Version 2.4.3
-------------

*(Released Fri, 5 Aug 2011)*

Many bug-fixes and a few small features:

- Fixed argument order in ``ReserveLV`` and ``ReserveMAC`` which caused
  issues when you tried to add an instance with two MAC addresses in one
  request
- KVM: fixed per-instance stored UID value
- KVM: configure bridged NICs at migration start
- KVM: Fix a bug where instance will not start with never KVM versions
  (>= 0.14)
- Added OS search path to ``gnt-cluster info``
- Fixed an issue with ``file_storage_dir`` where you were forced to
  provide an absolute path, but the documentation states it is a
  relative path, the documentation was right
- Added a new parameter to instance stop/start called ``--no-remember``
  that will make the state change to not be remembered
- Implemented ``no_remember`` at RAPI level
- Improved the documentation
- Node evacuation: don't call IAllocator if node is already empty
- Fixed bug in DRBD8 replace disks on current nodes
- Fixed bug in recreate-disks for DRBD instances
- Moved assertion checking locks in ``gnt-instance replace-disks``
  causing it to abort with not owning the right locks for some situation
- Job queue: Fixed potential race condition when cancelling queued jobs
- Fixed off-by-one bug in job serial generation
- ``gnt-node volumes``: Fix instance names
- Fixed aliases in bash completion
- Fixed a bug in reopening log files after being sent a SIGHUP
- Added a flag to burnin to allow specifying VCPU count
- Bugfixes to non-root Ganeti configuration


Version 2.4.2
-------------

*(Released Thu, 12 May 2011)*

Many bug-fixes and a few new small features:

- Fixed a bug related to log opening failures
- Fixed a bug in instance listing with orphan instances
- Fixed a bug which prevented resetting the cluster-level node parameter
  ``oob_program`` to the default
- Many fixes related to the ``cluster-merge`` tool
- Fixed a race condition in the lock monitor, which caused failures
  during (at least) creation of many instances in parallel
- Improved output for gnt-job info
- Removed the quiet flag on some ssh calls which prevented debugging
  failures
- Improved the N+1 failure messages in cluster verify by actually
  showing the memory values (needed and available)
- Increased lock attempt timeouts so that when executing long operations
  (e.g. DRBD replace-disks) other jobs do not enter 'blocking acquire'
  too early and thus prevent the use of the 'fair' mechanism
- Changed instance query data (``gnt-instance info``) to not acquire
  locks unless needed, thus allowing its use on locked instance if only
  static information is asked for
- Improved behaviour with filesystems that do not support rename on an
  opened file
- Fixed the behaviour of ``prealloc_wipe_disks`` cluster parameter which
  kept locks on all nodes during the wipe, which is unneeded
- Fixed ``gnt-watcher`` handling of errors during hooks execution
- Fixed bug in ``prealloc_wipe_disks`` with small disk sizes (less than
  10GiB) which caused the wipe to fail right at the end in some cases
- Fixed master IP activation when doing master failover with no-voting
- Fixed bug in ``gnt-node add --readd`` which allowed the re-adding of
  the master node itself
- Fixed potential data-loss in under disk full conditions, where Ganeti
  wouldn't check correctly the return code and would consider
  partially-written files 'correct'
- Fixed bug related to multiple VGs and DRBD disk replacing
- Added new disk parameter ``metavg`` that allows placement of the meta
  device for DRBD in a different volume group
- Fixed error handling in the node daemon when the system libc doesn't
  have major number 6 (i.e. if ``libc.so.6`` is not the actual libc)
- Fixed lock release during replace-disks, which kept cluster-wide locks
  when doing disk replaces with an iallocator script
- Added check for missing bridges in cluster verify
- Handle EPIPE errors while writing to the terminal better, so that
  piping the output to e.g. ``less`` doesn't cause a backtrace
- Fixed rare case where a ^C during Luxi calls could have been
  interpreted as server errors, instead of simply terminating
- Fixed a race condition in LUGroupAssignNodes (``gnt-group
  assign-nodes``)
- Added a few more parameters to the KVM hypervisor, allowing a second
  CDROM, custom disk type for CDROMs and a floppy image
- Removed redundant message in instance rename when the name is given
  already as a FQDN
- Added option to ``gnt-instance recreate-disks`` to allow creating the
  disks on new nodes, allowing recreation when the original instance
  nodes are completely gone
- Added option when converting disk templates to DRBD to skip waiting
  for the resync, in order to make the instance available sooner
- Added two new variables to the OS scripts environment (containing the
  instance's nodes)
- Made the root_path and optional parameter for the xen-pvm hypervisor,
  to allow use of ``pvgrub`` as bootloader
- Changed the instance memory modifications to only check out-of-memory
  conditions on memory increases, and turned the secondary node warnings
  into errors (they can still be overridden via ``--force``)
- Fixed the handling of a corner case when the Python installation gets
  corrupted (e.g. a bad disk) while ganeti-noded is running and we try
  to execute a command that doesn't exist
- Fixed a bug in ``gnt-instance move`` (LUInstanceMove) when the primary
  node of the instance returned failures during instance shutdown; this
  adds the option ``--ignore-consistency`` to gnt-instance move

And as usual, various improvements to the error messages, documentation
and man pages.


Version 2.4.1
-------------

*(Released Wed, 09 Mar 2011)*

Emergency bug-fix release. ``tools/cfgupgrade`` was broken and overwrote
the RAPI users file if run twice (even with ``--dry-run``).

The release fixes that bug (nothing else changed).


Version 2.4.0
-------------

*(Released Mon, 07 Mar 2011)*

Final 2.4.0 release. Just a few small fixes:

- Fixed RAPI node evacuate
- Fixed the kvm-ifup script
- Fixed internal error handling for special job cases
- Updated man page to specify the escaping feature for options


Version 2.4.0 rc3
-----------------

*(Released Mon, 28 Feb 2011)*

A critical fix for the ``prealloc_wipe_disks`` feature: it is possible
that this feature wiped the disks of the wrong instance, leading to loss
of data.

Other changes:

- Fixed title of query field containing instance name
- Expanded the glossary in the documentation
- Fixed one unittest (internal issue)


Version 2.4.0 rc2
-----------------

*(Released Mon, 21 Feb 2011)*

A number of bug fixes plus just a couple functionality changes.

On the user-visible side, the ``gnt-* list`` command output has changed
with respect to "special" field states. The current rc1 style of display
can be re-enabled by passing a new ``--verbose`` (``-v``) flag, but in
the default output mode special fields states are displayed as follows:

- Offline resource: ``*``
- Unavailable/not applicable: ``-``
- Data missing (RPC failure): ``?``
- Unknown field: ``??``

Another user-visible change is the addition of ``--force-join`` to
``gnt-node add``.

As for bug fixes:

- ``tools/cluster-merge`` has seen many fixes and is now enabled again
- Fixed regression in RAPI/instance reinstall where all parameters were
  required (instead of optional)
- Fixed ``gnt-cluster repair-disk-sizes``, was broken since Ganeti 2.2
- Fixed iallocator usage (offline nodes were not considered offline)
- Fixed ``gnt-node list`` with respect to non-vm_capable nodes
- Fixed hypervisor and OS parameter validation with respect to
  non-vm_capable nodes
- Fixed ``gnt-cluster verify`` with respect to offline nodes (mostly
  cosmetic)
- Fixed ``tools/listrunner`` with respect to agent-based usage


Version 2.4.0 rc1
-----------------

*(Released Fri,  4 Feb 2011)*

Many changes and fixes since the beta1 release. While there were some
internal changes, the code has been mostly stabilised for the RC
release.

Note: the dumb allocator was removed in this release, as it was not kept
up-to-date with the IAllocator protocol changes. It is recommended to
use the ``hail`` command from the ganeti-htools package.

Note: the 2.4 and up versions of Ganeti are not compatible with the
0.2.x branch of ganeti-htools. You need to upgrade to
ganeti-htools-0.3.0 (or later).

Regressions fixed from 2.3
~~~~~~~~~~~~~~~~~~~~~~~~~~

- Fixed the ``gnt-cluster verify-disks`` command
- Made ``gnt-cluster verify-disks`` work in parallel (as opposed to
  serially on nodes)
- Fixed disk adoption breakage
- Fixed wrong headers in instance listing for field aliases

Other bugs fixed
~~~~~~~~~~~~~~~~

- Fixed corner case in KVM handling of NICs
- Fixed many cases of wrong handling of non-vm_capable nodes
- Fixed a bug where a missing instance symlink was not possible to
  recreate with any ``gnt-*`` command (now ``gnt-instance
  activate-disks`` does it)
- Fixed the volume group name as reported by ``gnt-cluster
  verify-disks``
- Increased timeouts for the import-export code, hopefully leading to
  fewer aborts due network or instance timeouts
- Fixed bug in ``gnt-node list-storage``
- Fixed bug where not all daemons were started on cluster
  initialisation, but only at the first watcher run
- Fixed many bugs in the OOB implementation
- Fixed watcher behaviour in presence of instances with offline
  secondaries
- Fixed instance list output for instances running on the wrong node
- a few fixes to the cluster-merge tool, but it still cannot merge
  multi-node groups (currently it is not recommended to use this tool)


Improvements
~~~~~~~~~~~~

- Improved network configuration for the KVM hypervisor
- Added e1000 as a supported NIC for Xen-HVM
- Improved the lvmstrap tool to also be able to use partitions, as
  opposed to full disks
- Improved speed of disk wiping (the cluster parameter
  ``prealloc_wipe_disks``, so that it has a low impact on the total time
  of instance creations
- Added documentation for the OS parameters
- Changed ``gnt-instance deactivate-disks`` so that it can work if the
  hypervisor is not responding
- Added display of blacklisted and hidden OS information in
  ``gnt-cluster info``
- Extended ``gnt-cluster verify`` to also validate hypervisor, backend,
  NIC and node parameters, which might create problems with currently
  invalid (but undetected) configuration files, but prevents validation
  failures when unrelated parameters are modified
- Changed cluster initialisation to wait for the master daemon to become
  available
- Expanded the RAPI interface:

  - Added config redistribution resource
  - Added activation/deactivation of instance disks
  - Added export of console information

- Implemented log file reopening on SIGHUP, which allows using
  logrotate(8) for the Ganeti log files
- Added a basic OOB helper script as an example


Version 2.4.0 beta1
-------------------

*(Released Fri, 14 Jan 2011)*

User-visible
~~~~~~~~~~~~

- Fixed timezone issues when formatting timestamps
- Added support for node groups, available via ``gnt-group`` and other
  commands
- Added out-of-band framework and management, see :doc:`design
  document <design-oob>`
- Removed support for roman numbers from ``gnt-node list`` and
  ``gnt-instance list``.
- Allowed modification of master network interface via ``gnt-cluster
  modify --master-netdev``
- Accept offline secondaries while shutting down instance disks
- Added ``blockdev_prefix`` parameter to Xen PVM and HVM hypervisors
- Added support for multiple LVM volume groups
- Avoid sorting nodes for ``gnt-node list`` if specific nodes are
  requested
- Added commands to list available fields:

  - ``gnt-node list-fields``
  - ``gnt-group list-fields``
  - ``gnt-instance list-fields``

- Updated documentation and man pages

Integration
~~~~~~~~~~~

- Moved ``rapi_users`` file into separate directory, now named
  ``.../ganeti/rapi/users``, ``cfgupgrade`` moves the file and creates a
  symlink
- Added new tool for running commands on many machines,
  ``tools/ganeti-listrunner``
- Implemented more verbose result in ``OpInstanceConsole`` opcode, also
  improving the ``gnt-instance console`` output
- Allowed customisation of disk index separator at ``configure`` time
- Export node group allocation policy to :doc:`iallocator <iallocator>`
- Added support for non-partitioned md disks in ``lvmstrap``
- Added script to gracefully power off KVM instances
- Split ``utils`` module into smaller parts
- Changed query operations to return more detailed information, e.g.
  whether an information is unavailable due to an offline node. To use
  this new functionality, the LUXI call ``Query`` must be used. Field
  information is now stored by the master daemon and can be retrieved
  using ``QueryFields``. Instances, nodes and groups can also be queried
  using the new opcodes ``OpQuery`` and ``OpQueryFields`` (not yet
  exposed via RAPI). The following commands make use of this
  infrastructure change:

  - ``gnt-group list``
  - ``gnt-group list-fields``
  - ``gnt-node list``
  - ``gnt-node list-fields``
  - ``gnt-instance list``
  - ``gnt-instance list-fields``
  - ``gnt-debug locks``

Remote API
~~~~~~~~~~

- New RAPI resources (see :doc:`rapi`):

  - ``/2/modify``
  - ``/2/groups``
  - ``/2/groups/[group_name]``
  - ``/2/groups/[group_name]/assign-nodes``
  - ``/2/groups/[group_name]/modify``
  - ``/2/groups/[group_name]/rename``
  - ``/2/instances/[instance_name]/disk/[disk_index]/grow``

- RAPI changes:

  - Implemented ``no_install`` for instance creation
  - Implemented OS parameters for instance reinstallation, allowing
    use of special settings on reinstallation (e.g. for preserving data)

Misc
~~~~

- Added IPv6 support in import/export
- Pause DRBD synchronization while wiping disks on instance creation
- Updated unittests and QA scripts
- Improved network parameters passed to KVM
- Converted man pages from docbook to reStructuredText


Version 2.3.1
-------------

*(Released Mon, 20 Dec 2010)*

Released version 2.3.1~rc1 without any changes.


Version 2.3.1 rc1
-----------------

*(Released Wed, 1 Dec 2010)*

- impexpd: Disable OpenSSL compression in socat if possible (backport
  from master, commit e90739d625b, see :doc:`installation guide
  <install-quick>` for details)
- Changed unittest coverage report to exclude test scripts
- Added script to check version format


Version 2.3.0
-------------

*(Released Wed, 1 Dec 2010)*

Released version 2.3.0~rc1 without any changes.


Version 2.3.0 rc1
-----------------

*(Released Fri, 19 Nov 2010)*

A number of bugfixes and documentation updates:

- Update ganeti-os-interface documentation
- Fixed a bug related to duplicate MACs or similar items which should be
  unique
- Fix breakage in OS state modify
- Reinstall instance: disallow offline secondaries (fixes bug related to
  OS changing but reinstall failing)
- plus all the other fixes between 2.2.1 and 2.2.2


Version 2.3.0 rc0
-----------------

*(Released Tue, 2 Nov 2010)*

- Fixed clearing of the default iallocator using ``gnt-cluster modify``
- Fixed master failover race with watcher
- Fixed a bug in ``gnt-node modify`` which could lead to an inconsistent
  configuration
- Accept previously stopped instance for export with instance removal
- Simplify and extend the environment variables for instance OS scripts
- Added new node flags, ``master_capable`` and ``vm_capable``
- Added optional instance disk wiping prior during allocation. This is a
  cluster-wide option and can be set/modified using
  ``gnt-cluster {init,modify} --prealloc-wipe-disks``.
- Added IPv6 support, see :doc:`design document <design-2.3>` and
  :doc:`install-quick`
- Added a new watcher option (``--ignore-pause``)
- Added option to ignore offline node on instance start/stop
  (``--ignore-offline``)
- Allow overriding OS parameters with ``gnt-instance reinstall``
- Added ability to change node's secondary IP address using ``gnt-node
  modify``
- Implemented privilege separation for all daemons except
  ``ganeti-noded``, see ``configure`` options