Reviewed-by: ultrotter

In cfgupgrade, we need to extract parts of and build new version numbers.

Reviewed-by: iustinp

cfgupgrade will not work with relative paths at all, but rather get them
from constants.py.

Reviewed-by: iustinp

Currently there isn't any easy way to list all nodes or instance and
their tags; you have to query each node in turn, or list all the tags
via something like “gnt-cluster search-tags '.*'”. Of course, this is
not optimal.

The patch adds a new fields to “gnt-instance list” and “gnt-node list”
called ‘tags’, that will list the tags of the object in comma-separated
form. This field will be empty if there are no tags (when using a
separator this output can still be parsed by other scripts).

At opcode level, there is a new fields called ‘tags’ that returns a
(python) list of the object tags.

Reviewed-by: ultrotter

Currently the generic handling of ganeti errors in cli.py (GenericMain
and FormatError) only handles the core ganeti errors, and not the client
protocol errors (which live in a separate hierarchy).

This patch adds handling of luxi errors too, and also adds another luxi
error for the case when the master is not running. This gives us a nice:

  gnta1:~# gnt-node list
  Cannot communicate with the master daemon.
  Is it running and listening on '/var/run/ganeti-master.sock'?

error message instead of a traceback.

Reviewed-by: amishchenko

This patch adds rpc layer calls (in rpc.py and the equivalent in
ganeti-noded) to close a list of block devices, and the wrapper in
backend.py that takes a list of Disk objects, identifies them and
returns correctly formatted results.

The reason why this very basic call was missing until now from the rpc
layer is that we usually don't care about device closes (though we
should, and will do so in the future) as only drbd has a meaningful
Close() operation; right now we directly do Shutdown().

The patch is clean enough that it's actually independent of the live
migration implementation.

Reviewed-by: imsnah

This change allows us to use cleaner dependencies between
directories. The build system is basically rewritten in large parts
and may contain bugs.

Reviewed-by: iustinp

Currently, compute the status of a drbd8 device in GetSyncStatus and
return only the values that we need (and fit in the framework of
GetSyncStatus). However, the full status details are useful (and needed)
in other places, so the patch attempts to improve this situation.

We abstract the status of a device outside in a separate class, that
knows how to parse contents from /proc/drbd and set easily accessible
attributes. We then simplify the GetSyncStatus to use this and return
the values that it needs, and add a separate method that returns the
full status object.

The move to a separate class cleans up a little bit the old
sync-progress computation from GetSyncStatus, but it's still many
regexes.

The patch also adds unittests for a few statuses, and modifies one
BaseDRBD call to accept a custom filename instead of '/proc/drbd' to
ease unittests.

Reviewed-by: imsnah

Reviewed-by: iustinp

- Make closing file optional: Required by ganeti-watcher to keep
  file open after writing it. Changes return value of utils.WriteFile
  if "close" parameter evaluates to True.
- Pre- and post-write functions: Can be used to lock files. This
  will be used by ganeti-watcher to lock the temporary file before
  renaming.

Reviewed-by: iustinp

- Log timestamp for all messages
- Write everything to logfile and optionally to stderr
- Log messages are no longer buffered, allowing a user to see progress

Reviewed-by: ultrotter

Also fix the regular expression to not remove newlines. The simplejson
module puts whitespace at line endings when using indentation. Remove
unnecessary import of ConfigParser module.

Reviewed-by: ultrotter

Currently, the way to customize a DRBD disk from (node name 1, node name
2, port) to (ip1, port, ip2, port) is to use the ConfigWriter method
SetDiskID. However, since this needs a ConfigWriter object, it can be
run only on the master, and therefore disk object can't be passed to
more than one node unchanged. This, coupled with the rpc layer
limitation that all nodes in a multi-node call receive the same
arguments, prevent any kind of multi-node operation that has disks as an
argument.

This patch takes the SetDiskID method from ConfigWriter and ports it to
the disk object itself, and instead of the full node configuration it
uses a simple {node_name: replication_ip} mapping for all the nodes
involved in the disk tree (currently we only pass primary and secondary
node since we don't support nested drbd devices).

This allows us to send disks to both the primary and secondary nodes at
once and perform synchronized drbd activation on primary/secondary
nodes.

Note that while for the 1.2 branch this will not change old methods, it
is worth to investigate and possible replace all such calls on the
master to the nodes themselves for the 2.0 branch.

Reviewed-by: ultrotter

There is a mistake in handling grow-disk for an invalid disk. This patch
fixes it.

Reviewed-by: imsnah

This patch adds a new opcode and LU for growing an instance's disk.

The opcode allows growing only one disk at time, and will throw an error
if the operation fails midway (e.g. on the primary node after it has
been increased on the secondary node). As such, it might actually leave
different sized LVs on different nodes, but this will not create
problems.

Reviewed-by: imsnah

This patch adds a method that implements updating of a disk
(object.Disk) size, together with its children.

While this will not track the exact disk size, it allows at least an
approximate size to be recorded in the configuration (and queried).

Reviewed-by: imsnah

This simple patch exposes the block device grow operation at the rpc
layer. It does not increase the protocol version as it has been recently
changed by the live failover rpc call.

Reviewed-by: imsnah

This patch adds a wrapper over the block device grow operation that
converts the input and output parameters as needed for the rpc layer.

Reviewed-by: imsnah

This patch implements disk resize at the bdev level for the LVM and
DRBD8 disk types. It is not implemented for DRBD7 and MD since the way
MD works with its underlaying devices makes it harder and this
combination is also deprecated.

The LVM resize operation is tried three times, with different allocation
policies:
  - contiguous first, since this is best for allocation purposes (it
    won't fragment too much the PV)
  - cling, which is supported only by more recent LVM versions, will try
    to place the new extents on the same PV as the rest of the LV
  - and finally normal, which is the default

Reviewed-by: imsnah

Before we used to be able to update SimpleStore by just calling SetKey, this
feature is now moved to an external class, which inherits from it. In this
patch the new WritableSimpleStore class is also put to use, in the LUs that
need it. Rather than making each LU instantiate it, we have a new LogicalUnit
flag REQ_WSSTORE which defaults to False, but when declared to be True asks the
LogicalUnit to be initialized with a writeable version of the SimpleStore.
LUMasterFailover and LURenameCluster are then changed to use it.

InitCluster is also changed to instantiate a WritableSimpleStore, rather
than a normal one.

Reviewed-by: imsnah

This patch adds the migration rpc call and its implementation in the
backend. The patch does not deal with the correct activation of disks.

Because of the new RPC, the protocol version is increased.

Reviewed-by: imsnah

This is just the hypervisor-level migration (e.g. “xm migrate”) not the
whole node coordination work.

Reviewed-by: ultrotter

When replacing disks or evacuating nodes with instances administratively
down ganeti fails because the instance disks are not active. This patch
activates them, performs the replacement, and shuts them down again.
Changing this also fixes the same issue on gnt-node evacuate.

Reviewed-by: iustinp

We're not adding a new instance, just making configuration changes to
the one we're working on.

Reviewed-by: imsnah

There is a mistake in the error message generated when we can't reach a
node for checking for available disk space. Without it, the error
message is:
Failure: prerequisites not met for this operation:
Cannot get current information from node '{u'gnte2.lab.k1024.org':
{'cpu_total': 1, 'memory_free': 480, 'vg_size': 131068, 'memory_total':
504, 'bootid': '2176dd3b-2f96-42f0-8b6e-2873ecaf5f9c', 'memory_dom0':
134, 'vg_free': 130172}, u'gnte1.lab.k1024.org': False}'

instead of the expected:
Failure: prerequisites not met for this operation:
Cannot get current information from node 'gnte2.lab.k1024.org'

Reviewed-by: imsnah

- Shorter code
- Reorder arguments to logger.SetupLogging calls to make more sense

Reviewed-by: iustinp

s/result/op_result/ (this code was never used, so this wasn't caught)

Reviewed-by: iustinp

This allows us to initialize a new cluster. The code certainly contains
bugs and hooks aren't implemented yet.

Reviewed-by: iustinp

This is required for splitting the cluster initialization code.

Reviewed-by: iustinp

This is required for the split of the cluster initialization code.

Reviewed-by: iustinp, ultrotter

It's not needed anymore now that all opcodes require a cluster. Cluster
initialization was the only exception.

Reviewed-by: iustinp

Reviewed-by: iustinp

…/run/master.sock is not specific enough.

Reviewed-by: iustinp

SimpleStore is instantiated anew most of the times it's used, so having
a cache inside it serves no purpose. Removing it.

Reviewed-by: iustinp

The remote API will use JSON for the foreseable future, so it's better
to put the serialization format in the function name. We can still
use another serialization format for Ganeti's core.

Reviewed-by: amishchenko, schreiberal

Currently the check for enough memory is done only on instance start
command and failover command. But we also start an instance in instance
create, therefore we need to check this instead of failing to start in
the hypervisor phase.

The patch adds a check for node memory in the case the creation command
specifies that the instance should be started. It is allowed for the
memory to be less than needed if the instance will not be started, in
order to allow migration and other such cases.

Reviewed-by: imsnah

Author: schreiberal
Reviewed-by: iustinp

This fixes gnt-instance modify so it actually works with the
new HVM parameters for Ganeti 1.2

Author: schreiberal
Reviewed-by: iustinp

This patch modifies the code for "gnt-instance info .." to only display
instance parameters that actually apply to that instance, i.e. for PVM
instances no HVM parameters are shown and vice versa.

Author: schreiberal
Reviewed-by: iustinp

This patch adds hypervisor support for the extended HVM features.

Author: schreiberal
Reviewed-by: iustinp