This patch changes the definition of a job and introduces per-opcode
results.

First, the result and status fields of a job are condensed into a single
'status' attribute. Then, we introduce an opcode status and one result
list, that allow jobs to return values.

The gnt-job script is also modified to allow these new fields to be
queried.

Note that the patch changes the opcode field to op_list, and it changes
its return value from string to a list of (serialized) opcodes.

Reviewed-by: ultrotter

Based on the previous OS search code changes, we can now move the OS
search code into a generic look-for-file function in utils.py. This
means that the allocator code can use the same function.

Reviewed-by: ultrotter

Currently, the function backend._OSSearch() returns the (first) base dir
in which this OS can be found. Thereafter the full actual path to the OS
dir is built in the backend.OSFromDisk() function.

This patch changes this so that _OSSearch() always returns the full path
to the OS directory, and OSFromDisk uses that as returned (it will only
build it if it gets a base dir in the first place).

This patch is needed before we can abstract the _OSSearch into a generic
'look for file object' functionality that can be used for allocator
plugins search too.

Reviewed-by: ultrotter

Currently, the example scripts (the init.d and the cron script) do not
depend on the Makefile, so if configure is rerun with different
parameters these scripts are not rebuild.

The patch adds Makefile as a dependency so changes to installation paths
or other parameters trigger the rebuild.

Reviewed-by: imsnah

The list of instances passed to _VerifyInstance is the one coming from
self.cfg.GetInstanceList(). So there's no point, inside that function, in
checking whether the current instance is a member of that list. Moreover
orphaned instance verification is already done in a separate step.

Reviewed-by: imsnah

The instance configuration is grabbed both in the _VerifyInstance function and
in the loop that calls it. Clean this up by passing the configuration as a
parameter.

Reviewed-by: imsnah

Currently if ganeti-noded doesn't respond on a node gnt-cluster verify will die
when verifying primary instances for that node. Fix this by just emitting an
error message if no information about running instances is returned from the
node.

Reviewed-by: iustinp

All ERROR messages in cluster verify are indented by four spaces, this one is
indented by two. Fixing this skew.

Reviewed-by: imsnah, iustinp

Of course instance creation don't have any modem, and the comment was just
talking about modes. Sorry to everybody expecting whistles.

Reviewed-by: imsnah

This patch adds the previously introduced lvm storage toggling
info to the gnt-cluster man page. It adds some info the the
gnt-cluster init -g option and adds the gnt-cluster modify command.

Reviewed-by: iustinp

Reviewed-by: imsnah

os.path.join does not like None as argument and fails with
AttributeError: 'NoneType' object has no attribute 'startswith'.

This patch makes sure the passed argument is a string in any case.

Reviewed-by: iustinp

Reviewed-by: imsnah

Reviewed-by: ultrotter

Reviewed-by: ultrotter, iustinp

This patch does two things:
- Modify LURenameInstance.Exec to rename directory
  when a file-based instance is renamed
- Modify config.RenameInstance() to replace the directory name in
  config.data for file devices

Reviewed-by: iustinp

- Modfiy _GenerateDiskTemplate to support file-based disk template
- Modify _CreateDisks to create directory needed for file-based
  instances before creating the actual files
- Modify _RemoveDisks to delete directory for file-based instances
  after deleting their VBDs
- Add Prereq-check to check if given file-driver is valid
- Add Prereq-check to check if given file-storage-dir path is relative

Reviewed-by: iustinp

Reviewed-by: iustinp

Having the individual parts in the code allows us to build version
numbers like "1.2" while leaving "3" out in a clean fashion, that is
without regular expressions or the like. This might be used together
with configuration format versions.

Why m4 code? AM_INIT_AUTOMAKE, which could take a shell variable, is
considered deprecated[1] and should be replaced by AC_INIT. Unfortunately,
AC_INIT is expanded at build time, so one has to use m4 to build
composite values like this version number[2].

[1] http://www.gnu.org/software/libtool/manual/automake/Public-macros.html
[2] http://www.mail-archive.com/autoconf@gnu.org/msg16720.html

Reviewed-by: iustinp

The driver in the xen config file needs to be changed when dealing with
files rather then bdevs.

This patch does two things:
- Add _GetConfigFileDiskData to XenHypervisor which returns the correct
  disk xen config line. This function checks the logical disk type of
  every given block device, such that also hybrid (e.g. mixed drbd and
  file VBDs) are possible
- Make Xen[Pvm|Hvm]Hypervisor._WriteConfigFile() a classmethod to be
  able to call the helper function _GetConfigFileDiskData() in their
  parent XenHypervisor

Reviewed-by: iustinp

Reviewed-by: iustinp

Reviewed-by: iustinp

Reviewed-by: iustinp

This patch implements query-ing of only selected jobs instead of all.

Reviewed-by: ultrotter

The idea of cli.py and luxi.py is that all protocol checks should be in
luxi, and cli.py should just offer some helpful shortcuts for the
command line scripts.

This patch removes the result checks from cli and adds some other checks
to luxi. It does no longer check the success/failure since it's not yet
clear how that should be handled - probably exceptions.

Reviewed-by: ultrotter

This small patch fixed the opcodes.OpCode.LoadOpcode capitalization to
what was intented to be (as the comment says): LoadOpCode.

Reviewed-by: ultrotter

This patch adds checks for the master role and daemonize support to
ganeti-masterd.

The patch modifies the startup/shutdown of the server because:
  - we want bind()/listen() to the master socket to occur before forking
    so that we can return a correct exit code and write messages to
    stderr
  - but we want thread startup to occur after fork(), otherwise python
    threading gets confused

The patch also has some small cleanups:
  - remove the unix socket after closing it, so we don't need to remove
    it manually
  - instead of just telling the threads to terminate via the new_queue,
    we also join() them so that the logs show what thread clinging to
    life
  - the daemon logs to its own logfile now
  - there is command line parameter support :)

Reviewed-by: imsnah

This is the representation of file VBDs on the backend. It's the first
implementation an supports only raw files.

Reviewed-by: iustinp

Reviewed-by: ultrotter

Reviewed-by: ultrotter

Add _[Create,Remove,Rename]FileStorageDir function which are needed for
file-based instance management. These function check whether the given
directory to operate on is under the cluster-wide defined default file
storage dir. If this is not the case the won't do anything and return
False. This is to prevent cluster manipulation or damage.

Reviewed-by: ultrotter

Reviewed-by: imsnah

This patch implements an optional parameter to utils.Daemonize() which
allows that function to not close some file descriptors.

This will allow the master daemon to open the listening socket before
fork - in order to be able to notify errors and return a meaningful exit
code, and then when we fork we don't close that fd.

Reviewed-by: imsnah

This patch adds a very basic gnt-job script that allows job querying.
This goes on top of the previous master daemon patches.

Currently, because of the not-changed cmd lock, you can't query the jobs
as long as a job is running - you have to rm the cmd lock and then you
can query the jobs.

Reviewed-by: imsnah

Currently, in ganeti-noded we have the createDaemon function. Since
we'll need the same in other daemons, we move this function to utils.py

With the move, a few changes were also done:
  - change the name to Daemonize()
  - add a parameter, logfile, as different daemons will want to log to
    different files
  - remove the try.. except.. around the fork calls, since they were
    only re-raising the OS exception with less data; unless we want to
    actually handle fork error (not just re-raising), these try blocks
    are not useful
  - change the return style at the end of the function

Reviewed-by: imsnah

This new version includes all the possible failure options.

Reviewed-by: iustinp

_CheckDiskConsistency outputs "Can't get any data from node NODE" when no drbd
is found on the target node. This causes a misleading error message to be
output for example on failover (when the primary node is down, or the instance
is not running), stating that no data could be got from the secondary node,
which scares the user and misleads him. Changing this to "Disk degraded or not
found on node %s" still reports that something is missing, but on the other
hand doesn't make the user think the node is down, or has no data at all...

Reviewed-by: imsnah

Right now if you try to failover an instance which is not marked as up the
operation will fail unless you pass the --ignore-consistency flag because the
disks won't be considered to be consistent. Allow them to be if we know the
instance shouldn't be up.

Reviewed-by: imsnah

Currently gnt-backup export chains the ShutdownInstance and StartupInstance
opcodes to itself. This works but (a) it's suboptimal, because there's no need
to deactivate the instance's disks as we are about to restart it anyway, and
(b) doesn't take care of instances which are already down (and should be). This
patch takes care of this by just calling the shutdown rpc function instead of
the whole opcode, and just starting up the instance if it's configured as up in
the first place.

Reviewed-by: imsnah

ConfigParser.SafeConfigParser doesn't support unicode string objects.
Unicode string objects are returned by simplejson.

Reviewed-by: iustinp