This is an extract of commit 1166 on the 1.2 branch (Add a rpc call for
drbd network reconfiguration), but only the blockdev_close part.

The patch changes the blockdev_close call to take the instance so that
it can remove the symlinks of the instance.

Originally-Reviewed-by: imsnah

This is a forward-port of commit 1163 on the 1.2 branch:
  This fixes the removal of the instance symlinks (probably breakage from
  the glob changes).

  Reviewed-by: imsnah

This is a forward-port of commits 1150 and 1151 on the 1.2 branch:
  Add _RemoveBlockDevLinks auxiliary function, called when an instance
  fails to start and when it is shut down.

  Reviewed-by: iustinp

and:
  Fix cut&paste error when removing symlinks

  It's just whitespace... isn't it? uhm... :) Anyway, fixing an error made
  when reformatting the code for the new "safer" behaviour.

  Reviewed-by: iustinp

Original-Author: ultrotter

This is a forward-port of commit 1149 on the 1.2 branch:
  _GatherAndLinkBlockDevs used to raise the errors.BlockDeviceError
  exception when it failed to create a block device, and with this patch
  set it does so also when it fails to create a symlink to it.

  With this patch we move the call to this function into a pre-existing
  try-except block in the code, and catch the BlockDeviceError exception,
  logging a message and returning a failure state if it happens.

  Reviewed-by: iustinp

The changes are related to the new hypervisor and logging syntax.

Original-Author: ultrotter

This is a forward-port of commit 1148 on the 1.2 branch:
  Change the _GatherBlockDevs private function, called only one time by
  StartInstance, to _GatherAndLinkBlockDevs, and make it transform the
  device returned even more by calling the new _SimlinkBlockDev auxiliary
  function.

  This makes sure that every time an instance is started symlinks to its
  block devices are created, and the instance is started off them, rather
  than the underlying block devices.

  Reviewed-by: iustinp

The changes we make to the patch is related to newer function signatures
in 2.0, and to the fact that iv_name is deprecated and we use instead
disk%d based on the disk index.

Original-Author: ultrotter

This is a partial forward-port of commit 1136 on the 1.2 branch:

  The hypervisor doesn't need to be passed the whole block device
  structure, so we'll just give it the block device name on the local
  node, and the name as seen by the instance. This will make it easier to
  manipulate it later without messing with the block devices (eg. by
  changing the system name to a symlink to the name itself).

  Since the HVM hypervisor changes the "virtual" name a note is added
  calling for a redesign that doesn't need this change, as different
  hypervisors and emulation types will anyway have different names for
  exported devices.

  Reviewed-by: iustinp

The changes in this patch compared to the original are:
  - we keep passing the original disk object, not for its iv_name, but
    for it's physical_id which is needed by the file driver (this could
    be fixed maybe)
  - we don't use the iv_name anymore, since in 2.0 we already use the
    index of the device

Original-Author: ultrotter

Commit 2117 changed _AssembleInstanceDisks to correctly parse the
failure status of the new RpcResult structure, but it didn't fix the
storing of only the result payload. Since RpcResult is not JSON
serializable, LUActivateInstanceDisks is failing.

Reviewed-by: ultrotter

Two bad indentation cases and a missing variable.

Reviewed-by: imsnah

Reviewed-by: amishchenko

Reviewed-by: amishchenko

Reviewed-by: amishchenko

Reviewed-by: amishchenko

Previously, we set the permissions only after writing the key. This
gave other users on the system a small window during which they could
read the key.

Reviewed-by: amishchenko

Reviewed-by: amishchenko

Reviewed-by: amishchenko

Passwords are stored in "$localstatedir/lib/ganeti/rapi_users". User
options specify the access permissions of a user (see docstring for
ganeti.http.ReadPasswordFile), for which only "write" is supported
to grant write access. Every other user has read-only access.

Reviewed-by: amishchenko

This will be used to evaluate access permissions to resources.

Reviewed-by: amishchenko

Lines in the password file are of the following format:

  <username> <password> [options]

Fields are separated by whitespace. Username and password are
mandatory, options are optional and separated by comma (",").
Empty lines and comments ("#") are ignored.

Reviewed-by: amishchenko

Reviewed-by: amishchenko

As per RFC2617.

Reviewed-by: amishchenko

The authentication class will override PreHandleRequest.

Reviewed-by: amishchenko

Reviewed-by: ultrotter

Reducing the actual number of RPC calls will come in another patch.

Reviewed-by: ultrotter

With a large job queue, auto-archiving jobs can take a very long time,
causing timeouts on the luxi RPC layer. With this change, auto-
archive returns after half of the RPC timeout has passed. The user
will see how many jobs are left unchecked.

Reviewed-by: ultrotter

This is done by passing the job object to _ArchiveJobUnlocked instead
of only the job ID. Also return whether job was actually archived.

Reviewed-by: ultrotter

As it turned out, having many files in a single directory can be
very painful. With this patch, only 10'000 files are stored in a
directory for the job queue archive. With 10'000 directries, this
allows for up to 100 million jobs be archived without having large
numbers of files in a single directories. Not that it is realistic,
anyway.

Reviewed-by: ultrotter

Unfortunately, os.makedirs in Python 2.4 is not safe against multiple
processes creating the same directory tree at the same time. This is
only fixed in Python 2.5 and up. Adding more checks in our code doesn't
make it any better.

Reviewed-by: iustinp

They're cheap to instantiate and doing this changes makes the code
a bit simpler.

Reviewed-by: ultrotter

All other exceptions are named after the error name in RFC2616 (HTTP/1.1).

Reviewed-by: amishchenko

Reviewed-by: amishchenko

Reviewed-by: amishchenko

Reviewed-by: amishchenko

More complex unittests will need some refactoring in the HTTP code.

Reviewed-by: amishchenko

Reviewed-by: iustinp

A job queue with too many jobs can increase memory usage and/or make
the master daemon slow. The current limit is just an arbitrary number.
A "soft" limit for automatic job archival is prepared.

Reviewed-by: iustinp

Sometimes the proc filesystem doesn't reflect the current status of
a process. By calling waitpid(), we make sure to get the current
information, at least for child processes. The timeout is still
kept for child processes to make sure the proc filesystem is updated.

Reviewed-by: iustinp

Reviewed-by: iustinp

The primary node is part of the instance, not of the opcode.

Reviewed-by: iustinp

poll() can be interrupted. rather than failing we retry until it
returns.

Reviewed-by: iustinp

Call socat with a full path specified at configure time, rather than
just by its name, and check for the binary to exist at hypervisor
verify.

Reviewed-by: iustinp