Currently we launch processes via the shell in a few places only to
redirect standard output and error to a log file ("&> $file"). It is
better to do such redirection from within RunCmd itself.

This patch splits RunCmd in two parts, the setup and the execution part,
the latter being implemented in two different functions depending on
whether we write to a file or not.

We also update the unittests with this new case.

Reviewed-by: imsnah

We change some messages which are obviously intended for the user to use
the proc.Log* functions instead of (only) the logging ones.

We also fix some wrong uses of feedback_fn.

Reviewed-by: imsnah

Currently we have two methods in ganeti.bdev_unittest.py of computing
the test data file name - and, of course, they don't give the same
results.

The patch moves the functions to compute the test file name and reading
of its contents to the GanetiTestCase class in testutils, which allows
running the tests from the command line as well.

We also change assertFileContent to use utils.ReadFile.

Reviewed-by: imsnah

During cluster init, set the default hypervisor to be used for instances.
Ensure that the default hypervisor belongs to the set enabled hypervisors
for this cluster. Also fix a small bug with setting the default enabled
hypervisor list.

Reviewed-by: imsnah

We also add two function for printing messages, so that scripts won't
have to import logger to get these. They are a simple extension over the
logger ones, as they accept the call style from logging:
 ToStdout("Message: %s", msg)
(instead of requiring formatting by the client.

Reviewed-by: imsnah

All users of this code have been migrated to the new and shiny HttpServer
class.

It also fixes a typo in the ApacheLogfile unittests. It has not yet been
decided whether we should keep ApacheLogfile or not, hence leaving it in.

Reviewed-by: ultrotter

I broke it with rev 1864. This patch also removes whitespace at
line endings.

Reviewed-by: amishchenko

So, instead of any timeouts for synchronization, we move to actual
events. We still have one (huge) timeout for pathological cases (just to
cleanup), but we don't sleep at all in normal usage.

The patch moves the zombie/cleaned up child creation out from setUp(),
and to on-demand by the functions. We remove all signal handler usage,
instead we rely on the fact that when a process is cleaned up, its FDs
are closed, so the read end of a pipe will get an event.

Reviewed-by: imsnah

This patch adds paramter removal in SplitKeyVal, by prefixing a
value-less key with "-"; this is needed in resetting parameters back to
cluster defaults, but care must be applied now that None can come from
the parser.

Reviewed-by: imsnah

For the new 2.0-style command line options, we need to parse strings of
the type:
  ident:key=val[,...]
and
  key=val[,...]

This patch adds two new option builders for these two, which return
(ident, {key=val,}) and {key=val,} for the above two formats. It also
handles specially constructs of type “key” (val is set to True) and
“no_key” (val is set to False, and the ‘no_’ prefix is stripped).

Reviewed-by: imsnah

Currently, we check if we have a given ip address (i.e. it's alive on
one of our interfaces) but manually calling TcpPing(source=localhost).
This works, but having it spread all over the code makes it hard to
change the implementation.

The patch abstracts this into a separate utils.OwnIpAddress(addr)
function. We add a rpc call for it, which we use instead of the
(single-use of) call_node_tcp_ping. We leave node_tcp_ping in, as seems
useful and eventually it should be removed in a separate patch.

Reviewed-by: imsnah

This big patch changes the call model used in internode-rpc from
standalong function calls in the rpc module to via a RpcRunner class,
that holds all the methods. This can be used in the future to enable
smarter processing in the RPC layer itself (some quick examples are not
setting the DiskID from cmdlib code, but only once in each rpc call,
etc.).

There are a few RPC calls that are made outside of the LU code, and
these calls are left as staticmethods, so they can be used without a
class instance (which requires a ConfigWriter instance).

Reviewed-by: imsnah

Reviewed-by: iustinp

This function will be used for auto-archiving jobs via the command line.
The function is pretty simple, we only support up to weeks since months
and higher are not 'precise' entities, and dealing with them would
require us to start using calendar functions.

Reviewed-by: imsnah

Remove leftovers from ssconf.

Reviewed-by: iustinp

Get rid of ssconf and convert to configuration instead.

Reviewed-by: iustinp

Replacing ssconf with configuration.

Reviewed-by: iustinp

Reviewed-by: iustinp

ssconf will become write-only from ganeti-masterd's point of view,
therefore all settings in there need to go into the main configuration
file.

Reviewed-by: iustinp

Future patches will add even more variables to the cluster config.
Adding more parameters wouldn't make the function easier to use and
it doesn't make sense to pass them to another function, as it's
only done once in bootstrap.py on cluster initialization.

Reviewed-by: iustinp

This patch bans add() on a half-acquired set. This behavior was
previously possible, but created a deadlock if someone tried to acquire
the set-lock in the meantime, and thus is now forbidden. The
testAddRemove unit test is fixed for this new behavior, and includes a
few more lines of testing and a new testConcurrentSetLockAdd function
tests its behavior in the concurrent case.

Reviewed-by: imsnah

If the set-lock is acquired, currently, the _names function will fail on
a double acquire of a non-recursive lock. This patch fixes the behavior,
and some lines of code added to the testAcquireSetLock test check that
this and other functioins behave properly.

Reviewed-by: imsnah

SplitTime didn't round the same on different platforms. This patch changes
it to use microseconds and not care about rounding.

Reviewed-by: iustinp

Reported by Iustin.

It used to return this:
>>> utils.SplitTime(1234.999999999999)
(1234, 1000)

while it should've returned this:
>>> utils.SplitTime(1234.999999999999)
(1235, 0)

Reviewed-by: ultrotter

A few more tests written while bug-hunting. One of them shows a real
issue, at last. :)

Reviewed-by: imsnah

I was hunting for a bug in my code and thought the culprit was in the
locking library, so I added a test to check. Unfortunately turns out it
wasn't. :( Committing the test anyway, while still trying to figure out
what's wrong...

Reviewed-by: imsnah

These will be used for job logs.

Reviewed-by: ultrotter

This class is a wrapper around fcntl.flock and abstracts opening and
closing the lockfile. It'll used for the job queue.

(The patch also removes a duplicate import of tempfile into the unittest)

Reviewed-by: iustinp

Add instance tag handling, improved error logging.
...oh, yes adopt instance listing for RAPI2!

Reviewed-by: iustinp

Since we need to compute this from outside utils.py, we change this to a
public function.

Reviewed-by: ultrotter

We cannot depend on all environments to have a start-stop-daemon or
similar tool. We instead implement a KillProcess function that behaves
similar to “start-stop-daemon --retry”.

Note that the attached unittest can hang in foreground if the child
misbehaves (doesn't write to the internal pipe). Since unittest are
either run in the foreground or are run with a timeout from an automated
framework, I think this is an acceptable trade-off (against of using
hardcoded timeouts in the test).

Reviewed-by: imsnah

We already have a function to test if a PID is alive, so it makes more
sense to use function composition that force calling (since we need to
read PIDs from files in other places too). Now IsProcessAlive returns
False for PIDs <= 0, since this is the error return from ReadPidFile.

The patch also adds a unittest for checking that WriteFile raises the
correct exception, and checks that an invalid or missing file causes
ReadPidFile to return zero. The unittest tearDown method will try to
cleanup the temp directory too (otherwise it leaves stuff after it).

Reviewed-by: ultrotter

The RESTHTTPServer module went the way of the dodo.

Reviewed-by: iustinp

Reviewed-by: imsnah

The submitted WritePidFile, RemovePidfile and IsPidFileAlive functions
miss unit tests. Adding a simple one which covers their basic
functionality.

Reviewed-by: iustinp

An implementation mistake from the original design caused nodes to be
locked before instances, rather than after. This patch inverts the level
numbering, changing also the relevant unittests and the recursive
locking function starting point.

Reviewed-by: iustinp

Reviewed-by: iustinp

Reviewed-by: imsnah

Reviewed-by: imsnah

This removes (hopefully) all traces of the old locking functions and
uses.

Reviewed-by: imsnah