This new asyncore dispatcher can be used to force a thread running the
asyncore loop to awake from the select, by signaling it on one of its
selected sockets.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Not much changes with this patch. The main loop for the IOServer is
repaced by mainloop.Run() and the main thread now uses asyncore to
handle connections to the master socket. Once it accepts them, though,
it just pushes them to the current infrastructure, and everything
proceeds as before.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Unittests cover AsyncStreamServer and AsyncTerminatedMessageStream with
both tcp and unix sockets.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is the counterpart of the AsyncStreamServer can be used to handle
connected sockets returned from connected clients if the protocol is a
terminator separated message stream. Nothing in this class is server
specific though: it can be used as a client as well, if the client is
implemented inside an asyncore daemon.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is a new asyncore server which handles listening stream sockets by
calling a non-implemented function for each connection it accepts. It's
the stream-oriented cousing of the AsyncUDPSocket.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Update ganeti-watcher so that it tests the master's RAPI port with a
simple test (in this case GetVersion). If it fails, make one attempt
at restarting ganeti-rapi and retest.

- daemons/ganeti-watcher: Test rapi and make one attempt at restarting it.
- lib/utils.py: add StopDaemon() function.

Signed-off-by: Tom Limoncelli <tlim@google.com>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

* origin/devel-2.1:
  Explicitely return None from IgnoreSignals
  AsyncUDPSocket: fix IgnoreSignals usage and test
  Add KVM chroot feature
  Fix and Improve TryToRoman unittest

Conflicts:
	test/ganeti.daemon_unittest.py
	  - trivial

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Same result, but what happens is clearer.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This bug was found in the asyncore master patch series, but actually
applies to 2.1 for AsyncUDPSocket as well.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds a new boolean hypervisor parameter to the KVM hypervisor,
named 'use_chroot'.
If it's turned on for an instance, than KVM is started in "chroot mode":
Ganeti creates an empty directory for the instance and passes the path
of this dir to KVM via the -chroot flag.
KVM changes its root to this directory after starting up.

It also adds a "quarantine" feature for moving any unexpected files to
a separate directory for later analysis.

This has been backported from master,
commit 84c08e4e

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This will be used to avoid a race condition between starting a program (dd
for import/export) and sending signals to it.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

1) Don't break when the roman module is not found
2) Test that not finding the roman module doesn't make TryToRoman fail
(currently that is the case)

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Until now, move-instance used different status variables: “success”,
“abort” and “error_message”. With this patch, everything is changed
to use “error_message” only. This simplifies the code a bit.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The cluster domain secret file was not distributed to other nodes.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.1:
  Convert gnt-instance list and info to use roman
  gnt-cluster info --roman
  FormatUidPool: provide optional roman conversion
  gnt-node: remove latinfriendlyfields
  Move roman conversion to compat
  Add a new opcode timestamp field
  Fix IgnoreSignals on socket.error
  RAPI client should convert urllib2.URLError to GanetiApiError
  KVM: Migration bandwidth and downtime control
  Make utils.EnsureDirs() ignore umask
  Fix two race conditions in reboot instance
  Support for latin friendly output in node list

Conflicts:
	man/gnt-instance.sgml
	  - trivial

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Finally gnt-instance has roman support as well.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Convert to roman (if so the user wishes) the following:
  - cluster candidate size
  - uid pool
  - any integer be or hv parameter

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The convert= option of compat.tryToRoman is used to do optional
conversion without duplicating formatting code.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Rather than relying on a static list of fields, we opportunistically
convert all integers.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The new TryToRoman function provides optional easy to use roman
conversion. Nunc cum demonstrationi unitati.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Since we impose a maximum limit when reading ssconf files, let's error
out when trying to write them too big, so we don't pretend everything is
ok, and make mistakes when we actually read partial files.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Since the current start_timestamp opcode attribute refers to the inital
start time, before locks are acquired, it's not useful to determine the
actual execution order of two opcodes/jobs competing for the same lock.

This patch adds a new field, exec_timestamp, that is updated when the
opcode moves from OP_STATUS_WAITLOCK to OP_STATUS_RUNNING, thus allowing
a clear view of the execution history. The new field is visible in the
job output via the 'opexec' field.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Some confusion arose handling EINTR on this function: in python 2.6
socket.error is an IOError, and thus:
  - It's an EnvironmentError
  - It has an .errno member

In 2.4 and 2.5 it's not, and so its errno variable must be extracted
from the args tuple. This patch fixes both the function, and the
unittests.

This is a cherry-pick of master commit
965d0e5b with the portions not relevant
to 2.1 removed (changes to the RetryOnSignals function).

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Some confusion arose handling EINTR on those functions: in python 2.6
socket.error is an IOError, and thus:
  - It's an EnvironmentError
  - It has an .errno member

In 2.4 and 2.5 it's not, and so its errno variable must be extracted
from the args tuple. This patch fixes both functions, and the unittests.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Since the hash functions' changed their module name between python 2.4
and 2.6, and we have to do an try/import/except trick, we'll do it just
once, for both hash functions, and in compat.py. This also fixes a use
of md5 in the utils unittests which didn't use the trick before, and
generated a deprecation warning under 2.6.

In compat we keep both a ganeti-wide non-version-specific version to be
used by other ganeti modules, and a python-version specific that can be
passed to python modules which expect a hash function for their input
but call it differently under different versions of python (hmac, for
example).

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This makes sure that any unforeseen error raises an exception rather
then just increasing a counter. It makes unittest debugging a lot
easier.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This builds up on the "Master core scalability design doc" detailing the
critical situations in the job queue and proposing how to fix them. The
bulleted point list at the beginning is changed to subparagraph, as the
job queue part is quite longer and more detailed, then the remediation
section is updated removing the generic "we'll fix it somehow" paragraph
to propose a real solution.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This initial design still lacks information about the job queue lock
contention decrease.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Tom Limoncelli <tlim@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Introduce 2 new hypervisor options, migration_bandwidth and migration_downtime
and implement KVM migration bandwidth and downtime control.

migration_bandwidth controls KVM's maximal bandwidth during migration, in
MiB/s. Default value is 32 MiB/s, same as KVM's internal default. This option
is a global hypervisor option.

migration_downtime sets the amount of time (in ms) a KVM instance is allowed to
freeze while copying memory pages. This is useful when migrating busy guests,
as KVM's internal default of 30ms is too low for the page-copying algorithm to
converge. This is a per-instance option, with a default of 30ms, same as KVM's
internal default.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

EnsureDirs() should create directories with the exact mode requested
in the arguments, but it currently applies the umask.
This patch makes it independent from the umask.

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The code parsing the child process' output is moved to a separate
class in the impexpd module. As more programs are added, it'll
become more complex and should be separated.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The import/export daemon code is already large. Moving some code
to a separate module will make it smaller and easier to test.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of passing around many variables for building the executed
command, they're now kept as instance variables.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds a new boolean hypervisor parameter to the KVM hypervisor,
named 'use_chroot'.
If it's turned on for an instance, than KVM is started in "chroot mode":
Ganeti creates an empty directory for the instance and passes the path
of this dir to KVM via the -chroot flag.
KVM changes its root to this directory after starting up.

It also adds a "quarantine" feature for moving any unexpected files to
a separate directory for later analysis.

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

If the instance crashes between backend.InstanceReboot checks the list
of running instances and the execution of hv_xen.RebootInstance,
ini_info will be None. And if the instance doesn't reboot fast enough,
new_info will be None. Both cases lead to “TypeError: unsubscriptable
object”. Too bad pylint doesn't detect such cases.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.1:
  Test for errors during inotify callback
  SingleFileEventHandler: Remove try/except blocks
  ErrorLoggingAsyncNotifier
  daemon.GanetiBaseAsyncoreDispatcher

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

- Create a new _MyErrorLoggingAsyncNotifier class which registers
  error counts, rather than logging them
- Add an additional ERR notifier to test with
- Check that no error was returned, for tests that weren't supposed to
- Add a new test case for a callback that's supposed to raise an
  exception

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>