Since we impose a maximum limit when reading ssconf files, let's error
out when trying to write them too big, so we don't pretend everything is
ok, and make mistakes when we actually read partial files.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Some confusion arose handling EINTR on those functions: in python 2.6
socket.error is an IOError, and thus:
  - It's an EnvironmentError
  - It has an .errno member

In 2.4 and 2.5 it's not, and so its errno variable must be extracted
from the args tuple. This patch fixes both functions, and the unittests.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Since the hash functions' changed their module name between python 2.4
and 2.6, and we have to do an try/import/except trick, we'll do it just
once, for both hash functions, and in compat.py. This also fixes a use
of md5 in the utils unittests which didn't use the trick before, and
generated a deprecation warning under 2.6.

In compat we keep both a ganeti-wide non-version-specific version to be
used by other ganeti modules, and a python-version specific that can be
passed to python modules which expect a hash function for their input
but call it differently under different versions of python (hmac, for
example).

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This makes sure that any unforeseen error raises an exception rather
then just increasing a counter. It makes unittest debugging a lot
easier.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This builds up on the "Master core scalability design doc" detailing the
critical situations in the job queue and proposing how to fix them. The
bulleted point list at the beginning is changed to subparagraph, as the
job queue part is quite longer and more detailed, then the remediation
section is updated removing the generic "we'll fix it somehow" paragraph
to propose a real solution.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This initial design still lacks information about the job queue lock
contention decrease.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The code parsing the child process' output is moved to a separate
class in the impexpd module. As more programs are added, it'll
become more complex and should be separated.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The import/export daemon code is already large. Moving some code
to a separate module will make it smaller and easier to test.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of passing around many variables for building the executed
command, they're now kept as instance variables.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds a new boolean hypervisor parameter to the KVM hypervisor,
named 'use_chroot'.
If it's turned on for an instance, than KVM is started in "chroot mode":
Ganeti creates an empty directory for the instance and passes the path
of this dir to KVM via the -chroot flag.
KVM changes its root to this directory after starting up.

It also adds a "quarantine" feature for moving any unexpected files to
a separate directory for later analysis.

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.1:
  Test for errors during inotify callback
  SingleFileEventHandler: Remove try/except blocks
  ErrorLoggingAsyncNotifier
  daemon.GanetiBaseAsyncoreDispatcher

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

- Create a new _MyErrorLoggingAsyncNotifier class which registers
  error counts, rather than logging them
- Add an additional ERR notifier to test with
- Check that no error was returned, for tests that weren't supposed to
- Add a new test case for a callback that's supposed to raise an
  exception

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Since now we use the SingleFileEventHandler together with an error
handling asyncore dispatcher, we don't need the internal try/except
anymore.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This mixes AsyncNotifier with GanetiBaseAsyncoreDispatcher to provide an
AsyncNotifier which will log errors, rather than bail out.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Abstract a few common functionalities between all ganeti asyncore
dispatchers:
  - Handle errors by logging them, and then continue
  - By default check sockets only for readability

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.1:
  TestSingleFileEventHandler: abstract notifier type
  Mainloop: handle SIGINT as well (and terminate)
  SingleFileEventHandler: update comments

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Rather than hardcode that we have two notifiers, and notifier 0 is the
terminating one, we abstract this with class level constants. This makes
it easier to add more, with different features.

The only real change is that now the callback class takes as input the
whole test object, rather than just the notified array, to have access
to those constants.

The rest is just replacing of hardcoded 0s and 1s with
self.NOTIFIER_TERM and self.NOTIFIER_NORM, and of notifier_count with
len(self.NOTIFIERS).

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

This is needed if daemons are in the foreground, and get ctrl+c-ed by
the user. Also add unittests to make sure the correct signals terminate
the mainloop.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The comments in the SingleFileEventHandler are still confd-specific.
Update them to make them generic for any single-file monitoring.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

For exports to/imports from the same machine, compression will
not be used anymore.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The X509 key name and CA are passed from cmdlib all the way to
the backend import/export daemon. With the addition of an option
to choose the compression method, another parameter would have
to be passed all the way. By moving these options to a separate
object, adding new ones will become much easier.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

For example, exports on the same node shouldn't be compressed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

* devel-2.1:
  Add asyncnotifier unittests
  Make ConfdInotifyEventHandler a library function
  Comment on AsyncNotifier upstream's availability
  Remove errors.ConfdFatalError
  TestMainloop: test scheduler priorities as well
  Fix race condition in mainloop unittest
  TestMainloop.testReRun
  Add AsyncUDPSocket tests
  Add initial mainloop unittests
  devel/review: make the range argument optional

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Cut&Paste, plus the following changes:
  - The class is renamed to SingleFileEventHandler
  - The monitored filename must be passed in and doesn't default to the
    ganeti cluster config file
  - A small docstring is added to the class
  - Pylint disables for except: and method names are added

This makes it possible to write unittests for this class.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Since we contributed AsyncNotifier back to the upstream pyinotify
project, we'll be able, one day, to remove the ganeti version of that
code. For now we still need it to support older distributions, but
having a note about when we'll be able to remove it is nice.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This exception is caught, but never thrown. It became useless when we
moved confd from on/off to enabled/disabled, but always running on all
nodes. Removing its definition and the code catching it can do no harm.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

By using enterabs we can schedule events at the same time, which will
then be sorted by priority.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, in testDeferredCancel if the self._CancelEvent are entered
more than 0.3 seconds after the _SendSig have been entered, the test
could fail. This is unlikely but may happen. To avoid it we use
enterabs, to use absolute times.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently we never do that, but it's nice to know we can restart the
mainloop, after termination.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rather than calling devel/review with from..to target we can just use
target, and assume target..HEAD as the range. This makes it a lot
easier, as now you just have to say something like "devel/review
devel-2.1" if you're already on the branch you want to commit, and want
to commit it to devel-2.1.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is a first version of the instance move tool and it supports moving
1..N instances from one cluster to another. When moving a single instance,
the instance can be renamed, allowing for moves within the same cluster
(not really useful in practice, but good for testing). Documentation is
updated to describe this new tool and its usage.

The “move-instance” tool uses the workerpool to support parallel moves of
instances. Supporting them was simple as threads were required anyway due
to the synchronous RAPI client.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Two new resources are added:
- /2/instances/$name/prepare-export
- /2/instances/$name/export

The documentation for the existing resource for creating instances is updated
for remote imports. The RAPI client is extended for the new resources.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

To prepare a remote export, the X509 key and certificate need to be generated.
A handshake value is also returned for an easier check whether both clusters
share the same cluster domain secret.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.1:
  Abstract the LUXI eom into a constant
  KVM: vhost net acceleration support

Conflicts:
	lib/luxi.py
	  - trivial

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>