This patch series basically adds a new parameter 'family' to the constructors
of daemon.AsyncUDPSocket and confd.client.ConfdUDPClient. This enables the
users of these two classes to support IPv6.

In ganeti-confd.ConfdAsyncUDPClient a method to check the address families of
all peers is added.

Furthermore it adds unittests for the added functionality.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch moves network utility functions to a dedicated module.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently the message stream can process any number of messages in
parallel (if they get dispatched to different threads or processes).
In order to limit their number we only handle messages and read from
the socket if we're under a certain limit of unanswered ones.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This function adds the ability for a AsyncTerminatedMessageStream to
have a thread-safe message delivery function.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This makes sure the out_socket can only be used for writing, and the
in_socket for reading.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This new asyncore dispatcher can be used to force a thread running the
asyncore loop to awake from the select, by signaling it on one of its
selected sockets.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is the counterpart of the AsyncStreamServer can be used to handle
connected sockets returned from connected clients if the protocol is a
terminator separated message stream. Nothing in this class is server
specific though: it can be used as a client as well, if the client is
implemented inside an asyncore daemon.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is a new asyncore server which handles listening stream sockets by
calling a non-implemented function for each connection it accepts. It's
the stream-oriented cousing of the AsyncUDPSocket.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This bug was found in the asyncore master patch series, but actually
applies to 2.1 for AsyncUDPSocket as well.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Abstract a few common functionalities between all ganeti asyncore
dispatchers:
  - Handle errors by logging them, and then continue
  - By default check sockets only for readability

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is needed if daemons are in the foreground, and get ctrl+c-ed by
the user. Also add unittests to make sure the correct signals terminate
the mainloop.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Luca Bigliardi <shammash@google.com>

Remove duplicate code between a couple of asyncore related function by
having a function in charge of handling EINTR errors. Unittests included.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

By overriding the default asyncore handle_error (which closes the
socket) with our own version, which logs what happened but tries to
proceed, we can get rid of a couple of try/except blocks. The resulting
churn is deindentation of the internal code.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The docstring reports a wrong type for the "dirs" argument. Fixing.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Commit dfdc4060 added WaitForFdCondition which uses utils.Retry without
handling timeout exceptions. This breaks any nested retry loops.

This patch fixes the above function, and also changes utils.Retry to
detect and warn future similar cases. In addition, we add a few small
unittests for utils.Retry.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This function allows receiving socket data synchronously.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This basically implements read handling, without catching all
exceptions. When using the socket in synchronous mode, it's useful to
avoid losing exception data (which, in an async daemon, can only be
logged)

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch adds a configure-time parameter that will set the defaults
used by all programs, and command-line parameters in the daemons that
allow overriding it.

Syslog 'yes' enables syslog in addition to file-based logging, 'only'
enables syslog and disables file-based syslog.

The log entries will be of the form:
Jan 27 08:45:04 node2 ganeti-noded[14504]: INFO 172.24.227.5:50850 PUT
/jobqueue_update HTTP/1.0 200
Jan 27 08:45:05 node2 ganeti-noded[14505]: INFO 172.24.227.5:50853 PUT
/lv_list HTTP/1.0 200

and (for a multi-threaded program):
Jan 27 08:51:48 node1 ganeti-masterd[15491]: (MainThread) INFO
ganeti-masterd daemon startup
Jan 27 08:51:49 node1 ganeti-masterd[15491]: (MainThread) INFO
Inspecting job queue

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch should have only:

- pylint disables
- docstring changes
- whitespace changes

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Pass SSL certificate and key paths from ganeti-* instead of using a dict. The
patch also improves the --ssl-{key,cert} options by giving the default in
--help output and changes the validation a bit.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Passing it in as a parameter seems more logical.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A newer version of pylint, more warnings…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Found using pylint and epydoc.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

It does “print >> sys.stderr, …” but there is no import sys.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently we read maximum 4K packets, and don't check packets when
sending them. With this patch we move to a well defined maximum size of
60K.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This allows this extended asyncore+udp module to be used also in other
daemons, and in the confd client library

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This time we use the standard python sched module, rather than doing it
all by ourselves. The scheduler in mainloop can be manipulated directly
by callers, to enter new events.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

We can avoid most of the Mainloop.Run() code if we use asyncore
for delivering I/O events, and just concentrate on what's missing in
asyncore: singnal handling and timers. This way confd can be ported to
use Mainloop as well.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Mainloop currently calls all handlers with the signal number harcoded to
sigchild. Fixing this.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

This makes the function a lot simpler, since it used to install two
different signal handlers, which are now transparently handled by the
decorator. The code is unindented, but remains unchanged apart from the
part that checks the signal handlers (which is now collapsed toghether).

The option of not installing signal handlers, which was unused, is now
removed.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

IO is never unregistered in our current usage, so dropping that
functionality for now. Also putting the poller outside of the Run()
function allows us to avoid the double step of adding tuples to an
_io_wait_add queue and adding them later in the main loop.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Timers are currently unused, slightly buggy (for example timeout doesn't
get updated correctly), and unneeded in the current form. Confd will
have timeouts in more simple fashion, and is not a mainloop user
anyway.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>

With three ganeti daemons, and one or two more coming, the daemon's main
function started becoming too much cut&pasted code. Collapsing most of
it in a daemon.GenericMain function. Some more code could be collapsed
between the two http-based daemons, but since the new daemons won't be
http-based we won't do it right now.

As a bonus a functionality for overriding the network port on the
command line for all network based nodes is added.

Signed-off-by: Guido Trotter <ultrotter@google.com>