Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

confd.server and daemons/ganeti-confd import a few modules they don't
actually use. Clean them up.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>

ganeti-confd is a simple asynchronous daemon, which listens on a UDP
port, passes each packet to a processor, and sends back to the client
the result.

It also listens on an inotify socket, in order to reload its
configuration when the ganeti config file changes.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch changes the return type for backend.BlockdevGetmirrorstatus from
a list of tuples to a list of objects.BlockDevStatus instances.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch changes the return type for backend.BlockdevFind to an object
(objects.BlockDevStatus). Before a tuple was used. Adding more values to
this tuple causes a lot of work.  Converting the result to an object with
properties will make this a bit simpler.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Note that this exports the disk size as bdev returns it, in bytes. The
value will be converted to MiB in cmdlib.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

When the parameter is set to True and start_daemons is also True,
ganeti-masterd will be started with the new --no-voting --yes-do-it
options.

This new option is set to True only on masterfailover, when no_voting is
used. This changed the behavior from 2.0, where we didn't start the
master daemon at all, when this option was used.

The manpage is also updated to remove the 2.0 only change.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>

With three ganeti daemons, and one or two more coming, the daemon's main
function started becoming too much cut&pasted code. Collapsing most of
it in a daemon.GenericMain function. Some more code could be collapsed
between the two http-based daemons, but since the new daemons won't be
http-based we won't do it right now.

As a bonus a functionality for overriding the network port on the
command line for all network based nodes is added.

Signed-off-by: Guido Trotter <ultrotter@google.com>

The <DAEMON>_PID constants were created to reference a daemon pid file,
but actually contain a daemon's name, because the various functions that
work with pidfiles abstract the filename from the daemon name
themselves. Removing the constants and using the actual daemon name
constants in their place.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The original LOG_<DAEMON_NAME> constants for daemon logfiles are gone.
In their place there is a DAEMONS_LOGFILES dict, indexed by daemon name.

This is a minor change with the objective to uniform most of the
daemon's main() functions code, which is very similar one to the other.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently rapi is the only daemon which accepts a port option, rather
than querying its own port from services, and failing back to the
default if not found. Changing this to conform to what other daemons do.

Also update the ganeti-rapi(8) manpage

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

GetNodeDaemonPort is used to lookup the node daemon port in the services
file, and if not found to return the default one. We make it a generic
function, which accepts the daemon name in input, so that it can be used
by confd as well, to lookup its own udp port.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Various modules set it to True when called in debugging mode, but the
utils module supports no such global.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

substitute exit(1) with exit(constants.EXIT_FAILURE).
Also fix a wrongly indented line.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

On machines without the ssl file noded exists '5'.
Changing this to constants.EXIT_NOTCLUSTER.

Also utils.GetNodeDaemonPort hasn't risen errors.ConfigurationError for
a while, so removing that try/except block.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

As a workaround for the job submit timeouts that we have, this patch
adds a new luxi call for multi-job submit; the advantage is that all the
jobs are added in the queue and only after the workers can start
processing them.

This is definitely faster than per-job submit, where the submission of
new jobs competes with the workers processing jobs.

On a pure no-op OpDelay opcode (not on master, not on nodes), we have:
  - 100 jobs:
    - individual: submit time ~21s, processing time ~21s
    - multiple:   submit time 7-9s, processing time ~22s
  - 250 jobs:
    - individual: submit time ~56s, processing time ~57s
                  run 2:      ~54s                  ~55s
    - multiple:   submit time ~20s, processing time ~51s
                  run 2:      ~17s                  ~52s

which shows that we indeed gain on the client side, and maybe even on
the total processing time for a high number of jobs. For just 10 or so I
expect the difference to be just noise.

This will probably require increasing the timeout a little when
submitting too many jobs - 250 jobs at ~20 seconds is close to the
current rw timeout of 60s.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
(cherry picked from commit 2971c913)

SimpleStore is a lot less heavyweight than SimpleConfigReader, and to
just get the master name we can use that. This is the only usage of
SimpleConfigReader currently, but we're not going to delete the class,
as new usages will come in for ganeti-confd (in 2.1). Using it there,
though, will make the class even more heavy to load, so it makes sense
for this simple usage to be converted.

Signed-off-by: Guido Trotter <ultrotter@google.com>

When the parameter is set to True and start_daemons is also True,
ganeti-masterd will be started with the new --no-voting --yes-do-it
options.

This new option is set to True only on masterfailover, when no_voting is
used. This changed the behavior from 2.0, where we didn't start the
master daemon at all, when this option was used.

The manpage is also updated to remove the 2.0 only change.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This will be used by ganeti-noded to start ganeti-masterd in a
--no-voting masterfailover.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

If a user used ^Z to stop the program, poll() in socket.recv would return
EAGAIN due to SIGSTOP. This patch changes luxi.Transport.Recv to ignore EAGAIN.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

There are volume-related rpc calls. This patch renames the ‘volume_list’
call to ‘lv_list’ to make more clear its purpose.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Since now all functions fail via _Fail, the return True, … is redundant
as all normal return paths have it, and thus the True value can be added
in the ganeti-noded handler.

This means that all functions can now forget about the special result
type, and instead return normally, but signal all failures via _Fail().

Only a few functions must be handled specially (the recursive ones).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Since all rpc calls were converted, we can now:
  - enforce result type to (status, data)
  - convert all unhandled exceptions to (False, str(err))

This makes sure that all unhandled errors are reported to rpc users.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently the OSes have a special, customized error handling: the OS
object can represent either a valid OS, or an invalid OS. The associated
function, instead of raising other exception or failing, create custom
OS objects representing failed OSes.

While this was good when no other RPC had failure handling, it's
extremely different from how other function in backend.py expect
failures to be signalled.

This patch reworks this completely:
  - the OS object always represents valid OSes (the next patch will
    remove the valid/invalid field and associated constants)
  - the call_os_diagnose returns instead of a list of OS objects, a list
    of (name, path, status, diagnose_msg); the status is then used in
    cmdlib to determine validity and the status and diagnose_msg values
    are used in gnt-os for display
  - call_os_get returns either a valid OS or a RPC remote failure (with
    the error message)
  - the other functions in backend.py now just call backend.OSFromDisk()
    which will return either a valid OS object or raise an exception
  - the bulk of the OSFromDisk was moved to _TryOSFromDisk which returns
    status, value for the functions which don't want an exception raised

The gnt-os list and diagnose commands still work after this patch.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch converts the job queue rpc calls to the new style result.
It's done in a single patch as there are helper function (in both jqueue
and backend) that are used by multiple rpcs and need synchronized
change.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This also removes custom post-processing from rpc.py; since this call
has only one user, it was simple to move it back to the caller.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This also cleans up its single use in cmdlib.py.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch converts this rpc call to the new style result, and also
changes in the process the meaning of the QuitGanetiException's
arguments and the node daemon rpc call exception handler.

The problem with the exception handler is that we used a two-stage one,
and the inner used to catch all exception (including this one), so in
the logs we always had an exception logged, instead of the normal
'leaving cluster message'. The patch also adds logging of the
exception's arguments, so that we have a trail in the logs about the
shutdown mode.

The exception's arguments were reversed from the normal RPC results
style. While it makes somewhat more sense for this exception, we change
them such that they match the rpc result format.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is used in multiple places outside cmdlib.py, so it's a more
interesting patch.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This should actually have a function in backend, but it's fine for now.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Since backend.GetInstanceList() is used both as RPC endpoint and as
internal function, it can't return (status, value). Instead it returns
only valid instance info, and failures are denoted by exceptions; and
the ganeti-noded function adds the (True,) status.

The patch also fixes a typo.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a big change, because we need to cleanup its users too.

The call and thus LUVerifyDisks LU used to differentiate between failure
at node level and failure at LV level, by returning different types in
the RPC result. This is way too complicated for our needs.

The patch changes to new style result (easy change), and then:
  - changes LUVerifyDisks.Exec() to return a tuple of 3-elements
    instead of 4-elements; we collapse the «nodes not reachable» and
    «nodes with LVM errors» in a single dict
  - changes gnt-cluster to parse 3-element results and simplifies the
    different by-error handling code

Note that the status is added in ganeti-noded, and not in the function
itself, as the function is used in other places too.

This was tested with down nodes and broken VGs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This also removes some code from ganeti-noded and rpc.py, which should
not do such processing of data (and be simply glue code). (Or
alternatively they could, if we had better infrastructure).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds a simple failure reporting tool, similar to bdev's
_ThrowError. In backend, we move towards the new-style RPC results (of
type (status, payload)) and thus functions which use this style can very
easily log and return the error message using this new function.

The exception is declared here and not in errors.py since it's local to
the node-daemon/backend combination.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>