This patch exports all the way from backend a new field ‘api_version’
which holds the list of support API versions, and exposes the (already
computed) ‘parameters’ field.

The patch also reworks (again) the field calculation in its Exec()
method. All callers of LUDiagnoseOS pass in the 'valid' and 'variants'
parameters, thus having the special casing of whether to compute or not
the validity seems overkill. We move to a model where we always compute
these across-nodes arguments, in order to simplify the code, and we also
change the parameters set to be intersection of all node's values (which
means a change in description will drop the parameter from the list of
parameters).

Additionally, we update scripts/gnt-os, which was broken for multi-dir
OSes since the introduction of variants…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Nothing special here, just copy/adjust the beparams code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The patch also modifies the instance RPC calls to fill the osparameters
correctly with the cluster defaults, and exports the OS parameters in
the instance/OS environment.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

While we only support the 'parameters' check today, the RPC call is
generic enough that will be able to support other checks in the future.
The backend function will both validate the parameters list (so as to
make sure we don't pass in extra parameters that the OS validation
doesn't care about) and the parameter values, via the OS verify script.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The patch also modifies the internal methods in LUDiagnoseOS and gnt-os
to deal with the format change of call_os_diagnose.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Among all users, turns out just one *may* need the output to be sorted.
All the others can cope without.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Since we'll need to be able to generate the OS-specific environment
separately from the instance one, we move it to a separate function. We
also add a new OS_NAME env. var which is identical to the INSTANCE_OS
one (which won't exist for OS-only environments).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In case an OS has inconsistent declarations, we might get into a case
where one node reports a valid variants list (with OS API >=15), and
another node has OS API < 15, in which case its supported_variants gets
the default value of None. This leads to the same variable having
inconsistent data types, which leads to subtle bugs later: instead of
reporting something like "Inconsistent OS API versions", the LU exits
with a run-time exception. Furthermore, in another datapath, variants is
initialized to '[]' in case of OS diagnose failures.

The patch changes _TryOSFromDisk to initialize variants to '[]' for
OS api level below 15, and changes the variants calculation in
DiagnoseOS to be more readable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Upon sending signals, ESRCH can be reported when the target no
longer exists.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This call was introduced but never used. In two years.
Since it's just creating/removing a file it can also be in simpler ways,
without a special rpc call, if/when we need it again. In the meantime,
let's give it to history.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The cluster domain secret file was not distributed to other nodes.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

For exports to/imports from the same machine, compression will
not be used anymore.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The X509 key name and CA are passed from cmdlib all the way to
the backend import/export daemon. With the addition of an option
to choose the compression method, another parameter would have
to be passed all the way. By moving these options to a separate
object, adding new ones will become much easier.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This also updates a comment in the unittest for utils.py. We unittest
the new function for two things: correct reporting on real case (for
localhost), and correct reporting with a mocked-out TcpPing that returns
false.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Disable warnings for:
- except Exception,
- use of __errno_location,
- redeclaration of handleError()

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

63bcea2a added file checks for import/export, but unfortunately they
were broken.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Those directories must exist for the node daemon to run (it's in the
node daemon's list of ensured directories) and those functions are only
called by the node daemon, so there's no point in those checks+mkdir
calls.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Once the import/export daemon uses separate users, the node daemon file (which
is used for intra-cluster transfers) might not be readable anymore. Always
writing it to a daemon-specific file will make this easier.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Both of these functions, “snapshot_export” and “instance_os_import”,
have been replaced by the instance import/export daemon.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This will be used to stop the daemon without doing complete cleanup (yet).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is useful in combination with utils.StartDaemon and will be used for
reading the import/export daemon's PID file.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently some hypervisors (namely kvm) need to do some cleanup after
making sure an instance is stopped. With the moving of the retry cycle
in backend those cleanups were never done. In order to solve this we add
a new optional hypervisor function, CleanupInstance, which gets called
at the end of the shutdown procedure, and which interested hypervisors
can implement to be sure not to miss cleanup operations.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

These RPC calls can be used to start, monitor and stop the instance data
import/export daemon.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently the backend parameters are not exported automatically, but
only a few directly in the '[instance]' section. Hypervisor type and
hypervisor parameters are not exported at all.

This patch creates two separate sections for the be and hv parameters,
and stores the parameters (including ones that come from the cluster
defaults, but not the hypervisor globals for example) in the export.
The import code is not changed yet.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The patch tries to export all params (based on the dict defined in
constants), using None for missing keys.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

In case LVM is broken, backend.GetVolumeList will raise an RPC exception
(as expected since it's a function exposed over RPC). Therefore we must
be prepared to catch any such exceptions, so that we don't fail the
whole verify call in this case. cmdlib is already prepared to handle
string results for this response key.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Certificates and keys generated using these functions will be used for
inter-cluster instance moves. As per design, the private key should never
leave the node.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

- Pass keyword parameter as such
- Replace “not x == y” with “x != y”

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, the ganeti-confd's HMAC key is called “cluster HMAC key” or
simply “HMAC key” everywhere. With the implementation of inter-cluster
instance moves, another HMAC key will be introduced for signing critical
data. They can not be the same, so this patch clarifies the purpose of the
“cluster HMAC key” by renaming it. The actual file name is not changed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

To be consistent with RAPI_CERT_FILE, the rather generic named
“SSL_CERT_FILE” constant is renamed to “NODED_CERT_FILE”. The actual file
name is not changed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Seconds since the Epoch are not easily readable by a human. Using a
formatted timestamp makes it easier (e.g.
“….backup-2010-03-12_14_02_43.…”). This patch also makes OS logfiles use
this formatted timestamp.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In case the hypervisor has issues on one node, currently
backend.VerifyNode will exit via an exception (two exit paths possible,
one via HypervisorError from hypervisor.Verify(), and one via RPCFail
from GetInstanceList). This is bad as it invalidates all other checks of
that node.

This patch catches these two errors and allows the rest of the
VerifyNode function to run. This leads to a more complete verify cluster
run, for example now only real missing LVs are reported, not all of
them.

The cluster verify is not perfect as it will skip some tests even if it
has data, but this will require a more complete rewrite (see issue 90).

Also, the patch fixes and improves some error messages in cmdlib.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently backend.NodeVolumes() drops everything except the first PV,
thus we get a truncated result. The patch is not the nicest, as Python
doesn't have a simple `concat' function, so I had to change the list
comprehension to an explicit loop.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This should remove most of the remaining constructs which can be
replaced by PathJoin.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Since in objects we don't have access to utils.py, we add a warning that
the result value from objects.Disk.StaticDevPath might not be a valid
path, and change its only caller to validate the path.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rationale: the file-based storage backend can add/remove files under a
certain directory. However, the master node is also controlling the
setting of the file-based root directory, so basically it means we can't
prevent arbitrary modifications by the master of the node's filesystem.

In order to mitigate this for setups where the file-based storage is not
used, we introduce a new setting at ./configure time, that controls the
enable/disable of file-based storage. Since this is not modifiable by
the master (over RPC), it is now possible in this case to prevent
unintended modifications of the node's filesystem from the master.

The new setting is used in bdev.py to not expose the file-based storage
at all, and in cmdlib.py to prevent attempts at creation of such
instances.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>