This backend daemon for instance import and export will be used to
transfer instance data to other machines. It is implemented in a generic
way to support different ways of data input and output. The third-party
program “socat”, which is already used by the KVM hypervisor abstraction,
is used to connect to remote machines using SSL/TLS. After starting the
child processes in a separate process group, the import/export daemon
monitors their output and updates a status file regularily. This status
file can then be read by ganeti-noded (not in this patch).

Three I/O methods are supported: Raw disk, file and script. Each of these
can be used for import and export.

Similar to daemon-util, an incomplete set of tests written in Bash is
included.

Two future enhancements are planned:
- Run parts of the command chain as a dedicated user (privilege
  separation).
- Currently users of this daemon have to poll the status file while data
  is transferred. This is inefficient and creates unnecessary delays. By
  adding “dd” into the chain and sending it SIGUSR1 regularily, we can get
  some statistics, optimize the polling frequenc and even provide the user
  with an ETA (which isn't available with all current methods to
  import/export instance data).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.1:
  backend: Consolidate code opening real block device

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.1:
  utils: Add class to split string stream into lines
  Fix cluster behaviour with disabled file storage
  Update docstrings in tools/ and enable epydoc
  Forward-port the ganeti 2.0 cfgupgrade
  Add a new tool: sanitize-config
  Fix cfgupgrade with non-default DATA_DIR
  Improving the RAPI documentation
  Mark cluster-merge as executable
  QA: Make the rapi credential handling less involving

Conflicts:
	lib/bootstrap.py (merge cds and new file names)
	lib/cmdlib.py    (trivial, kept 2.1 version for disabled file
	                  storage)
	lib/utils.py     (trivial, new imports)
	tools/cfgupgrade (trivial, new cds parameter)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This will be used by the new import/export daemon to split
output by its child processes into lines.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

There are a few issues with disabled file storage:
- cluster initialization is broken by default, as it uses the 'no'
  setting which is not a valid path
- some other parts of the code require the file storage dir to be a
  valid path; we workaround by skipping such code paths when it is
  disabled

A side effect is that we abstract the storage type checks into a
separate function and add validation in RepairNodeStorage (previously a
luxi client which didn't use cli.py and submitted an invalid type would
get "storage units of type 'foo' can not be repaired").

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch updates the docstrings in tools/ (mostly in lvmstrap, which
is very very old code-base) and then enabled the tools in this directory
for 'make apidoc' too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

2.1's cfgupgrade doesn't support upgrading from 1.2, only from 2.0.
However, it's trivial to forward-port the 2.0 cfgupgrade to work with
2.1, thus providing an upgrade path for 1.2 users directly to 2.1,
without the intermediate step of installing 2.0.

The new cfgupgrade12 file has just a few trivial changes from 2.0, and
skipping the load of the config file.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This can be used for two purposes:

- safety copy of the config file, with just the secrets changed
- cleanup of the config file (full randomization), so that (e.g.) users
  could send a broken config file to the devel-list

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Commit 43575108 added bootstrap.GenerateclusterCrypto and commit
7506a7f1 changed cfgupgrade to use it. However, this lost the
functionality of upgrading in non-default DATA_DIR.

To fix this, we enhance bootstrap.GenerateclusterCrypto to accept custom
file paths for the three files it modifies. If more files will be needed
in the future, we could just pass in modified DATA_DIR, but for now it
suffices.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

* Be more clear about when POST and PUT should be used
* Be more verbose on parameters
* Describe allowed parameter values (if not self explaining)

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

* devel-2.1:
  Update import documentation for the recent changes
  Add a identify-defaults options for import
  Fix create/import verification of hvparams
  objects.Cluster: add method to get hv defaults
  Reuse NIC information from export
  Reuse backend parameters from export
  Reuse disk information from export
  Reuse hypervisor parameters in import
  Read disk template from export info
  CreateInstance: separate the reading of the export
  Move code from ExpandNames to CheckPrereq
  CreateInstance: Move some code to CheckArguments
  Export more instance parameters in instance export
  Export the nicparams too during instance export
  Handle errors better for wrong nic_count in export
  QA: Make sure RAPI credentials are setup on cluster init

Conflicts:
	lib/cmdlib.py (strange conflict, HEAD had no changes)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

When importing an instance, all the saved valued will be used as
explicitly specified values, overriding the cluster defaults. This means
export+import will change the status (from default to explicitly
specified) of parameters.

This patch adds a new option that changes the behaviour to identify
parameter values which are equal to the current cluster defaults and
mark them as such. It does this for hv, be and nic parameters.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently the instance creation checks the cluster hv defaults + the new
parameters for validity, ignoring the os-specific hvparams (this was an
oversight during the implementation of the os hvp). This patch uses the
new GetHVDefaults method to provide the defaults, instead of reading
directly the attributes of the cluster object.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently the FillHV method is the one that does the cluster hvparams +
os hvparams merger. However, in some cases we need to do just this,
without adding the instance hvparams on top.

This patch adds a function to compute just this (hv + os hvp
combination) default dict, and modifies FillHV to use it to build the
final dict.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

If the user doesn't pass any nics in import, do not use a default
one-nic, but instead read the nics from the export file as is.

Fortunately the export and the way nics are read from the command line
are compatible…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Similar to the previous patches, if we're missing some parameters and
the export has them (either in the new style or old-style), we reuse
them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

If the user doesn't pass the disk information on import, automatically
reuse the number and size of disks. This loses the iv_name attribute,
but that is only cosmetic and cannot be changed by the user.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

If available, we reuse the parameters from the export info.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch changes the instance import to read the disk template
automatically from the export info, if the opcode doesn't already
specify a disk template.

To do this, we have a couple of additional changes:

- change from required parameter to optional one for disk_template
- move check for disabled file storage at ./configure time to the
  generic _CheckDiskTemplate checker
- move checks of the disk template from CheckArguments to CheckPrereq

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

We move the reading of the export to a separate function, to simplify
CheckPrepreq and also read it earlier. This will allow building the
missing opcode parameters from the export information, instead of
requiring all of them on the command line.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is needed since only in CheckPrereq we have the nodes locked, and
future import enhancements will need to have access to the export info
during the parameter build.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

ExpandNames holds too much non-locking code (first LU to be converted to
ExpandNames, and we didn't have CheckArguments at that poin), and this
patch moves the checks that are lock-independent to CheckArguments.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently the backend parameters are not exported automatically, but
only a few directly in the '[instance]' section. Hypervisor type and
hypervisor parameters are not exported at all.

This patch creates two separate sections for the be and hv parameters,
and stores the parameters (including ones that come from the cluster
defaults, but not the hypervisor globals for example) in the export.
The import code is not changed yet.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The patch tries to export all params (based on the dict defined in
constants), using None for missing keys.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This fixes an old 'FIXME' entry.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch makes sure that the Ganeti RAPI credentials are setup,
if any, on cluster init time.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.1:
  burnin: only remove instances we actually added
  burnin.ExecOrQueue: add post-process function
  burnin.ExecOrQueue: remove variable argument list
  Fix new pylint errors
  Rename the confd_client unittest (to confd.client)
  Make watcher request the max coverage
  ConfdClient.SendRequest: allow max coverage
  Document the watcher node maintenance feature
  Watcher: automatic shutdown of orphan resources
  Export the maintain_node_health option in ssconf
  Add a new cluster parameter maintain_node_health
  Add a new confd callback (StoreResultCallback)
  ConfdClient: add synchronous wait for replies mode
  ConfdClient: unify some internal variables

Conflicts:
	daemons/ganeti-masterd
	  change moved to luxi.py

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently burnin, if proceding in parallel, will remove all instances
which were passed, even if they failed to add. This is bad because it
will also remove instances which existed before burnin started. By
adding the instances to the removal queue only if their creation was
successful (passing the action as a post processing action to
ExecOrQueue) we guarantee pre-existing instances are saved.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

If a post-process function is passed to ExecOrQueue it is executed if
and only if the job is successful. This happens immediately if we're
proceding iteratively, and at the end, when we collect all job results,
if we're proceding in parallel.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In order to later add an optional parameter we transform the variable
ops argument list in an explicit list.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Under squeeze pylint reports the following errors:
************* Module ganeti.serializer
E1103:155:LoadSignedJson: Instance of 'False' has no 'get' member (but some types could not be inferred)
************* Module ganeti-masterd
E1103:166:ClientRqHandler.handle: Instance of 'False' has no 'get' member (but some types could not be inferred)
E1103:167:ClientRqHandler.handle: Instance of 'False' has no 'get' member (but some types could not be inferred)
************* Module gnt-instance
E1103:431:BatchCreate: Instance of 'False' has no 'keys' member (but some types could not be inferred)

For the first two cases it's actually wrong: we had checked before that
the variable on which "get" is called is actually a dict. In the third
case though such check doesn't exist, so we add it. Then we silence the
error all three times.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is to keep same naming across all tests (modules separate with dot,
followed by _unittest.py).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Since the actions are potentially destructive, we should try to get a
consistent view of the cluster, so it's better to get the most coverage
possible.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch changes the coverage parameter to allow specification of max
coverage (via -1), versus auto-computation (default, 0) and manual
specification.

Unittests are updated for this case too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The patch changes significantly the watcher man page, as it was very
simplistic.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch changes the watcher so that it maintains (on all nodes) the
list of instances and DRBD devices by shutting down ones that confd
daemons indicate should not be running on this node.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>