File "../test/ganeti.hooks_unittest.py", line 239, in setUp
  self.lu = FakeLU(FakeProc(), self.op, self.context, None)
File "…/ganeti/cmdlib.py", line 92, in __init__
  self.LogStep = processor.LogStep
AttributeError: FakeProc instance has no attribute 'LogStep'

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This parameter is now mandatory for the cluster config to work.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently we have both a default_hypervisor and an enabled_hypervisors
list. The former is only settable at cluster init time, while the latter
can be changed with cluster modify.

This becomes cumbersome in a few ways: at cluster init time for example
if we pass in a list of enabled hypervisors which doesn't include the
"default" xen-pvm one, we're also forced to pass a default hypervisor,
or an error will be reported. It is also currently possible to disable
the default hypervisor in cluster-modify (with unknown results).

In order to avoid this we get rid of this field altogether, and define
the "first" enabled hypervisor as the default one. This allows ease of
changing which one is the default, and at the same time maintains
coherency.

At configuration upgrade we make sure that the old default is first in
the list, so that 2.0 cluster defaults are preserved.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch includes HMAC authenticated json messages to the serializer.
The new interface works on any json-encodable data type, and can sign it
with a private key and an optional salt. The same private key must be
used upon message loading to verify the message.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

If an ident member of an IdentKeyVal relationship starts with no_ or -,
handle it the same way we do for a key. Some unittests are added to
check that check_ident_key_val behaves as expected.

This patch also changes ForceDictType to, for now, fail on such an
entry, and the same to happen when creating an instance or modifying its
nics or disks.

This behavior will be used later on to allow deletion of os entries in
os parameters.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

If an empty string is passed to _SplitKeyVal, we should return {},
rather than {'': True}. Also test for the correct behavior.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Patch "Simplify the RPC result framework in backend.py" changed all
RPCs, and thus the hooks results where also changed. This needs changes
to the hooks unittests too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This also converts (and fixes) unittests and mock objects to deal with
this change, and the custom hook verifier in cmdlib.LUClusterVerify.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Only the custom handling in rpc.py for call_node_info used this, so now
it can be removed.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The target test cluster should have nic params, rather than a default
bridge.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This function will be used to check the NIC parameters for validity.
Unittests are included.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

We used to refer to normalized paths as "normal" which might be
confusing. This fixes the syntax in all current IsNormAbsPath users and
in the docstring.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently most of the time we check for absolute path, but that doesn't
protect us from some invalid paths. In some places we should be more
strict, and this function should help us to.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently we miss in hooks the instance's hypervisor, hypervisor
parameters and backend parameters. This forces hooks to query back into
ganeti, which is dangerous due to possible luxi sockets exhaustion.

This patch adds these three as INSTANCE_HYPERVISOR, INSTANCE_HV_*,
INSTANCE_BE_*. The hook environment prefixes all keys with “GANETI”, so
a default settings for a xen-pvm instance would be:

  GANETI_INSTANCE_HV_initrd_path=
  GANETI_INSTANCE_HV_kernel_args=ro
  GANETI_INSTANCE_HV_kernel_path=/boot/vmlinuz-2.6-xenU
  GANETI_INSTANCE_HV_root_path=/dev/sda1

Any dashes in parameter names are changed to underscores, since
variables with dashes are not easy to access from the shell
(alternatively we could deny those via an unittest for constants.py).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently we have bad behaviour in SafeEncode:
  - binary strings are actually not handled correctly (ahem)
  - the encoding is not stable, due to use of string_escape

For this reason, we replace the use of string_escape with part of the
code of string escape (PyString_Repr in Objects/stringobject.c); we
don't escape backslashes or single quotes, since that is that makes it
nonstable. Furthermore, we only use the encode('ascii', ...) for unicode
inputs.

The patch also adds unittests for the function that test basic
behaviour.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

DRBD 8.3 changes two more things compared to 8.2:
  - /proc/drbd format changed in multiple ways; the part we're
    interested is the ‘st:’ to ‘ro:‘ change (in the changelog named as
    “Renamed 'state' to 'role'”
  - “drbdsetup /dev/drbdN show” changed the ‘device’ stanza from:
      device "/dev/drbd0";
    to:
      device                  minor 0;

This patch fixes these both and adds data files and unittests for DRBD
8.3.1.

Signed-off-by: Iustin Pop <iustin@google.com>

If /proc/drbd can't be opened, this raises an IOError, but all the
error-handling behaviour in backend treats only BlockDeviceErrors. This
creates a plain failure in cluster verify and in other RPC calls.

This patch simply converts EnvironmentErrors into BlockDeviceErrors, and
also changes the RPC result for NV_DRBDLIST and its handling to be able
to show the error. The other RPC calls work by default now, due the
existing error handling.

Reviewed-by: ultrotter

In the current Ganeti version when modifying /etc/hosts we mistakenly
give it the permissions of the temporary file we create to define its
content, which is by default 0600. This breaks most non-root
applications, and thus must be corrected. This patch forces the mode to
be 0644 (but we might decide to just use the mode of the previous
/etc/hosts, if we want to be more polite against any eventual
administrative choice). We also add a new assertFileMode() method for
unit tests and actually check in the SetEtcHostsEntry and
RemoveEtcHostsEntry tests that the mode is correct, to be sure not to
reintroduce this bug again. Also, a FIXME is added in the original
functions stating that it would be nice to use WriteFile+fn() rather
than reimplementing its functionality again.

Reviewed-by: iustinp

We want all the hv/be parameters to have a known type, rather than a
random mix of empty string, boolean values, and None, so we declare the
type of each variable and we enforce/convert it.

- Add some new constants for enforceable value types
- Add new constants dicts HVS_PARAMETER_TYPES and BES_PARAMETER_TYPES
  holding not only the valid parameters but also their types
- Drop the old HVS_PARAMETERS and BES_PARAMETERS constants and calculate
  the values from the type dict
- Convert all the default parameters to a valid type value
- Create a new ForceDictType utils function, to check/enforce a dict's
  element value types, with relevant unit tests
- Drop a few custom functions to check/convert the BE param types in
  utils and cli, in favor of ForceDictType
- Double-check the parameter types using ForceDictType in both scripts
  and LogicalUnits, when possible.

As a bonus:
- Remove some old commented-out code in gnt-instance
- Remove some already fixed FIXME
- Fix a bug which prevented VALUE_DEFAULT to be applied to BE parameters
  in SetInstanceParams because the value was checked for validity before
  that transformation was made
- Fix a bug which prevented initing a cluster and passing hvparams to
  work at all
- ForceDictType allows an allowed_values for exceptions, which makes us
  able to do the checking even when some values must not be
  converted/typechecked (for example the 'default' string in
  SetInstanceParameters)

Reviewed-by: iustinp

Due to the fact that we sanitize now the output from environment
scripts, the unittest needs to be adjusted. My bad for not checking it.

Reviewed-by: imsnah

The resources we still need moved to rlib2.

Reviewed-by: iustinp

Reviewed-by: iustinp

It is impossible to keep backward compatibility due to
significant changes in the Ganeti core.

Reviewed-by: iustinp

This patch enhances the duplicate DRBD minors checks (currently just a
few) and adds automatic checks of configuration consistency at
configuration file writing time.

In order to do so and show meaningful error messages, the
_UnlockedComputeDRBDMap function is changed to not raise errors in case
of duplicates, but instead return both the minors map and the duplicate
list, and its callers now raise the error. This allows the VerifyConfig
function to return a complete list of duplicates.

The new checks required some small updates to the unittests for the
config module.

Reviewed-by: ultrotter

In order to prevent errors with old, in-use DRBD minors, we check and
abort at create time if our minor is already in use. For this we need to
also modify DRBD8Status to be able to parse cs:Unconfigured devices.

Reviewed-by: ultrotter

This patch adds a tail file function, to be used for parsing and returning in
the job log OS installation failures.

Reviewed-by: ultrotter

This patch adds unified temporary file handling to the
testutils.GanetiTestCase class, which adds easy creation and automated
cleanup of temporary files.

The patch allows a simpler handling in a couple of test cases but
requires all child classes to call the parent setUp and tearDown
methods.

Reviewed-by: ultrotter

Reviewed-by: amishchenko

Unfortunately, os.makedirs in Python 2.4 is not safe against multiple
processes creating the same directory tree at the same time. This is
only fixed in Python 2.5 and up. Adding more checks in our code doesn't
make it any better.

Reviewed-by: iustinp

More complex unittests will need some refactoring in the HTTP code.

Reviewed-by: amishchenko

It should be consistent.

Reviewed-by: amishchenko

We don't need it anymore and it wouldn't work as it is, anyway.

Reviewed-by: ultrotter

Reviewed-by: iustinp

Reviewed-by: imsnah

This patch enables by default the old 'human-readable' option, but in a
slightly different model.

The option is now called "units" and takes either:
 - 'h' for automatic formatting
 - 'm', 'g' or 't' for mebi/gibi/tebibytes

If 'h' is used, we add a unit suffix, otherwise nothing is added so that
parsing is easy.

The default value of this unit is:
  - 'h' if a separator is not passed
  - 'm' if a separator is passed

Reviewed-by: ultrotter

Since we can use the FieldSet class in cli.py to nicely format disk
sizes and such, we move it to utils.py and also move its associated
unittest. I didn't remove the cmdlib.py unittest file as that's not the
good direction :)

Reviewed-by: ultrotter

The waiting sharer blocks exclusive is another not-possible to test
right now.

Sorry for missing it the first time; there are no other testWaiting...
tests.

Reviewed-by: ultrotter

This patch adds a _FieldSet class that can be used for the new variable
parameter sets: e.g. the sda_size will change to disk/0.size (or
similar) and we need to both check validity and extract the index of the
parameter easily.

The patch also sorts the unittest list in Makefile.am.

Reviewed-by: ultrotter

Until we have a way to properly test this, we'll skip the test.

Reviewed-by: imsnah

Currently the locking tests are using timeouts to ensure an event will
'never happen'. However, this is suboptimal.

The patch converts all of these to instead: not wait, but sequence the
operations logically and expect that they execute as such. In case of
not so, they will timeout with a big (60s) timeout.

The 'never happen' is tested by multiple repetitions of the same test -
this is not perfect, but again 'X will never happen' is not actually
testable.

This patch reduces the runtime of the tests from ~5.4 to ~0.8 seconds
(with 8 repetitions of each test thas has 'never happen' situations).

Reviewed-by: imsnah