* devel-2.5:
  Fix wrong option names in QA and cluster-merge
  Bump version to 2.5.0~rc5, update NEWS
  Add UnescapeAndSplit unittest for multi-escapes
  Fix a bug in command line option parsing code
  cli: Disable abbreviation matching for options
  Bump version to 2.5.0~rc5, update NEWS
  Add UnescapeAndSplit unittest for multi-escapes
  Fix a bug in command line option parsing code

Conflicts:
	lib/cli.py: Trivial

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* stable-2.5:
  Bump version to 2.5.0~rc5, update NEWS
  Add UnescapeAndSplit unittest for multi-escapes
  Fix a bug in command line option parsing code

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This would have caught the bug in the first place. Argh,
hand-generated test cases!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Fix bug affecting command line options of "keyval" type. Although
escaping commands with \ is supported, it is is not applied to the
input recursively.

Signed-off-by: Nikos Skalkotos <skalkoto@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Python's “optparse” module does option name prefix matching by default.
Since this can lead to confusing behaviour, e.g. by specifying “--force”
for a command which only has a “--force-multi” option, this patch
disables the prefix matching.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add new --maxmem-size and --minmem-size options.
The old --mem-size option allows setting both at the same time.
If --minmem-size is lower than --maxmem-size execute a runtime memory
change on all instances, bringing them to their minimum memory size.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Currently they have no memory limits set.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is set to the same amount as the memory for now, but will allow
starting instances with less memory than their maximum.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This new hypervisor call sets the memory of an instance to a new value,
through ballooning.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Currently, noded requires PUT, even though the semantics of the RPC
calls do not match a PUT. We change the code accept both PUT and POST,
with the intention to remove the PUT support in a later version.

On the client side, the RPC code is changed to send POST requests.

Additionally, we add a message to the HttpBadRequest exception to make
clear the failure mode (not seeing any error message was what made me
send this patch…). This was the only description-less use of this
exception, by the way.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

After looking again at the QA speeds, it turned out that instance
rename was run 12 times via the command line and 12 times via RAPI;
outside of start/stop which we need for maintenance during QA, this is
the most used test.

This patch does a few changes with regard to instance rename:

- it changes rename functions to rename the instance back to the
  original name
- which allows us not to run said functions twice, A→B and then B→A

This brings down the number of rename tests from 12 to 8 (for both
cmdline and RAPI).

Furthermore, the CLI rename test was doing too many actual instance
renames (at Ganeti level), so we rework it to be simpler (brings down
the average duration from ~27s to ~21s), and we reorganise the run of
various tests such that we do fewer instances start/stop tests (10
instead of 14).

Overall we only gain about 3 minutes (~7%), but still it's a speedup.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

It seems that QuickCheck will happily generate huge data structures,
if that's what one requests. For this test, where we ask for [[Char]],
it could happen that we get a huge number of chars, so let's limit
them to some (arbitrary, but low) limits. This doesn't "ruin" the
test, since the implementation of sepSplit/commaJoin is not
size-based.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Otherwise we test very big lists (slow) without any advantage in doing
so.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

These two tests are very similar, so let's run them together instead
of generating the data twice. Also improve the test a bit.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

After reorganizing the way we process options, we can finally add a
test that all program personalities accept --help and --version and
that they exit early for these two.

This trivial patch has a higher than expected impact on the coverage,
as the use of short options finally enforces evaluation of the short
option names, thus it affects more than just the help/version options.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

… a mini-module, actually. This is only done for unit-testing, as we
don't want to duplicate the personalities list in the tests.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch moves the call to getArgs/parseOpts into htools.hs, as
opposed to the current situation where htools.hs only calls the main
file of each program, which in turn calls these two functions.

By doing it only once we remove some duplicate code and we also export
each program's options, which allows us to test some common behaviour
in the future.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Just tests that passing a string argument returns it correctly. We're
not interested in the flags themselves that much just that the
function itself parses some options at least correctly.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This will allow unittesting the CLI options much more easily. The
patch also fixes an inconsistency: usage help was displayed with an
extra new line in the error case (but not in --help).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Also fix a tiny typo in the docstring.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

'zip' finishes on the shortest list, so parseISpecString accepted any
string of 3 values or more, which is not good. Fix this and add test
for both successful and failed parsing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This only test successful parsing, but for this case it tests
reasonably well the resulting cluster properties, I think (coverage
is >70% for expressions).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Following on the lessons from this week (commit 7806125e), this patch
replaces generation of instances `suchThat` they are smaller than on
node with direct generation of instances with upper bounds on
memory/disk/cpu.

For a particular case I was investigating, this reduces the overall
duration of tests by almost 50%. I don't expect it to speed up this
nice in most cases, but it's too good an optimisation to pass up.

The patch also does a trivial simplification (if … then True else
False, d'oh).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This simplifies the building of profiling-enabled htools binaries;
there are two targets, one used for the initial two-stage build, and
one that can be used for incremental builds.

The patch adds a new internal variable for the htools/test binary,
which should also simplify custom builds for the non-profiling
targets.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Learning, learning. It turns out that for objects with even just five
fields, generating them and filtering can be costly, especially when
the filtering is nested (as it was in this case, arbitrary1 →
arbitrary2 + filter on arbitrary1 → arbitrary plus filter on
aribitrary2).

Changing the code to generate directly the good values speeds things
up significantly. Additionally, we also generate correctly unique
lists of disk templates (again, this should be a set, not a list,
but…).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

It could happen that the node is generated with 0 disk available, in
which case the operation will fail due to disk issues, not CPU. Fix
this by using the recently-introduced genOnlineNode helper.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

As a followup from "Remove extraneous check in policy creation", there
are more places where we build an ipolicy, and then manually check for
its validity. This is very bad style, as it duplicates the
verification code across many places.

This patch removes all such explicit checks (except for one in
cmdlib.py which is correct), and instead does a bit more validation in
the builder functions or in the actual dedicated verification
functions. It also fixes cluster init which used the wrong,
non-completed ipolicy (this was not detected before as we did call
check on it, but otherwise we ignored it), and fixes a too-strong
assert (due to the call chain, we first create the ipolicy from
cmdline params, and only then we fill it).

Finally, it removes an extraneous logging.info which I forgot from
debugging.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Another "good" seed value manages to generate nodes with identical
names, which of course makes the sets be of smaller value than the
lists. Avoid this issue by generating unique node names.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

While wondering why Cluster.tryReloc is not unit-tested, I realised
that this is no longer uses. Apparently, last summer's new IAllocator
modes and related changes have silently deprecated this function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch fixes two ugly but non-critical issues:

- the getFQDN wrongly generated only the first component as
  limited-length name, the rest could be any length, and that makes
  the generated name bad for display purposes
- the random generator had real issues in finding an offline instance
  with the right properties when starting from certain seeds (found
  accidentally); this meant the test took 1 minute, instead of ~300ms,
  for the node_addSecOffline

Furthermore, for the same evil seed, the Container_findByName test
wasn't able to generate enough good data, so we change its
implementation to directly generate an uniques list, instead of
generating random lists and filtering the unique ones.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>