Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:

- bump version in the docs
- silence some new checks that are wrong due to our indent=2 instead of 4
- fix lots of errors in the code where the indentation was wrong by 1
  or 2 spaces
- fix a few cases of == True, False, None and replace with 'is'
- re-indent some cases where the code is OK, but pep8 complains

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Since this is an inheritable parameter, having it as a MABYE_STRING
creates only problems (per our derivation rules). We change it to
STRING, with the default "", meaning no program. Note that most of the
code already accepts this as valid for "no program", and some comments
even say that this is the expected value.

We have some other parameters like this, I'll have to investigate
whether they need to be changed too. But right now I need this for the
hconfd changes (it's a prerequisite for them, I forgot to send it in
that patch series).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This is the design doc for the bulk instance creation. You can more
details in the doc itself.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is a trivial code change, but it allows us to finally test the
send-receive code on both client and server sides via a simple
in-process server.

The unittest works, but it won't handle timeouts very nicely; it will
wait until the actual Luxi timeout expires, instead of using much
shorter timeouts as we could in the same process.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

I'm doing this change for future performance optimisations. Currently
we use the Luxi interface just as a client, so not in the hot path,
but when we'll use this as a server interface, we're interested to
both reduce the space and time consumption of the interface.

We have to simultaneous changes here:

- switch from using socket-related function (sendto, recv, etc.) to
  handle-based functions, since the standard network library doesn't
  work with sockets
- switch from using Strings for the internal buffer to strict
  ByteStrings; the only downside is that we now have the issue of
  decoding/encoding from binary to UTF-8 strings, a fact which brings
  its own issues into the mix (we have to check for failed decodings,
  etc.); but this is similar to what we'll have to handle on the
  Python side when moving to Python 3.x

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This has been a long-standing cleanup item, which we've always
refrained from doing due to the high estimated effort needed.

In reality, it turned out that after some infrastructure improvements
(the previous patches), the actual job queue-related changes are quite
small.

We will need to update the NEWS file later, but so far the RAPI
documentation doesn't mention that the job ID is a string (it only
says it is "a number"), so it doesn't look like it needs update.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Currently, all the CLI helpers in qlang.py and cli.py assume that all
namefields are string, which leads to various breakage in case the're
actually not.

To improve the flexibility of the helpers, we add a bit of
infrastructure for accepting so called "numeric" namefields; this is a
bit of a hack, as a proper fix would actually add QFT_* support to the
helpers, and case for example the regex/globbing on QFT_TEXT, etc. But
that's left for (eventual) later improvement.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This patch removes/abstracts some hardcoded values in the jqueue
unittests. Currently we have a per-resource type name field, but the
name field values are hardcoded at the call sites, instead of being
abstracted into separate variables.

This will become a problem later, so let's just introduce some new
vars holding these; modifying the tests later will be therefore
easier.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

It wasn't clear for which tests we were ignoring the errors.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

A while back, we did cleanup the code and ensured (manually) that use
of OpPrereqError includes an errors.ECODE_* field as second
argument. Since we cannot automate the check for this, it turns out
that more and more such usage has crept over the years, including in
the master code (the use on the CLI side is not as important).

Note that this also uncovered a few errors in ovf.py where the errors
messages were wrongly constructed.

Still looking for a way to automate this check…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Both the job id and submit job result parsing are abstracted into
separate functions, so that later changes are more localised.

Also, this makes submitManyJobs itself easier to read.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This will be used for easier change later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

QA for recreate-disks called it twice with a list of node; the first call
now uses an allocator.

Signed-off-by: Adam Ingrassia <api@google.com>
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The recreate-disks command no longer requires an explicit list of nodes.

Signed-off-by: Adam Ingrassia <api@google.com>
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This affects only the backend.

Signed-off-by: Adam Ingrassia <api@google.com>
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Ideally this should have used relocation mode, but to due lacking of
support in current allocators, allocation mode has been used instead.

Signed-off-by: Adam Ingrassia <api@google.com>
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

_CheckInstanceNodeGroups behaves in the old way with the default value
of the new parameter.

Signed-off-by: Adam Ingrassia <api@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is just begging to be converted to a standard replace_vars_sed
rule, instead of custom sed calls.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A rule of type "a/%: a/%.in" will also match "a/b/%: a/b/%.in", so no
need for the explicit examples/hooks rule. As for the man rules, they
are identical and thus can be collapsed.

We still have the problem that globally, not all our %.in to %
transformations are identical; this is suboptimal and should be
cleaned sometime…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Generic rules like:

  %: %.in

have the downside that the source (%.in) itself matches the target
(via %.in: %.in.in). This leads to things like:

  Looking for a rule with intermediate file `doc/examples/hooks/ipsec.in.in'.
   Trying pattern rule with stem `ipsec.in.in'.
   Trying implicit prerequisite `doc/examples/hooks/ipsec.in.in.in'.
   Trying pattern rule with stem `ipsec.in.in'.
   Trying implicit prerequisite `doc/examples/hooks/ipsec.in.in.in'.
   Looking for a rule with intermediate file `doc/examples/hooks/ipsec.in.in.in'.
  Rejecting impossible implicit prerequisite `doc/examples/hooks/ipsec.in.in'.

To fix this, we need to tell make that such rules are terminating, so
that it doesn't recurse into them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

GNU Make contains some (ancient) implicit rules, that try to
_automatically_ extract source files from RCS/SCCS version control
systems. This is unneeded, and it pollutes the make -d output
significantly: after removing these rules (by defining empty targets
for their patterns), make -d line count goes from 5.3K to 3.3K.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit dc7d2c49 introduced the use of BUILT_SOURCES to work around
missing dependencies. However, on closer reading of the gmake manual,
BUILT_SOURCES is mainly used in cases where the dependencies are not
know before the build starts (e.g. with auto-generated C header
files). Additionally, there are a number of drawbacks to
BUILT_SOURCES, which we already had to work around (e.g. in commit
eb732fb5).

Since we know all our dependencies statically, there's no need to use
this special variable. Let's rename it to GENERATED_FILES, which
doesn't have a special meaning to make, and add a few missing
dependencies (one of which was already broken by make -j on a clean
tree).

After this change, running make in a fully built tree is finally "clean":

  $ make
  make: Nothing to be done for `all'.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, if one runs 'make' in an already fully-built tree, this is
the result:

  cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; }
  make  all-am
  make[1]: Entering directory `/tmp/test'
  cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; }
  make[1]: Leaving directory `/tmp/test'

This is because commit dc7d2c49 added 'ganeti' (which is a PHONY
target) to BUILT_SOURCES, and since that is a dependency of other,
real targets, it means the ganeti target is always remade.

To fix this, we keep ganeti as a PHONY target, but we remove it from
the 'built_base_sources' target, and instead we only remake it
manually in the stamp-directories target. A make run now is just:

  make  all-am
  make[1]: Entering directory `/tmp/test'
  make[1]: Nothing to be done for `all-am'.
  make[1]: Leaving directory `/tmp/test'

Note that we can't get rid of the all-am since we use BUILT_SOURCES.

We also remove the comment of BUILT_SOURCES since it no longer depends
on PHONY targets.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit c964d962 changed the way we create directories, by two things:

- unifying all dependencies and ad-hoc directory creation into a
  single target (all_dirfiles)
- changing how directories are created from a stamp file to .dir files
  in each directory

The first item is a very good one, but the second item is debatable:
there's no per-se advantage of .dir files versus a single one,
top-level, since both the .dir file and stamp-directories creation are
depending on Makefile, which is the only one which can introduce new
directories.

On the other hand, moving back from .dir files to stamp-directories
has an advantage: "make -d | wc -l" does from ~8.7K lines to ~5.3K
lines, because we eliminate the many .dir files and their multiple
implicit and explicit dependencies (the %/.dir files fall under
multiple patterns).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Seen while debugging make rules.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Reading the automake documentation, it seems that explicit calls to
$(MAKE) need $(AM_MAKEFLAGS) passed, so let's do that.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This removes "pass" from a few definitions, making all errors
consistent.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This adds supports for the (non-validated) cpu parameter to KVM.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Fixes: http://code.google.com/p/ganeti/issues/detail?id=251


Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* stable-2.6: (21 commits)
  Release Ganeti 2.6.0 rc4
  Prepare NEWS for Ganeti 2.6.0rc4
  Add some rudimentary node group ipolicy checks
  Fix setting ipolicy on node groups
  Fix --no-headers for the new list-drbd command
  Add a simple QA test for gnt-node list-drbd
  Add a new gnt-node command list-drbd
  Implement a node to drbd minors query function
  Add a new unused confd query
  Add support for computing instance all/secondary nodes
  Add disk logical ID support in Objects.hs
  Fix a docstring in bdev's DRBD8 class
  Extend the Template Haskell loadFn model
  Change how customFields are built
  Implement lookup of partial names in Config.hs
  Reorganise the lookup functions
  Remove an unused function
  Ensure that disk.params is always defined (and a dict)
  Another small consistency fix with if branches
  Fix inconsistency in the LUXI protocol w.r.t. args
  ...

Conflicts:
        htools/Ganeti/HTools/QC.hs (imports changes, trivial)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

By a strange accident, this file seems to be tabified, instead of our
own 2-space rule. Re-indent it and also break one long line.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Another release candidate is coming.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

On node groups we don't have the std field. However, the InstancePolicy
object always verifies that the std value is within a given range. As we
fill it up with defaults if not set (as it happens to be on node groups)
and the min value is higher than the default std value (taken from
constants.py) we fail.

We overcome this situation by simply let the function know if we want to
verify the std value at all. If we don't want to verify std, we just set
it to a compliant value (min_v) and continue.

We also slightly adapt the error message provided, as we don't have std
values on groups.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Setting a high prefix discourages the bridge from adopting the tap's
MAC. Xen is not affected by this since the MAC is forced to
"fe:ff:ff:ff:ff:ff".

This addresses issue #217.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Sorry, I forgot that with GenerateTable this needs to be handled
manually. Fixed now and tested in both ways.

(But to be honest, this should be abstracted in GenerateTable, instead
of the 'if' test in all its callers.)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This will be run only if 'haskell-confd' QA configuration has been
enabled.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This uses confd to query the DRBD minors, which is very special; no
other command currently does so.

Since the backend is only implemented in the Haskell version of confd,
we have checks that 1) confd is enable, and 2) hs confd is also
enabled. If by mistake people do manage to query Python confd, the
error message will be clean:

  Query gave non-ok status '2': not implemented

So nothing breaks in an "ugly" way.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This can be queried remotely since it's a pure configuration query.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>