* stable-2.6:
  Disable E1101 on ganeti/http/server.py:424
  Fix live migration under xl
  Don't check for xend port when using xl

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Until now the only way to make live migration work in conjunction with
"xl" was to add ssh known_hosts keys for every node's secondary ip on
every other node.

With this command we remove the target key verification: this is not
worse than what we were doing before with "xm", and allows the migration
to happen under either toolstack, without extra manual work. Of course
the full security of ssh is not used by live migration, then.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

If the toolstack is set to "xl" we shouldn't ping xend for liveness
before attempting a live migration.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* stable-2.6:
  Fix typo in gnt-instance man page
  jqueue: Return jobs to queue when shutting down
  gnt-debug delay: Add "--submit" option
  Make hostname checks uniform between instance rename and add
  Improve logging of new job submissions
  Improve handling of lock exceptions
  Add note about developing on a production machine
  Fix runtime memory increases
  Fix validation of vgname in OpClusterSetParams
  Fix removal of storage directory on shared file storage
  Switch non-redundant check to disk template-based
  Document the new --yes-do-it option for master-failover
  Add option to force master-failover without voting
  Update instance modify message
  Force tap's MAC prefix to "fe"
  Fix disk adoption interaction with ipolicy checks
  Better NEWS file for 2.6.1

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

When a job is still waiting for locks and the queue is shutting down,
they should be returned and not actually start processing. Until now
jobs which transitioned from “queued” to “waiting” were already
considered to be running as far as the shutdown code was concerned.

This fixes issue 296.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, we have instance rename doing extra checks on the host
name, to prevent accidental wrong renames; however, instance create
doesn't do these checks (issue 291), which (if DNS is misconfigured)
can lead to hard to diagnose errors.

This patch abstracts the name checking from LUInstanceRename into a
separate function, which is then reused in both instance rename and
instance create.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This addresses issue 290: when receiving new jobs, logging is
incomplete, and we don't have the job ID(s) and/or summaries
logged. Only later, when the job is queried for or being processed, we
know more.

This is not good when troubleshooting, so let's improve the initial
logging.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

There are two issues with lock exceptions right now:

- first, we don't log the original error; this is fine for now
  (locking.py always returns the same error here), but in general is
  brittle: if locking.py would start returning more information, we'd
  completely miss that

- second, an actual honest lock conflict is not an internal error;
  it's simply an optimistic lock failing, and as such we should not
  return internal error, but rather resource_not_unique

This addresses issue 287.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This is the bit of documentation missing for issue 170. Doing
development on a machine which already has Ganeti installed kind of
works, but only when the installed and the developed version are very
similar, and even then it can be problematic.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Commit 2c0af7da which added the runtime memory changes functionality
had a small typo (wrong name); I've rewritten this to only compute the
delta once, for simplicity.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This variable can be empty, when we want to disable LVM, so we can't
use TMaybeString.

Fixes issue 285.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch makes _RemoveDisks symmetric to _CreateDisks with respect
to file-based storage: _CreateDisks uses "in constants.DTS_FILEBASED",
whereas _RemoveDisks was not update and only uses "==
constants.DT_FILE". This results in stale directories left on the
filesystem.

Fixes issue 262.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the warning/notice about non-redundant instances in cluster
verify is based non empty secondaries list (how old is this?); the
proper way to check this nowadays is via DTS_MIRRORED.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Sorry, this should have gone in the previous commit, I forgot about
it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This fixes issue 282.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Currently the message does not say explicitly that instance-initiated
reboots are useless to trigger the use of new parameters, per the
thread on the user mailing list. Let's improve it a bit.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Setting a high prefix discourages the bridge from adopting the tap's
MAC. Xen is not affected by this since the MAC is forced to
"fe:ff:ff:ff:ff:ff".

This addresses issue #217.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit be3f52f3)
(this is critical enough that we want it on stable-2.6)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In Ganeti 2.6, disk adoption is broken due to the ipolicy checks being
done before we read volume size from remote nodes. We fix this by
simply moving these checks to after the disk adoption code which
updates the disk size; it's not that nice that we fail a (almost)
config-level check after we've reserved the LVs, etc., but we need to
do so in order to validate the ipolicy correctly.

Tested:

- normal instance creation
- creation via adoption with good size (pass)
- creation via adoption with wrong LV size (fail as expected)
- QA in progress

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Before they would just be silently ignored.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

No string replacements are used, so doubling of the percent sign is not
necessary.

Before: Usage: %ensure-dirs [--full-run]
After: Usage: ensure-dirs [--full-run]

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

s/it/if/

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Dato Simó <dato@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Better formatting of text, past tense used when appropriate.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This commit adds a Group.allTags field to store the tags of node groups,
and teaches each loader backend in HTools to populate it (additionally, the
IAllocator class in lib/cmdlib.py now includes tags for groups too). Test
data is updated to include an empty set of tags for node groups in all
affected test cases.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The mergeData function in Loader.hs included a step to filter an instance's
tags to include only the exclusion tags (as specified via the commandline,
or cluster-level tags). Later on, code in Node.hs assumed Instance.tags to
contain only tags to be used for exclusion.

Because in the future we will need to access the full list of an instance's
tags (and not only exclusion tags), this commits deprecates the 'tags'
field, and introduces Instance.exclTags and Instance.allTags.

Instance.allTags is now populated from the different backends (Text, Luxi,
Rapi, etc.), and Instance.exclTags is only populated from Loader.mergeData,
as was done previously. This means that loading tags from e.g. Text or Simu
and assuming that they'll be used as exclusion tags without going through
Loader.hs will no longer work; but this was already the case with other
fields, and 'mergeData' or 'loadExternalData' continue to be the only entry
points to get a consistent view of the cluster. (Additionally, there were
no tests that made this assumption that I could find.)

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

In preparation for future modifications in the exclusion tags field, add a
test that verifies that exclusion tags are being honored: in a test cluster
with two instances of the same exclusion group in each node, hbal should
shuffle instances around to improve the score.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

A socket shouldn't have its executable bit set.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Quite some things were out of date. Some formatting was also updated.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* stable-2.6:
  Update NEWS and bump version to 2.6.1

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This is a small bug-fix only release.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The comments in parseData had become out of date with the implementations
of load{Group,Node,Inst}. This commit updates the field list in comments to
match the implementations.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* stable-2.6:
  verify-disks: Explicitely state nothing has to be done
  Add list of design documents implemented in version 2.6
  Better list of replace-disks arguments + typos fixed
  jqueue: Look at archived jobs when watching
  Show old primary/secondary node on disk replacement
  gnt-instance reinstall: Don't always exit with success
  LUClusterVerify: Ignore /proc/drbd if DRBD is disabled
  Fixed typos in devnotes.rst
  Always_failover doesn't require --allow-failover anymore
  bash_completion: Enable extglob while parsing file
  rpc: Remove duplicated logic, fix unittests
  Annotate disk params on instance_start
  cmdlib: Handle locking.ALL_SET correctly when copying locks

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Example output:
$ gnt-cluster verify-disks
Submitted jobs 4327
Waiting for job 4327 ...
No disks need to be activated.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Each version should have its dedicated list.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The man page and the bultin-in help for gnt-instance replace-disks were
inconsistent. Also fixed some typos in man pages.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

First: This enables the use of “gnt-job watch $id” for archived jobs.

Now, the reason for actually making this work is that during
sufficiently large group or node evacuations jobs are archived before
the client gets to poll for their output. This led to situations where
the jobs would finish successfully, but the client reported an error
because it couldn't see the job anymore.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
(cherry picked from commit 04569469)

People unfamiliar with Ganeti's internals might be confused with the
different hostnames showing up later in the process.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>