hansmi helped me with merging the conflict. Thanks

Conflicts:
	lib/workerpool.py

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

And provide unittests for them

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.1:
  Increase version to 2.1.7
  Update NEWS for the 2.1.7 release

Conflicts:
	NEWS
	  - merge
	configure.ac
	  - keep the 2.2 version

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

To add job priorities, the worker pool underlying the job queue must
support priorities per task. This patch adds them to the worker pool.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

While writing unittests for the new lock monitor, I made a small mistake and
didn't synchronize acquiring locks properly. This patch takes care of this
issue by adding additional synchronization.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch adds an initial implementation of a lock monitor, accessible
for the user through “gnt-debug locks”. It currently shows all resource
locks: BGL, nodes and instances. Config and job queue locks could be
shown too, but wouldn't be of much help.  The current owner(s) and mode
are also shown.

Showing pending acquires will require further changes on the SharedLock
internals and is not yet implemented.

Example output:
$ gnt-debug locks -o name,mode,owner
Name            Mode      Owner
BGL/BGL         shared    JobQueue19/Job147
instances/inst1 exclusive JobQueue19/Job147
instances/inst2 -         -
instances/inst3 -         -
instances/inst4 -         -
nodes/node1     exclusive JobQueue19/Job147
nodes/node2     exclusive JobQueue19/Job147

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

With this patch, the task name is added to the thread name and will show up in
logs. Log messages from jobs will look like “pid=578/JobQueue14/Job13 mcpu:289
DEBUG LU locks acquired/cluster/BGL/shared”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

When removing the field in commit 9bdab621, I forgot to remove some parts.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Update NEWS and version number.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

* devel-2.2:
  setup-ssh: fix updating of authorized_keys
  setup-ssh: Also use keys from the ssh-agent
  setup-ssh: try to use key auth first
  setup-ssh: redo the logging levels
  setup-ssh: only read the ssh port once
  setup-ssh: fix the logging error message
  Use Sphinx' :rfc: extension to refer to RFCs
  Document non-standard usage of JSON in RAPI
  Fix small spelling mistake
  Explicitly add dry-run to some commands
  Stop adding the dry-run option by default
  Fix a few commands behaviour with dry-run
  jqueue: Remove lock status field
  QA: Run simple job queue test
  Don't ignore secondary node silently
  etags: force Python as a language

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

By doing this we delegate the task of finding the correct address family
to the FormatAddress method.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Due to what seems like a bug (or inconsistency) in paramiko, files
opened with a+ over SFTP need a seek() in order for the user to be able
to read data from them. We implement this, and rely on the fact that we
do iterate over all lines before writing and that the file is opened in
append mode (which at least on Linux should work correctly).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, setup-ssh only uses one disk-based key. This means that any
setup where we use keys from ssh-agent (which do not necessarily exist
on disk) will break when moving from the old method to setup-ssh.

This patch moves the SSH key handling to separate functions, and uses
both the disk key (first) and the agent keys for login.

The patch also fixes the root_logger setup level (I tried to hard to
reduce noise and broke the debug level, sorry).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch changes the setup-ssh workflow to try key authentication
first, and then fall-back to password authentication. The password is
also read lazily, with no prompts if we can authenticate via keys.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

There is not enough logged by default on stderr, and way too much
information in the log file.

Since we don't want to debug paramiko itself, we filter only warnings
and above.

Otherwise, we make WARNINGS the default level, with verbose meaning INFO
and debug, well, DEBUG.

Additionally, a message is improved.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Then reuse it. With certain setups, this could be slow when pre-seeding
an entire cluster.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This took a bit of interesting debugging, but the issue seems to be that
on older python/logging/paramiko (I'm not sure which is at fault here),
the other threads are finishing later than the main thread, so the
logging module closes the logging file (but doesn't unregister it)
before the paramiko thread(s) get a change to finish their logging.

Tested on both old versions (Dapper) and new versions (Lucid) of these
three pieces of the puzzle.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This was requested in issue 118.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

My last setup-ssh patch series removed this option. Though, this is
just a temporary fix as a patch series is coming which revert that
behaviour to the old state

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patches changes the StartMaster method to consult the cluster
primary ip version when deciding whether to use arping or ndisc6 after
activating the master ip.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch restores the behaviour of Hostname (previously known as
HostInfo) to always use fqdn. This was broken due to the fact that the
now used getaddrinfo does not return an fqdn in contrast to
gethostbyname_ex().

This patch adds Hostname.GetFqdn() which is always used when
initializing a Hostname object.

A problem this solves is when adding a node and not using its fqdn, node
verify won't complain.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Based on manual inspection (that the command only does a submit of some
jobs/opcodes), we re-add the dry-run option on a subset of the existing
commands.

A few more commands could use dry-run, but the code doesn't deal nicely
with no results, so fixing those is left for later patches.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently cli.py unconditionally adds the dry-run option. This patch
disables this, and exports dry-run as a normal option.

The other alternative I tried to implement (adding a new fake option for
disabling the auto-add per individual command) would require changes in
more places, as the list of options is no longer a homogeneous list.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

These commands use or display the result from the LU, so in case of
dry-run, they will crash or display just 'None'. At least checking that
the result is 'true' (in the boolean sense) will make them work better.

As for gnt-os modify, it didn't pass the 'opts' parameter properly to
SubmitOpCode, so the dry-run option was silently ignored.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

With the job queue changes for Ganeti 2.2, watched and queried jobs are
loaded directly from disk, rendering the in-memory “lock_status” field
useless. Writing it to disk would be possible, but has a huge cost at
runtime (when tested, processing 1'000 opcodes involved 4'000 additional
writes to job files, even with replication turned off).

Using an additional in-memory dictionary to just manage this field turned
out to be a complicated task due to the necessary locking.

The plan is to introduce a more generic lock debugging mechanism in the
near future. Hence the decision is to remove this field now instead of
spending a lot of time to make it working again.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This avoids duplicate effort and has been a TODO for a long time.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>