If a device has entries in its holder directory
(/sys/block/$name/holders), it means that some kernel system "uses"
that device, and hence should not be considered available.

This patch adds a new 'in-use' check based on this sysfs test, and
introduces an overal InUse function for devices.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch, originally written by Marc Schmitt <mschmitt@google.com>,
adds support for MD devices (used in a non-partitioned mode). I
abstracted all the original startswith('md') checks into separate
functions, and also moved the supported disk types to a list.

Proper "in-use" detection also needs another check, which will come in
a subsequent patch.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

- Check hostname and abort if it doesn't match contents of
  “ssconf_master_node”, can be overridden using “--ignore-hostname”
  parameter.
- Clarify confirmation question and don't mention instances anymore.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

- Remove non-ASCII character from manpage
- Reformat docstring for epydoc in script

These caused build breakage on some but not all distributions.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This tool was used and worked on internally for quite a long time. We
decided to include it in Ganeti.

Known issues:
- Code doesn't match rest of Ganeti (e.g. using “print” all over the
  place, hardcoded calls to sys.exit deep in functions)
- Code duplication from Ganeti library (e.g. PingByTcp/netutils.TcpPing,
  GetHosts/utils.ReadFile)
- Using ssh-agent doesn't work with more than one worker (Paramiko keeps
  the socket open and the file descriptor is used from different
  workers)
  - No clear separation between parent and child process in code
- Uses getopt instead of optparse

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This reduces the number of notifications in “ganeti-rapi”. Until now it
was notified for every change in …/lib/ganeti and had to check whether
the users file was affected. A symlink is always created in cfgupgrade
to not break tools referring to the old name.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Also update cfgupgrade and NEWS.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Together with Michael we refactored the code to make it better and
easier error reporting. Without printing backtraces for authentication
and verification issues.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

When moving a single instance within the same cluster, the NIC
is not allowed to re-use an existing MAC address. To avoid this,
NIC parameters must be overridden. BE, HV, OS and NIC parameters
can be overridden after applying this patch.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch changes the behaviour of OpDiagnoseOS with regards to the
'valid' field to be similar to the one for the hidden/blacklisted
fields: unless this field is requested, invalid OSes are filtered out.

The rationale is that, except for the gnt-os info/diagnose, all other
users of this opcode are requesting the valid field just to filter out
invalid OSes, and not for any other use. Thus, changing this behaviour
makes these callers simpler.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This checks if the host might possibily belong already to another
cluster. If this is the case we abort without any further action
unless we are forced by --force-join.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This reverts commit 2c9cf6bb.
The change is better done in objects.py

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch fixes a bug, which caused the primary and secondary
destination nodes to be revered on the destination cluster after an
instance move.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This was previously done in setup-ssh which was the wrong place

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Due to what seems like a bug (or inconsistency) in paramiko, files
opened with a+ over SFTP need a seek() in order for the user to be able
to read data from them. We implement this, and rely on the fact that we
do iterate over all lines before writing and that the file is opened in
append mode (which at least on Linux should work correctly).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Currently, setup-ssh only uses one disk-based key. This means that any
setup where we use keys from ssh-agent (which do not necessarily exist
on disk) will break when moving from the old method to setup-ssh.

This patch moves the SSH key handling to separate functions, and uses
both the disk key (first) and the agent keys for login.

The patch also fixes the root_logger setup level (I tried to hard to
reduce noise and broke the debug level, sorry).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch changes the setup-ssh workflow to try key authentication
first, and then fall-back to password authentication. The password is
also read lazily, with no prompts if we can authenticate via keys.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

There is not enough logged by default on stderr, and way too much
information in the log file.

Since we don't want to debug paramiko itself, we filter only warnings
and above.

Otherwise, we make WARNINGS the default level, with verbose meaning INFO
and debug, well, DEBUG.

Additionally, a message is improved.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Then reuse it. With certain setups, this could be slow when pre-seeding
an entire cluster.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This took a bit of interesting debugging, but the issue seems to be that
on older python/logging/paramiko (I'm not sure which is at fault here),
the other threads are finishing later than the main thread, so the
logging module closes the logging file (but doesn't unregister it)
before the paramiko thread(s) get a change to finish their logging.

Tested on both old versions (Dapper) and new versions (Lucid) of these
three pieces of the puzzle.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

If we call burning with only existing instance, then it will fail to
create any of them, and thus in the removal phase it won't have anything
to remove. Since calling luxi.SUBMIT_MULTIPLE_JOBS with an empty job set
is an error (and will raise an exception), this creates a very strange
error in burnin (which is unfortunately hidden by ExecJobSet()).

As such, we modify CommitQueue to return immediately if it has an empty
op queue.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

By changing it to a normal parameter, which must be a sequence, we can
start using keyword parameters.

Before this patch all arguments to “AddTask(self, *args)” were passed as
arguments to the worker's “RunTask” method. Priorities, which should be
optional and will be implemented in a future patch, must be passed as a keyword
parameter. This means “*args” can no longer be used as one can't combine *args
and keyword parameters in a clean way:

>>> def f(name=None, *args):
...   print "%r, %r" % (args, name)
...
>>> f("p1", "p2", "p3", name="thename")
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 TypeError: f() got multiple values for keyword argument 'name'

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is breakage from the original 'live' parameter changes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This prepares the remote node to be joined into a cluster

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Just calling utils.SafeEncode on the log message failed when it
wasn't of the type ELOG_MESSAGE and not a string. Now non-message
log entries are formatted using repr().

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

After some more investigation, only the soft reboot type fails for Xen
3.4 (due to the reboot/uptime time counter). As such, it's better to
allow selective testing, since we do want to test in general these
opcodes/the command line script.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Luca Bigliardi <shammash@google.com>

Currently the RAPI client uses the urllib2 and httplib modules from
Python's standard library. They're used with pyOpenSSL in a very fragile
way, and there are known issues when receiving large responses from a RAPI
server.

By switching to PycURL we leverage the power and stability of the
widely-used curl library (libcurl). This brings us much more flexibility
than before, and timeouts were easily implemented (something that would
have involved a lot of work with the built-in modules).

There's one small drawback: Programs using libcurl have to call
curl_global_init(3) (available as pycurl.global_init) while exactly one
thread is running (e.g. before other threads) and are supposed to call
curl_global_cleanup(3) (available as pycurl.global_cleanup) upon exiting.
See the manpages for details. A decorator is provided to simplify this.

Unittests for the new code are provided, increasing the test coverage of
the RAPI client from 74% to 89%.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is needed in a new test script for cfgupgrade where it's better
to only use the minimum parts necessary for testing cfgupgrade.
Including full configuration data to ensure verification doesn't fail
would make maintenance difficult.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In the condition for checking the configuration directory, one “or” should have
been an “and”. This bug was in cfgupgrade since commit 95e4a814 (June 2008).
Found thanks to a test script.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>