Replacing ssconf with utility functions.

Reviewed-by: iustinp

Replacing ssconf with configuration.

Reviewed-by: iustinp

Replacing ssconf with configuration.

Reviewed-by: iustinp

The configuration version is now again in the configuration file.

Reviewed-by: iustinp

Replacing ssconf with simpleconfig.

Reviewed-by: iustinp

This can be used to retrieve certain cluster config values from
within clients.

OpDumpClusterConfig was not used anywhere, hence I'm just reusing
it. The way ConfigWriter.DumpConfig returned the configuration
was not thread-safe, anyway (no deepcopy).

Reviewed-by: iustinp

These functions will be used to access config values instead of using
ssconf.

Reviewed-by: iustinp

This will be used to read the configuration file in the node daemon.
The write functionality is needed for master failover.

Reviewed-by: iustinp

The watcher didn't handle the down nodes, fix this by ignoring (in
secondary node reboot checks) any node that doesn't return a boot id.

Reviewed-by: imsnah

The watcher was using conflicting attributes of the instance:
  - it queried the admin_/oper_state, which are booleans
  - but it compared those to the status (which is a text field)

The code was changed to query the aggregated 'status' field, as that
will also return indication of node problems, and we can use this only
one field for all decisions. We still ask for the admin_state field as
that is needed for the activate disks check (in secondary node restart).

The patch also touches the watcher in some other parts:
  - log exceptions nicer
  - convert a method to @staticmethod
  - remove unused imports

Reviewed-by: imsnah

The watcher has one last use of ganeti commands as opposed to sending
requests via luxi. The patch changes this to use the cli functions.

The patch also has two other changes:
  - fix the docstring for OpVerifyDisks (found out while converting
    this)
  - enable stderr logging on the watcher when “-d” is passes

Reviewed-by: imsnah

Reviewed-by: iustinp

ssconf will become write-only from ganeti-masterd's point of view,
therefore all settings in there need to go into the main configuration
file.

Reviewed-by: iustinp

Future patches will add even more variables to the cluster config.
Adding more parameters wouldn't make the function easier to use and
it doesn't make sense to pass them to another function, as it's
only done once in bootstrap.py on cluster initialization.

Reviewed-by: iustinp

Curently PollJob accepts a generic job, but will return (history
artifact) only the first opcode result. This is wrong, as it doesn't
allow polling of a job with multiple results.

Its only caller (for now) is also changed, so no functional changes
should happen.

Reviewed-by: ultrotter, amishchenko

Reviewed-by: iustinp

This patch also adds the design documents to Makefile.am.

Reviewed-by: iustinp

I didn't realize that my zip will break when no args are passed...

Reviewed-by: imsnah

This patch adds posibility of selection of job/opcode timestamps in
gnt-job list and info.

The code handling the possible cases (None or a valid timestamps) are
ugly though...

Reviwed-by: imsnah

This patch adds start, stop, and received timestamp for jobs (and allows
querying of them), and allows querying of the opcode timestamps.

Reviewed-by: imsnah

It said CLI/RAPI will talk to master using HTTP, which isn't true. Add
a reference to job queue design document. Small typos.

Reviewed-by: iustinp

Reviewed-by: iustinp

Reviewed-by: iustinp

Before we were discussing this possible future feature, and its
drawbacks, but not its usefulness. This patch corrects this.

Reviewed-by: iustinp

Reviewed-by: iustinp

Reviewed-by: iustinp

Currently we format the timestamp inside the gnt-job info function. We
will need this more times in the future, so move it to cli.py as a
separate, exported function.

Reviewed-by: imsnah

Reviewed-by: iustinp

This will be an overview document, enumerating the changes without going
into details and pointing to the actual documents.

Reviewed-by: ultrotter

This patch adds the job execution log in “gnt-job info” and also allows
its selection in “gnt-job list” (however here it's not very useful as
it's not easy to parse). It does this by adding a new field in the query
job call, named ‘oplog’.

With this, one can get a very clear examination of the job. What remains
to be added would be timestamps for start/stop of the processing for the
job itself and its opcodes.

Reviewed-by: imsnah

For now we only use the ‘C’ protocol so we can put it in constants.py
instead of hardcoding it.

Reviewed-by: imsnah

This patch enables the use of the shared secrets for DRBD8 disks, using
(hardcoded in constants.py) the md5 digest algorithm.

For making this more flexible, either we implement a cluster parameter
(once the new model is in place), or we can make it ./configure-time
selectable.

Reviewed-by: imsnah

This patch, which is similar to r1679 (Extend DRBD disks with minors
attribute), extends the logical and physical id of the DRBD disks with a
shared secret attribute. This is generated at disk creation time and
saved in the config file.

The generation of the secret is done so that we don't have duplicates in
the configuration (otherwise the goal of preventing cross-connection
will not be reached), so we add to config.py more than just a simple
call to utils.GenerateSecret().

The patch does not yet enable the use of the secrets.

Reviewed-by: imsnah

Currently, it is hard to examine a job in detail; the output of ‘gnt-job
list’ is not easy to parse.

The patch adds a ‘gnt-job info’ command that is (vaguely) similar to
‘gnt-instance info’ in that it shows in a somewhat easy to understand
format the details of a job.

The result formatter is the most complicated part, since the results are
not standardized; the code attempts to format nicely the most common
result types (as taken from a random job list), via a generic algorithm.

Reviewed-by: imsnah

It is not currently possibly to show a summary of the job in the output
of “gnt-job list”. The closes is listing the whole opcode(s), but that
is too verbose. Also, the default output (id, status) is not very
useful, unless one looks for (and knows about) an exact job ID.

The patch adds a “summary” description of a job composed of the list of
OP_ID of the individual opcodes. Moreover, if an opcode has a ‘logical’
target in a certain opcode field (e.g. start instance has the instance
name as the target), then it is included in the formatting also. It's
easier to explain via a sample output:

gnt-job list
ID Status  Summary
1  error   NODE_QUERY
2  success NODE_ADD(gnta2)
3  success CLUSTER_QUERY
4  success NODE_REMOVE(gnta2.example.com)
5  error   NODE_QUERY
6  success NODE_ADD(gnta2)
7  success NODE_QUERY
8  success OS_DIAGNOSE
9  success INSTANCE_CREATE(instance1.example.com)
10 success INSTANCE_REMOVE(instance1.example.com)
11 error   INSTANCE_CREATE(instance1.example.com)
12 success INSTANCE_CREATE(instance1.example.com)
13 success INSTANCE_SHUTDOWN(instance1.example.com)
14 success INSTANCE_ACTIVATE_DISKS(instance1.example.com)
15 error   INSTANCE_CREATE(instance2.example.com)
16 error   INSTANCE_CREATE(instance2.example.com)
17 success INSTANCE_CREATE(instance2.example.com)
18 success INSTANCE_ACTIVATE_DISKS(instance1.example.com)
19 success INSTANCE_ACTIVATE_DISKS(instance2.example.com)
20 success INSTANCE_SHUTDOWN(instance1.example.com)
21 success INSTANCE_SHUTDOWN(instance2.example.com)

This is done by a simple change to the opcode classes, which allows an
opcode to format itself. The additional function is small enough that it
can go in opcodes.py, where it could also be used by a client if needed.

Reviewed-by: imsnah

Unless we decide to change the job identifiers to integer, we should at
least sort the list returned by _GetJobIDsUnlocked.

Reviewed-by: imsnah

The bootstrap code needs a pseudo-secret and this is currently generated
inside the InitGanetiServerSetup function. Since more users will need
this, move it to utils.py

Reviewed-by: ultrotter

When the node does not yet have any minors allocated, the first minor
(0) will not be entered in the ConfigWriter._temporary_drbds structure.
This does not happen for our current usage, since we always ask for two
minors (so the next call will not match this case), but it will be
triggered if we only ask for one minor, and then ask again before adding
the instance to the config file.

Reviewed-by: ultrotter

In case the config file is manually modified, or in case of bugs, the
tcp/udp ports could be reused, which will create various problems
(instances not able to start, or drbd disks not able to communicate).

This patch extends the ConfigWriter.VerifyConfig() method (which is used
in cluster verify) to check for duplicates between:
  - the ports used for DRBD disks
  - the ports used for network console
  - the ports marked as free in the config file

Also, if the cluster parameter ‘highest_used_port’ is actually lower
than the computed highest used port, this is also flagged as an error.

The output from gnt-cluster verify will show (output manually wrapped):

node1 # gnt-cluster verify
* Verifying global settings
  - ERROR: tcp/udp port 11006 has duplicates: instance3.example.com/network port,
instance2.example.com/drbd disk sda
  - ERROR: tcp/udp port 11017 has duplicates: instance3.example.com/drbd disk sda,
instance3.example.com/drbd disk sdb, cluster/port marked as free
  - ERROR: Highest used port mismatch, saved 11010, computed 11017
* Gathering data (2 nodes)
...

Reviewed-by: ultrotter

This patch adds update of the cluster serial number for:
  - add/remove node (as the cluster's node list is changed)
  - add/remove/rename instance (as the cluster's instance list is changed)
  - change the volume group name

The rule for updating this attribute is when cluster-wide properties are
changed, but not individual node/instance ones.

There are other remaining cases to handle, pending on the ssconf
changes.

Reviewed-by: ultrotter