This commit expands the monitoring agent design document improving the
description of the general report file format.

There can be two kinds of data collectors (performance and status collectors).

Every collector can now be part of a category of collectors (such as
storage, instance, daemon, hypervisor, etc.) that defines a minimum basic
structure that has to be common to all the collectors part of the same class.

Also, such basic structure is defined for all the currently envisioned class
of collectors.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is a long standing bug in Ganeti. Add a small design on how we plan
to fix this for Ganeti 2.7.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

* devel-2.7:
  Rename lib/objectutils to outils.py
  Fix typo in gnt-group manpage
  Fix wrong type in a docstring of the RAPI subsystem
  Finish the remote→restricted commands rename
  Enable use of the priority option in hbal
  Add CLI-level option to override the priority
  Add functions to parse CLI-level format of priorities
  Add a function to change an OpCode's priority
  Make hbal opcode annotation more generic
  Add unit tests for RADOSBLockDevice
  Fix rbd showmapped output parsing
  Change default xen root path to /dev/xvda1
  Removes check for conflicts from NetworkDisconnect
  If _UnlockedLookupNetwork() fails raise error
  Force conflicts check in LUNetworkDisconnect

Also updated objects.py with more outils renames.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This reverts commit fb251c2c. On the
master branch we want to continue to have them enabled.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Back when this was introduced, I mentioned that it breaks heavily tab
completion (ob<TAB> doesn't work anymore), but at that moment I didn't
have a suggestion what to name it. I think outils is good and short
enough, and doesn't conflict with anything else, so here it goes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Hroller in 2.7 was implemented (as an experiment) but never explicitly
designed. This design mentions the current functionality, and explains
the changes that will happen for the next release to support live
maintenance scheduling (rolling reboot).

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The --ipol-disk-templates parameter should actually be
--ipolicy-disk-templates.

Signed-off-by: Ben Lipton <ben.lipton@gmail.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Use a method on the instance object instead of going through
“qa_config.GetInstanceTemplate”. This provides for better encapsulation.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Call the wrapped instance method directly.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Disk template names shouldn't be hardcoded.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Lines removed in devel-2.7 (250a9404) weren't removed in the merge with
master.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

The documentation still points to /etc/ganeti/remote-commands,
although the code is already using restricted-command. Update the
documentation and a few docstrings accordingly.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds the option to hbal, and uses it to tweak the submitted
jobs. There are also two small shelltests for testing the parsing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This just defined the new priority, with the same name as the Python one.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The current serialisation format for submit priorities is
integer-based, same as the opcode json serialisation. But for CLI
level, we need to support a string-based format, so we add functions
to parse and format this representation.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This simply updates the metaopcode submit priority.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, hbal code always uses annotateOpCode function, which means
we would have to pass the options data to all function in the call
chain if we wanted to make this more flexible.

By abstracting the type of the annotator and passing it as an argument
to the function, we can be more flexible.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add unit tests for the RADOSBlockDevice JSON output parsing function,
and modify the unit tests for the plain output parsing function to
work with the new code.

Signed-off-by: Stratos Psomadakis <psomas@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

'rbd showmapped' output formatting differs between older and newer versions of
the ceph tools. Try to use json output formatting if available (currently
available only in the ceph master branch). For bobtail, argonaut and older
releases fallback to manually parsing the 'rbd showmapped' output, handling the
differences in the output format for each rbd version correctly.

Signed-off-by: Stratos Psomadakis <psomas@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

All recent-enough versions of linux see the xen paravirtual device as
/dev/xvd*.

This doesn't break old installations, as the default is only used on new
clusters.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

This removes the check for conflicts from the Haskell
version of the OpCode NetworkDisconnect. This alignes
the Haskell code with the patch
"Force conflicts check in LUNetworkDisconnect" (which
is currently under review). I will submit these patches
together then.

This is a cherry-pick of commit 0ae4b355.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Make _UnlockedLookupNetwork() raise OpPrereqError (instead of returning
None) in case it does not find the requested network. Remove useless and
duplicate code such as:

if net_uuid is None:
  raise...

This is a cherry-pick of commit 1cce2c47.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

Until now if one disconnects a network with --no-conflicts-check
and then remove it, there is a possibility to leave instances with NICs
referencing non-existing networks. This causes network queries,
instance removal and modification to fail.

This patch allows a network to be disconnected from a nodegroup,
only if instances residing in the nodegroup do not have NICs attached
to the network. Otherwise OpPrereqError is raised. The
--no-conflicts-check option is removed from the gnt-network disconnect
command as well.

This is a cherry-pick of commit 6bb939a2.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

* devel-2.7: (23 commits)
  QA: Support additional arguments for initialization
  qa_utils: Fix order of arguments passed to _AssertRetCode
  Improve reporting on errors.AddressPoolError exceptions
  Add note about lv-tags rename
  Make use of HooksDict() for networks
  Remove family and size from network objects
  Remove network_type slot (Issue 363)
  Moved uniformity check for exclusive_storage flag
  "exclusive_storage" cannot be changed on single nodes
  Upgrades made on loading the configuration are always saved
  Show correct daemon name on Luxi connect errors
  Update the security document for Ganeti 2.7
  OS environment: add network information
  ConfigData: run UpgradeConfig on network objects
  Make ParticalNic's network field of type String
  Make gnt-os list work with no OSes
  Fix OCF files installation in devel/upload
  baserlib: Fix two mistakes in docstring
  Workaround hlint behaviour with no warnings/errors
  Remove use of 'head' and add hlint warning for it
  ...

Conflicts:
	qa/qa_cluster.py: Trivial
	qa/qa_node.py: Node attributes
	src/Ganeti/Types.hs: Network cleanup
	test/hs/Test/Ganeti/Objects.hs: Network cleanup

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

In some scenarios it can be useful to add more arguments to “gnt-cluster
init”, such as “--no-drbd-storage”, if DRBD is not installed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch improves the error messages given when a
“errors.AddressPoolError” exception is caught. Includes some small style
fixes.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is not included on the 2.6 branch, so update the NEWS for 2.7.0 beta1.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This can be used in hypervisor code as well. For consistency
export *NETWORK_NAME and not *NETWORK throughout the code.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This info is not used by Ganeti and therefore is removed.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This slot was not used by Ganeti so the same info can be
provided via tags. In order not to break configuration data
we add a FromDict() method in Network config object that
removes the deprecated network_type (if found) and then invoke
the parent's method (which does the validation).

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

These are no longer used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

This is no longer needed now that instance/node objects in QA have
attributes. Use a helper function instead to retrieve an entity's
identifying attribute.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

The new objects support attributes. Item access (like a dictionary) is
going away in a couple of patches.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

The new objects support attributes, which are cleaner than
dictionary-style access. Item access (like a dictionary) is going away
in a couple of patches. Also, pylint is better at checking attributes
than dictionary entries.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Instead of having a module-level function in “qa_config” to release a
node, a new method on node objects is used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Cluster-verify used to check that the value of exclusive_storage is uniform
within node groups. Now, it's impossible to change the flag for a single
node, so that check has been removed and an equivalent one has been added
in ConfigWriter.VerifyConfig(). Unit test provided. The unit test covers
only the new check, not the whole VerifyConfig method.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

There's never been support for a configuration where nodes in the same node
group have different values of the exclusive_storage flag. This patch
disables the possibility to change the flag for individual nodes.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>