Commit af2ae1c0 authored by Iustin Pop's avatar Iustin Pop
Browse files

Merge branch 'devel-2.1'



* devel-2.1:
  utils: Add class to split string stream into lines
  Fix cluster behaviour with disabled file storage
  Update docstrings in tools/ and enable epydoc
  Forward-port the ganeti 2.0 cfgupgrade
  Add a new tool: sanitize-config
  Fix cfgupgrade with non-default DATA_DIR
  Improving the RAPI documentation
  Mark cluster-merge as executable
  QA: Make the rapi credential handling less involving

Conflicts:
	lib/bootstrap.py (merge cds and new file names)
	lib/cmdlib.py    (trivial, kept 2.1 version for disabled file
	                  storage)
	lib/utils.py     (trivial, new imports)
	tools/cfgupgrade (trivial, new cds parameter)
Signed-off-by: default avatarIustin Pop <iustin@google.com>
Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
parents 30198f04 339be5a8
......@@ -236,8 +236,10 @@ dist_tools_SCRIPTS = \
tools/burnin \
tools/cfgshell \
tools/cfgupgrade \
tools/cfgupgrade12 \
tools/cluster-merge \
tools/lvmstrap
tools/lvmstrap \
tools/sanitize-config
pkglib_SCRIPTS = \
daemons/daemon-util
......
......@@ -1304,6 +1304,9 @@ Ganeti versions. Point-releases are usually transparent for the admin.
More information about the upgrade procedure is listed on the wiki at
http://code.google.com/p/ganeti/wiki/UpgradeNotes.
There is also a script designed to upgrade from Ganeti 1.2 to 2.0,
called ``cfgupgrade12``.
cfgshell
++++++++
......@@ -1350,6 +1353,28 @@ systems. Depending on the passed options, it can also test that the
instance OS definitions are executing properly the rename, import and
export operations.
sanitize-config
+++++++++++++++
This tool takes the Ganeti configuration and outputs a "sanitized"
version, by randomizing or clearing:
- DRBD secrets and cluster public key (always)
- host names (optional)
- IPs (optional)
- OS names (optional)
- LV names (optional, only useful for very old clusters which still have
instances whose LVs are based on the instance name)
By default, all optional items are activated except the LV name
randomization. When passing ``--no-randomization``, which disables the
optional items (i.e. just the DRBD secrets and cluster public keys are
randomized), the resulting file can be used as a safety copy of the
cluster config - while not trivial, the layout of the cluster can be
recreated from it and if the instance disks have not been lost it
permits recovery from the loss of all master candidates.
Other Ganeti projects
---------------------
......
......@@ -71,6 +71,33 @@ HTTP Basic authentication as per RFC2617_ is supported.
.. _REST: http://en.wikipedia.org/wiki/Representational_State_Transfer
.. _RFC2617: http://tools.ietf.org/rfc/rfc2617.txt
PUT or POST?
------------
According to RFC2616 the main difference between PUT and POST is that
POST can create new resources but PUT can only create the resource the
URI was pointing to on the PUT request.
Unfortunately, due to historic reasons, the Ganeti RAPI library is not
consistent with this usage, so just use the methods as documented below
for each resource.
For more details have a look in the source code at
``lib/rapi/rlib2.py``.
Generic parameter types
-----------------------
A few generic refered parameter types and the values they allow.
``bool``
++++++++
A boolean option will accept ``1`` or ``0`` as numbers but not
i.e. ``True`` or ``False``.
Generic parameters
------------------
......@@ -89,9 +116,9 @@ themselves.
``dry-run``
+++++++++++
The optional *dry-run* argument, if provided and set to a positive
integer value (e.g. ``?dry-run=1``), signals to Ganeti that the job
should not be executed, only the pre-execution checks will be done.
The boolean *dry-run* argument, if provided and set, signals to Ganeti
that the job should not be executed, only the pre-execution checks will
be done.
This is useful in trying to determine (without guarantees though, as in
the meantime the cluster state could have changed) if the operation is
......@@ -278,9 +305,9 @@ Example::
}
]
If the optional *bulk* argument is provided and set to a true value (i.e
``?bulk=1``), the output contains detailed information about instances
as a list.
If the optional bool *bulk* argument is provided and set to a true value
(i.e ``?bulk=1``), the output contains detailed information about
instances as a list.
Example::
......@@ -317,11 +344,10 @@ Example::
Creates an instance.
If the optional *dry-run* argument is provided and set to a positive
integer valu (e.g. ``?dry-run=1``), the job will not be actually
executed, only the pre-execution checks will be done. Query-ing the job
result will return, in both dry-run and normal case, the list of nodes
selected for the instance.
If the optional bool *dry-run* argument is provided, the job will not be
actually executed, only the pre-execution checks will be done. Query-ing
the job result will return, in both dry-run and normal case, the list of
nodes selected for the instance.
Returns: a job ID that can be used later for polling.
......@@ -372,8 +398,18 @@ It supports the following commands: ``POST``.
Reboots the instance.
The URI takes optional ``type=hard|soft|full`` and
``ignore_secondaries=False|True`` parameters.
The URI takes optional ``type=soft|hard|full`` and
``ignore_secondaries=0|1`` parameters.
``type`` defines the reboot type. ``soft`` is just a normal reboot,
without terminating the hypervisor. ``hard`` means full shutdown
(including terminating the hypervisor process) and startup again.
``full`` is like ``hard`` but also recreates the configuration from
ground up as if you would have done a ``gnt-instance shutdown`` and
``gnt-instance start`` on it.
``ignore_secondaries`` is a bool argument indicating if we start the
instance even if secondary disks are failing.
It supports the ``dry-run`` argument.
......@@ -405,8 +441,8 @@ It supports the following commands: ``PUT``.
Startup an instance.
The URI takes an optional ``force=False|True`` parameter to start the
instance if even if secondary disks are failing.
The URI takes an optional ``force=1|0`` parameter to start the
instance even if secondary disks are failing.
It supports the ``dry-run`` argument.
......@@ -438,6 +474,12 @@ Takes the parameters ``mode`` (one of ``replace_on_primary``,
``replace_auto``), ``disks`` (comma separated list of disk indexes),
``remote_node`` and ``iallocator``.
Either ``remote_node`` or ``iallocator`` needs to be defined when using
``mode=replace_new_secondary``.
``mode`` is a mandatory parameter. ``replace_auto`` tries to determine
the broken disk(s) on its own and replacing it.
``/2/instances/[instance_name]/activate-disks``
+++++++++++++++++++++++++++++++++++++++++++++++
......@@ -449,7 +491,7 @@ It supports the following commands: ``PUT``.
``PUT``
~~~~~~~
Takes the parameter ``ignore_size``. When set ignore the recorded
Takes the bool parameter ``ignore_size``. When set ignore the recorded
size (useful for forcing activation when recorded size is wrong).
......@@ -674,7 +716,7 @@ It supports the following commands: ``POST``.
~~~~~~~~
To evacuate a node, either one of the ``iallocator`` or ``remote_node``
parameters must be passed:
parameters must be passed::
evacuate?iallocator=[iallocator]
evacuate?remote_node=[nodeX.example.com]
......@@ -689,7 +731,8 @@ It supports the following commands: ``POST``.
``POST``
~~~~~~~~
No parameters are required, but ``live`` can be set to a boolean value.
No parameters are required, but the bool parameter ``live`` can be set
to use live migration (if available).
migrate?live=[0|1]
......@@ -725,7 +768,7 @@ Change the node role.
The request is a string which should be PUT to this URI. The result will
be a job id.
It supports the ``force`` argument.
It supports the bool ``force`` argument.
``/2/nodes/[node_name]/storage``
++++++++++++++++++++++++++++++++
......
......@@ -8,7 +8,7 @@ output: html
# note: the wildcards means the directories should be cleaned up after each
# run, otherwise there will be stale '*c' (compiled) files that will not be
# parsable and will break the epydoc run
modules: ganeti, scripts/gnt-*, daemons/ganeti-confd, daemons/ganeti-masterd, daemons/ganeti-noded, daemons/ganeti-rapi, daemons/ganeti-watcher
modules: ganeti, scripts/gnt-*, daemons/ganeti-confd, daemons/ganeti-masterd, daemons/ganeti-noded, daemons/ganeti-rapi, daemons/ganeti-watcher, tools/*
graph: all
......
......@@ -77,7 +77,10 @@ def GenerateHmacKey(file_name):
def GenerateClusterCrypto(new_cluster_cert, new_rapi_cert, new_confd_hmac_key,
new_cds, rapi_cert_pem=None, cds=None):
new_cds, rapi_cert_pem=None, cds=None,
nodecert_file=constants.NODED_CERT_FILE,
rapicert_file=constants.RAPI_CERT_FILE,
hmackey_file=constants.CONFD_HMAC_KEY):
"""Updates the cluster certificates, keys and secrets.
@type new_cluster_cert: bool
......@@ -92,39 +95,42 @@ def GenerateClusterCrypto(new_cluster_cert, new_rapi_cert, new_confd_hmac_key,
@param rapi_cert_pem: New RAPI certificate in PEM format
@type cds: string
@param cds: New cluster domain secret
@type nodecert_file: string
@param nodecert_file: optional override of the node cert file path
@type rapicert_file: string
@param rapicert_file: optional override of the rapi cert file path
@type hmackey_file: string
@param hmackey_file: optional override of the hmac key file path
"""
# noded SSL certificate
cluster_cert_exists = os.path.exists(constants.NODED_CERT_FILE)
cluster_cert_exists = os.path.exists(nodecert_file)
if new_cluster_cert or not cluster_cert_exists:
if cluster_cert_exists:
utils.CreateBackup(constants.NODED_CERT_FILE)
utils.CreateBackup(nodecert_file)
logging.debug("Generating new cluster certificate at %s",
constants.NODED_CERT_FILE)
utils.GenerateSelfSignedSslCert(constants.NODED_CERT_FILE)
logging.debug("Generating new cluster certificate at %s", nodecert_file)
utils.GenerateSelfSignedSslCert(nodecert_file)
# confd HMAC key
if new_confd_hmac_key or not os.path.exists(constants.CONFD_HMAC_KEY):
logging.debug("Writing new confd HMAC key to %s", constants.CONFD_HMAC_KEY)
GenerateHmacKey(constants.CONFD_HMAC_KEY)
if new_confd_hmac_key or not os.path.exists(hmackey_file):
logging.debug("Writing new confd HMAC key to %s", hmackey_file)
GenerateHmacKey(hmackey_file)
# RAPI
rapi_cert_exists = os.path.exists(constants.RAPI_CERT_FILE)
rapi_cert_exists = os.path.exists(rapicert_file)
if rapi_cert_pem:
# Assume rapi_pem contains a valid PEM-formatted certificate and key
logging.debug("Writing RAPI certificate at %s",
constants.RAPI_CERT_FILE)
utils.WriteFile(constants.RAPI_CERT_FILE, data=rapi_cert_pem, backup=True)
logging.debug("Writing RAPI certificate at %s", rapicert_file)
utils.WriteFile(rapicert_file, data=rapi_cert_pem, backup=True)
elif new_rapi_cert or not rapi_cert_exists:
if rapi_cert_exists:
utils.CreateBackup(constants.RAPI_CERT_FILE)
utils.CreateBackup(rapicert_file)
logging.debug("Generating new RAPI certificate at %s",
constants.RAPI_CERT_FILE)
utils.GenerateSelfSignedSslCert(constants.RAPI_CERT_FILE)
logging.debug("Generating new RAPI certificate at %s", rapicert_file)
utils.GenerateSelfSignedSslCert(rapicert_file)
# Cluster domain secret
if cds:
......@@ -174,6 +180,38 @@ def _WaitForNodeDaemon(node_name):
" 10 seconds" % node_name)
def _InitFileStorage(file_storage_dir):
"""Initialize if needed the file storage.
@param file_storage_dir: the user-supplied value
@return: either empty string (if file storage was disabled at build
time) or the normalized path to the storage directory
"""
if not constants.ENABLE_FILE_STORAGE:
return ""
file_storage_dir = os.path.normpath(file_storage_dir)
if not os.path.isabs(file_storage_dir):
raise errors.OpPrereqError("The file storage directory you passed is"
" not an absolute path.", errors.ECODE_INVAL)
if not os.path.exists(file_storage_dir):
try:
os.makedirs(file_storage_dir, 0750)
except OSError, err:
raise errors.OpPrereqError("Cannot create file storage directory"
" '%s': %s" % (file_storage_dir, err),
errors.ECODE_ENVIRON)
if not os.path.isdir(file_storage_dir):
raise errors.OpPrereqError("The file storage directory '%s' is not"
" a directory." % file_storage_dir,
errors.ECODE_ENVIRON)
return file_storage_dir
def InitCluster(cluster_name, mac_prefix,
master_netdev, file_storage_dir, candidate_pool_size,
secondary_ip=None, vg_name=None, beparams=None,
......@@ -242,24 +280,7 @@ def InitCluster(cluster_name, mac_prefix,
" you are not using lvm" % vgstatus,
errors.ECODE_INVAL)
file_storage_dir = os.path.normpath(file_storage_dir)
if not os.path.isabs(file_storage_dir):
raise errors.OpPrereqError("The file storage directory you passed is"
" not an absolute path.", errors.ECODE_INVAL)
if not os.path.exists(file_storage_dir):
try:
os.makedirs(file_storage_dir, 0750)
except OSError, err:
raise errors.OpPrereqError("Cannot create file storage directory"
" '%s': %s" % (file_storage_dir, err),
errors.ECODE_ENVIRON)
if not os.path.isdir(file_storage_dir):
raise errors.OpPrereqError("The file storage directory '%s' is not"
" a directory." % file_storage_dir,
errors.ECODE_ENVIRON)
file_storage_dir = _InitFileStorage(file_storage_dir)
if not re.match("^[0-9a-z]{2}:[0-9a-z]{2}:[0-9a-z]{2}$", mac_prefix):
raise errors.OpPrereqError("Invalid mac prefix given '%s'" % mac_prefix,
......
......@@ -560,6 +560,17 @@ def _CheckNodeHasOS(lu, node, os_name, force_variant):
_CheckOSVariant(result.payload, os_name)
def _RequireFileStorage():
"""Checks that file storage is enabled.
@raise errors.OpPrereqError: when file storage is disabled
"""
if not constants.ENABLE_FILE_STORAGE:
raise errors.OpPrereqError("File storage disabled at configure time",
errors.ECODE_INVAL)
def _CheckDiskTemplate(template):
"""Ensure a given disk template is valid.
......@@ -568,9 +579,20 @@ def _CheckDiskTemplate(template):
msg = ("Invalid disk template name '%s', valid templates are: %s" %
(template, utils.CommaJoin(constants.DISK_TEMPLATES)))
raise errors.OpPrereqError(msg, errors.ECODE_INVAL)
if template == constants.DT_FILE and not constants.ENABLE_FILE_STORAGE:
raise errors.OpPrereqError("File storage disabled at configure time",
if template == constants.DT_FILE:
_RequireFileStorage()
def _CheckStorageType(storage_type):
"""Ensure a given storage type is valid.
"""
if storage_type not in constants.VALID_STORAGE_TYPES:
raise errors.OpPrereqError("Unknown storage type: %s" % storage_type,
errors.ECODE_INVAL)
if storage_type == constants.ST_FILE:
_RequireFileStorage()
def _CheckInstanceDown(lu, instance, reason):
......@@ -3079,17 +3101,14 @@ class LUQueryNodeStorage(NoHooksLU):
REQ_BGL = False
_FIELDS_STATIC = utils.FieldSet(constants.SF_NODE)
def ExpandNames(self):
storage_type = self.op.storage_type
if storage_type not in constants.VALID_STORAGE_TYPES:
raise errors.OpPrereqError("Unknown storage type: %s" % storage_type,
errors.ECODE_INVAL)
def CheckArguments(self):
_CheckStorageType(self.op.storage_type)
_CheckOutputFields(static=self._FIELDS_STATIC,
dynamic=utils.FieldSet(*constants.VALID_STORAGE_FIELDS),
selected=self.op.output_fields)
def ExpandNames(self):
self.needed_locks = {}
self.share_locks[locking.LEVEL_NODE] = 1
......@@ -3178,10 +3197,7 @@ class LUModifyNodeStorage(NoHooksLU):
def CheckArguments(self):
self.opnode_name = _ExpandNodeName(self.cfg, self.op.node_name)
storage_type = self.op.storage_type
if storage_type not in constants.VALID_STORAGE_TYPES:
raise errors.OpPrereqError("Unknown storage type: %s" % storage_type,
errors.ECODE_INVAL)
_CheckStorageType(self.op.storage_type)
def ExpandNames(self):
self.needed_locks = {
......@@ -5819,6 +5835,8 @@ def _GenerateDiskTemplate(lu, template_name,
if len(secondary_nodes) != 0:
raise errors.ProgrammerError("Wrong template configuration")
_RequireFileStorage()
for idx, disk in enumerate(disk_info):
disk_index = idx + base_index
disk_dev = objects.Disk(dev_type=constants.LD_FILE, size=disk["size"],
......@@ -6640,15 +6658,18 @@ class LUCreateInstance(LogicalUnit):
else:
network_port = None
# this is needed because os.path.join does not accept None arguments
if self.op.file_storage_dir is None:
string_file_storage_dir = ""
else:
string_file_storage_dir = self.op.file_storage_dir
if constants.ENABLE_FILE_STORAGE:
# this is needed because os.path.join does not accept None arguments
if self.op.file_storage_dir is None:
string_file_storage_dir = ""
else:
string_file_storage_dir = self.op.file_storage_dir
# build the full file storage dir path
file_storage_dir = utils.PathJoin(self.cfg.GetFileStorageDir(),
string_file_storage_dir, instance)
# build the full file storage dir path
file_storage_dir = utils.PathJoin(self.cfg.GetFileStorageDir(),
string_file_storage_dir, instance)
else:
file_storage_dir = ""
disks = _GenerateDiskTemplate(self,
self.op.disk_template,
......@@ -7667,6 +7688,8 @@ class LURepairNodeStorage(NoHooksLU):
def CheckArguments(self):
self.op.node_name = _ExpandNodeName(self.cfg, self.op.node_name)
_CheckStorageType(self.op.storage_type)
def ExpandNames(self):
self.needed_locks = {
locking.LEVEL_NODE: [self.op.node_name],
......
......@@ -21,6 +21,20 @@
"""Remote API version 2 baserlib.library.
PUT or POST?
------------
According to RFC2616 the main difference between PUT and POST is that
POST can create new resources but PUT can only create the resource the
URI was pointing to on the PUT request.
To be in context of this module for instance creation POST on
/2/instances is legitim while PUT would be not, due to it does create a
new entity and not just replace /2/instances with it.
So when adding new methods, if they are operating on the URI entity itself,
PUT should be prefered over POST.
"""
# pylint: disable-msg=C0103
......
......@@ -48,6 +48,7 @@ import OpenSSL
import datetime
import calendar
import hmac
import collections
from cStringIO import StringIO
......@@ -3141,6 +3142,47 @@ class FileLock(object):
"Failed to unlock %s" % self.filename)
class LineSplitter:
"""Splits data chunks into lines separated by newline.
Instances provide a file-like interface.
"""
def __init__(self, line_fn, *args):
"""Initializes this class.
@type line_fn: callable
@param line_fn: Function called for each line, first parameter is line
@param args: Extra arguments for L{line_fn}
"""
assert callable(line_fn)
if args:
# Python 2.4 doesn't have functools.partial yet
self._line_fn = \
lambda line: line_fn(line, *args) # pylint: disable-msg=W0142
else:
self._line_fn = line_fn
self._lines = collections.deque()
self._buffer = ""
def write(self, data):
parts = (self._buffer + data).split("\n")
self._buffer = parts.pop()
self._lines.extend(parts)
def flush(self):
while self._lines:
self._line_fn(self._lines.popleft().rstrip("\r\n"))
def close(self):
self.flush()
if self._buffer:
self._line_fn(self._buffer)
def SignalHandled(signums):
"""Signal Handled decoration.
......
......@@ -38,6 +38,8 @@ import qa_rapi
import qa_tags
import qa_utils
from ganeti import utils
def RunTest(fn, *args):
"""Runs a test after printing a header.
......@@ -70,12 +72,15 @@ def RunEnvTests():
RunTest(qa_env.TestGanetiCommands)
def SetupCluster():
def SetupCluster(rapi_user, rapi_secret):
"""Initializes the cluster.
@param rapi_user: Login user for RAPI
@param rapi_secret: Login secret for RAPI
"""
if qa_config.TestEnabled('create-cluster'):
RunTest(qa_cluster.TestClusterInit)
RunTest(qa_cluster.TestClusterInit, rapi_user, rapi_secret)
RunTest(qa_node.TestNodeAddAll)
else:
# consider the nodes are already there
......@@ -274,8 +279,12 @@ def main():
qa_config.Load(config_file)
rapi_user = "ganeti-qa"
rapi_secret = utils.GenerateSecret()
qa_rapi.OpenerFactory.SetCredentials(rapi_user, rapi_secret)
RunEnvTests()
SetupCluster()
SetupCluster(rapi_user, rapi_secret)
RunClusterTests()
RunOsTests()
......
......@@ -55,10 +55,17 @@ def _CheckFileOnAllNodes(filename, content):
content)
def TestClusterInit():
def TestClusterInit(rapi_user, rapi_secret):
"""gnt-cluster init"""
master = qa_config.GetMasterNode()
# First create the RAPI credentials
cred_string = "%s %s write" % (rapi_user, rapi_secret)
cmd = ("echo %s > %s" %
(utils.ShellQuote(cred_string),
utils.ShellQuote(constants.RAPI_USERS_FILE)))
AssertEqual(StartSSH(master['primary'], cmd).wait(), 0)
cmd = ['gnt-cluster', 'init']
if master.get('secondary', None):
......@@ -78,21 +85,6 @@ def TestClusterInit():
AssertEqual(StartSSH(master['primary'],
utils.ShellQuoteArgs(cmd)).wait(), 0)
# Create RAPI credentials
rapi_user = qa_config.get("rapi-user", default=None)
rapi_pass = qa_config.get("rapi-pass", default=None)
if rapi_user and rapi_pass:
cmds = []
cred_string = "%s %s write" % (rapi_user, rapi_pass)
cmds.append(("echo %s >> %s" %
(utils.ShellQuote(cred_string),
utils.ShellQuote(constants.RAPI_USERS_FILE))))
cmds.append("%s stop-master" % constants.DAEMON_UTIL)
cmds.append("%s start-master" % constants.DAEMON_UTIL)
AssertEqual(StartSSH(master['primary'], ' && '.join(cmds)).wait(), 0)
def TestClusterRename():
"""gnt-cluster rename"""
......
......@@ -44,6 +44,16 @@ class OpenerFactory:
"""
_opener = None
_rapi_user = None
_rapi_secret = None
@classmethod
def SetCredentials(cls, rapi_user, rapi_secret):
"""Set the credentials for authorized access.