Commit 9189c902 authored by Michael Hanselmann's avatar Michael Hanselmann
Browse files

Update query2 design document



While starting to implement this, I found a number of deficiencies:

- Drop regular expressions. As it turned out, only very few fields for
  instances used them, all of which can easily be turned into static
  field names.
- Use two separate calls with a request and response description each.
- Add forgotten list of fields to query request.
- Add value status for case where a field is not available for an item,
  e.g. NIC 3 for an instance with only one network interface.
- Add "timestamp" field type.
- Updated examples.
Signed-off-by: default avatarMichael Hanselmann <hansmi@google.com>
Reviewed-by: default avatarIustin Pop <iustin@google.com>
parent c300f9c9
...@@ -57,26 +57,70 @@ In Python code, the objects described below will be implemented using ...@@ -57,26 +57,70 @@ In Python code, the objects described below will be implemented using
subclasses of ``objects.ConfigObject``, providing existing facilities subclasses of ``objects.ConfigObject``, providing existing facilities
for de-/serializing. for de-/serializing.
.. _query-request: Regular expressions
+++++++++++++++++++
Query request
+++++++++++++ As it turned out, only very few fields for instances used regular
expressions, all of which can easily be turned into static field names.
Each query operation will take a single parameter, a query request Therefore their use in field names is dropped. Reasons:
dictionary with the following properties:
- When regexps are used and a field name is not listed as a simple
``kind`` string in the field dictionary, all keys in the field dictionary have
Denotes request kind. One of the following strings: to be checked whether they're a regular expression object and if so,
matched (see ``utils.FindMatch``).
``query`` - Code becomes simpler. There would be no need anymore to care about
Execute a query on items, optionally filtered (see below). For a regular expressions as field names—they'd all be simple strings, even
description of the result see :ref:`query-result`. if there are many more. The list of field names would be static once
``fields`` built at module-load time.
Return list of supported fields as :ref:`field definitions - There's the issue of formatting titles for the clients. Should it be
<field-def>`. For a complete description of the result see done in the server? In the client? The field definition's title would
:ref:`fields-result`. contain backreferences to the regexp groups in the field name
(``re.MatchObject.expand`` can be used). With just strings, the field
``filter`` definitions can be passed directly to the client. They're static.
- Only a side note: In the memory consumed for 1'000
``_sre.SRE_Pattern`` objects (as returned by ``re.compile`` for an
expression with one group) one can easily store 10'000 strings of the
same length (the regexp objects keep the expression string around, so
compiling the expression always uses more memory).
.. _item-types:
Item types
++++++++++
The proposal is to implement this new interface for the following
items:
``instance``
Instances
``node``
Nodes
``job``
Jobs
``lock``
Locks
.. _data-query:
Data query
++++++++++
.. _data-query-request:
Request
^^^^^^^
The request is a dictionary with the following entries:
``kind`` (string, required)
An :ref:`item type <item-types>`.
``fields`` (list of strings, required)
List of names of fields to return. Example::
["name", "mem", "nic0.ip", "disk0.size", "disk1.size"]
``filter`` (optional)
This will be used to filter queries. In this implementation only names This will be used to filter queries. In this implementation only names
can be filtered to replace the previous ``names`` parameter to can be filtered to replace the previous ``names`` parameter to
queries. An empty filter (``None``) will return all items. To retrieve queries. An empty filter (``None``) will return all items. To retrieve
...@@ -98,33 +142,37 @@ Support for synchronous queries, currently available in the interface ...@@ -98,33 +142,37 @@ Support for synchronous queries, currently available in the interface
but disabled in the master daemon, will be dropped. Direct calls to but disabled in the master daemon, will be dropped. Direct calls to
opcodes have to be used instead. opcodes have to be used instead.
.. _query-result: .. _data-query-response:
New query result format Response
+++++++++++++++++++++++ ^^^^^^^^
The result is a dictionary with the following entries: The result is a dictionary with the following entries:
``fields`` ``fields`` (list of :ref:`field definitions <field-def>`)
In-order list of a :ref:`field definition <field-def>` for each In-order list of a :ref:`field definition <field-def>` for each
requested field, unknown fields are returned with the kind requested field, unknown fields are returned with the kind
``unknown``. Length must be equal to number of requested fields. ``unknown``. Length must be equal to number of requested fields.
``data`` ``data`` (list of lists of tuples)
A list of lists, one list for each item found. Each item's list must List of lists, one list for each item found. Each item's list must
have one entry for each field listed in ``fields``. Each field entry have one entry for each field listed in ``fields`` (meaning their
is a tuple of ``(status, value)``. ``status`` must be one of the length is equal). Each field entry is a tuple of ``(status, value)``.
following values: ``status`` must be one of the following values:
Normal (numeric 0) Normal (numeric 0)
Value is available and matches the kind in the :ref:`field Value is available and matches the kind in the :ref:`field
definition <field-def>`. definition <field-def>`.
Value unavailable (numeric 1) Unknown field (numeric 1)
Field for this column is not known. Value must be ``None``.
No data (numeric 2)
Exact meaning depends on query, e.g. node is unreachable or marked Exact meaning depends on query, e.g. node is unreachable or marked
offline. Value must be ``None``. offline. Value must be ``None``.
Unknown field (numeric 2) Value unavailable for item (numeric 3)
Field for this column is not known. Value must be ``None``. Used if, for example, NIC 3 is requested for an instance with only
one network interface. Value must be ``None``.
Example:: Example response after requesting the fields ``name``, ``mfree``,
``xyz``, ``mtotal``, ``nic0.ip``, ``nic1.ip`` and ``nic2.ip``::
{ {
"fields": [ "fields": [
...@@ -133,24 +181,53 @@ Example:: ...@@ -133,24 +181,53 @@ Example::
# Unknown field # Unknown field
{ "name": "xyz", "title": None, "kind": "unknown", }, { "name": "xyz", "title": None, "kind": "unknown", },
{ "name": "mtotal", "title": "MemTotal", "kind": "unit", }, { "name": "mtotal", "title": "MemTotal", "kind": "unit", },
{ "name": "nic0.ip", "title": "Nic.IP/0", "kind": "text", },
{ "name": "nic1.ip", "title": "Nic.IP/1", "kind": "text", },
{ "name": "nic2.ip", "title": "Nic.IP/2", "kind": "text", },
], ],
"data": [ "data": [
[(0, "node1"), (0, 128), (2, None), (0, 4096)], [(0, "node1"), (0, 128), (1, None), (0, 4096),
# Node not available (0, "192.0.2.1"), (0, "192.0.2.2"), (3, None)],
[(0, "node2"), (1, None), (2, None), (1, None)], [(0, "node2"), (0, 96), (1, None), (0, 5000),
(0, "192.0.2.21"), (0, "192.0.2.39"), (3, "192.0.2.90")],
# Node not available, can't get "mfree" or "mtotal"
[(0, "node3"), (2, None), (1, None), (2, None),
(0, "192.0.2.30"), (3, None), (3, None)],
], ],
} }
.. _fields-result: .. _fields-query:
Fields query
++++++++++++
.. _fields-query-request:
Request
^^^^^^^
The request is a dictionary with the following entries:
``kind`` (string, required)
An :ref:`item type <item-types>`.
``fields`` (list of strings, optional)
List of names of fields to return. If not set, all fields are
returned. Example::
["name", "mem", "nic0.ip", "disk0.size", "disk1.size"]
Field query result format .. _fields-query-response:
+++++++++++++++++++++++++
Response
^^^^^^^^
The result is a dictionary with the following entries: The result is a dictionary with the following entries:
``fields`` ``fields`` (list of :ref:`field definitions <field-def>`)
List of :ref:`field definitions <field-def>` for each available field. List of a :ref:`field definition <field-def>` for each field. If
``fields`` was set in the request and contained an unknown field, it
is returned as type ``unknown``.
Example:: Example::
...@@ -159,6 +236,16 @@ Example:: ...@@ -159,6 +236,16 @@ Example::
{ "name": "name", "title": "Name", "kind": "text", }, { "name": "name", "title": "Name", "kind": "text", },
{ "name": "mfree", "title": "MemFree", "kind": "unit", }, { "name": "mfree", "title": "MemFree", "kind": "unit", },
{ "name": "mtotal", "title": "MemTotal", "kind": "unit", }, { "name": "mtotal", "title": "MemTotal", "kind": "unit", },
{ "name": "nic0.ip", "title": "Nic.IP/0", "kind": "text", },
{ "name": "nic1.ip", "title": "Nic.IP/1", "kind": "text", },
{ "name": "nic2.ip", "title": "Nic.IP/2", "kind": "text", },
{ "name": "nic3.ip", "title": "Nic.IP/3", "kind": "text", },
# …
{ "name": "disk0.size", "title": "Disk.Size/0", "kind": "unit", },
{ "name": "disk1.size", "title": "Disk.Size/1", "kind": "unit", },
{ "name": "disk2.size", "title": "Disk.Size/2", "kind": "unit", },
{ "name": "disk3.size", "title": "Disk.Size/3", "kind": "unit", },
# …
] ]
} }
...@@ -169,19 +256,15 @@ Field definition ...@@ -169,19 +256,15 @@ Field definition
A field definition is a dictionary with the following entries: A field definition is a dictionary with the following entries:
``name`` ``name`` (string)
The field name as a regular expression. The latter is necessary to Field name. Must only contain characters matching ``[a-z0-9/._]``.
represent dynamic fields (e.g. NIC 3 of an instance). ``title`` (string)
``title``
Human-readable title to use in output. Must not contain whitespace. Human-readable title to use in output. Must not contain whitespace.
``kind`` ``kind`` (string)
Field type, one of the following: Field type, one of the following:
.. TODO: Investigate whether there are fields with floating point
.. numbers
``unknown`` ``unknown``
Unknown field (only used for :ref:`data queries <query-request>`) Unknown field
``text`` ``text``
String String
``bool`` ``bool``
...@@ -190,6 +273,8 @@ A field definition is a dictionary with the following entries: ...@@ -190,6 +273,8 @@ A field definition is a dictionary with the following entries:
Numeric Numeric
``unit`` ``unit``
Numeric, in megabytes Numeric, in megabytes
``timestamp``
Unix timestamp in seconds since the epoch
``other`` ``other``
Free-form type, depending on query Free-form type, depending on query
...@@ -197,6 +282,9 @@ A field definition is a dictionary with the following entries: ...@@ -197,6 +282,9 @@ A field definition is a dictionary with the following entries:
formatting any unknown types the same way as "other", which should be formatting any unknown types the same way as "other", which should be
a string representation in most cases. a string representation in most cases.
.. TODO: Investigate whether there are fields with floating point
.. numbers
Example 1 (item name):: Example 1 (item name)::
{ {
...@@ -227,19 +315,33 @@ Old result format ...@@ -227,19 +315,33 @@ Old result format
+++++++++++++++++ +++++++++++++++++
To limit the amount of code necessary, the :ref:`new result format To limit the amount of code necessary, the :ref:`new result format
<query-result>` will be converted for older clients. Unavailable values <data-query-response>` will be converted for clients calling the old
are set to ``None``. If unknown fields were requested, the whole query methods. Unavailable values are set to ``None``. If unknown fields were
fails as the client expects exactly the fields it requested. requested, the whole query fails as the client expects exactly the
fields it requested.
.. _luxi:
LUXI LUXI
++++ ++++
Currently query calls take a number of parameters, e.g. names, fields Currently query calls take a number of parameters, e.g. names, fields
and whether to use locking. These will continue to work and return the and whether to use locking. These will continue to work and return the
:ref:`old result format <old-result-format>`. To use the new query :ref:`old result format <old-result-format>`. Only clients using the
requests, the same calls must be invoked with a single parameter as the new calls described below will be able to make use of new features such
:ref:`query object <query-request>`. Only clients using the new call as filters. Two new calls are introduced:
syntax will be able to make use of new features such as filters.
``Query``
Execute a query on items, optionally filtered. Takes a single
parameter, a :ref:`query object <data-query-request>` encoded as a
dictionary and returns a :ref:`data query response
<data-query-response`.
``QueryFields``
Return list of supported fields as :ref:`field definitions
<field-def>`. Takes a single parameter, a :ref:`fields query object
<fields-query-request>` encoded as a dictionary and returns a
:ref:`fields query response <fields-query-response>`.
Python Python
++++++ ++++++
...@@ -258,6 +360,24 @@ parameter to allow clients to execute the requests described in this ...@@ -258,6 +360,24 @@ parameter to allow clients to execute the requests described in this
proposal directly and to receive the unmodified result. The new formats proposal directly and to receive the unmodified result. The new formats
are a lot more verbose, flexible and extensible. are a lot more verbose, flexible and extensible.
.. _cli-programs:
CLI programs
++++++++++++
Command line programs might have difficulties to display the verbose
status data to the user. There are several options:
- Use colours to indicate missing values
- Display status as value in parentheses, e.g. "(unavailable)"
- Hide unknown columns from the result table and print a warning
- Exit with non-zero code to indicate failures and/or missing data
Some are better for interactive usage, some better for use by other
programs. It is expected that a combination will be used. The column
separator (``--separator=…``) can be used to differentiate between
interactive and programmatic usage.
Other discussed solutions Other discussed solutions
------------------------- -------------------------
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment