Skip to content

GitLab

Commit 4a469d38 authored by Stavros Sachtouris's avatar Stavros Sachtouris
Browse files

Merge branch 'feature-image-meta-record' into develop

parents 90099c14 c4aefeaf
Expand all Hide whitespace changes
Inline Side-by-side
Changelog
View file @ 4a469d38
......@@ -6,6 +6,7 @@ Bug Fixes:
- Shell can manage all valid command line arguments [#3716]
- Restore 2nd level command syntax in shell [#3736]
- Allow copy of deleted objects by refering to older version [#3737]
- Add image.add_member missing content-length header
Changes:
......@@ -18,6 +19,24 @@ Changes:
- Disallow moving deleted objects by version [#3737]
This operation was implemented by accident, due to the symetry between
move and copy
- Rename file-meta commands to file-metadata
- Rename image-[add|del]member commands to members-[add|delete]
- Remove update option from imagre-register
- In image-compute split properties to properties-list and properties-get
- Add optional output to methods[#3756, #3732]:
- file:
mkdir, touch, create, move, copy, move, append, truncate, overwrite,
manifest, upload, delete, purge, unpublish, permissions set/delete, info,
metadata set/delete, containerlimit set, versioning set, group set/delete,
upload, overwrite
- image:
unregister, members add/delete/set
-image compute:
delete, properties delete
- Transliterate methods to list-get-set-delete command groups:
- file: permissions, versioning, group and metadata
- image: members, member
- image compute: properties
Features:
......@@ -34,4 +53,8 @@ Features:
- Add enumeration to all listing commands, make it optional [#3739]
- Add a download_to_string method in pithos client [#3608]
- Add an upload_from_string method in pithos client [#3608]
- Add pithos client method create_container [#3756]
- Store image properties on remote location after image registration [#3769]
- Add runtime args to image register for forcing or unsettitng property
storage [#3769]
docs/commands.rst
View file @ 4a469d38
......@@ -64,22 +64,26 @@ image (Plankton commands + Compute Image subcommands)
.. code-block:: text
addmember : Add a member to an image
addproperty: Add an image property
delmember : Remove a member from an image
list : List images accessible by user
members : Get image members
meta : Get image metadata
register : (Re)Register an image
setmembers : Set the members of an image
unregister : Unregister an image (does not delete the image file)
shared : List shared images
compute : Compute Image API commands
list : List images
delete : Delete image
info : Get image details
properties : Get image properties
delproperty: Delete an image property
setproperty: Update an image property
properties : Manage properties related to OS installation in an image
add : Add a property to an image
delete: Delete a property from an image
get : Get an image property
list : List all image properties
set : Add / update a set of properties for an image
members : Manage members (users who can modify an image)
add : Add a member to an image
delete : Remove a member from an image
list : List members of an image
set : Set the members of an image
Showcase: Pick an image and list the properties
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
......@@ -311,38 +315,42 @@ file (Storage/Pithos+)
append : Append local file to remote
cat : Print a file to console
copy : Copy an object
containerlimit: Container size limit commands
set : Set container data limit
get : Get container data limit
create : Create a container
delete : Delete a container [or an object]
delgroup : Delete a user group
delmeta : Delete an existing metadatum for an account [, container [or object]]
delpermissions: Delete all sharing permissions
download : Download a file or directory
group : Get user groups details
group : Manage access groups and group members
delete: Delete a user group
get : Get groups and group members
set : Set a user group
hashmap : Get the hashmap of an object
info : Get information for account [, container [or object]]
list : List containers, object trees or objects in a directory
manifest : Create a remote file with uploaded parts by manifestation
meta : Get custom meta-content for account [, container [or object]]
metadata : Metadata are attached on objects (key:value pairs)
delete: Delete metadata with given key
get : Get metadatum
set : Set a piece of metadata
mkdir : Create a directory
move : Copy an object
overwrite : Overwrite part (from start to end) of a remote file
permissions : Get object read/write permissions
permissions : Manage user and group accessibility for objects
delete: Delete all permissions set on object
get : Get read and write permissions of an object
set : Set permissions for an object
publish : Publish an object
purge : Purge a container
quota : Get quota for account
setgroup : Create/update a new user group
setmeta : Set a new metadatum for account [, container [or object]]
setpermissions: Set sharing permissions
containerlimit: Container size limit commands
set : Set container data limit
get : Get container data limit
setversioning : Set new versioning (auto, none) for account [or container]
sharers : List the accounts that share objects with default account
touch : Create an empty object (file)
truncate : Truncate remote file up to a size
unpublish : Unpublish an object
upload : Upload a file or directory
versioning : Get versioning for account [or container ]
versioning : Manage the versioning scheme of current pithos user account
get: Get versioning for account or container
set: Set versioning mode (auto, none) for account or container
versions : Get the version list of an object
Showcase: Upload and download a file
......
docs/man/kamaki.rst
View file @ 4a469d38
......@@ -140,27 +140,26 @@ flavor commands
image commands
**************
* addmember Add a member to an image
* addproperty Add an OS-related property to an image
* delete Delete an image (image file remains intact)
* delmember Remove a member from an image
* delproperty Delete a property of an image
* info Get detailed information on an image
* members Get image members
* meta Get image metadata
* properties Get properties related to OS installation in an image
* list List images accessible by user
* register (Re)Register an image
* setmembers Set the members of an image
* setproperty Update an existing property in an image
* shared List images shared by a member
* compute Compute Image API commands
* list List images
* delete Delete image
* info Get image details
* properties Get image properties
* delproperty Delete an image property
* setproperty Update an image property
* list List images accessible by user
* meta Get image metadata
* register (Re)Register an image
* unregister Unregister an image (does not delete the image file)
* shared List shared images
* compute Compute Image API commands
* list List images
* delete Delete image
* info Get image details
* properties Manage properties related to OS installation in an image
* add Add a property to an image
* delete Delete a property from an image
* get Get an image property
* list List all image properties
* set Add / update a set of properties for an image
* members Manage members (users who can modify an image)
* add Add a member to an image
* delete Remove a member from an image
* list List members of an image
* set Set the members of an image
network commands
......@@ -178,41 +177,46 @@ network commands
file commands
**************
* append Append local file to (existing) remote object
* cat Print remote file contents to console
* copy Copy an object from container to (another) container
* create Create a container
* delete Delete a container [or an object]
* delgroup Delete a user group
* delmeta Delete metadata from account, container or object
* delpermissions Delete all permissions set on object
* download Download remote object(s) as local file(s)
* group Get groups and group members
* hashmap Get the hash-map of an object
* info Get detailed info for account, containers or objects
* list List containers, object trees or objects in a directory
* manifest Create a remote file of uploaded parts by manifestation
* meta Get metadata for account, containers or objects
* mkdir Create a directory
* move Copy an object
* overwrite Overwrite part (from start to end) of a remote file
* permissions Get read and write permissions of an object
* publish Publish the object and print the public url
* purge Delete a container and release related data blocks
* quota Get quota (in KB) for account or container
* setgroup Set a user group
* setmeta Set a piece of metadata for account, container or object
* setpermissions Set permissions for an object
* containerlimit set Set new limit for container
* containerlimit get Get container limit
* setversioning Set versioning mode (auto, none) for account or container
* sharers List the accounts that share objects with current user
* touch Create an empty object (file)
* truncate Truncate remote file up to a size
* unpublish Unpublish an object
* upload Upload a file or directory
* versioning Get versioning for account or container
* versions Get the list of object versions
* append Append local file to remote
* cat Print a file to console
* copy Copy an object
* containerlimit Container size limit commands
* set Set container data limit
* get Get container data limit
* create Create a container
* delete Delete a container [or an object]
* download Download a file or directory
* group Manage access groups and group members
* delete Delete a user group
* get Get groups and group members
* set Set a user group
* hashmap Get the hashmap of an object
* info Get information for account [, container [or object]]
* list List containers, object trees or objects in a directory
* manifest Create a remote file with uploaded parts by manifestation
* metadata Metadata are attached on objects (key:value pairs)
* delete Delete metadata with given key
* get Get metadatum
* set Set a piece of metadata
* mkdir Create a directory
* move Copy an object
* overwrite Overwrite part (from start to end) of a remote file
* permissions Manage user and group accessibility for objects
* delete Delete all permissions set on object
* get Get read and write permissions of an object
* set Set permissions for an object
* publish Publish an object
* purge Purge a container
* quota Get quota for account
* sharers List the accounts that share objects with default account
* touch Create an empty object (file)
* truncate Truncate remote file up to a size
* unpublish Unpublish an object
* upload Upload a file or directory
* versioning Manage the versioning scheme of current pithos user account
* get Get versioning for account or container
* set Set versioning mode (auto, none) for account or container
* versions Get the version list of an object
test commands (hidden)
......
kamaki/cli/commands/__init__.py
View file @ 4a469d38
......@@ -32,6 +32,8 @@
# or implied, of GRNET S.A.command
from kamaki.logger import get_logger
from kamaki.cli.utils import print_json, print_items
from kamaki.cli.argument import FlagArgument
log = get_logger(__name__)
......@@ -41,10 +43,11 @@ class _command_init(object):
def __init__(self, arguments={}):
if hasattr(self, 'arguments'):
arguments.update(self.arguments)
if isinstance(self, _optional_output_cmd):
arguments.update(self.oo_arguments)
self.arguments = dict(arguments)
try:
self.config = self['config']
#self.config = self.get_argument('config')
except KeyError:
pass
......@@ -124,3 +127,17 @@ class _command_init(object):
:raises KeyError: if argterm not in self.arguments of this object
"""
return self[argterm]
class _optional_output_cmd(object):
oo_arguments = dict(
with_output=FlagArgument('show response headers', ('--with-output')),
json_output=FlagArgument('show headers in json', ('-j', '--json'))
)
def _optional_output(self, r):
if self['json_output']:
print_json(r)
elif self['with_output']:
print_items([r] if isinstance(r, dict) else r)
kamaki/cli/commands/errors.py
View file @ 4a469d38
......@@ -358,6 +358,22 @@ class plankton(object):
'* get a list of image ids: /image list',
'* details of image: /flavor info <image id>']
remote_image_file = [
'Suggested usage:',
' /image register <image container>:<uploaded image file path>',
'To set "image" as image container and "my_dir/img.diskdump" as',
'the remote image file path, try one of the following:',
'- <image container>:<remote path>',
' e.g. image:/my_dir/img.diskdump',
'- <remote path> -C <image container>',
' e.g. /my_dir/img.diskdump -C image',
'To check if the image file is accessible to current user:',
' /file list <image container>',
'If the file is located under a different user id "us3r1d"',
' use the --fileowner=us3r1d argument e.g.:',
' /image register "my" image:my_dir/img.diskdump --fileowner=us3r1d',
'Note: The form pithos://<userid>/<container>/<path> is deprecated']
@classmethod
def connection(this, foo):
return generic._connection(foo, 'image.url')
......@@ -386,7 +402,7 @@ class plankton(object):
def _raise(self, *args, **kwargs):
key = kwargs.get('key', None)
try:
foo(self, *args, **kwargs)
return foo(self, *args, **kwargs)
except ClientError as ce:
ce_msg = ('%s' % ce).lower()
if ce.status == 404 or (
......@@ -396,6 +412,20 @@ class plankton(object):
raise
return _raise
@classmethod
def image_file(this, foo):
def _raise(self, name, container_path):
try:
return foo(self, name, container_path)
except ClientError as ce:
if ce.status in (400,):
raiseCLIError(
ce,
'Nonexistent location for %s' % container_path,
importance=2, details=this.remote_image_file)
raise
return _raise
class pithos(object):
container_howto = [
......
kamaki/cli/commands/image.py
View file @ 4a469d38
This diff is collapsed.
kamaki/cli/commands/pithos.py
View file @ 4a469d38
This diff is collapsed.
kamaki/cli/utils.py
View file @ 4a469d38
......@@ -35,6 +35,7 @@ from sys import stdout, stdin
from re import compile as regex_compile
from time import sleep
from os import walk, path
from json import dumps
from kamaki.cli.errors import raiseCLIError
......@@ -100,6 +101,14 @@ def pretty_keys(d, delim='_', recurcive=False):
return new_d
def print_json(data):
"""Print a list or dict as json in console
:param data: json-dumpable data
"""
print(dumps(data, indent=2))
def print_dict(
d, exclude=(), ident=0,
with_enumeration=False, recursive_enumeration=False):
......
kamaki/clients/compute/__init__.py
View file @ 4a469d38
......@@ -233,7 +233,8 @@ class ComputeClient(ComputeRestClient):
"""
:param image_id: (str)
"""
self.images_delete(image_id)
r = self.images_delete(image_id)
return r.headers
def get_image_metadata(self, image_id, key=''):
"""
......@@ -280,4 +281,5 @@ class ComputeClient(ComputeRestClient):
:param key: (str) metadatum key
"""
command = path4url('meta', key)
self.images_delete(image_id, command)
r = self.images_delete(image_id, command)
return r.headers
kamaki/clients/image/__init__.py
View file @ 4a469d38
......@@ -99,7 +99,7 @@ class ImageClient(Client):
return reply
def register(self, name, location, params={}, properties={}):
"""Register image put at location
"""Register an image that is uploaded at location
:param name: (str)
......@@ -133,9 +133,12 @@ class ImageClient(Client):
"""Unregister an image
:param image_id: (str)
:returns: (dict) response headers
"""
path = path4url('images', image_id)
self.delete(path, success=204)
r = self.delete(path, success=204)
return r.headers
def list_members(self, image_id):
"""
......@@ -165,7 +168,9 @@ class ImageClient(Client):
:param member: (str) user to allow access to current user's images
"""
path = path4url('images', image_id, 'members', member)
self.put(path, success=204)
self.set_header('Content-Length', len(member))
r = self.put(path, success=204)
return r.headers
def remove_member(self, image_id, member):
"""
......@@ -174,7 +179,8 @@ class ImageClient(Client):
:param member: (str) user to deprive from current user's images
"""
path = path4url('images', image_id, 'members', member)
self.delete(path, success=204)
r = self.delete(path, success=204)
return r.headers
def set_members(self, image_id, members):
"""
......@@ -184,4 +190,5 @@ class ImageClient(Client):
"""
path = path4url('images', image_id, 'members')
req = {'memberships': [{'member_id': member} for member in members]}
self.put(path, json=req, success=204)
r = self.put(path, json=req, success=204)
return r.headers
kamaki/clients/livetest/pithos.py
View file @ 4a469d38
......@@ -393,22 +393,22 @@ class Pithos(livetest.Generic):
def _test_0050_container_put(self):
self.client.container = self.c2
r = self.client.container_put()
self.assertEqual(r.status_code, 202)
r = self.client.create_container()
self.assertTrue(isinstance(r, dict))
r = self.client.get_container_limit(self.client.container)
cquota = r.values()[0]
newquota = 2 * int(cquota)
r = self.client.container_put(quota=newquota)
self.assertEqual(r.status_code, 202)
r = self.client.create_container(sizelimit=newquota)
self.assertTrue(isinstance(r, dict))
r = self.client.get_container_limit(self.client.container)
xquota = int(r.values()[0])
self.assertEqual(newquota, xquota)
r = self.client.container_put(versioning='auto')
self.assertEqual(r.status_code, 202)
r = self.client.create_container(versioning='auto')
self.assertTrue(isinstance(r, dict))
r = self.client.get_container_versioning(self.client.container)
nvers = r.values()[0]
......@@ -421,8 +421,8 @@ class Pithos(livetest.Generic):
nvers = r.values()[0]
self.assertEqual('none', nvers)
r = self.client.container_put(metadata={'m1': 'v1', 'm2': 'v2'})
self.assertEqual(r.status_code, 202)
r = self.client.create_container(metadata={'m1': 'v1', 'm2': 'v2'})
self.assertTrue(isinstance(r, dict))
r = self.client.get_container_meta(self.client.container)
self.assertTrue('x-container-meta-m1' in r)
......
kamaki/clients/pithos/__init__.py
View file @ 4a469d38
......@@ -71,15 +71,40 @@ class PithosClient(PithosRestClient):
def __init__(self, base_url, token, account=None, container=None):
super(PithosClient, self).__init__(base_url, token, account, container)
def create_container(
self,
container=None, sizelimit=None, versioning=None, metadata=None):
"""
:param container: (str) if not given, self.container is used instead
:param sizelimit: (int) container total size limit in bytes
:param versioning: (str) can be auto or whatever supported by server
:param metadata: (dict) Custom user-defined metadata of the form
{ 'name1': 'value1', 'name2': 'value2', ... }
:returns: (dict) response headers
"""
cnt_back_up = self.container
try:
self.container = container or cnt_back_up
r = self.container_put(
quota=sizelimit, versioning=versioning, metadata=metadata)
return r.headers
finally:
self.container = cnt_back_up
def purge_container(self, container=None):
"""Delete an empty container and destroy associated blocks
"""
cnt_back_up = self.container
try:
self.container = container or cnt_back_up
self.container_delete(until=unicode(time()))
r = self.container_delete(until=unicode(time()))
finally:
self.container = cnt_back_up
return r.headers
def upload_object_unchunked(
self, obj, f,
......@@ -811,25 +836,30 @@ class PithosClient(PithosRestClient):
ret = [''] * num_of_blocks
self._init_thread_limit()
flying = dict()
for blockid, blockhash in enumerate(remote_hashes):
start = blocksize * blockid
is_last = start + blocksize > total_size
end = (total_size - 1) if is_last else (start + blocksize - 1)
(start, end) = _range_up(start, end, range_str)
if start < end:
self._watch_thread_limit(flying.values())
flying[blockid] = self._get_block_async(obj, **restargs)
for runid, thread in flying.items():
if (blockid + 1) == num_of_blocks:
thread.join()
elif thread.isAlive():
continue
if thread.exception:
raise thread.exception
ret[runid] = thread.value.content
self._cb_next()
flying.pop(runid)
return ''.join(ret)
try:
for blockid, blockhash in enumerate(remote_hashes):
start = blocksize * blockid
is_last = start + blocksize > total_size
end = (total_size - 1) if is_last else (start + blocksize - 1)
(start, end) = _range_up(start, end, range_str)
if start < end:
self._watch_thread_limit(flying.values())
flying[blockid] = self._get_block_async(obj, **restargs)
for runid, thread in flying.items():
if (blockid + 1) == num_of_blocks:
thread.join()
elif thread.isAlive():
continue
if thread.exception:
raise thread.exception
ret[runid] = thread.value.content
self._cb_next()
flying.pop(runid)
return ''.join(ret)
except KeyboardInterrupt:
sendlog.info('- - - wait for threads to finish')
for thread in activethreads():
thread.join()
#Command Progress Bar method
def _cb_next(self, step=1):
......@@ -893,7 +923,8 @@ class PithosClient(PithosRestClient):
:param usernames: (list)
"""
self.account_post(update=True, groups={group: usernames})
r = self.account_post(update=True, groups={group: usernames})
return r
def del_account_group(self, group):
"""
......@@ -949,13 +980,15 @@ class PithosClient(PithosRestClient):
:param metapairs: (dict) {key1:val1, key2:val2, ...}
"""
assert(type(metapairs) is dict)
self.account_post(update=True, metadata=metapairs)
r = self.account_post(update=True, metadata=metapairs)
return r.headers
def del_account_meta(self, metakey):
"""
:param metakey: (str) metadatum key
"""
self.account_post(update=True, metadata={metakey: ''})
r = self.account_post(update=True, metadata={metakey: ''})
return r.headers
"""
def set_account_quota(self, quota):
......@@ -969,7 +1002,8 @@ class PithosClient(PithosRestClient):
"""
"param versioning: (str)
"""
self.account_post(update=True, versioning=versioning)
r = self.account_post(update=True, versioning=versioning)
return r.headers
def list_containers(self):
"""
......@@ -1001,6 +1035,7 @@ class PithosClient(PithosRestClient):
raise ClientError(
'Container "%s" is not empty' % self.container,
r.status_code)
return r.headers
def get_container_versioning(self, container=None):
"""
......@@ -1072,25 +1107,29 @@ class PithosClient(PithosRestClient):
:param metapairs: (dict) {key1:val1, key2:v