Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
itminedu
kamaki
Commits
4a469d38
Commit
4a469d38
authored
May 21, 2013
by
Stavros Sachtouris
Browse files
Merge branch 'feature-image-meta-record' into develop
parents
90099c14
c4aefeaf
Changes
14
Expand all
Hide whitespace changes
Inline
Side-by-side
Changelog
View file @
4a469d38
...
...
@@ -6,6 +6,7 @@ Bug Fixes:
- Shell can manage all valid command line arguments [#3716]
- Restore 2nd level command syntax in shell [#3736]
- Allow copy of deleted objects by refering to older version [#3737]
- Add image.add_member missing content-length header
Changes:
...
...
@@ -18,6 +19,24 @@ Changes:
- Disallow moving deleted objects by version [#3737]
This operation was implemented by accident, due to the symetry between
move and copy
- Rename file-meta commands to file-metadata
- Rename image-[add|del]member commands to members-[add|delete]
- Remove update option from imagre-register
- In image-compute split properties to properties-list and properties-get
- Add optional output to methods[#3756, #3732]:
- file:
mkdir, touch, create, move, copy, move, append, truncate, overwrite,
manifest, upload, delete, purge, unpublish, permissions set/delete, info,
metadata set/delete, containerlimit set, versioning set, group set/delete,
upload, overwrite
- image:
unregister, members add/delete/set
-image compute:
delete, properties delete
- Transliterate methods to list-get-set-delete command groups:
- file: permissions, versioning, group and metadata
- image: members, member
- image compute: properties
Features:
...
...
@@ -34,4 +53,8 @@ Features:
- Add enumeration to all listing commands, make it optional [#3739]
- Add a download_to_string method in pithos client [#3608]
- Add an upload_from_string method in pithos client [#3608]
- Add pithos client method create_container [#3756]
- Store image properties on remote location after image registration [#3769]
- Add runtime args to image register for forcing or unsettitng property
storage [#3769]
docs/commands.rst
View file @
4a469d38
...
...
@@ -64,22 +64,26 @@ image (Plankton commands + Compute Image subcommands)
.. code-block:: text
addmember : Add a member to an image
addproperty: Add an image property
delmember : Remove a member from an image
list : List images accessible by user
members : Get image members
meta : Get image metadata
register : (Re)Register an image
setmemb
er
s
:
Set the members of an image
unregist
er :
Unregister an image (does not delete the image file)
shared : List shared images
compute : Compute Image API commands
list : List images
delete : Delete image
info : Get image details
properties : Get image properties
delproperty: Delete an image property
setproperty: Update an image property
properties : Manage properties related to OS installation in an image
add : Add a property to an image
delete: Delete a property from an image
get : Get an image property
list : List all image properties
set : Add / update a set of properties for an image
members : Manage members (users who can modify an image)
add : Add a member to an image
delete : Remove a member from an image
list : List members of an image
set : Set the members of an image
Showcase: Pick an image and list the properties
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
...
@@ -311,38 +315,42 @@ file (Storage/Pithos+)
append : Append local file to remote
cat : Print a file to console
copy : Copy an object
containerlimit: Container size limit commands
set : Set container data limit
get : Get container data limit
create : Create a container
delete : Delete a container [or an object]
delgroup : Delete a user group
delmeta : Delete an existing metadatum for an account [, container [or object]]
delpermissions: Delete all sharing permissions
download : Download a file or directory
group : Get user groups details
group : Manage access groups and group members
delete: Delete a user group
get : Get groups and group members
set : Set a user group
hashmap : Get the hashmap of an object
info : Get information for account [, container [or object]]
list : List containers, object trees or objects in a directory
manifest : Create a remote file with uploaded parts by manifestation
meta : Get custom meta-content for account [, container [or object]]
metadata : Metadata are attached on objects (key:value pairs)
delete: Delete metadata with given key
get : Get metadatum
set : Set a piece of metadata
mkdir : Create a directory
move : Copy an object
overwrite : Overwrite part (from start to end) of a remote file
permissions : Get object read/write permissions
permissions : Manage user and group accessibility for objects
delete: Delete all permissions set on object
get : Get read and write permissions of an object
set : Set permissions for an object
publish : Publish an object
purge : Purge a container
quota : Get quota for account
setgroup : Create/update a new user group
setmeta : Set a new metadatum for account [, container [or object]]
setpermissions: Set sharing permissions
containerlimit: Container size limit commands
set : Set container data limit
get : Get container data limit
setversioning : Set new versioning (auto, none) for account [or container]
sharers : List the accounts that share objects with default account
touch : Create an empty object (file)
truncate : Truncate remote file up to a size
unpublish : Unpublish an object
upload : Upload a file or directory
versioning : Get versioning for account [or container ]
versioning : Manage the versioning scheme of current pithos user account
get: Get versioning for account or container
set: Set versioning mode (auto, none) for account or container
versions : Get the version list of an object
Showcase: Upload and download a file
...
...
docs/man/kamaki.rst
View file @
4a469d38
...
...
@@ -140,27 +140,26 @@ flavor commands
image commands
**************
* addmember Add a member to an image
* addproperty Add an OS-related property to an image
* delete Delete an image (image file remains intact)
* delmember Remove a member from an image
* delproperty Delete a property of an image
* info Get detailed information on an image
* members Get image members
* meta Get image metadata
* properties Get properties related to OS installation in an image
* list List images accessible by user
* register (Re)Register an image
* setmembers Set the members of an image
* setproperty Update an existing property in an image
* shared List images shared by a member
* compute Compute Image API commands
* list List images
* delete Delete image
* info Get image details
* properties Get image properties
* delproperty Delete an image property
* setproperty Update an image property
* list List images accessible by user
* meta Get image metadata
* register (Re)Register an image
* unregister Unregister an image (does not delete the image file)
* shared List shared images
* compute Compute Image API commands
* list List images
* delete Delete image
* info Get image details
* properties Manage properties related to OS installation in an image
* add Add a property to an image
* delete Delete a property from an image
* get Get an image property
* list List all image properties
* set Add / update a set of properties for an image
* members Manage members (users who can modify an image)
* add Add a member to an image
* delete Remove a member from an image
* list List members of an image
* set Set the members of an image
network commands
...
...
@@ -178,41 +177,46 @@ network commands
file commands
**************
* append Append local file to (existing) remote object
* cat Print remote file contents to console
* copy Copy an object from container to (another) container
* create Create a container
* delete Delete a container [or an object]
* delgroup Delete a user group
* delmeta Delete metadata from account, container or object
* delpermissions Delete all permissions set on object
* download Download remote object(s) as local file(s)
* group Get groups and group members
* hashmap Get the hash-map of an object
* info Get detailed info for account, containers or objects
* list List containers, object trees or objects in a directory
* manifest Create a remote file of uploaded parts by manifestation
* meta Get metadata for account, containers or objects
* mkdir Create a directory
* move Copy an object
* overwrite Overwrite part (from start to end) of a remote file
* permissions Get read and write permissions of an object
* publish Publish the object and print the public url
* purge Delete a container and release related data blocks
* quota Get quota (in KB) for account or container
* setgroup Set a user group
* setmeta Set a piece of metadata for account, container or object
* setpermissions Set permissions for an object
* containerlimit set Set new limit for container
* containerlimit get Get container limit
* setversioning Set versioning mode (auto, none) for account or container
* sharers List the accounts that share objects with current user
* touch Create an empty object (file)
* truncate Truncate remote file up to a size
* unpublish Unpublish an object
* upload Upload a file or directory
* versioning Get versioning for account or container
* versions Get the list of object versions
* append Append local file to remote
* cat Print a file to console
* copy Copy an object
* containerlimit Container size limit commands
* set Set container data limit
* get Get container data limit
* create Create a container
* delete Delete a container [or an object]
* download Download a file or directory
* group Manage access groups and group members
* delete Delete a user group
* get Get groups and group members
* set Set a user group
* hashmap Get the hashmap of an object
* info Get information for account [, container [or object]]
* list List containers, object trees or objects in a directory
* manifest Create a remote file with uploaded parts by manifestation
* metadata Metadata are attached on objects (key:value pairs)
* delete Delete metadata with given key
* get Get metadatum
* set Set a piece of metadata
* mkdir Create a directory
* move Copy an object
* overwrite Overwrite part (from start to end) of a remote file
* permissions Manage user and group accessibility for objects
* delete Delete all permissions set on object
* get Get read and write permissions of an object
* set Set permissions for an object
* publish Publish an object
* purge Purge a container
* quota Get quota for account
* sharers List the accounts that share objects with default account
* touch Create an empty object (file)
* truncate Truncate remote file up to a size
* unpublish Unpublish an object
* upload Upload a file or directory
* versioning Manage the versioning scheme of current pithos user account
* get Get versioning for account or container
* set Set versioning mode (auto, none) for account or container
* versions Get the version list of an object
test commands (hidden)
...
...
kamaki/cli/commands/__init__.py
View file @
4a469d38
...
...
@@ -32,6 +32,8 @@
# or implied, of GRNET S.A.command
from
kamaki.logger
import
get_logger
from
kamaki.cli.utils
import
print_json
,
print_items
from
kamaki.cli.argument
import
FlagArgument
log
=
get_logger
(
__name__
)
...
...
@@ -41,10 +43,11 @@ class _command_init(object):
def
__init__
(
self
,
arguments
=
{}):
if
hasattr
(
self
,
'arguments'
):
arguments
.
update
(
self
.
arguments
)
if
isinstance
(
self
,
_optional_output_cmd
):
arguments
.
update
(
self
.
oo_arguments
)
self
.
arguments
=
dict
(
arguments
)
try
:
self
.
config
=
self
[
'config'
]
#self.config = self.get_argument('config')
except
KeyError
:
pass
...
...
@@ -124,3 +127,17 @@ class _command_init(object):
:raises KeyError: if argterm not in self.arguments of this object
"""
return
self
[
argterm
]
class
_optional_output_cmd
(
object
):
oo_arguments
=
dict
(
with_output
=
FlagArgument
(
'show response headers'
,
(
'--with-output'
)),
json_output
=
FlagArgument
(
'show headers in json'
,
(
'-j'
,
'--json'
))
)
def
_optional_output
(
self
,
r
):
if
self
[
'json_output'
]:
print_json
(
r
)
elif
self
[
'with_output'
]:
print_items
([
r
]
if
isinstance
(
r
,
dict
)
else
r
)
kamaki/cli/commands/errors.py
View file @
4a469d38
...
...
@@ -358,6 +358,22 @@ class plankton(object):
'* get a list of image ids: /image list'
,
'* details of image: /flavor info <image id>'
]
remote_image_file
=
[
'Suggested usage:'
,
' /image register <image container>:<uploaded image file path>'
,
'To set "image" as image container and "my_dir/img.diskdump" as'
,
'the remote image file path, try one of the following:'
,
'- <image container>:<remote path>'
,
' e.g. image:/my_dir/img.diskdump'
,
'- <remote path> -C <image container>'
,
' e.g. /my_dir/img.diskdump -C image'
,
'To check if the image file is accessible to current user:'
,
' /file list <image container>'
,
'If the file is located under a different user id "us3r1d"'
,
' use the --fileowner=us3r1d argument e.g.:'
,
' /image register "my" image:my_dir/img.diskdump --fileowner=us3r1d'
,
'Note: The form pithos://<userid>/<container>/<path> is deprecated'
]
@
classmethod
def
connection
(
this
,
foo
):
return
generic
.
_connection
(
foo
,
'image.url'
)
...
...
@@ -386,7 +402,7 @@ class plankton(object):
def
_raise
(
self
,
*
args
,
**
kwargs
):
key
=
kwargs
.
get
(
'key'
,
None
)
try
:
foo
(
self
,
*
args
,
**
kwargs
)
return
foo
(
self
,
*
args
,
**
kwargs
)
except
ClientError
as
ce
:
ce_msg
=
(
'%s'
%
ce
).
lower
()
if
ce
.
status
==
404
or
(
...
...
@@ -396,6 +412,20 @@ class plankton(object):
raise
return
_raise
@
classmethod
def
image_file
(
this
,
foo
):
def
_raise
(
self
,
name
,
container_path
):
try
:
return
foo
(
self
,
name
,
container_path
)
except
ClientError
as
ce
:
if
ce
.
status
in
(
400
,):
raiseCLIError
(
ce
,
'Nonexistent location for %s'
%
container_path
,
importance
=
2
,
details
=
this
.
remote_image_file
)
raise
return
_raise
class
pithos
(
object
):
container_howto
=
[
...
...
kamaki/cli/commands/image.py
View file @
4a469d38
This diff is collapsed.
Click to expand it.
kamaki/cli/commands/pithos.py
View file @
4a469d38
This diff is collapsed.
Click to expand it.
kamaki/cli/utils.py
View file @
4a469d38
...
...
@@ -35,6 +35,7 @@ from sys import stdout, stdin
from
re
import
compile
as
regex_compile
from
time
import
sleep
from
os
import
walk
,
path
from
json
import
dumps
from
kamaki.cli.errors
import
raiseCLIError
...
...
@@ -100,6 +101,14 @@ def pretty_keys(d, delim='_', recurcive=False):
return
new_d
def
print_json
(
data
):
"""Print a list or dict as json in console
:param data: json-dumpable data
"""
print
(
dumps
(
data
,
indent
=
2
))
def
print_dict
(
d
,
exclude
=
(),
ident
=
0
,
with_enumeration
=
False
,
recursive_enumeration
=
False
):
...
...
kamaki/clients/compute/__init__.py
View file @
4a469d38
...
...
@@ -233,7 +233,8 @@ class ComputeClient(ComputeRestClient):
"""
:param image_id: (str)
"""
self
.
images_delete
(
image_id
)
r
=
self
.
images_delete
(
image_id
)
return
r
.
headers
def
get_image_metadata
(
self
,
image_id
,
key
=
''
):
"""
...
...
@@ -280,4 +281,5 @@ class ComputeClient(ComputeRestClient):
:param key: (str) metadatum key
"""
command
=
path4url
(
'meta'
,
key
)
self
.
images_delete
(
image_id
,
command
)
r
=
self
.
images_delete
(
image_id
,
command
)
return
r
.
headers
kamaki/clients/image/__init__.py
View file @
4a469d38
...
...
@@ -99,7 +99,7 @@ class ImageClient(Client):
return
reply
def
register
(
self
,
name
,
location
,
params
=
{},
properties
=
{}):
"""Register image
put
at location
"""Register
an
image
that is uploaded
at location
:param name: (str)
...
...
@@ -133,9 +133,12 @@ class ImageClient(Client):
"""Unregister an image
:param image_id: (str)
:returns: (dict) response headers
"""
path
=
path4url
(
'images'
,
image_id
)
self
.
delete
(
path
,
success
=
204
)
r
=
self
.
delete
(
path
,
success
=
204
)
return
r
.
headers
def
list_members
(
self
,
image_id
):
"""
...
...
@@ -165,7 +168,9 @@ class ImageClient(Client):
:param member: (str) user to allow access to current user's images
"""
path
=
path4url
(
'images'
,
image_id
,
'members'
,
member
)
self
.
put
(
path
,
success
=
204
)
self
.
set_header
(
'Content-Length'
,
len
(
member
))
r
=
self
.
put
(
path
,
success
=
204
)
return
r
.
headers
def
remove_member
(
self
,
image_id
,
member
):
"""
...
...
@@ -174,7 +179,8 @@ class ImageClient(Client):
:param member: (str) user to deprive from current user's images
"""
path
=
path4url
(
'images'
,
image_id
,
'members'
,
member
)
self
.
delete
(
path
,
success
=
204
)
r
=
self
.
delete
(
path
,
success
=
204
)
return
r
.
headers
def
set_members
(
self
,
image_id
,
members
):
"""
...
...
@@ -184,4 +190,5 @@ class ImageClient(Client):
"""
path
=
path4url
(
'images'
,
image_id
,
'members'
)
req
=
{
'memberships'
:
[{
'member_id'
:
member
}
for
member
in
members
]}
self
.
put
(
path
,
json
=
req
,
success
=
204
)
r
=
self
.
put
(
path
,
json
=
req
,
success
=
204
)
return
r
.
headers
kamaki/clients/livetest/pithos.py
View file @
4a469d38
...
...
@@ -393,22 +393,22 @@ class Pithos(livetest.Generic):
def
_test_0050_container_put
(
self
):
self
.
client
.
container
=
self
.
c2
r
=
self
.
client
.
container
_put
()
self
.
assert
Equal
(
r
.
status_code
,
202
)
r
=
self
.
client
.
create_
container
()
self
.
assert
True
(
isinstance
(
r
,
dict
)
)
r
=
self
.
client
.
get_container_limit
(
self
.
client
.
container
)
cquota
=
r
.
values
()[
0
]
newquota
=
2
*
int
(
cquota
)
r
=
self
.
client
.
container
_put
(
quota
=
newquota
)
self
.
assert
Equal
(
r
.
status_code
,
202
)
r
=
self
.
client
.
create_
container
(
sizelimit
=
newquota
)
self
.
assert
True
(
isinstance
(
r
,
dict
)
)
r
=
self
.
client
.
get_container_limit
(
self
.
client
.
container
)
xquota
=
int
(
r
.
values
()[
0
])
self
.
assertEqual
(
newquota
,
xquota
)
r
=
self
.
client
.
container
_put
(
versioning
=
'auto'
)
self
.
assert
Equal
(
r
.
status_code
,
202
)
r
=
self
.
client
.
create_
container
(
versioning
=
'auto'
)
self
.
assert
True
(
isinstance
(
r
,
dict
)
)
r
=
self
.
client
.
get_container_versioning
(
self
.
client
.
container
)
nvers
=
r
.
values
()[
0
]
...
...
@@ -421,8 +421,8 @@ class Pithos(livetest.Generic):
nvers
=
r
.
values
()[
0
]
self
.
assertEqual
(
'none'
,
nvers
)
r
=
self
.
client
.
container
_put
(
metadata
=
{
'm1'
:
'v1'
,
'm2'
:
'v2'
})
self
.
assert
Equal
(
r
.
status_code
,
202
)
r
=
self
.
client
.
create_
container
(
metadata
=
{
'm1'
:
'v1'
,
'm2'
:
'v2'
})
self
.
assert
True
(
isinstance
(
r
,
dict
)
)
r
=
self
.
client
.
get_container_meta
(
self
.
client
.
container
)
self
.
assertTrue
(
'x-container-meta-m1'
in
r
)
...
...
kamaki/clients/pithos/__init__.py
View file @
4a469d38
...
...
@@ -71,15 +71,40 @@ class PithosClient(PithosRestClient):
def
__init__
(
self
,
base_url
,
token
,
account
=
None
,
container
=
None
):
super
(
PithosClient
,
self
).
__init__
(
base_url
,
token
,
account
,
container
)
def
create_container
(
self
,
container
=
None
,
sizelimit
=
None
,
versioning
=
None
,
metadata
=
None
):
"""
:param container: (str) if not given, self.container is used instead
:param sizelimit: (int) container total size limit in bytes
:param versioning: (str) can be auto or whatever supported by server
:param metadata: (dict) Custom user-defined metadata of the form
{ 'name1': 'value1', 'name2': 'value2', ... }
:returns: (dict) response headers
"""
cnt_back_up
=
self
.
container
try
:
self
.
container
=
container
or
cnt_back_up
r
=
self
.
container_put
(
quota
=
sizelimit
,
versioning
=
versioning
,
metadata
=
metadata
)
return
r
.
headers
finally
:
self
.
container
=
cnt_back_up
def
purge_container
(
self
,
container
=
None
):
"""Delete an empty container and destroy associated blocks
"""
cnt_back_up
=
self
.
container
try
:
self
.
container
=
container
or
cnt_back_up
self
.
container_delete
(
until
=
unicode
(
time
()))
r
=
self
.
container_delete
(
until
=
unicode
(
time
()))
finally
:
self
.
container
=
cnt_back_up
return
r
.
headers
def
upload_object_unchunked
(
self
,
obj
,
f
,
...
...
@@ -811,25 +836,30 @@ class PithosClient(PithosRestClient):
ret
=
[
''
]
*
num_of_blocks
self
.
_init_thread_limit
()
flying
=
dict
()
for
blockid
,
blockhash
in
enumerate
(
remote_hashes
):
start
=
blocksize
*
blockid
is_last
=
start
+
blocksize
>
total_size
end
=
(
total_size
-
1
)
if
is_last
else
(
start
+
blocksize
-
1
)
(
start
,
end
)
=
_range_up
(
start
,
end
,
range_str
)
if
start
<
end
:
self
.
_watch_thread_limit
(
flying
.
values
())
flying
[
blockid
]
=
self
.
_get_block_async
(
obj
,
**
restargs
)
for
runid
,
thread
in
flying
.
items
():
if
(
blockid
+
1
)
==
num_of_blocks
:
thread
.
join
()
elif
thread
.
isAlive
():
continue
if
thread
.
exception
:
raise
thread
.
exception
ret
[
runid
]
=
thread
.
value
.
content
self
.
_cb_next
()
flying
.
pop
(
runid
)
return
''
.
join
(
ret
)
try
:
for
blockid
,
blockhash
in
enumerate
(
remote_hashes
):
start
=
blocksize
*
blockid
is_last
=
start
+
blocksize
>
total_size
end
=
(
total_size
-
1
)
if
is_last
else
(
start
+
blocksize
-
1
)
(
start
,
end
)
=
_range_up
(
start
,
end
,
range_str
)
if
start
<
end
:
self
.
_watch_thread_limit
(
flying
.
values
())
flying
[
blockid
]
=
self
.
_get_block_async
(
obj
,
**
restargs
)
for
runid
,
thread
in
flying
.
items
():
if
(
blockid
+
1
)
==
num_of_blocks
:
thread
.
join
()
elif
thread
.
isAlive
():
continue
if
thread
.
exception
:
raise
thread
.
exception
ret
[
runid
]
=
thread
.
value
.
content
self
.
_cb_next
()
flying
.
pop
(
runid
)
return
''
.
join
(
ret
)
except
KeyboardInterrupt
:
sendlog
.
info
(
'- - - wait for threads to finish'
)
for
thread
in
activethreads
():
thread
.
join
()
#Command Progress Bar method
def
_cb_next
(
self
,
step
=
1
):
...
...
@@ -893,7 +923,8 @@ class PithosClient(PithosRestClient):
:param usernames: (list)
"""
self
.
account_post
(
update
=
True
,
groups
=
{
group
:
usernames
})
r
=
self
.
account_post
(
update
=
True
,
groups
=
{
group
:
usernames
})
return
r
def
del_account_group
(
self
,
group
):
"""
...
...
@@ -949,13 +980,15 @@ class PithosClient(PithosRestClient):
:param metapairs: (dict) {key1:val1, key2:val2, ...}
"""
assert
(
type
(
metapairs
)
is
dict
)
self
.
account_post
(
update
=
True
,
metadata
=
metapairs
)
r
=
self
.
account_post
(
update
=
True
,
metadata
=
metapairs
)
return
r
.
headers
def
del_account_meta
(
self
,
metakey
):
"""
:param metakey: (str) metadatum key
"""
self
.
account_post
(
update
=
True
,
metadata
=
{
metakey
:
''
})
r
=
self
.
account_post
(
update
=
True
,
metadata
=
{
metakey
:
''
})
return
r
.
headers
"""
def set_account_quota(self, quota):
...
...
@@ -969,7 +1002,8 @@ class PithosClient(PithosRestClient):
"""
"param versioning: (str)
"""
self
.
account_post
(
update
=
True
,
versioning
=
versioning
)
r
=
self
.
account_post
(
update
=
True
,
versioning
=
versioning
)
return
r
.
headers
def
list_containers
(
self
):
"""
...
...
@@ -1001,6 +1035,7 @@ class PithosClient(PithosRestClient):
raise
ClientError
(
'Container "%s" is not empty'
%
self
.
container
,
r
.
status_code
)
return
r
.
headers
def
get_container_versioning
(
self
,
container
=
None
):
"""
...
...
@@ -1072,25 +1107,29 @@ class PithosClient(PithosRestClient):
:param metapairs: (dict) {key1:val1, key2:v