Commit 919232f8 authored by Sofia Papagiannaki's avatar Sofia Papagiannaki

Merge branch 'master' of https://code.grnet.gr/git/pithos

parents e5401a16 b4b29ad7
......@@ -27,7 +27,8 @@ Document Revisions
========================= ================================
Revision Description
========================= ================================
0.9 (Feb 15, 2012) Change permissions model.
0.9 (Feb 17, 2012) Change permissions model.
\ Do not include user-defined metadata in account/container/object listings.
0.8 (Jan 24, 2012) Update allowed versioning values.
\ Change policy/meta formatting in JSON/XML replies.
\ Document that all non-ASCII characters in headers should be URL-encoded.
......@@ -277,7 +278,7 @@ The reply is a list of container names. Account headers (as in a ``HEAD`` reques
Cross-user requests are not allowed to use ``until`` and only include the account/container modification dates in the reply.
If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
For each container, the information will include all container metadata (names will be in lower case and with hyphens replaced with underscores):
For each container, the information will include all container metadata, except user-defined (names will be in lower case and with hyphens replaced with underscores):
=========================== ============================
Name Description
......@@ -287,8 +288,7 @@ count The number of objects inside the container
bytes The total size of the objects inside the container
last_modified The last container modification date (regardless of ``until``)
x_container_until_timestamp The last container modification date until the timestamp provided
x_container_policy_* Container behavior and limits
x_container_meta_* Optional user defined metadata
x_container_policy Container behavior and limits
=========================== ============================
Example ``format=json`` reply:
......@@ -299,8 +299,7 @@ Example ``format=json`` reply:
"bytes": 62452,
"count": 8374,
"last_modified": "2011-12-02T08:10:41.565891+00:00",
"x_container_policy": {"quota": "53687091200", "versioning": "auto"},
"x_container_meta": {"a": "b", "1": "2"}}, ...]
"x_container_policy": {"quota": "53687091200", "versioning": "auto"}}, ...]
Example ``format=xml`` reply:
......@@ -317,15 +316,11 @@ Example ``format=xml`` reply:
<key>quota</key><value>53687091200</value>
<key>versioning</key><value>auto</value>
</x_container_policy>
<x_container_meta>
<key>a</key><value>b</value>
<key>1</key><value>2</value>
</x_container_meta>
</container>
<container>...</container>
</account>
For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns all fields. Policy and metadata values are grouped and returned as key-value pairs.
For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns policy fields, grouped as key-value pairs.
=========================== =====================
Return Code Description
......@@ -471,7 +466,7 @@ Last-Modified The last container modification date
=========================== ===============================
If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.
For each object, the information will include all object metadata (names will be in lower case and with hyphens replaced with underscores):
For each object, the information will include all object metadata, except user-defined (names will be in lower case and with hyphens replaced with underscores). User-defined metadata includes ``X-Object-Meta-*``, ``X-Object-Manifest``, ``Content-Disposition`` and ``Content-Encoding`` keys. Also, sharing directives will only be included with the actual shared objects (inherited permissions are not calculated):
========================== ======================================
Name Description
......@@ -480,23 +475,18 @@ name The name of the object
hash The ETag of the object
bytes The size of the object
content_type The MIME content type of the object
content_encoding The encoding of the object (optional)
content-disposition The presentation style of the object (optional)
last_modified The last object modification date (regardless of version)
x_object_hash The Merkle hash
x_object_uuid The object's UUID
x_object_version The object's version identifier
x_object_version_timestamp The object's version timestamp
x_object_modified_by The user that committed the object's version
x_object_manifest Object parts prefix in ``<container>/<object>`` form (optional)
x_object_sharing Object permissions (optional)
x_object_shared_by Object inheriting permissions (optional)
x_object_allowed_to Allowed actions on object (optional)
x_object_public Object's publicly accessible URI (optional)
x_object_meta_* Optional user defined metadata
========================== ======================================
Sharing metadata will only be returned if there is no ``until`` parameter defined.
Sharing metadata and last modification timestamp will only be returned if there is no ``until`` parameter defined.
Extended replies may also include virtual directory markers in separate sections of the ``json`` or ``xml`` results.
Virtual directory markers are only included when ``delimiter`` is explicitly set. They correspond to the substrings up to and including the first occurrence of the delimiter.
......@@ -512,7 +502,6 @@ Example ``format=json`` reply:
"hash": "d41d8cd98f00b204e9800998ecf8427e",
"content_type": "application/octet-stream",
"last_modified": "2011-12-02T08:10:41.565891+00:00",
"x_object_meta": {"asdf": "qwerty"},
"x_object_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"x_object_uuid": "8ed9af1b-c948-4bb6-82b0-48344f5c822c",
"x_object_version": 98,
......@@ -531,9 +520,6 @@ Example ``format=xml`` reply:
<hash>d41d8cd98f00b204e9800998ecf8427e</hash>
<content_type>application/octet-stream</content_type>
<last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
<x_object_meta>
<key>asdf</key><value>qwerty</value>
</x_object_meta>
<x_object_hash>e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</x_object_hash>
<x_object_uuid>8ed9af1b-c948-4bb6-82b0-48344f5c822c</x_object_uuid>
<x_object_version>98</x_object_version>
......@@ -543,7 +529,7 @@ Example ``format=xml`` reply:
<object>...</object>
</container>
For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns all fields. Metadata values are grouped and returned as key-value pairs.
For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns more fields that should help with synchronization.
=========================== ===============================
Return Code Description
......@@ -1076,7 +1062,7 @@ List of differences from the OOS API:
* Headers ``X-Container-Block-*`` at the container level, exposing the underlying storage characteristics.
* All metadata replies, at all levels, include latest modification information.
* At all levels, a ``HEAD`` or ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers.
* Container/object lists include all associated metadata if the reply is of type JSON/XML. Some names are kept to their OOS API equivalents for compatibility.
* Container/object lists include more fields if the reply is of type JSON/XML. Some names are kept to their OOS API equivalents for compatibility.
* Option to include only shared containers/objects in listings.
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object.
* Multi-range object ``GET`` support as outlined in RFC2616.
......
......@@ -166,18 +166,13 @@ def account_list(request):
if x == request.user_uniq:
continue
try:
meta = request.backend.get_account_meta(request.user_uniq, x, 'pithos')
meta = request.backend.get_account_meta(request.user_uniq, x, 'pithos', include_user_defined=False)
groups = request.backend.get_account_groups(request.user_uniq, x)
except NotAllowedError:
raise Forbidden('Not allowed')
else:
rename_meta_key(meta, 'modified', 'last_modified')
rename_meta_key(meta, 'until_timestamp', 'x_account_until_timestamp')
m = dict([(k[15:], v) for k, v in meta.iteritems() if k.startswith('X-Account-Meta-')])
for k in m:
del(meta['X-Account-Meta-' + k])
if m:
meta['X-Account-Meta'] = printable_header_dict(m)
if groups:
meta['X-Account-Group'] = printable_header_dict(dict([(k, ','.join(v)) for k, v in groups.iteritems()]))
account_meta.append(printable_header_dict(meta))
......@@ -288,7 +283,7 @@ def container_list(request, v_account):
for x in containers:
try:
meta = request.backend.get_container_meta(request.user_uniq, v_account,
x, 'pithos', until)
x, 'pithos', until, include_user_defined=False)
policy = request.backend.get_container_policy(request.user_uniq,
v_account, x)
except NotAllowedError:
......@@ -298,11 +293,6 @@ def container_list(request, v_account):
else:
rename_meta_key(meta, 'modified', 'last_modified')
rename_meta_key(meta, 'until_timestamp', 'x_container_until_timestamp')
m = dict([(k[17:], v) for k, v in meta.iteritems() if k.startswith('X-Container-Meta-')])
for k in m:
del(meta['X-Container-Meta-' + k])
if m:
meta['X-Container-Meta'] = printable_header_dict(m)
if policy:
meta['X-Container-Policy'] = printable_header_dict(dict([(k, v) for k, v in policy.iteritems()]))
container_meta.append(printable_header_dict(meta))
......@@ -695,9 +685,13 @@ def object_read(request, v_account, v_container, v_object):
response['ETag'] = meta['checksum']
return response
hashmap_reply = False
if 'hashmap' in request.GET and request.serialization != 'text':
hashmap_reply = True
sizes = []
hashmaps = []
if 'X-Object-Manifest' in meta:
if 'X-Object-Manifest' in meta and not hashmap_reply:
try:
src_container, src_name = split_container_object_string('/' + meta['X-Object-Manifest'])
objects = request.backend.list_objects(request.user_uniq, v_account,
......@@ -735,7 +729,7 @@ def object_read(request, v_account, v_container, v_object):
raise ItemNotFound('Version does not exist')
# Reply with the hashmap.
if 'hashmap' in request.GET and request.serialization != 'text':
if hashmap_reply:
size = sum(sizes)
hashmap = sum(hashmaps, [])
d = {
......
......@@ -648,7 +648,10 @@ class ObjectWrapper(object):
# Get the data from the block.
bo = self.offset % self.backend.block_size
bl = min(self.length, len(self.block) - bo)
bs = self.backend.block_size
if self.block_index == len(self.hashmaps[self.file_index]) - 1:
bs = self.sizes[self.file_index] % self.backend.block_size
bl = min(self.length, bs - bo)
data = self.block[bo:bo + bl]
self.offset += bl
self.length -= bl
......@@ -753,11 +756,10 @@ def hashmap_md5(request, hashmap, size):
md5 = hashlib.md5()
bs = request.backend.block_size
for bi, hash in enumerate(hashmap):
data = request.backend.get_block(hash)
data = request.backend.get_block(hash) # Blocks come in padded.
if bi == len(hashmap) - 1:
bs = size % bs
pad = bs - min(len(data), bs)
md5.update(data + ('\x00' * pad))
data = data[:size % bs]
md5.update(data)
return md5.hexdigest().lower()
def simple_list_response(request, l):
......
......@@ -74,7 +74,7 @@ class BaseBackend(object):
"""
return []
def get_account_meta(self, user, account, domain, until=None):
def get_account_meta(self, user, account, domain, until=None, include_user_defined=True):
"""Return a dictionary with the account metadata for the domain.
The keys returned are all user-defined, except:
......@@ -195,7 +195,7 @@ class BaseBackend(object):
"""
return []
def get_container_meta(self, user, account, container, domain, until=None):
def get_container_meta(self, user, account, container, domain, until=None, include_user_defined=True):
"""Return a dictionary with the container metadata for the domain.
The keys returned are all user-defined, except:
......@@ -345,7 +345,7 @@ class BaseBackend(object):
"""Return a dict mapping paths to public ids for objects that are public under a container."""
return {}
def get_object_meta(self, user, account, container, name, domain, version=None):
def get_object_meta(self, user, account, container, name, domain, version=None, include_user_defined=True):
"""Return a dictionary with the object metadata for the domain.
The keys returned are all user-defined, except:
......
......@@ -74,6 +74,9 @@ class Blocker(object):
self.hashlen = len(emptyhash)
self.emptyhash = emptyhash
def _pad(self, block):
return block + ('\x00' * (self.blocksize - len(block)))
def _get_rear_block(self, blkhash, create=0):
filename = hexlify(blkhash)
dir = join(self.blockpath, filename[0:2], filename[2:4], filename[4:6])
......@@ -116,7 +119,7 @@ class Blocker(object):
for h in hashes:
if h == self.emptyhash:
append('')
append(self._pad(''))
continue
with self._get_rear_block(h, 0) as rbl:
if not rbl:
......@@ -125,7 +128,7 @@ class Blocker(object):
break # there should be just one block there
if not block:
break
append(block)
append(self._pad(block))
return blocks
......@@ -145,36 +148,26 @@ class Blocker(object):
return hashlist, missing
def block_delta(self, blkhash, offdata=()):
def block_delta(self, blkhash, offset, data):
"""Construct and store a new block from a given block
and a list of (offset, data) 'patches'. Return:
and a data 'patch' applied at offset. Return:
(the hash of the new block, if the block already existed)
"""
if not offdata:
return None, None
blocksize = self.blocksize
if offset >= blocksize or not data:
return None, None
block = self.block_retr((blkhash,))
if not block:
return None, None
block = block[0]
newblock = ''
idx = 0
size = 0
trunc = 0
for off, data in offdata:
if not data:
trunc = 1
break
newblock += block[idx:off] + data
size += off - idx + len(data)
if size >= blocksize:
break
off = size
if not trunc:
newblock += block[size:len(block)]
newblock = block[:offset] + data
if len(newblock) > blocksize:
newblock = newblock[:blocksize]
elif len(newblock) < blocksize:
newblock += block[len(newblock):]
h, a = self.block_stor((newblock,))
return h[0], 1 if a else 0
......
......@@ -76,7 +76,7 @@ class Store(object):
return hashes[0]
def block_update(self, hash, offset, data):
h, e = self.blocker.block_delta(hash, ((offset, data),))
h, e = self.blocker.block_delta(hash, offset, data)
return h
def block_search(self, map):
......
......@@ -154,7 +154,7 @@ class ModularBackend(BaseBackend):
return allowed[start:start + limit]
@backend_method
def get_account_meta(self, user, account, domain, until=None):
def get_account_meta(self, user, account, domain, until=None, include_user_defined=True):
"""Return a dictionary with the account metadata for the domain."""
logger.debug("get_account_meta: %s %s %s", account, domain, until)
......@@ -180,7 +180,7 @@ class ModularBackend(BaseBackend):
meta = {'name': account}
else:
meta = {}
if props is not None:
if props is not None and include_user_defined:
meta.update(dict(self.node.attribute_get(props[self.SERIAL], domain)))
if until is not None:
meta.update({'until_timestamp': tstamp})
......@@ -316,7 +316,7 @@ class ModularBackend(BaseBackend):
return self.node.latest_attribute_keys(node, domain, before, CLUSTER_DELETED, allowed)
@backend_method
def get_container_meta(self, user, account, container, domain, until=None):
def get_container_meta(self, user, account, container, domain, until=None, include_user_defined=True):
"""Return a dictionary with the container metadata for the domain."""
logger.debug("get_container_meta: %s %s %s %s", account, container, domain, until)
......@@ -337,7 +337,9 @@ class ModularBackend(BaseBackend):
if user != account:
meta = {'name': container}
else:
meta = dict(self.node.attribute_get(props[self.SERIAL], domain))
meta = {}
if include_user_defined:
meta.update(dict(self.node.attribute_get(props[self.SERIAL], domain)))
if until is not None:
meta.update({'until_timestamp': tstamp})
meta.update({'name': container, 'count': count, 'bytes': bytes})
......@@ -491,7 +493,7 @@ class ModularBackend(BaseBackend):
return public
@backend_method
def get_object_meta(self, user, account, container, name, domain, version=None):
def get_object_meta(self, user, account, container, name, domain, version=None, include_user_defined=True):
"""Return a dictionary with the object metadata for the domain."""
logger.debug("get_object_meta: %s %s %s %s %s", account, container, name, domain, version)
......@@ -509,7 +511,9 @@ class ModularBackend(BaseBackend):
raise NameError('Object does not exist')
modified = del_props[self.MTIME]
meta = dict(self.node.attribute_get(props[self.SERIAL], domain))
meta = {}
if include_user_defined:
meta.update(dict(self.node.attribute_get(props[self.SERIAL], domain)))
meta.update({'name': name,
'bytes': props[self.SIZE],
'type': props[self.TYPE],
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment