Rooba
/
perkeep
mirror of https://github.com/perkeep/perkeep.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity

website: convert schema docs to markdown 

Refs #720

Change-Id: Iccfaf7384cffa78a51606868f83f4019e9441791
This commit is contained in:
Will Norris 2016-04-26 09:03:41 -07:00
parent 066db0952c
commit 2f79a55baa
33 changed files with 472 additions and 505 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
doc/schema/README.md Normal file
View File

@ -0,0 +1,34 @@
# Schema
At the lowest layer, Camlistore doesn't care what you put in it (everything is
just dumb bytes) and you're free to adopt your own data model. However, the
upper layers of Camlistore standardize on a common schema to represent various
classes of data.
Schema blobs are JSON objects with at least two attributes always set:
`camliVersion`, which is always 1, and `camliType`, which tells you the type of
metadata the blob contains.
Here are some of the data types we've started to formalize a
[JSON](http://json.org/) schema for:
* [Files](files/): traditional filesystems. Files, directories, inodes,
symlinks, etc. Uses the `file`, `directory`, `symlink`, and `inode`
camliTypes.
* [Permanodes](permanode.md): the immutable root "anchor" of mutable Camlistore
objects (see [terminology](terms.md)). Users create signed
[claim](permanode.md#claim) schema blobs which reference a permanode and
define some mutation for the permanode.
Permanodes are used to model many kinds of mutable data, including
mutable files, dynamic directories, and more.
Uses the `permanode` and `claim` camliTypes.
* [Static Sets](objects/static-set.md): Immutable lists of other blobs by
their refs. Indicated by the `static-set` camliType.
* ["Keep" claims](objects/keep.md): Normally, any object that isn't referenced
by a permanode could theoretically be garbage collected. Keep claims prevent
that from happening. Indicated by the `keep` camliType.

23
doc/schema/blob-magic.md Normal file
View File

@ -0,0 +1,23 @@
# Camli Blob Magic
[Note: not totally happy with this yet...]
Ideal Camli JSON blobs should begin with the following 15 bytes:
{"camliVersion"
However, it's acknowledged that some JSON serialization libraries will format
things differently, so additional whitespace should be tolerated.
An ideal camli serializer will strive for the above header, though, by doing
something like:
- removing the "camliVersion" from the object, noting its value (and requiring
it to be present)
- serializing the JSON with an existing JSON serialization library,
- removing the serialized JSON's leading "{" character and prepending the 15
byte header above, as well as the colon and saved version and comma (which can
have whitespace as desired)

26
doc/schema/blob-magic.txt
View File

@ -1,26 +0,0 @@
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Camli Blob Magic
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
[Note: not totally happy with this yet...]
Ideal Camli JSON blobs should begin with the following 15 bytes:
{"camliVersion"
However, it's acknowledged that some JSON serialization libraries will
format things differently, so additional whitespace should be
tolerated.
An ideal camli serializer will strive for the above header, though, by
doing something like:
-- removing the "camliVersion" from the object, noting its value
(and requiring it to be present)
-- serializing the JSON with an existing JSON serialization library,
-- removing the serialized JSON's leading "{" character and prepending
the 15 byte header above, as well as the colon and saved version
and comma (which can have whitespace as desired)

39
doc/schema/bytes.md Normal file
View File

@ -0,0 +1,39 @@
# Bytes Blob
Description of a series of bytes.
A "bytes" is a metadata (JSON) blob to describe blobs. It's a recursive
definition that's able to describe a hash tree, describing very large
blobs (or "files").
A "bytes" blob can be used on its own, but is also used by things like
a "file" schema blob.
{"camliVersion": 1,
"camliType": "bytes",
// Required. Array of contiguous regions of bytes. Zero or more elements.
//
// Each element must have:
// "size": the number of bytes that this element contributes to array of bytes.
// Required, and must be greater than zero.
//
// At most one of:
// "blobRef": where to get the raw bytes from. if this and "bytesRef"
// are missing, the bytes are all zero (e.g. a sparse file hole)
// "bytesRef": alternative to blobRef, where to get the range's bytes
// from, but pointing recursively at a "bytes" schema blob
// describing the range, recursively. large files are made of
// these in a hash tree. it is an error if both "bytesRef"
// and "blobRef" are specified.
//
// Optional:
// "offset": the number of bytes into blobRef or bytesRef to skip to
// get the necessary bytes for the range. usually zero (unspecified)
"parts": [
{"blobRef": "digalg-blobref", "size": 1024},
{"bytesRef": "digalg-blobref", "size": 5000000, "offset": 492 },
{"size": 1000000},
{"blobRef": "digalg-blobref", "size": 10},
]
}

38
doc/schema/bytes.txt
View File

@ -1,38 +0,0 @@
Description of a series of bytes.
A "bytes" is a metadata (JSON) blob to describe blobs. It's a recursive
definition that's able to describe a hash tree, describing very large
blobs (or "files").
A "bytes" blob can be used on its own, but is also used by things like
a "file" schema blob.
{"camliVersion": 1,
"camliType": "bytes",
// Required. Array of contiguous regions of bytes. Zero or more elements.
//
// Each element must have:
// "size": the number of bytes that this element contributes to array of bytes.
// Required, and must be greater than zero.
//
// At most one of:
// "blobRef": where to get the raw bytes from. if this and "bytesRef"
// are missing, the bytes are all zero (e.g. a sparse file hole)
// "bytesRef": alternative to blobRef, where to get the range's bytes
// from, but pointing recursively at a "bytes" schema blob
// describing the range, recursively. large files are made of
// these in a hash tree. it is an error if both "bytesRef"
// and "blobRef" are specified.
//
// Optional:
// "offset": the number of bytes into blobRef or bytesRef to skip to
// get the necessary bytes for the range. usually zero (unspecified)
"parts": [
{"blobRef": "digalg-blobref", "size": 1024},
{"bytesRef": "digalg-blobref", "size": 5000000, "offset": 492 },
{"size": 1000000},
{"blobRef": "digalg-blobref", "size": 10},
]
}

5
doc/schema/claims/README.md Normal file
View File

@ -0,0 +1,5 @@
# Claims
* [Permanode Attributes](attributes.md)
* [Delete Claim](delete.md)
* [Share Claim](share.md)

153
doc/schema/claims/TODO
View File

@ -1,90 +1,99 @@
TODO:
-----
# TODO
Clean this up and/or break into separate files.
{"camliVersion": 1,
"camliType": "claim",
"camliSigner": "....",
"claimDate": "2010-07-10T17:20:03.9212Z", // redundant with data in ascii armored "camliSig",
// but required. more legible. takes precedence over
// any date inferred from camliSig
"permaNode": "dig-xxxxxxx", // what is being modified
"claimType": "set-attribute",
"attribute": "camliContent",
"value": "dig-yyyyyyy",
"camliSig": .........}
{"camliVersion": 1,
"camliType": "claim",
"camliSigner": "....",
"claimDate": "2010-07-10T17:20:03.9212Z", // redundant with data in ascii armored "camliSig",
// but required. more legible. takes precedence over
// any date inferred from camliSig
"permaNode": "dig-xxxxxxx", // what is being modified
"claimType": "set-attribute",
"attribute": "camliContent",
"value": "dig-yyyyyyy",
"camliSig": .........}
claimTypes:
-----------
"add-attribute" (adds a value to a multi-valued attribute (e.g. "tag"))
"set-attribute" (set a single-valued attribute. equivalent to "del-attribute" of "attribute" and then add-attribute)
"del-attribute" (deletes all values of "attribute", if no "value" given, or just the provided "value" if multi-valued)
* "add-attribute" (adds a value to a multi-valued attribute (e.g. "tag"))
* "set-attribute" (set a single-valued attribute. equivalent to "del-attribute"
of "attribute" and then add-attribute)
* "del-attribute" (deletes all values of "attribute", if no "value" given, or
just the provided "value" if multi-valued)
"multi".. atomically do multiple add/set/del from above on potentially different permanodes. looks like:
"multi".. atomically do multiple add/set/del from above on potentially different
permanodes. looks like:
{"camliVersion": 1,
"camliType": "claim",
"claimType": "multi",
"claimDate": "2013-02-24T17:20:03.9212Z",
"claims": [
{"claimType": "set-attribute",
"permanode": "dig-xxxxxx",
"attribute": "foo",
"value": "fooValue"},
{"claimType": "add-attribute",
"permanode": "dig-yyyyy",
"attribute": "tag",
"value": "funny"}
],
"camliSig": .........}
{"camliVersion": 1,
"camliType": "claim",
"claimType": "multi",
"claimDate": "2013-02-24T17:20:03.9212Z",
"claims": [
{"claimType": "set-attribute",
"permanode": "dig-xxxxxx",
"attribute": "foo",
"value": "fooValue"},
{"claimType": "add-attribute",
"permanode": "dig-yyyyy",
"attribute": "tag",
"value": "funny"}
],
"camliSig": .........}
Attribute names:
----------------
camliContent: a permanode "becoming" something. value is pointer to what it is now.
## Attribute names
camliContent: a permanode "becoming" something. value is pointer to what it is
now.
Old notes from July 2010 doc:
-----------------------------
## Old notes from July 2010 doc
Claim types:
permanode-become:
-- implies either:
1) switching from typeless/lifeless virgin pnode into something (dynamic set, filesystem tree, etc)
2) changing versions of that base metadata (new filesystem snapshot)
-- ‘permaNode’ is the thing that is changing
-- ‘contents’ is the current node that represents what permaNode changes to
set-membership: add a blobref to a dynamic set
-- "permaNode" is blobref of the dynamic set
delete-claim: delete another claim (target is claim to delete)
-- "contents" is the claim blobref you’re deleting
{set,add}-attribute:
-- attach a piece of metadata to something.
-- use set-attribute for single-valued attributes only: highest dated claim wins (of trusted person) e.g. "title", "description"
-- use add-attribute for multi-valued things. e.g. "tag"
* permanode-become:
* implies either:
1. switching from typeless/lifeless virgin pnode into something (dynamic
set, filesystem tree, etc)
2. changing versions of that base metadata (new filesystem snapshot)
* 'permaNode' is the thing that is changing
* 'contents' is the current node that represents what permaNode changes to
* set-membership: add a blobref to a dynamic set
* "permaNode" is blobref of the dynamic set
* delete-claim: delete another claim (target is claim to delete)
* "contents" is the claim blobref you’re deleting
* {set,add}-attribute:
* attach a piece of metadata to something.
* use set-attribute for single-valued attributes only: highest dated claim
wins (of trusted person) e.g. "title", "description"
* use add-attribute for multi-valued things. e.g. "tag"
Tagging something:
{"claimType": "add-attribute", //
"attribute": "tag", // utf-8, should have list of valid attributes names, preferrably not made up by us (open social spec?)
"value": "funny", // value that doesn’t have lasting value
"valueRef": "sha1-blobref", // hefty reference to a lasting value
"claimer?": "sha1-of-the-dude-who’s-signing",
"claimDate": "2010-07-10T17:20:03.9212Z",
"claimType", "permanode-become",
"permaNode": "sha1-pnode",
}
{"claimType": "add-attribute", //
"attribute": "tag", // utf-8, should have list of valid attributes names, preferrably not made up by us (open social spec?)
"value": "funny", // value that doesn’t have lasting value
"valueRef": "sha1-blobref", // hefty reference to a lasting value
"claimer?": "sha1-of-the-dude-who’s-signing",
"claimDate": "2010-07-10T17:20:03.9212Z",
"claimType", "permanode-become",
"permaNode": "sha1-pnode",
}
filesystem root claim:
{
"camliVersion": 1,
"camliType": "claim",
// Stuff for camliType "claim":
"claimDate": "2010-07-10T17:20:03.9212Z", // redundant with data in ascii armored "camliSig". TODO: resolve
"claimType", "permanode-become",
{
"camliVersion": 1,
"camliType": "claim",
// Stuff for "permanode-become":
"permaNode": "sha1-pnode",
"contents": "sha1-fs-node"
// Stuff for camliType "claim":
"claimDate": "2010-07-10T17:20:03.9212Z", // redundant with data in ascii armored "camliSig". TODO: resolve
"claimType", "permanode-become",
,"camliSigner": "digalg-blobref-to-ascii-armor-public-key-of-signer",
"camliSig": "......"}
// Stuff for "permanode-become":
"permaNode": "sha1-pnode",
"contents": "sha1-fs-node"
,"camliSigner": "digalg-blobref-to-ascii-armor-public-key-of-signer",
"camliSig": "......"}

2
doc/schema/claims/attributes.txt → doc/schema/claims/attributes.md
View File

@ -1,4 +1,4 @@
Permanode Attributes
# Permanode Attributes
While a permanode can have any arbitrary attributes and values (and
each value can be single-valued or multi-valued), the following are

18
doc/schema/claims/delete.md Normal file
View File

@ -0,0 +1,18 @@
# Delete Claim
A claim can delete a permanode or another claim.
(Un)Deletions are not considered as modifications, so the claimDate of a delete
claim is never considered as a modtime in the context of time constrained
searches.
Example:
{"camliVersion": 1,
"camliType": "claim",
"camliSigner": "sha1-f2b0b7da718b97ce8c31591d8ed4645c777f3ef4",
"claimDate": "2010-07-10T17:20:03.9212Z",
"claimType": "delete",
"target": "sha1-ab6dacb972eeee72df2a846aab7d751b5856a1a0", // the permanode or claim being deleted.
<REQUIRED-JSON-SIGNATURE>
}

13
doc/schema/claims/delete.txt
View File

@ -1,13 +0,0 @@
A claim can delete a permanode or another claim.
(Un)Deletions are not considered as modifications, so the claimDate of a delete claim
is never considered as a modtime in the context of time constrained searches.
-----
{"camliVersion": 1,
"camliType": "claim",
"camliSigner": "sha1-f2b0b7da718b97ce8c31591d8ed4645c777f3ef4",
"claimDate": "2010-07-10T17:20:03.9212Z",
"claimType": "delete",
"target": "sha1-ab6dacb972eeee72df2a846aab7d751b5856a1a0", // the permanode or claim being deleted.
<REQUIRED-JSON-SIGNATURE>
}

34
doc/schema/claims/share.md Normal file
View File

@ -0,0 +1,34 @@
# Share Claim
A share claim makes blob(s) available to others. (that is, parties who are not
the owner of the Camlistore instance).
Example:
{"camliVersion": 1,
// Type of authentication required to access the share. Currently only haveref
// is supported, which means that anyone with the claim blobref can access.
"authType": "haveref",
"camliType": "claim",
"camliSigner": "sha1-f2b0b7da718b97ce8c31591d8ed4645c777f3ef4",
"claimDate": "2014-09-04T20:04:09.193945801Z",
"claimType": "share",
// The blob or search to share. Exactly one of these must be present. It is an
// error to set neither or both.
"target": "sha1-543fbdfdbcb1297af8a4dc7d299c0cb90e2bea0f",
"search": <JSON object of type *search.SearchQuery>,
// If true, anything recursively reachable from target or search is also
// shared. Edges that are guaranteed to be followed for purposes of
// reachability are:
// - blobRef and bytesRef values of camliType="blob|file"
// - members of camliType="static-set"
// Currently reachability is implemented more loosely, but clients should not
// depend on that.
"transitive": false,
<REQUIRED-JSON-SIGNATURE>
}

31
doc/schema/claims/share.txt
View File

@ -1,31 +0,0 @@
A share claim makes blob(s) available to others. (that is, parties who are not
the owner of the Camlistore instance).
-----
{"camliVersion": 1,
// Type of authentication required to access the share. Currently only haveref
// is supported, which means that anyone with the claim blobref can access.
"authType": "haveref",
"camliType": "claim",
"camliSigner": "sha1-f2b0b7da718b97ce8c31591d8ed4645c777f3ef4",
"claimDate": "2014-09-04T20:04:09.193945801Z",
"claimType": "share",
// The blob or search to share. Exactly one of these must be present. It is an
// error to set neither or both.
"target": "sha1-543fbdfdbcb1297af8a4dc7d299c0cb90e2bea0f",
"search": <JSON object of type *search.SearchQuery>,
// If true, anything recursively reachable from target or search is also
// shared. Edges that are guaranteed to be followed for purposes of
// reachability are:
// - blobRef and bytesRef values of camliType="blob|file"
// - members of camliType="static-set"
// Currently reachability is implemented more loosely, but clients should not
// depend on that.
"transitive": false,
<REQUIRED-JSON-SIGNATURE>
}

9
doc/schema/files/README.md Normal file
View File

@ -0,0 +1,9 @@
# Files
* [Common Attributes](common.md)
* [Directory](directory.md)
* [FIFO](fifo.md)
* [File](file.md)
* [Inode](inode.md)
* [Socket](socket.md)
* [Symlink](symlink.md)

26
doc/schema/files/common.md Normal file
View File

@ -0,0 +1,26 @@
# Common Schema Fields
Fields common to files, directories, symlinks, FIFOs and sockets.
{"camliVersion": 1,
"camliType": "...", // one of "file", "directory", "symlink", "fifo", "socket"
// At most one of these may be set. (zero may be present only for large files' subranges,
// represented as a tree of file schemas) But exactly one of these is required for
// top-level files, directories, symlinks, FIFOs, sockets, e.t.c.
"fileName": "if-it-is-utf8.txt", // only for utf-8
"fileNameBytes": [65, 234, 234, 192, 23, 123], // if unknown charset (not recommended)
// Optional:
"unixPermission": "0755", // no octal in JSON, so octal as string
"unixOwnerId": 1000,
"unixOwner": "bradfitz",
"unixGroupId": 500,
"unixGroup": "camliteam",
"unixXattrs": [....], // TBD
"unixMtime": "2010-07-10T17:14:51.5678Z", // UTC-- ISO 8601, as many significant digits as known
"unixCtime": "2010-07-10T17:20:03.9212Z", // UTC-- ISO 8601, best-effort to match unix meaning
// Not recommended to include, but if you must: (atime is a bit silly)
"unixAtime": "2010-07-10T17:14:22.1234Z", // UTC-- ISO 8601
}

12
doc/schema/files/directory.md Normal file
View File

@ -0,0 +1,12 @@
# Directory Schema
{"camliVersion": 1,
"camliType": "directory",
//
// INCLUDE ALL REQUIRED & ANY OPTIONAL FIELDS FROM common.md
//
// Required:
"entries": "digalg-blobref-to-static-set",
}

13
doc/schema/files/directory.txt
View File

@ -1,13 +0,0 @@
Directory schema
{"camliVersion": 1,
"camliType": "directory",
//
// INCLUDE ALL REQUIRED & ANY OPTIONAL FIELDS FROM file-common.txt
//
// Required:
"entries": "digalg-blobref-to-static-set",
}

9
doc/schema/files/fifo.md Normal file
View File

@ -0,0 +1,9 @@
# FIFO Schema
{"camliVersion": 1,
"camliType": "fifo",
//
// INCLUDE ALL REQUIRED & ANY OPTIONAL FIELDS FROM common.md
//
}

9
doc/schema/files/fifo.txt
View File

@ -1,9 +0,0 @@
fifo schema
{"camliVersion": 1,
"camliType": "fifo",
//
// INCLUDE ALL REQUIRED & ANY OPTIONAL FIELDS FROM file-common.txt
//
}

24
doc/schema/files/file-common.txt
View File

@ -1,24 +0,0 @@
Fields common to files, directories, symlinks, FIFOs and sockets
{"camliVersion": 1,
"camliType": "...", // one of "file", "directory", "symlink", "fifo", "socket"
// At most one of these may be set. (zero may be present only for large files' subranges,
// represented as a tree of file schemas) But exactly one of these is required for
// top-level files, directories, symlinks, FIFOs, sockets, e.t.c.
"fileName": "if-it-is-utf8.txt", // only for utf-8
"fileNameBytes": [65, 234, 234, 192, 23, 123], // if unknown charset (not recommended)
// Optional:
"unixPermission": "0755", // no octal in JSON, so octal as string
"unixOwnerId": 1000,
"unixOwner": "bradfitz",
"unixGroupId": 500,
"unixGroup": "camliteam",
"unixXattrs": [....], // TBD
"unixMtime": "2010-07-10T17:14:51.5678Z", // UTC-- ISO 8601, as many significant digits as known
"unixCtime": "2010-07-10T17:20:03.9212Z", // UTC-- ISO 8601, best-effort to match unix meaning
// Not recommended to include, but if you must: (atime is a bit silly)
"unixAtime": "2010-07-10T17:14:22.1234Z", // UTC-- ISO 8601
}

14
doc/schema/files/file.md Normal file
View File

@ -0,0 +1,14 @@
# File Schema
{"camliVersion": 1,
"camliType": "file",
// #include "common.md" # metadata about the file
// #include "../bytes.md" # describes the bytes of the file
// Optional, if linkcount > 1, for representing hardlinks properly.
"inodeRef": "digalg-blobref", // to "inode" blobref, when the link count > 1
}
TODO: Mac/NTFS-style resource forks? perhaps just a "streams" array of
recursive file objects?

15
doc/schema/files/file.txt
View File

@ -1,15 +0,0 @@
File schema
{"camliVersion": 1,
"camliType": "file",
// #include "file-common.txt" # metadata about the file
// #include "../bytes.txt" # describes the bytes of the file
// Optional, if linkcount > 1, for representing hardlinks properly.
"inodeRef": "digalg-blobref", // to "inode" blobref, when the link count > 1
}
// TODO: Mac/NTFS-style resource forks? perhaps just a "streams"
// array of recursive file objects?

14
doc/schema/files/inode.txt → doc/schema/files/inode.md
View File

@ -1,11 +1,11 @@
Inode schema.
# Inode Schema
{"camliVersion": 1,
"camliType": "inode",
"inodeId": 12345 // st_ino
"deviceId": 53, // st_dev
"numLinks": 3, // st_nlink
}
{"camliVersion": 1,
"camliType": "inode",
"inodeId": 12345 // st_ino
"deviceId": 53, // st_dev
"numLinks": 3, // st_nlink
}
This is optional and probably rarely used, but lets two+ files be
represented as hardlinks with each other. If both files point to the

9
doc/schema/files/socket.md Normal file
View File

@ -0,0 +1,9 @@
# Socket Schema
{"camliVersion": 1,
"camliType": "socket",
//
// INCLUDE ALL REQUIRED & ANY OPTIONAL FIELDS FROM common.md
//
}

9
doc/schema/files/socket.txt
View File

@ -1,9 +0,0 @@
socket schema
{"camliVersion": 1,
"camliType": "socket",
//
// INCLUDE ALL REQUIRED & ANY OPTIONAL FIELDS FROM file-common.txt
//
}

18
doc/schema/files/symlink.md Normal file
View File

@ -0,0 +1,18 @@
# Symlink Schema
{"camliVersion": 1,
"camliType": "symlink",
//
// INCLUDE ALL REQUIRED & ANY OPTIONAL FIELDS FROM common.md
//
// Exactly one of:
// If UTF-8:
"symlinkTarget": "../foo/blah",
// If unknown charset & have raw 8-bit filenames and can't convert
// to UTF-8. The array is a mix of UTF-8 and/or non-UTF-8 bytes (0-255).
"symlinkTargetBytes": ["../foo/Am", 233, "lie.jpg"], // e.g. Amélie in ISO-8859-1 when charset unknown
}

18
doc/schema/files/symlink.txt
View File

@ -1,18 +0,0 @@
Symlink schema
{"camliVersion": 1,
"camliType": "symlink",
//
// INCLUDE ALL REQUIRED & ANY OPTIONAL FIELDS FROM file-common.txt
//
// Exactly one of:
// If UTF-8:
"symlinkTarget": "../foo/blah",
// If unknown charset & have raw 8-bit filenames and can't convert
// to UTF-8. The array is a mix of UTF-8 and/or non-UTF-8 bytes (0-255).
"symlinkTargetBytes": ["../foo/Am", 233, "lie.jpg"], // e.g. Amélie in ISO-8859-1 when charset unknown
}

57
doc/schema/index.html
View File

@ -1,57 +0,0 @@
<h1>Schema</h1>
<p>
At the lowest layer, Camlistore doesn't care what you put in it
(everything is just dumb bytes) and you're free to adopt your own
data model. However, the upper layers of Camlistore standardize on
a <a href="/gw/doc/schema">common schema</a> to represent various
classes of data.
</p>
<p>
Schema blobs are JSON objects with at least two attributes always
set: <code>camliVersion</code>, which is always 1,
and <code>camliType</code>, which tells you the type of metadata the
blob contains.
</p>
<p>
Here are some of the data types we've started to formalize
a <a href="http://json.org/">JSON</a> schema for:
</p>
<ul>
<li>
<a href="http://camlistore.org/gw/doc/schema/files">Files</a>:
traditional filesystems. Files, directories, inodes, symlinks,
etc. Uses the <code>file</code>, <code>directory</code>, <code>symlink</code>,
and <code>inode</code> camliTypes.
</li>
<li>
<a href="schema/permanode">Permanodes</a>: the immutable root
"anchor" of mutable Camlistore objects
(see <a href="terms">terminology</a>). Users create
signed <a href="/docs/schema/permanode#claim">claim</a> schema
blobs which reference a permanode and define some mutation for the
permanode.
<br>
Permanodes are used to model many kinds of mutable data, including
mutable files, dynamic directories, and more.
<br>
Uses the <code>permanode</code> and <code>claim</code> camliTypes.
</li>
<li>
<a href="/gw/doc/schema/objects/static-set.txt">Static Sets</a>:
Immutable lists of other blobs by their refs. Indicated by
the <code>static-set</code> camliType.
</li>
<li>
<a href="/gw/doc/schema/objects/keep.txt">"Keep" claims</a>:
Normally, any object that isn't referenced by a permanode could
theoretically be garbage collected. Keep claims prevent that from
happening. Indicated by the <code>keep</code> camliType.
</li>
</ul>

4
doc/schema/objects/README.md Normal file
View File

@ -0,0 +1,4 @@
# Objects
* [Keep Claims](keep.md)
* [Static Set](static-set.md)

10
doc/schema/objects/keep.txt → doc/schema/objects/keep.md
View File

@ -1,3 +1,5 @@
# Keep Object
A signed "keep" edge for GC/indexing purposes. Expresses a user's
intent to keep an object.
@ -9,8 +11,8 @@ those permanodes)
This is just the most explicit way when you're not modeling the data
with permanodes.
{"camliVersion": 1,
"camliType": "keep",
"target": "digalg-blobref-of-thing-to-keep",
<REQUIRED-JSON-SIGNATURE>}
{"camliVersion": 1,
"camliType": "keep",
"target": "digalg-blobref-of-thing-to-keep",
<REQUIRED-JSON-SIGNATURE>}

15
doc/schema/objects/permanode.txt
View File

@ -1,15 +0,0 @@
The idea of a permanode is that it's the anchor from which you build
mutable objects. To serve as a reliable (consistently nameable)
object it must have no mutable state itself.
{"camliVersion": 1,
"camliType": "permanode",
// Required. Any random string, to force the sha1 of this
// node to be unique. Note that the date in the ASCII-armored
// GPG JSON signature will already help it be unique, so this
// doesn't need to be a great random.
"random": "615e05c68c8411df81a2001b639d041f"
<REQUIRED-JSON-SIGNATURE>}

25
doc/schema/objects/static-set.md Normal file
View File

@ -0,0 +1,25 @@
# Static Set schema
Example:
{"camliVersion": 1,
"camliType": "static-set",
// Required.
// May be ordered to unordered, depending on context/needs. If unordered,
// it's recommended but not required to sort the blobrefs.
"members": [
"digalg-blobref-item1", // maybe a file?
"digalg-blobref-item2", // maybe a directory?
"digalg-blobref-item3", // maybe a symlink?
"digalg-blobref-item4", // maybe a permanode?
"digalg-blobref-item5", // ... don't know until you fetch it
"digalg-blobref-item6", // ... and what's valid depends on context
"digalg-blobref-item7", // ... a permanode in a directory would
"digalg-blobref-item8" // ... be invalid, for instance.
]
}
Note: dynamic sets are structured differently, using a permanode and
membership claim nodes. The above is just for presenting a snapshot
of members.

23
doc/schema/objects/static-set.txt
View File

@ -1,23 +0,0 @@
Static set schema
{"camliVersion": 1,
"camliType": "static-set",
// Required.
// May be ordered to unordered, depending on context/needs. If unordered,
// it's recommended but not required to sort the blobrefs.
"members": [
"digalg-blobref-item1", // maybe a file?
"digalg-blobref-item2", // maybe a directory?
"digalg-blobref-item3", // maybe a symlink?
"digalg-blobref-item4", // maybe a permanode?
"digalg-blobref-item5", // ... don't know until you fetch it
"digalg-blobref-item6", // ... and what's valid depends on context
"digalg-blobref-item7", // ... a permanode in a directory would
"digalg-blobref-item8" // ... be invalid, for instance.
]
}
Note: dynamic sets are structured differently, using a permanode and
membership claim nodes. The above is just for presenting a snapshot
of members.

228
doc/schema/permanode.md
View File

@ -1,146 +1,114 @@
<h1>Permanodes</h1>
# Permanodes
<p>
Permanodes are how Camlistore models mutable data on top of an
immutable, content-addressable datastore. The data is modeled using
nodes with two camliTypes: <code>permanode</code> and <code>claim</code>.
</p>
Permanodes are how Camlistore models mutable data on top of an immutable,
content-addressable datastore. The data is modeled using nodes with two
camliTypes: `permanode` and `claim`.
<h2 id="permanode">Permanode</h2>
<p>
A permanode is an anchor from which you build mutable objects. To
serve as a reliable (consistently nameable) object, it must have no
mutable state itself. In fact, a permanode is really just a
<a href="/gw/doc/json-signing/json-signing.txt">signed</a> random
number.
</p>
<pre>
{"camliVersion": 1,
"camliType": "permanode",
## Permanode
// Required. Any random string, to force the digest of this
// node to be unique. Note that the date in the ASCII-armored
// GPG JSON signature will already help it be unique, so this
// doesn't need to be a great random.
"random": "615e05c68c8411df81a2001b639d041f"
A permanode is an anchor from which you build mutable objects. To serve as a
reliable (consistently nameable) object, it must have no mutable state itself.
In fact, a permanode is really just a [signed](../json-signing/) random number.
<a href="/gw/doc/json-signing/json-signing.txt">&lt;REQUIRED-JSON-SIGNATURE&gt;</a>}
</pre>
{"camliVersion": 1,
"camliType": "permanode",
<h2 id="claim">Claim</h2>
<p>
A claim is any signed JSON schema blob. One common use is modifying
"attributes" on a permanode. The state of a permanode is the result
of combining all attribute-modifying claims which reference it, in
order. Claim nodes look something like this:
</p>
<pre>
{"camliVersion": 1,
"camliType": "claim",
"camliSigner": "....",
"claimDate": "2010-07-10T17:20:03.9212Z", // redundant with data in ascii armored "camliSig",
// but required. more legible. takes precedence over
// any date inferred from camliSig
"permaNode": "sha1-xxxxxxx", // what is being modified
"claimType": "set-attribute",
"attribute": "camliContent",
"value": "sha1-yyyyyyy",
"camliSig": .........}
</pre>
<p>
All claims must be <a href="/gw/doc/json-signing/json-signing.txt">signed</a>.
<p>
The anagrammatical property <code>claimType</code> defines what the
claim does, and is one of the following:
</p>
<ul>
<li>
<code>add-attribute</code>: adds a value to a multi-valued attribute
(e.g. "tag")
</li>
// Required. Any random string, to force the digest of this
// node to be unique. Note that the date in the ASCII-armored
// GPG JSON signature will already help it be unique, so this
// doesn't need to be a great random.
"random": "615e05c68c8411df81a2001b639d041f"
<li>
<code>set-attribute</code>: set a single-valued attribute. equivalent
to "del-attribute" of "attribute" and then add-attribute.
</li>
<REQUIRED-JSON-SIGNATURE>}
<li>
<code>del-attribute</code>: deletes all values of "attribute", if no
"value" given, or just the provided "value" if multi-valued
</li>
## Claim
<li>
<code>multi</code>: atomically do multiple add/set/del from above on
potentially different permanodes. looks like:
<pre>
{"camliVersion": 1,
"camliType": "claim",
"claimType": "multi",
"claimDate": "2013-02-24T17:20:03.9212Z",
"claims": [
{"claimType": "set-attribute",
"permanode": "sha1-xxxxxx",
"attribute": "foo",
"value": "fooValue"},
{"claimType": "add-attribute",
"permanode": "sha1-yyyyy",
"attribute": "tag",
"value": "funny"}
],
"camliSig": .........}
</pre>
</li>
</ul>
A claim is any signed JSON schema blob. One common use is modifying
"attributes" on a permanode. The state of a permanode is the result
of combining all attribute-modifying claims which reference it, in
order. Claim nodes look something like this:
<h2 id="attributes">Attributes</h2>
<p>
A permanode can have any attribute you like, but here are the ones
that currently mean something to Camlistore:
</p>
<ul>
<li>
<code>tag</code>: A set of zero or more keywords (or phrases) indexed
completely, for searching by tag. No HTML.
</li>
<li>
<code>title</code>: A name given to the permanode. No HTML.
</li>
<li>
<code>description</code>: An account of the permanode. It may include but is
not limited to: an abstract, a table of contents, or a free-text
account of the resource. No HTML.
</li>
<li>
<code>camliContent</code>: A reference to another blob. If a permanode
has this attribute, it's considered a pointer to its camliContent
value.
</li>
<li>
<code>camliMember</code>: A reference to another permanode. This
indicates that the referenced permanode is a dynamic set, and
we're a part of it.
</li>
<li>
<code>camliPath*</code>: camliPath attributes are set on permanodes
which represent dynamic directories. If a permanode has attributes:
<pre>
camliPath:dir2 = $blobref_dir2_permandode
camliPath:bar.txt = $blobref_bartxt_permanode
</pre>
{"camliVersion": 1,
"camliType": "claim",
"camliSigner": "....",
"claimDate": "2010-07-10T17:20:03.9212Z", // redundant with data in ascii armored "camliSig",
// but required. more legible. takes precedence over
// any date inferred from camliSig
"permaNode": "sha1-xxxxxxx", // what is being modified
"claimType": "set-attribute",
"attribute": "camliContent",
"value": "sha1-yyyyyyy",
"camliSig": .........}
All claims must be [signed](../json-signing/).
The anagrammatical property `claimType` defines what the claim does, and is one
of the following:
* `add-attribute`: adds a value to a multi-valued attribute (e.g. "tag")
* `set-attribute`: set a single-valued attribute. equivalent to "del-attribute"
of "attribute" and then add-attribute.
* `del-attribute`: deletes all values of "attribute", if no "value" given, or
just the provided "value" if multi-valued
* `multi`: atomically do multiple add/set/del from above on potentially
different permanodes. looks like:
{"camliVersion": 1,
"camliType": "claim",
"claimType": "multi",
"claimDate": "2013-02-24T17:20:03.9212Z",
"claims": [
{"claimType": "set-attribute",
"permanode": "sha1-xxxxxx",
"attribute": "foo",
"value": "fooValue"},
{"claimType": "add-attribute",
"permanode": "sha1-yyyyy",
"attribute": "tag",
"value": "funny"}
],
"camliSig": .........}
## Attributes
A permanode can have any attribute you like, but here are the ones that
currently mean something to Camlistore:
* `tag`: A set of zero or more keywords (or phrases) indexed completely, for
searching by tag. No HTML.
* `title`: A name given to the permanode. No HTML.
* `description`: An account of the permanode. It may include but is not limited
to: an abstract, a table of contents, or a free-text account of the resource.
No HTML.
* `camliContent`: A reference to another blob. If a permanode has this
attribute, it's considered a pointer to its camliContent value.
* `camliMember`: A reference to another permanode. This indicates that the
referenced permanode is a dynamic set, and we're a part of it.
* `camliPath`: camliPath attributes are set on permanodes which represent
dynamic directories. If a permanode has attributes:
camliPath:dir2 = $blobref_dir2_permandode
camliPath:bar.txt = $blobref_bartxt_permanode
It will appear as a directory containing "dir2" and
"bar.txt".
<br>
These are used by a few things, including the web UI, the
"publish" code (declaring you want a photo at a URL and then the
HTTP front end resolving each directory link in
http://myhostname.com/pics/x/y/x/funny.jpg), and the FUSE
<http://myhostname.com/pics/x/y/x/funny.jpg>), and the FUSE
read/write filesystem code.
</li>
<li>
<code>camliRoot</code>: A root name for the permanode. This will cause it to
show up as a named folder in the FUSE filesystem under <code>roots/</code>.
Creating a directory in <code>roots/</code> will cause a new permanode to be
created with this attr set. You can also browse roots in the web UI.
</li>
</ul>
* `camliRoot`: A root name for the permanode. This will cause it to show up as a
named folder in the FUSE filesystem under <code>roots/</code>. Creating a
directory in <code>roots/</code> will cause a new permanode to be created with
this attr set. You can also browse roots in the web UI.