2011-05-03 23:40:50 +00:00
<!DOCTYPE html>
<!--
2011-05-06 12:36:59 +00:00
Google I/O 2011 HTML slides template
2011-05-03 23:40:50 +00:00
URL: http://code.google.com/p/io-2011-slides/
-->
< html >
< head >
< title > Camlistore< / title >
< meta charset = 'utf-8' / >
< script src = 'slides.js' > < / script >
< / head >
< style >
/* Your individual styles here, or just use inline styles if that’
what you want. */
2011-05-04 23:11:51 +00:00
.smaller {
font-size: 80%;
}
2011-05-03 23:40:50 +00:00
2011-05-04 23:11:51 +00:00
ul li ul {
margin-top: 1.5em;
margin-bottom: 1em;
}
ul li ul li {
margin-top: 1em;
font-size: 80%;
}
ul li ul.dense li {
margin-top: 0em;
margin-bottom: 0em;
font-size: 80%;
}
h1.center {
text-align: center;
font-style: italic;
}
2011-05-03 23:40:50 +00:00
< / style >
< body style = 'display: none' >
< section class = 'slides layout-regular' >
<!-- Your slides (<article>s) go here. Delete or comment out the
slides below. -->
< article >
< h1 >
Camlistore
< / h1 >
< p >
Brad Fitzpatrick
< br >
2011-05-07
< / p >
2011-05-29 22:30:01 +00:00
< p > < i > < small > (use arrow keys or PgUp/PgDown to move slides)< / small > < / i > < / p >
2011-05-03 23:40:50 +00:00
< / article >
2011-05-04 23:11:51 +00:00
< article >
< h3 >
Who am I?
< / h3 >
< ul class = 'nobuild' >
< li >
Brad Fitzpatrick < brad@danga.com>
< / li >
< li > Perl Hacker since 1994< / li >
< li > Projects:
< table > < tr valign = 'top' >
< th > Danga / 6A (Perl)< / th >
< th > Google< / th >
< / tr >
< td class = 'nobuild' >
< div > LiveJournal< / div >
< div > memcached< / div >
< div > Perlbal< / div >
< div > MogileFS< / div >
< div class = 'blue' > OpenID< / div >
< / td >
< td class = 'nobuild' >
< div > < nobr > Social Graph API (< span class = 'blue' > XFN / FOAF< / a > )< / nobr > < / div >
< div class = 'blue' > WebFinger< / div >
< div class = 'blue' > PubSubHubbub< / div >
< div > Android< / div >
< div > Go< / div >
< / td >
< / table >
< div style = 'font-size: 70%; margin-top: 1em' > * < span class = 'blue' > decentralized social< / span > < / div >
< / li >
< / ul >
< / article >
< article >
< h3 >
But why am I in Brazil?
< / h3 >
< ul class = 'nobuild' >
< li >
< i > "Hey, want to come speak at a Perl conference in Brazil?"< / i >
< / li >
< li > "Yes, totally, but... I don't write much Perl these days. :-("< / li > <!-- " -->
< li style = "margin-top: 2em" > < i > "You could speak on memcached."< / i > < / li >
< li > "But that's an old topic, no?"< / li >
< li style = "margin-top: 2em" > < i > "You have any new project you're excited about?"< / i > < / li >
< / li >
< / ul >
< / article >
< article >
< h1 align = 'center' >
Camlistore!
< / h1 >
< / article >
< article >
< h3 >
Camlistore
< / h1 >
< ul >
< li > New open source project< / li >
< li > Almost a year old< / li >
< li > Still in development< / li >
< li > Starting to be useful :-)< / li >
< li > Hard to easily describe...< / li >
< / article >
< article >
< q >
Camlistore is a way to store, sync, share, model and back up content
< / q >
< div class = 'author' >
camlistore.org
< / div >
< / article >
< article >
< h3 >
Motivation
< / h3 >
< ul >
< li > I've written too many Content Management Systems
< ul >
< li > blogs, comments, photos, emails, backups, scanned paperwork, ...< / li >
< li > is a scanned photo a scan, a photo, or a blog post? who cares.< / li >
< li > write < b > one CMS to rule them all< / b > < / li >
< li > ... or at least a good framework for higher-level CMSes< / li >
< / ul >
< / li >
< / ul >
< / article >
< article >
< h3 >
Motivation (cont)
< / h3 >
< ul >
2011-05-05 22:06:49 +00:00
< li > I still want to help solve the Decentralized Social Network Problem
2011-05-04 23:11:51 +00:00
< ul >
< li > protocols, not companies< / li >
< li > gmail, hotmail: hosted versions of SMTP, IMAP< / li >
< li > ... but I can run my own SMTP/IMAP server if I want.< / li >
< li > ... or change my SMTP/IMAP provider< / li >
< / ul >
< / li >
< / ul >
< / article >
< article >
< h3 >
Motivation (cont)
< / h3 >
< ul >
< li > I wanted something conceptually simple.< / li >
< li > HTTP interfaces, not language-specific< / li >
< li > I use lots of machines; don't want to think about sync or conflicts.< / li >
< li > Data archaeology: should be easy and obvious to
reconstruct in 20 or 100 years< / li >
< / ul >
< / article >
< article >
< h3 >
2011-05-04 23:43:50 +00:00
The Product
2011-05-04 23:11:51 +00:00
< / h3 >
< ul >
< li > one private dumping ground to store anything< / li >
2011-05-05 22:06:49 +00:00
< li > backups, filesystems, objects, photos, likes, bookmarks, shares, my website, ...< / li >
2011-05-04 23:11:51 +00:00
< li > live backup my phone< / li >
< li > live replicate / sync my dumping group between my house & laptop & Amazon & Google< / li >
< li > web UI (ala gmail, docs.google.com, etc) or FUSE filesystem< / li >
2011-05-04 23:43:50 +00:00
< li > Easy for end-users; powerful for dorks< / li >
2011-05-04 23:11:51 +00:00
< / ul >
< / article >
2011-05-05 22:06:49 +00:00
< article >
< h3 >
Security Model
< / h3 >
< ul >
< li > < i > < b > your< / b > < / i > private repo, for life< / li >
< li > everything private by default< / li >
< li > grant access to specific objects/trees with friends or the world< / li >
< li > web UI or CLI tools let you share< / li >
< / ul >
< / article >
2011-05-04 23:11:51 +00:00
< article >
< h1 class = 'center' >
So what's with the silly name?
< / h1 >
< / article >
< article >
< h3 >
Camlistore
< / h3 >
< ul >
< li > Content-< / li >
< li > Addressable< / li >
< li > Multi-< / li >
< li > Layer-< / li >
< li > Indexed< / li >
< li > Storage< / li >
< / ul >
< / article >
2011-05-03 23:40:50 +00:00
2011-05-04 23:11:51 +00:00
< article >
< h3 >
Content-Addressable
< / h3 >
< ul >
< li > At the core, everything is stored & addressed by its digest (e.g. SHA1, MD5, etc)< / li >
2014-03-19 00:46:17 +00:00
< li > e.g. < code class = 'smaller' > "sha1-0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33"< / code > for the blob < code class = 'smaller' > "foo"< / code > < / li >
2011-05-04 23:11:51 +00:00
< li > Great properties:
< ul >
< li > no versions of content: change it changes the new digest too< / li >
< li > no versions: no sync conflicts< / li >
< li > no versions: perfect caching (have it or don't)< / li >
< / ul >
< / li >
< / ul >
< / article >
2011-05-03 23:40:50 +00:00
2011-05-04 23:11:51 +00:00
< article >
< h3 >
Multi-Layer, Indexed
< / h3 >
< ul >
< li > Unix philosophy: small pieces with well-defined interfaces that can be chained or composed< / li >
< li > Camlistore pieces include:
< ul class = 'dense' >
< li style = 'margin-top: 1em' > Blob storage: memory, disk, S3, Google, MySQL index, etc< / li >
< li > Schema< / li >
< li > Signing< / li >
< li > Replication< / li >
< li > Indexing: (e.g. replicate from disk to MySQL index)< / li >
< li > Search< / li >
< li > HTML UI< / li >
< / ul >
< / li >
< / ul >
< / article >
2011-05-05 00:38:15 +00:00
< article >
< h3 > Logically< / h3 >
< img src = 'arch.png' width = '100%' / >
< / article >
< article >
< h3 > In reality< / h3 >
< ul >
< li > End-users: use a hosted version< / li >
< li > Dorks: single server binary with all the logical pieces< / li >
< / ul >
< / article >
2011-05-05 22:06:49 +00:00
< article >
< h2 >
From the bottom up...
< / h2 >
< / article >
< article >
< h2 >
Blob Server
< / h2 >
< / article >
< article >
< h3 > Blob Server: how dumb it is< / h3 >
< ul >
< li > "Blob" == zero or more bytes. < i class = 'red' > no< / i > meta-data< / li >
< li > private operations, to owner of data only:< / li >
< ul >
< li class = 'green' > get(blobref) → blob< / li >
< li class = 'green' > stat(blobref+) → [(blobref, size), ...]< / li >
< li class = 'green' > put(blobref, blob)< / li >
< li class = 'green' > enumerate(..) → [(blobref, size)...] (sorted by blobref)< / li >
< / ul >
< li > no public (non-owner) access< / li >
2014-03-19 00:46:17 +00:00
< li > HTTP interface: < code > GET /camli/sha1-xxxxxxx HTTP/1.1< / code > < / li >
2011-05-05 22:06:49 +00:00
< li > < span class = 'green' > delete(blobref)< / span > is disabled by default, privileged op for GC or replication queues only< / li >
< / ul >
< / article >
< article >
< h3 > Blob Server: seriously, no metadata< / h3 >
< ul >
< li > no filenames< / li >
< li > no "mime types"< / li >
< li > no "{create,mod,access} time"< / li >
< li > size is implicit< / li >
< li > blob: just some bytes< / li >
< li > metadata? layers above.< / li >
< / ul >
< / article >
< article >
< h1 class = 'center' >
Uh, what can you do with that?
< / h1 >
< / article >
< article >
< h3 > Uh, what can you do with that?< / h3 >
< ul >
< li > with just a blob server?< / li >
< li > not much< / li >
< li > but let's start with an easy example...< / li >
< / ul >
< / article >
< article >
< h1 class = 'center' >
Filesystem Backups
< / h1 >
< / article >
< article >
< h3 > Filesystem Backups< / h3 >
< ul >
< li > previous project: brackup
< ul >
< li > good: Perl, slide/dice/encrypt S3 backup, content-addressed, good iterative backups< / li >
< li > bad: large (several MB) "backup manifest" text files
< / ul >
< / li >
< li > fossil/venti, git, etc: directories content-addressed by content of their children, hash trees, etc< / li >
< li > git: "tree objects", "commmit objects", etc< / li >
< li > Camlistore: "schema blobs"< / li >
< / ul >
< / article >
< article >
< h3 > Schema: how to model your content< / h3 >
< ul >
< li > Camlistore defines < i > one possible< / i > schema< / li >
< li > but blobserver doesn't know about it all< / li >
< li > tools generate schema,< / li >
< li > indexer + search understand the schema.< / li >
< / ul >
< / article >
< article >
< h3 > Schema Blobs< / h3 >
< ul >
< li > so if all blobs are just dumb blobs of bytes with no metadata,< / li >
< li > how do you store metadata?< / li >
< li > as blobs themselves!< / li >
< / ul >
< / article >
< article >
< h3 > Minimal Schema Blob< / h3 >
< section >
< pre > {
"camliVersion": 1,
"camliType": "whatever"
}< / pre >
< / section >
< p > Whitespace doesn't matter. Just must be valid JSON in its
entirety. Use whatever JSON libraries you've got.< / p >
2014-03-19 00:46:17 +00:00
< p > That one is named< br / > < code class = 'smaller' > sha1-19e851fe3eb3d1f3d9d1cefe9f92c6f3c7d754f6< / code > < / p >
< p > or perhaps: < code class = 'smaller' > sha512-2c6746aba012337aaf113fd63c24d994a0703d33eb5d6ed58859e45dc4e02dcf< br / > dae5c4d46c5c757fb85d5aff342245fe4edb780c028a6f3c994c1295236c931e< / code > < / p >
2011-05-05 22:06:49 +00:00
< / article >
2011-05-03 23:40:50 +00:00
<!-- END -->
< article >
2011-05-05 22:06:49 +00:00
< h3 > Schema blob; type "file"< / h3 >
< section > < pre > {"camliVersion": 1,
< span style = 'background: #fff' > "camliType": "file",< / span >
"fileName": "foo.dat",
"unixPermission": "0644",
...,
"size": 6000133,
"contentParts": [
{"blobRef": "sha1-...dead", "size": 111},
{"blobRef": "sha1-...beef", "size": 5000000, "offset": 492 },
{"size": 1000000},
{"blobRef": "digalg-blobref", "size": 22},
]
}< / pre > < / section >
< / article >
< article >
< h3 > Schema blob; type "directory"< / h3 >
< section > < pre > {"camliVersion": 1,
< span style = 'background: #fff' > "camliType": "directory",< / span >
"fileName": "foodir",
"unixPermission": "0755",
...,
"entries": < span style = 'background: #fff' > "sha1-c3764bc2138338d5e2936def18ff8cc9cda38455"< / span >
}< / pre > < / section >
< / article >
< article >
< h3 > Schema blob; type "static-set"< / h3 >
< section > < pre > {"camliVersion": 1,
< span style = 'background: #fff' > "camliType": "static-set",< / span >
"members": [
"sha1-xxxxxxxxxxxx",
"sha1-xxxxxxxxxxxx",
"sha1-xxxxxxxxxxxx",
"sha1-xxxxxxxxxxxx",
"sha1-xxxxxxxxxxxx",
"sha1-xxxxxxxxxxxx",
]
}< / pre > < / section >
2011-05-03 23:40:50 +00:00
< / article >
< article >
< h3 >
2011-05-05 22:06:49 +00:00
Backup a directory...
2011-05-03 23:40:50 +00:00
< / h3 >
2011-05-05 22:06:49 +00:00
< section > < pre > $ camput --file $HOME
sha1-8659a52f726588dc44d38dfb22d84a4da2902fed< / pre > < / section >
< p > (like git/hg/fossil, that identifier represents everything down.)< / p >
< p > Iterative backups are cheap, easy identifier to share, etc< / p >
< p > But how will you remember that identifier? (later)< / p >
2011-05-03 23:40:50 +00:00
< / article >
2011-05-05 22:06:49 +00:00
< article >
2011-05-03 23:40:50 +00:00
< h3 >
2011-05-05 22:06:49 +00:00
But what about mutable data?
2011-05-03 23:40:50 +00:00
< / h3 >
2011-05-05 22:06:49 +00:00
< ul >
< li > immutable data is easy to represent & reference< / li >
< ul >
2014-03-19 00:46:17 +00:00
< li > < code class = 'smaller' > sha1-8659a52f726588dc44d38dfb22d84a4da2902fed< / code > is an immutable snapshot< / li >
2011-05-05 22:06:49 +00:00
< / ul >
< li > how to represent mutable data in an immutable, content-addressed world?< / li >
< li > how to share a reference to a mutable object when changing an object mutates its name?< / li >
< / ul >
< / article >
< article >
< h1 class = 'center' >
Objects & "Permanodes"
< / h1 >
< / article >
< article >
< h3 >
Terminology
< / h3 >
< ul >
< li > < span class = 'red' > blob< / span > : just dumb, immutable series of bytes< / li >
< li > < span class = 'red' > schema blob< / span > : a blob that's a valid JSON object w/ camliVersion & camliType< / li >
< li > < span class = 'red' > signed schema blob< / span > aka "< span class = 'red' > claim< / span > ": a schema blob with an embedded OpenPGP signature< / li >
< li > < span class = 'red' > object< / span > : something mutable. represented as an anchor "< span class = 'blue' > permanode< / span > " + a set of mutations (< span class = 'blue' > claims< / span > )< / li >
< li > < span class = 'red' > permanode< / span > : a stable reference. an anchor. just a < span class = 'blue' > signed schema blob< / span > , but of almost no content...< / li >
< / ul >
< / article >
< article >
< h3 >
Permanode
< / h3 >
< section > < pre > < span style = 'font-weight: bold' class = 'blue' > $ camput --permanode< / span >
sha1-ea799271abfbf85d8e22e4577f15f704c8349026
< span style = 'font-weight: bold' class = "blue" > $ camget sha1-ea799271abfbf85d8e22e4577f15f704c8349026< / span >
< span style = "background: #ff7" > {"camliVersion": 1,
"camliSigner": "sha1-c4da9d771661563a27704b91b67989e7ea1e50b8",
< span style = 'font-weight: bold' > "camliType": "permanode"< / span > ,
"random": "oj)r}$Wa/[J|XQThNdhE"< / span >
,"camliSig":"iQEcBAABAgAGBQJNRxceAAoJEGjzeDN/6vt8ihIH/Aov7FRIq4dODAPWGDwqL
1X9Ko2ZtSSO1lwHxCQVdCMquDtAdI3387fDlEG/ALoT/LhmtXQgYTt8QqDxVdu
EK1or6/jqo3RMQ8tTgZ+rW2cj9f3Q/dg7el0Ngoq03hyYXdo3whxCH2x0jajSt4RCc
gdXN6XmLlOgD/LVQEJ303Du1OhCvKX1A40BIdwe1zxBc5zkLmoa8rClAlHdqwo
gxYFY4cwFm+jJM5YhSPemNrDe8W7KT6r0oA7SVfOan1NbIQUel65xwIZBD0ah
CXBx6WXvfId6AdiahnbZiBup1fWSzxeeW7Y2/RQwv5IZ8UgfBqRHvnxcbNmScrzl
p3V3ZoY"}< / pre > < / section >
< / article >
< article >
< h3 >
Backup a directory...
< / h3 >
< section > < pre > < span style = 'font-weight: bold' > $ camput --file $HOME< / span >
sha1-8659a52f726588dc44d38dfb22d84a4da2902fed
< span style = 'font-weight: bold' > $ camput --permanode --file $HOME< / span >
sha1-ea799271abfbf85d8e22e4577f15f704c8349026
< span style = 'font-weight: bold' > $ camput --permanode --name="Brad's home directory" --file $HOME< / span >
sha1-ea799271abfbf85d8e22e4577f15f704c8349026< / pre > < / section >
< ul >
< li > all the file data blobs, file/dir schema blobs,< / li >
< li > a new permanode, owned by you< / li >
< li > a mutation: permanode's content attribute == directory root< / li >
< li > a mutation: permanode's name attribute == "Brad's home directory"< / li >
< / ul >
< / article >
< article class = 'fill' >
< p > < img src = "fsbackup.png" height = "100%" / > < / p >
< / article >
2011-05-07 02:18:53 +00:00
< article >
< h3 > Aside: Garbage Collection< / h3 >
< ul >
< li > Permanodes are (optionally) GC roots,< / li >
< li > or anything signed by you.< / li >
< li > If not a blob isn't reachable by a signed root, can be deleted.< / li >
< li > If you want to keep a plain "dumb" blob, you should create a "keep" claim for it, or a permanode.< / li >
< / ul >
< / article >
2011-05-05 22:06:49 +00:00
< article >
< h1 class = 'center' >
Modeling non-filesystem objects
< / h1 >
< / article >
< article >
< h3 > Example: a photo gallery< / h3 >
< ul >
< li > Photos are objects< / li >
< li > Galleries (sets) are objects< / li >
< li > Photos are members of galleries< / li >
< li > Photos & galleries have attributes (single-valued: "title", multi-valued: "tag")< / li >
< li > Photos might be updated over time:
< ul >
< li > EXIF GPS updated, cropping, white balance< / li >
< li > don't want to break links!< / li >
< / ul >
< / li >
< / ul >
< / article >
< article class = 'fill' >
< p > < img src = "blobjects.png" width = "100%" / > < / p >
< / article >
< article >
< h1 class = 'center' >
How to make sense of that?
< / h1 >
2011-05-03 23:40:50 +00:00
< / article >
2011-05-05 22:50:16 +00:00
< article >
< h1 class = 'center' >
Indexing & Search
< / h1 >
< / article >
2011-05-05 22:06:49 +00:00
2011-05-03 23:40:50 +00:00
< article >
< h3 >
2011-05-05 22:50:16 +00:00
Indexing: summary
2011-05-03 23:40:50 +00:00
< / h3 >
2011-05-05 22:50:16 +00:00
< p style = 'margin-top: 2em' > For each blob, build an index of:
2011-05-03 23:40:50 +00:00
< ul >
2011-05-05 22:50:16 +00:00
< li > directed graph of inter-blob references< / li >
< li > (permanode, time) => resolved attributes< / li >
< li > (permanode, time) => set memberships< / li >
< li > etc...< / li >
2011-05-03 23:40:50 +00:00
< / ul >
< / article >
< article >
< h3 >
2011-05-05 22:50:16 +00:00
Indexing & Replication
2011-05-03 23:40:50 +00:00
< / h3 >
2011-05-05 22:50:16 +00:00
< ul >
< li > indexing is real-time, no polling< / li >
< li > MySQL index speaks the blob server protocol< / li >
2011-05-07 02:18:53 +00:00
< li > just replicate < i > to< / i > the index (MySQL, etc) just like other blob servers (Amazon S3, etc)< / li >
2011-05-03 23:40:50 +00:00
< / ul >
2011-05-05 22:50:16 +00:00
< center > < img src = 'repl.png' / > < / center >
2011-05-03 23:40:50 +00:00
< / article >
2011-05-07 02:18:53 +00:00
< article >
< h3 >
Replication Implementation
< / h3 >
< ul >
2014-03-19 00:46:17 +00:00
< li > cold bootstrap: < code class = 'green' > enumerate()< / code > (sorted) all blobs from < code class = 'red' > src< / code > and < code class = 'red' > dst< / code > , copy all blobs that < code > dst< / code > doesn't have.< / code >
< li > more efficient: use multiple machines, starting at < code class = 'blue' > sha1-0*< / code > , < code class = 'blue' > sha1-1*< / code > , < code class = 'blue' > sha1-2*< / code > , ... etc< / li >
< li > once in-sync, for each < code class = 'red' > (src, dst)< / code > replication pair, keep a < code class = 'red' > src_to_dst_QUEUE< / code > namespace on < code class = 'red' > src< / code > ,< / li >
< li > all new blobs to < code class = 'red' > src< / code > also go into < code class = 'red' > src_to_dst_QUEUE< / code > (refcount, hardlink, etc)< / li >
< li > real-time watch < code class = 'red' > src_to_dst_QUEUE< / code > & replicate & delete from the queue. or re-enumerate just the queue.
2011-05-07 02:18:53 +00:00
< / ul >
< / article >
2011-05-05 22:50:16 +00:00
< article >
2011-05-03 23:40:50 +00:00
< h3 >
2011-05-05 22:50:16 +00:00
Search
2011-05-03 23:40:50 +00:00
< / h3 >
< ul >
2011-05-05 22:50:16 +00:00
< li > Permanodes created by $who, sorted by date desc, type "photo", tagged "funny"< / li >
< li > My recent backups with attribute "hostname" == "camlistore.org",< / l >
< li > All friends' galleries in which this photo appears,< / li >
< li > etc...< / li >
2011-05-03 23:40:50 +00:00
< / ul >
2011-05-05 22:50:16 +00:00
< p > ...similar to your email, or docs.google.com. "My stuff" or "My bookmarks".< / p >
2011-05-03 23:40:50 +00:00
< / article >
< article >
< h3 >
2011-05-05 22:50:16 +00:00
Privacy Model
2011-05-03 23:40:50 +00:00
< / h3 >
2011-05-05 22:50:16 +00:00
< ul >
< li > all your blobs & objects & searches are private< / li >
< li > nothing is public by default< / li >
< / ul >
2011-05-03 23:40:50 +00:00
< / article >
2011-05-05 22:50:16 +00:00
< article >
< h1 >
What if you want to share with friends, or globally publish something?
< / h1 >
< / article >
< article >
2011-05-03 23:40:50 +00:00
< h3 >
2011-05-05 22:50:16 +00:00
Sharing & Share Blobs
2011-05-03 23:40:50 +00:00
< / h3 >
2011-05-07 02:18:53 +00:00
< p > the act of sharing involves creating a new < span class = 'red' > share claim< / span > , just another blob, signed.< / p >
2014-03-19 00:46:17 +00:00
< p > here is: < a href = "http://camlistore.org:3179/camli/sha1-071fda36c1bd9e4595ed16ab5e2a46d44491f708" > < code class = 'smaller' > sha1-071fda36c1bd9e4595ed16ab5e2a46d44491f708< / code > < / a > :< / p >
2011-05-07 02:18:53 +00:00
< section > < pre > {"camliVersion": 1,
"authType": < span style = "background: #fff" > "haveref"< / span > ,
"camliSigner": "sha1-f019d17dd308eebbd49fd94536eb67214c2f0587",
"camliType": "share",
"target": "< a style = "background: #fff" href = "http://camlistore.org:3179/camli/sha1-0e5e60f367cc8156ae48198c496b2b2ebdf5313d" > sha1-0e5e60f367cc8156ae48198c496b2b2ebdf5313d< / a > ",
"transitive": < span style = "background: #fff" > true< / span >
,"camliSig":"iQEcBAABAgAGBQJNQJGuAAoJEIUeCLJL7Fq1EuAIAL/nGoX8caGaANnam0bcIQT7C61wXMRW4qCCaFW+w67ys5z4ztfnTPKwL9ErzMF8Hd32Xe/bVcF6ZL38x/axqI7ehxN8lneKGQNoEdZDA9i752aAr0fkAba6eDehoOj9F4XxOzk3iVrq445jEXtu/+twamHV3UfRozWK1ZQb57dM+cRff47M/Y6VIBRSgW2BrABjuBs8G6PiKxycgh1mb+RL8f9KG+HB/yFuK37YJqZ0zU2OTRp6ELiOgTxbeg99koV9Duy4f4mQgxQgli46077Sv/ujzIeVbmdFL3OenGEzQnyKG0fhf8fa5WkED0XfH7zibAHLiSq3O7x11Q0406U==ANug"}< / pre > < / section >
Target w/ ?via= parameter: < a href = "http://camlistore.org:3179/camli/sha1-0e5e60f367cc8156ae48198c496b2b2ebdf5313d?via=sha1-071fda36c1bd9e4595ed16ab5e2a46d44491f708" > sha1-0e5e60f?via=sha1-071fda< / a > & < a href = "http://camlistore.org:3179/camli/sha1-3dc1d1cfe92fce5f09d194ba73a0b023102c9b25?via=sha1-071fda36c1bd9e4595ed16ab5e2a46d44491f708,sha1-0e5e60f367cc8156ae48198c496b2b2ebdf5313d" > next hop< / a >
2011-05-03 23:40:50 +00:00
< / article >
2011-05-07 02:18:53 +00:00
< article >
2011-05-03 23:40:50 +00:00
< h3 >
2011-05-07 02:18:53 +00:00
Sharing Details & Implementation
2011-05-03 23:40:50 +00:00
< / h3 >
2011-05-07 02:18:53 +00:00
< ul >
< li > blobserver is private-only. the < span class = 'red' > frontend< / span > mediates access to the world, checks authentication, or lack thereof.< / li >
< li > all non-owner requests must present a share blob's blobref as an access token< / li >
< li > that share blob dictates:
< ul >
2014-03-19 00:46:17 +00:00
< li > what sort of authenticatation is required (or "< code class = "green" > haveref< / code > " for none, like a secret link)< / li >
< li > which blob(s) are granted access (the "< code class = 'green' > transitive< / code > " option)< / li >
2011-05-07 02:18:53 +00:00
< / ul >
< / li >
< li > requests for a blob must include the path to get there, from the share root< / li >
< / ul >
< / article >
2011-05-03 23:40:50 +00:00
2011-05-07 02:18:53 +00:00
< article >
2011-05-03 23:40:50 +00:00
< h3 >
2011-05-07 02:18:53 +00:00
What can be shared
2011-05-03 23:40:50 +00:00
< / h3 >
2011-05-07 02:18:53 +00:00
< ul >
< li > Share a single blob,
< li > Share a subtree,
< li > Share a < i > search query< / i > and its results' reachable blobs< / li >
< li > ... give out [world, girlfriend] access to all pictures you take on your phone, in real-time< / li >
< / ul >
< / article >
2011-05-03 23:40:50 +00:00
2011-05-07 02:18:53 +00:00
< article >
< h2 >
Project Status
< / h2 >
2011-05-03 23:40:50 +00:00
< / article >
2011-05-07 02:18:53 +00:00
< article >
2011-05-03 23:40:50 +00:00
< h3 >
2011-05-07 02:18:53 +00:00
Project Status
2011-05-03 23:40:50 +00:00
< / h3 >
2011-05-07 02:18:53 +00:00
< ul >
< li > Blobstore, Go (any OS), can store on disk, s3, mysqlindex
< li > Blobstore, Python (App Engine only) can store on Google
< li > Perl tests for two blob stores
< li > Android uploader (Java)
< li > Bunch of Go libraries / command-line tools: sync, put, get
< li > FUSE filesystem (read-only, currently)
< li > Search: basics working. more queries looks easy now.
< li > Simple, self-contained everything binary (blob storage, sharing, search, index, frontend) for early adopters: ~95%
< li > Web UI / JavaScript APIs: in progress
< / ul >
< / article >
< article >
< h3 >
In Review
< / h3 >
< ul >
< li > You own all your blobs; everything is private by default.< / li >
< li > Mutable objects are made of mutation claim blobs.< / li >
< li > Sync is trivial: either you have it or you don't< / li >
< li > Some blobs are signed< / li >
< li > Indexing & search to find your blobs / roots< / li >
< li > To share you must create a declaration of sharing ...< / li >
< li > ... and the system will only allow access if such claims exist.< / li >
< li > Decentralized, but hostable. You can run your own server (with no central
company or point of control), but you can also let somebody else do it for
you, like email.< / li >
< / ul >
< / article >
2011-05-03 23:40:50 +00:00
< article >
< h3 >
Thank you!
< / h3 >
2011-05-07 02:18:53 +00:00
< p class = 'smaller' > Brad Fitzpatrick, < a href = "mailto:brad@danga.com" > brad@danga.com< / a > ; Want to help? More info: < a href = "http://camlistore.org/" > camlistore.org< / a > < / p >
< img src = 'arch.png' width = '100%' / >
2011-05-03 23:40:50 +00:00
< / article >
< / section >
< / body >
< / html >
