Camlistore

Brad Fitzpatrick
2011-05-07

(use arrow keys or PgUp/PgDown to move slides)

Who am I?

Brad Fitzpatrick <brad@danga.com>
Perl Hacker since 1994

Projects:

Danga / 6A (Perl)	Google
LiveJournal memcached Perlbal MogileFS OpenID	Social Graph API (XFN / FOAF) WebFinger PubSubHubbub Android Go

* decentralized social

But why am I in Brazil?

"Hey, want to come speak at a Perl conference in Brazil?"
"Yes, totally, but... I don't write much Perl these days. :-("
"You could speak on memcached."
"But that's an old topic, no?"
"You have any new project you're excited about?"

Camlistore

New open source project
Almost a year old
Still in development
Starting to be useful :-)
Hard to easily describe...

Camlistore is a way to store, sync, share, model and back up content

camlistore.org

Motivation

I've written too many Content Management Systems
- blogs, comments, photos, emails, backups, scanned paperwork, ...
- is a scanned photo a scan, a photo, or a blog post? who cares.
- write one CMS to rule them all
- ... or at least a good framework for higher-level CMSes

Motivation (cont)

I still want to help solve the Decentralized Social Network Problem
- protocols, not companies
- gmail, hotmail: hosted versions of SMTP, IMAP
- ... but I can run my own SMTP/IMAP server if I want.
- ... or change my SMTP/IMAP provider

Motivation (cont)

I wanted something conceptually simple.
HTTP interfaces, not language-specific
I use lots of machines; don't want to think about sync or conflicts.
Data archaeology: should be easy and obvious to reconstruct in 20 or 100 years

The Product

one private dumping ground to store anything
backups, filesystems, objects, photos, likes, bookmarks, shares, my website, ...
live backup my phone
live replicate / sync my dumping group between my house & laptop & Amazon & Google
web UI (ala gmail, docs.google.com, etc) or FUSE filesystem
Easy for end-users; powerful for dorks

Security Model

your private repo, for life
everything private by default
grant access to specific objects/trees with friends or the world
web UI or CLI tools let you share

Camlistore

Content-
Addressable
Multi-
Layer-
Indexed
Storage

Content-Addressable

At the core, everything is stored & addressed by its digest (e.g. SHA1, MD5, etc)
e.g. "sha1-0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33" for the blob "foo"
Great properties:
- no versions of content: change it changes the new digest too
- no versions: no sync conflicts
- no versions: perfect caching (have it or don't)

Multi-Layer, Indexed

Unix philosophy: small pieces with well-defined interfaces that can be chained or composed
Camlistore pieces include:
- Blob storage: memory, disk, S3, Google, MySQL index, etc
- Schema
- Signing
- Replication
- Indexing: (e.g. replicate from disk to MySQL index)
- Search
- HTML UI

In reality

End-users: use a hosted version
Dorks: single server binary with all the logical pieces

Blob Server: how dumb it is

"Blob" == zero or more bytes. no meta-data
private operations, to owner of data only:

get(blobref) → blob
stat(blobref+) → [(blobref, size), ...]
put(blobref, blob)
enumerate(..) → [(blobref, size)...] (sorted by blobref)

no public (non-owner) access
HTTP interface: GET /camli/sha1-xxxxxxx HTTP/1.1
delete(blobref) is disabled by default, privileged op for GC or replication queues only

Blob Server: seriously, no metadata

no filenames
no "mime types"
no "{create,mod,access} time"
size is implicit
blob: just some bytes
metadata? layers above.

Uh, what can you do with that?

with just a blob server?
not much
but let's start with an easy example...

Filesystem Backups

previous project: brackup
- good: Perl, slide/dice/encrypt S3 backup, content-addressed, good iterative backups
- bad: large (several MB) "backup manifest" text files
fossil/venti, git, etc: directories content-addressed by content of their children, hash trees, etc
git: "tree objects", "commmit objects", etc
Camlistore: "schema blobs"

Schema: how to model your content

Camlistore defines one possible schema
but blobserver doesn't know about it all
tools generate schema,
indexer + search understand the schema.

Schema Blobs

so if all blobs are just dumb blobs of bytes with no metadata,
how do you store metadata?
as blobs themselves!

Minimal Schema Blob

{
  "camliVersion": 1,
  "camliType": "whatever"
}

Whitespace doesn't matter. Just must be valid JSON in its entirety. Use whatever JSON libraries you've got.

That one is named
sha1-19e851fe3eb3d1f3d9d1cefe9f92c6f3c7d754f6

or perhaps: sha512-2c6746aba012337aaf113fd63c24d994a0703d33eb5d6ed58859e45dc4e02dcf dae5c4d46c5c757fb85d5aff342245fe4edb780c028a6f3c994c1295236c931e

Schema blob; type "file"

{"camliVersion": 1,
 "camliType": "file",
 "fileName": "foo.dat",
 "unixPermission": "0644",
 ...,
 "size": 6000133,
 "contentParts": [
   {"blobRef": "sha1-...dead", "size": 111},
   {"blobRef": "sha1-...beef", "size": 5000000, "offset": 492 },
   {"size": 1000000},
   {"blobRef": "digalg-blobref", "size": 22},
  ]
}

Schema blob; type "directory"

{"camliVersion": 1,
 "camliType": "directory",
 "fileName": "foodir",
 "unixPermission": "0755",
  ...,
 "entries": "sha1-c3764bc2138338d5e2936def18ff8cc9cda38455"
}

Schema blob; type "static-set"

{"camliVersion": 1,
 "camliType": "static-set",
"members": [
     "sha1-xxxxxxxxxxxx",
     "sha1-xxxxxxxxxxxx",
     "sha1-xxxxxxxxxxxx",
     "sha1-xxxxxxxxxxxx",
     "sha1-xxxxxxxxxxxx",
     "sha1-xxxxxxxxxxxx",
  ]
}

Backup a directory...

$ camput --file $HOME
sha1-8659a52f726588dc44d38dfb22d84a4da2902fed

(like git/hg/fossil, that identifier represents everything down.)

Iterative backups are cheap, easy identifier to share, etc

But how will you remember that identifier? (later)

But what about mutable data?

immutable data is easy to represent & reference

sha1-8659a52f726588dc44d38dfb22d84a4da2902fed is an immutable snapshot

how to represent mutable data in an immutable, content-addressed world?
how to share a reference to a mutable object when changing an object mutates its name?

Terminology

blob: just dumb, immutable series of bytes
schema blob: a blob that's a valid JSON object w/ camliVersion & camliType
signed schema blob aka "claim": a schema blob with an embedded OpenPGP signature
object: something mutable. represented as an anchor "permanode" + a set of mutations (claims)
permanode: a stable reference. an anchor. just a signed schema blob, but of almost no content...

Permanode

$ camput --permanode
sha1-ea799271abfbf85d8e22e4577f15f704c8349026

$ camget sha1-ea799271abfbf85d8e22e4577f15f704c8349026
{"camliVersion": 1,
  "camliSigner": "sha1-c4da9d771661563a27704b91b67989e7ea1e50b8",
  "camliType": "permanode",
  "random": "oj)r}$Wa/[J|XQThNdhE"
,"camliSig":"iQEcBAABAgAGBQJNRxceAAoJEGjzeDN/6vt8ihIH/Aov7FRIq4dODAPWGDwqL
1X9Ko2ZtSSO1lwHxCQVdCMquDtAdI3387fDlEG/ALoT/LhmtXQgYTt8QqDxVdu
EK1or6/jqo3RMQ8tTgZ+rW2cj9f3Q/dg7el0Ngoq03hyYXdo3whxCH2x0jajSt4RCc
gdXN6XmLlOgD/LVQEJ303Du1OhCvKX1A40BIdwe1zxBc5zkLmoa8rClAlHdqwo
gxYFY4cwFm+jJM5YhSPemNrDe8W7KT6r0oA7SVfOan1NbIQUel65xwIZBD0ah
CXBx6WXvfId6AdiahnbZiBup1fWSzxeeW7Y2/RQwv5IZ8UgfBqRHvnxcbNmScrzl
p3V3ZoY"}

Backup a directory...

$ camput --file $HOME
sha1-8659a52f726588dc44d38dfb22d84a4da2902fed
$ camput --permanode --file $HOME
sha1-ea799271abfbf85d8e22e4577f15f704c8349026
$ camput --permanode --name="Brad's home directory" --file $HOME
sha1-ea799271abfbf85d8e22e4577f15f704c8349026

all the file data blobs, file/dir schema blobs,
a new permanode, owned by you
a mutation: permanode's content attribute == directory root
a mutation: permanode's name attribute == "Brad's home directory"

Aside: Garbage Collection

Permanodes are (optionally) GC roots,
or anything signed by you.
If not a blob isn't reachable by a signed root, can be deleted.
If you want to keep a plain "dumb" blob, you should create a "keep" claim for it, or a permanode.

Example: a photo gallery

Photos are objects
Galleries (sets) are objects
Photos are members of galleries
Photos & galleries have attributes (single-valued: "title", multi-valued: "tag")
Photos might be updated over time:
- EXIF GPS updated, cropping, white balance
- don't want to break links!

Indexing: summary

For each blob, build an index of:

directed graph of inter-blob references
(permanode, time) => resolved attributes
(permanode, time) => set memberships
etc...

Indexing & Replication

indexing is real-time, no polling
MySQL index speaks the blob server protocol
just replicate to the index (MySQL, etc) just like other blob servers (Amazon S3, etc)

Replication Implementation

cold bootstrap: enumerate() (sorted) all blobs from src and dst, copy all blobs that dst doesn't have.
more efficient: use multiple machines, starting at sha1-0*, sha1-1*, sha1-2*, ... etc
once in-sync, for each (src, dst) replication pair, keep a src_to_dst_QUEUE namespace on src,
all new blobs to src also go into src_to_dst_QUEUE (refcount, hardlink, etc)
real-time watch src_to_dst_QUEUE & replicate & delete from the queue. or re-enumerate just the queue.

Search

Permanodes created by $who, sorted by date desc, type "photo", tagged "funny"
My recent backups with attribute "hostname" == "camlistore.org",
All friends' galleries in which this photo appears,
etc...

...similar to your email, or docs.google.com. "My stuff" or "My bookmarks".

Privacy Model

all your blobs & objects & searches are private
nothing is public by default

What if you want to share with friends, or globally publish something?

Sharing & Share Blobs

the act of sharing involves creating a new share claim, just another blob, signed.

here is: sha1-071fda36c1bd9e4595ed16ab5e2a46d44491f708:

{"camliVersion": 1,
  "authType": "haveref",
  "camliSigner": "sha1-f019d17dd308eebbd49fd94536eb67214c2f0587",
  "camliType": "share",
  "target": "sha1-0e5e60f367cc8156ae48198c496b2b2ebdf5313d",
  "transitive": true
,"camliSig":"iQEcBAABAgAGBQJNQJGuAAoJEIUeCLJL7Fq1EuAIAL/nGoX8caGaANnam0bcIQT7C61wXMRW4qCCaFW+w67ys5z4ztfnTPKwL9ErzMF8Hd32Xe/bVcF6ZL38x/axqI7ehxN8lneKGQNoEdZDA9i752aAr0fkAba6eDehoOj9F4XxOzk3iVrq445jEXtu/+twamHV3UfRozWK1ZQb57dM+cRff47M/Y6VIBRSgW2BrABjuBs8G6PiKxycgh1mb+RL8f9KG+HB/yFuK37YJqZ0zU2OTRp6ELiOgTxbeg99koV9Duy4f4mQgxQgli46077Sv/ujzIeVbmdFL3OenGEzQnyKG0fhf8fa5WkED0XfH7zibAHLiSq3O7x11Q0406U==ANug"}

Target w/ ?via= parameter: sha1-0e5e60f?via=sha1-071fda & next hop

Sharing Details & Implementation

blobserver is private-only. the frontend mediates access to the world, checks authentication, or lack thereof.
all non-owner requests must present a share blob's blobref as an access token
that share blob dictates:
- what sort of authenticatation is required (or "haveref" for none, like a secret link)
- which blob(s) are granted access (the "transitive" option)
requests for a blob must include the path to get there, from the share root

What can be shared

Share a single blob,
Share a subtree,
Share a search query and its results' reachable blobs
... give out [world, girlfriend] access to all pictures you take on your phone, in real-time

Project Status

Blobstore, Go (any OS), can store on disk, s3, mysqlindex
Blobstore, Python (App Engine only) can store on Google
Perl tests for two blob stores
Android uploader (Java)
Bunch of Go libraries / command-line tools: sync, put, get
FUSE filesystem (read-only, currently)
Search: basics working. more queries looks easy now.
Simple, self-contained everything binary (blob storage, sharing, search, index, frontend) for early adopters: ~95%
Web UI / JavaScript APIs: in progress

In Review

You own all your blobs; everything is private by default.
Mutable objects are made of mutation claim blobs.
Sync is trivial: either you have it or you don't
Some blobs are signed
Indexing & search to find your blobs / roots
To share you must create a declaration of sharing ...
... and the system will only allow access if such claims exist.
Decentralized, but hostable. You can run your own server (with no central company or point of control), but you can also let somebody else do it for you, like email.

Thank you!

Brad Fitzpatrick, brad@danga.com; Want to help? More info: camlistore.org

Camlistore

Who am I?

But why am I in Brazil?

Camlistore!

Camlistore

Motivation

Motivation (cont)

Motivation (cont)

The Product

Security Model

So what's with the silly name?

Camlistore

Content-Addressable

Multi-Layer, Indexed

Logically

In reality

From the bottom up...

Blob Server

Blob Server: how dumb it is

Blob Server: seriously, no metadata

Uh, what can you do with that?

Uh, what can you do with that?

Filesystem Backups

Filesystem Backups

Schema: how to model your content

Schema Blobs

Minimal Schema Blob

Schema blob; type "file"

Schema blob; type "directory"

Schema blob; type "static-set"

Backup a directory...

But what about mutable data?

Objects & "Permanodes"

Terminology

Permanode

Backup a directory...

Aside: Garbage Collection

Modeling non-filesystem objects

Example: a photo gallery

How to make sense of that?

Indexing & Search

Indexing: summary

Indexing & Replication

Replication Implementation

Search

Privacy Model

What if you want to share with friends, or globally publish something?

Sharing & Share Blobs

Sharing Details & Implementation

What can be shared

Project Status

Project Status

In Review

Thank you!