There are two TODO lists. This file (good for airplanes) and the online bug tracker: https://github.com/perkeep/perkeep/issues Offline list: -- add a build tag to allow perkeepd to be compiled without any GCE support. (good for smaller binaries on Raspberry Pis or whatnot). Most code has moved to the osutil/gce package, and there's little gce usage now in perkeepd except for 2-3 things bethind "if env.OnGCE" checks. those things inside the checks can be func pointers registered by a file with a +build gce or !without_gce, depending on which way we decide to go. But make sure our test coverage builds both. -- fix the presubmit's gofmt to be happy about emacs: go fmt perkeep.org/cmd... perkeep.org/dev... perkeep.org/misc... perkeep.org/pkg... perkeep.org/server... stat pkg/blobserver/.#multistream_test.go: no such file or directory exit status 2 make: *** [fmt] Error 1 -- add HTTP handler for blobstreamer. stream a tar file? where to put continuation token? special file after each tar entry? special file at the end? HTTP Trailers? (but nobody supports them) -- reindexing: * add streaming interface to localdisk? maybe, even though not ideal, but really: migrate my personal instance from localdisk to blobpacked + maybe diskpacked for loose blobs? start by migrating to blobpacked and measuring size of loose. * add blobserver.EnumerateAllUnsorted (which could use StreamBlobs if available, else use EnumerateAll, else maybe even use a new interface method that goes forever and can't resume at a point, but can be canceled, and localdisk could implement that at least) * add buffered sorted.KeyValue implementation: a memory one (of configurable max size) in front of a real disk one. add a Flush method to it. also Flush when memory gets big enough. In progress: pkg/sorted/buffer -- stop using the "cond" blob router storage type in genconfig, as well as the /bs-and-index/ "replica" storage type, and just let the index register its own AddReceiveHook like the sync handler (pkg/server/sync.go). But whereas the sync handler only synchronously _enqueues_ the blob to replicate, the indexer should synchronously do the ReceiveBlob (ooo-reindex) on it too before returning. But the sync handler, despite technically only synchronously-enqueueing and being therefore async, is still very fast. It's likely the sync handler will therefore send a ReceiveBlob to the indexer at the ~same time the indexer is already indexing it. So the indexer should have some dup/merge suppression, and not do double work. singleflight should work. The loser should still consume the source io.Reader body and reply with the same error value. -- ditch the importer.Interrupt type and pass along a context.Context instead, which has its Done channel for cancelation. -- S3-only mode doesn't work with a local disk index (kvfile) because there's no directory for us to put the kv in. -- fault injection many more places with pkg/fault. maybe even in all handlers automatically somehow? -- sync handler's shard validation doesn't retry on error. only reports the errors now. -- export blobserver.checkHashReader and document it with the blob.Fetcher docs. -- "filestogether" handler, putting related blobs (e.g. files) next to each other in bigger blobs / separate files, and recording offsets of small blobs into bigger ones -- diskpacked doesn't seem to sync its index quickly enough. A new blob receieved + process exit + read in a new process doesn't find that blob. kv bug? Seems to need an explicit Close. This feels broken. Add tests & debug. -- websocket upload protocol. different write & read on same socket, as opposed to HTTP, to have multiple chunks in flight. -- extension to blobserver upload protocol to minimize fsyncs: maybe a client can say "no rush" on a bunch of data blobs first (which still don't get acked back over websocket until they've been fsynced), and then when the client uploads the schema/vivivy blob, that websocket message won't have the "no rush" flag, calling the optional blobserver.Storage method to fsync (in the case of diskpacked/localdisk) and getting all the "uploaded" messages back for the data chunks that were written-but-not-synced. -- measure FUSE operations, latency, round-trips, performance. see next item: -- ... we probaby need a "describe all chunks in file" HTTP handler. then FUSE (when it sees sequential access) can say "what's the list of all chunks in this file?" and then fetch them all at once. see next item: -- ... HTTP handler to get multiple blobs at once. multi-download in multipart/mime body. we have this for stat and upload, but not download. -- ... if we do blob fetching over websocket too, then we can support cancellation of blob requests. Then we can combine the previous two items: FUSE client can ask the server, over websockets, for a list of all chunks, and to also start streaming them all. assume a high-latency (but acceptable bandwidth) link. the chunks are already in flight, but some might be redundant. once the client figures out some might be redundant, it can issue "stop send" messages over that websocket connection to prevent dups. this should work on both "files" and "bytes" types. -- cacher: configurable policy on max cache size. clean oldest things (consider mtime+atime) to get back under max cache size. maybe prefer keeping small things (metadata blobs) too, and only delete large data chunks. -- UI: video, at least thumbnailing (use external program, like VLC or whatever nautilus uses?) -- rename server.ImageHandler to ThumbnailRequest or something? It's not really a Handler in the normal sense. It's not built once and called repeatedly; it's built for every ServeHTTP request. -- unexport more stuff from pkg/server. Cache, etc. -- look into garbage from openpgp signing -- make leveldb memdb's iterator struct only 8 bytes, pointing to a recycled object, and just nil out that pointer at EOF. -- bring in the google glog package to third_party and use it in places that want selective logging (e.g. pkg/index/receive.go) -- (Mostly done) verify all ReceiveBlob calls and see which should be blobserver.Receive instead, or ReceiveNoHash. git grep -E "\.ReceiveBlob\(" And maybe ReceiveNoHash should go away and be replaced with a "ReceiveString" method which combines the blobref-from-string and ReceiveNoHash at once. -- union storage target. sharder can be thought of a specialization of union. sharder already unions, but has a hard-coded policy of where to put new blobs. union could a library (used by sharder) with a pluggable policy on that. -- support for running pk-mount under perkeepd. especially for OS X, where the lifetime of the background daemon will be the same as the user's login session. -- website: add godoc for /server/perkeepd (also without a "go get" line) -- tests for all cmd/* stuff, perhaps as part of some integration tests. -- move most of pk-put into a library, not a package main. -- server cron support: full syncs, pk-put file backups, integrity checks. -- status in top right of UI: sync, crons. (in-progress, un-acked problems) -- finish metadata compaction on the encryption blobserver.Storage wrapper. -- get security review on encryption wrapper. (agl?) -- peer-to-peer server and blobserver target to store encrypted blobs on stranger's hardrives. server will be open source so groups of friends/family can run their own for small circles, or some company could run a huge instance. spray encrypted backup chunks across friends' machines, and have central server(s) present challenges to the replicas to have them verify what they have and how big, and also occasionally say what the SHA-1("challenge" + blob-data) is. -- sharing: make pk-get work with permanode sets too, not just "directory" and "file" things. -- sharing: when hitting e.g. http://myserver/share/sha1-xxxxx, if a web browser and not a smart client (Accept header? User-Agent?) then redirect or render a cutesy gallery or file browser instead, still with machine-readable data for slurping. -- rethink the directory schema so it can a) represent directories with millions of files (without making a >1MB or >16MB schema blob), probably forming a tree, similar to files. but rather than rolling checksum, just split lexically when nodes get too big. -- delete mostly-obsolete camsigd. see big TODO in camsigd.go. -- we used to be able live-edit js/css files in server/perkeepd/ui when running under the App Engine dev_appserver.py. That's now broken with my latest efforts to revive it. The place to start looking is: server/perkeepd/ui/fileembed_appengine.go -- should a "share" claim be not a claim but its own permanode, so it can be rescinded? right now you can't really unshare a "haveref" claim. or rather, TODO: verify we support "delete" claims to delete any claim, and verify the share system and indexer all support it. I think the indexer might, but not the share system. Also TODO: "pk-put delete" or "rescind" subcommand. Also TODO: document share claims in doc/schema/ and on website. -- make the -transitive flag for "pk-put share -transitive" be a tri-state: unset, true, false, and unset should then mean default to true for "file" and "directory" schema blobs, and "false" for other things. -- index: static directory recursive sizes: search: ask to see biggest directories? -- index: index dates in filenames ("yyyy-mm-dd-Foo-Trip", "yyyy-mm blah", etc). -- get webdav server working again, for mounting on Windows. This worked before Go 1 but bitrot when we moved pkg/fs to use the rsc/fuse. -- BUG: osutil paths.go on OS X: should use Library everywhere instead of mix of Library and ~/.camlistore? OLD: -- add CROS support? Access-Control-Allow-Origin: * + w/ OPTIONS http://hacks.mozilla.org/2009/07/cross-site-xmlhttprequest-with-cors/ -- brackup integration, perhaps sans GPG? (requires Perl client?) -- blobserver: clean up channel-closing consistency in blobserver interface (most close, one doesn't. all should probably close) Android: [ ] Fix wake locks in UploadThread. need to hold CPU + WiFi whenever something's enqueued at all and we're running. Move out of the Thread that's uploading itself. [ ] GPG signing of blobs (brad) http://code.google.com/p/android-privacy-guard/ http://www.thialfihar.org/projects/apg/ (supports signing in code, but not an Intent?) http://code.google.com/p/android-privacy-guard/wiki/UsingApgForDevelopment ... mailed the author. Client libraries: [X] Go [X] JavaScript [/] Python (Brett); but see https://github.com/tsileo/camlipy [ ] Perl [ ] Ruby [ ] PHP