Commit Graph

25 Commits

Author SHA1 Message Date
Tamás Gulácsi 97520583b8 Use 'uint32' instead of 'int64' for blob sizes everywhere.
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.

Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
2014-02-08 17:58:12 +01:00
Brad Fitzpatrick 76171ddb3d Change sorted.KeyValue.Find to take an optional end bound; add tests.
The new package sorted/kvtest provides a generic KeyValue test for all
implementations. Memory, SQLite, and kvfile now use it.

This speeds up the index slurping start-up of my personal Camlistore
server from 30 seconds (when it was doing 17,000+ queries in small
windows) to now just 5 seconds. That 5 seconds can be improved yet
further.

Change-Id: Idd55ba9ccd3ed12a26868a41db1af676aff7b67b
2013-12-07 08:43:18 -08:00
Brad Fitzpatrick b82b8efe4c Start of new context package and *context.Context type.
Will eventually be plumbed through lots of APIs, especially those requiring or benefiting from
cancelation notification and/or those needing access to the HTTP context (e.g. App Engine).

Change-Id: I591496725d620126e09d49eb07cade7707c7fc64
2013-12-02 13:20:51 -08:00
Brad Fitzpatrick 62fbbb61a0 encrypt: use key metaIndex for index, not meta; that was used. fixes devcam server.
Change-Id: I483c696dcc3402537bec136f8847c17767760b3b
2013-11-23 11:11:05 -08:00
Brad Fitzpatrick 90c1e48afe Rename index.Storage to sorted.KeyValue and move it into a new package.
Having index.Index and index.Storage both in the same package led to
confusing discussions about "an index". Better names now, and smaller
packages.
2013-11-22 23:24:54 -08:00
mpl 3a26c433db devcam: 'put' and 'get' commands, replace dev-camput and dev-camget
Also added -port flag, and untied the port to the
password in devcam server, always 'pass3179' now.

http://camlistore.org/issue/212

Change-Id: Iadf693951763d47907be17c53d4807a4706eb150
2013-09-01 10:25:09 -07:00
Brad Fitzpatrick b24cad68dd Cleanup: remove BlobHub and time.Duration waits from storage interface
Move up a layer to the HTTP.  Also, start to remove ContextWrapper
stuff.  We've done it differently for App Engine instead, and will do
it differently yet moving forward.

Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.

Bunch of other cleanup and TODO fixing too.

Much simpler and cleaner.

Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
2013-08-21 13:57:28 -07:00
Brad Fitzpatrick 0bdf20884b all: delete pkg/blobref; convert all from *blobref.BlobRef to new blob.Ref
Change-Id: Id2dfb7f19452bedf4f3c9310b36227fd8117b225
2013-08-03 19:54:30 -07:00
Brad Fitzpatrick 2c4d272ef9 encrypt: fix check for missing config
Change-Id: I4433c36bc55a475cfa59953594f3a554c2d6ec2e
2013-07-09 07:24:14 +10:00
Brad Fitzpatrick 62ad21ff2e Docs and minor cleanups
Change-Id: Ibf1d69c21f53cf4b5b576a0dfefa5d0e7b26264d
2013-07-07 16:09:17 -07:00
Brad Fitzpatrick c3e37c2b56 encrypt: start of tests, and notes/TODOs for metadata compaction
Change-Id: I78f060fca5e486585c16a4e33b04ba29a38dc71a
2013-07-07 13:07:06 -07:00
Brad Fitzpatrick d065aac5b7 Fix compilation on FreeBSD. Thanks to retzkek. Fixes Issue 168.
Change-Id: I554321932709c6961fc05533aa4255bd8127d323
2013-07-02 22:12:53 -07:00
Brad Fitzpatrick c4e516545f encrypt: scan all meta blobs into index on start.
Yet to do: allow configuring an index besides just in-memory, and meta blob compaction.

Change-Id: I76041f1bc01487d208a0231abfef6d874ab819d9
2013-06-18 11:16:13 -07:00
Brad Fitzpatrick 2e632793ca encrypt: start of index recovery/compaction from meta blobs on start-up
Change-Id: I8aae66194ee67b1d63704ff98868e02f8b943e17
2013-06-17 21:56:25 -07:00
Brad Fitzpatrick c6577d600c encrypt: writing, but not yet reading meta blobs
Change-Id: I395ec3cb19b7bee23e15fed231e2e0a10258c729
2013-06-15 21:49:46 -07:00
Brad Fitzpatrick 1fddf9579a encrypt: more paranoia in FetchStreaming
Change-Id: Ie072a56015eb8a921328eccc6684647d2a153df3
2013-06-15 20:37:33 -07:00
Brad Fitzpatrick 3176c5be43 encrypt: require big ugly I_AGREE configuration in order to use it, so people don't get misled
Change-Id: I4259e423d6a4b733f353a1e8792c6b8cb3c95c43
2013-06-15 20:08:37 -07:00
Brad Fitzpatrick 3111bd5e0e encrypt: enumerate support
Change-Id: Ia7a20b2501a53b52fb514763c57e0ed2d6468735
2013-06-15 18:04:34 -07:00
Brad Fitzpatrick 3b6b49cb89 encrypt: fetch support (decryption)
Change-Id: I17f36a30efd300874b1844e99038ff11a978feef
2013-06-15 17:20:10 -07:00
Brad Fitzpatrick 537005fa32 encrypt: stat support.
Can now upload encrypted blobs.

Change-Id: Ib39e3b3218c99f139b0ed2b6d4a232160b4e1e64
2013-06-15 16:44:49 -07:00
Brad Fitzpatrick 4bf389bb81 encrypt: actually write encrypted blob to backend
Change-Id: I889088481af141e07a97cd1e4b89c459c5eaaedf
2013-06-15 16:06:11 -07:00
Brad Fitzpatrick 8e3a432e53 encrypt: write encrypted blobs.
no meta or retrieval yet, though.

Change-Id: I9f3a53500ca6b96226fbcfe65e8278481618d3b0
2013-06-15 15:42:37 -07:00
Brad Fitzpatrick d12007af52 encrypt: enough of a shell to start up in dev-server, even if nothing works yet
Change-Id: I7e3c17d952a943afd4cfb57939f25bf181aaa6fd
2013-06-15 15:05:27 -07:00
Brad Fitzpatrick ee165ffaef encryption TODOs/notes
Change-Id: Ifbbae03a71096d3c99008280f4ffd9b7bf13b365
2013-06-15 14:27:10 -07:00
Brad Fitzpatrick 11c32c142b Start of encryption support.
Change-Id: If99ac252fffa8b50f0a17e94e8aa454295dd7e2b
2013-06-12 04:35:50 -07:00