Rooba
/
perkeep
mirror of https://github.com/perkeep/perkeep.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,361 Commits 44 Branches 11 Tags 59 MiB
Commit Graph

3361 Commits

Author SHA1 Message Date
Brad Fitzpatrick db46a44758 test: add TLog function to set log output to a *testing.T 
Change-Id: I68d945634ce7780acabf120fe27989083fb02a04
 2013-12-03 18:23:42 -08:00
Brad Fitzpatrick a7c7e717f6 search: optionally also describe matched blobs in same request 
Change-Id: Id9c16c19fa309d0780c6107b1dd29880d777fd05
 2013-12-02 20:29:50 -08:00
Brad Fitzpatrick bf4d2721ba search: efficient permanode queries, sorted, and with a limit. 
Change-Id: I7f8edfe3636ff3f1344ebb41e27d23073c630ff5
 2013-12-02 20:01:37 -08:00
Brad Fitzpatrick da1abea89e readerutil: fix atomic alignment for 32-bit platforms 
Change-Id: I10eb60a9abbbaf5aeccb1eba9592ff5d354d7985
 2013-12-02 19:01:44 -08:00
Brad Fitzpatrick b82b8efe4c Start of new context package and *context.Context type. 
Will eventually be plumbed through lots of APIs, especially those requiring or benefiting from
cancelation notification and/or those needing access to the HTTP context (e.g. App Engine).

Change-Id: I591496725d620126e09d49eb07cade7707c7fc64
 2013-12-02 13:20:51 -08:00
Brad Fitzpatrick 0c73a65f78 TODO: use uint32 for SizedRef, not int 
Change-Id: I6f8bdfd483c1be4e3ec933f2bf05686163ed2bd7
 2013-12-02 12:01:42 -08:00
Brad Fitzpatrick 7466336f44 storagetest: more tests, and simplify a bit 
Change-Id: Ic0d6c9718ef79ff82f05cf142286022862f30236
 2013-12-02 11:57:02 -08:00
Brad Fitzpatrick 6478e256f7 diskpacked: finish enumeration bug with 'after' param 
Change-Id: I391a33e7672cce6c7c9a4eebc0a09c2f03546b82
 2013-12-02 11:56:23 -08:00
Brad Fitzpatrick d324eecaba blob: add slice sorter types for []blob.Ref and []blob.SizedRef 
Change-Id: I23224a9408a9d6fd2bbd7dd0fa94dba64f4586c1
 2013-12-02 11:55:39 -08:00
Brad Fitzpatrick 88f4f309ac Merge "Finish implementing storagetest" 2013-12-02 19:01:27 +00:00
Aaron Boodman 0514b9fd3f Formatting cleanup in blob_item_container.js. 
Change-Id: Id6430c205f2d97e9439105ce7498ae9bc8861557
 2013-12-01 22:52:53 -08:00
Tamás Gulácsi f69306cbee Finish implementing storagetest 
Add use into localdisk (diskpacked already uses it).
Add ErrNotImplemented error for blobserver and mention the possibility
for RemoveBlobs (diskpacked deficit).

Change-Id: I6a50f263a58c8d3d1611ff9a060ea9fa4aee6163
 2013-12-01 21:05:37 +01:00
Aaron Boodman 31a6685f1d When updating spinner, don't relayout entire doc 
Change-Id: I004e691e015b8a5e64f95702dc1f08ac75c91c0f
 2013-12-01 11:32:31 -08:00
Aaron Boodman bffbb0b349 Merge "Adds camlistore.Spinner, a progress meter based on the safe icon." 2013-12-01 18:09:50 +00:00
Aaron Boodman 9104eefb7c Adds camlistore.Spinner, a progress meter based on the safe icon. 
This isn't used by anything yet, but will be used in subsequent changes.

Change-Id: I0bd3174b2e16f8cab814ce72425b5fa1fed8d07e
 2013-12-01 10:08:47 -08:00
Brad Fitzpatrick a666df55c6 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-12-01 08:49:46 -08:00
Brad Fitzpatrick 91d735df4b index: start of re-indexing smartly when dependent blobs are missing 
See https://camlistore.org/issue/102

Change-Id: Ia5f69475d8f47398bc228a96e7694d59edf277bf
 2013-11-30 23:15:17 -08:00
Brad Fitzpatrick f544114844 Fix diskpacked regression, add tests, start of storagetest. 
Regressed from rev cb6f423e. Eventually pkg storagetest should test all methods of blobserver.Storage
for all storage target types.

Change-Id: I2c1c93b76fd9280a3eb429b1d71c64a693ed1ace
 2013-11-30 13:06:04 -08:00
Brad Fitzpatrick 778f5cc6c4 index: show corpus building CPU usage, disabled blob.Parse cache, size maps 
Change-Id: I089053e8de46978573e4b5fe5cdc14ccac0d54a2
 2013-11-30 12:04:04 -08:00
Brad Fitzpatrick e0e0c9fead osutil: add CPUUsage function 
Change-Id: Iba56be21be27f8786d8d7ca847ed9f2b7c3c1390
 2013-11-30 11:59:36 -08:00
Brad Fitzpatrick 6e900be6d0 serverconfig: add CAMLI_PPROF_START 
Change-Id: Ia3d3e021710f7681de7d43de87c236f8e37e992c
 2013-11-30 10:01:32 -08:00
Brad Fitzpatrick ecc5759758 kvfile: log which db was closed 
Change-Id: I932d9b143db0842afe6df55aad8b8c17f85d1254
 2013-11-30 09:32:37 -08:00
Brad Fitzpatrick f07ec21f5d localdisk: faster enumerate. 
still not as fast as possible, compared to native unix tools (find, ls, etc).
but closer.

Change-Id: I0add6bd07257f04f232332079214312297ec689b
 2013-11-29 20:53:46 -08:00
mpl e288622e01 Merge "pkg/index: do not record a keySignerAttrValue on DelAttributeClaim" 2013-11-29 23:57:59 +00:00
mpl 6c75ceb8b5 pkg/index: do not record a keySignerAttrValue on DelAttributeClaim 
Change-Id: Ib1f81fe4879de2be7d484a5a40cc6bf0449893d5
 2013-11-30 00:56:09 +01:00
Brad Fitzpatrick 8c46afe490 Shrink camtypes.BlobMeta a bit. Saves 4 bytes of memory per blob. 
For me,

Before:
Corpus stats: 148.859 MiB mem: 799465 blobs (89.487 GiB) (282303 schema (6478 permanode, 150962 file (112224 image), ...)

After:
Corpus stats: 146.559 MiB mem: 799465 blobs (89.487 GiB) (282303 schema (6478 permanode, 150962 file (112224 image), ...)

Change-Id: Id421dfc9d72c2a24c228418c01f9a3f05900c941
 2013-11-29 13:41:39 -08:00
Brad Fitzpatrick 4eaeee9ae3 Reduce in-memory size of image info dimensions. 
For my personal server,

Before:
Corpus stats: 149.884 MiB mem: 787886 blobs (88.853 GiB) (279308 schema (6073 permanode, 150557 file (111819 image), ...)

After:
Corpus stats: 145.172 MiB mem: 787886 blobs (88.853 GiB) (279308 schema (6073 permanode, 150557 file (111819 image), ...)

Change-Id: I47484e46b1eb70c8c4b5ee356cdcb0ceecde94e7
 2013-11-29 11:01:41 -08:00
Brad Fitzpatrick 2d47f93d76 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-29 10:56:26 -08:00
Brad Fitzpatrick 7e24120c5e Intern blob.Refs in the in-memory Corpus. 
For me, saves 419054 blob.Refs and 10 MB of memory.  Before/after:

Before:
Corpus stats: 159.057 MiB mem: 787886 blobs (88.853 GiB) (279308 schema (6073 permanode, 150557 file (111819 image), ...)

After:
Corpus stats: 149.884 MiB mem: 787886 blobs (88.853 GiB) (279308 schema (6073 permanode, 150557 file (111819 image), ...)

Change-Id: Idf71011c14d0d9b7457beab86dd084651fc3b122
 2013-11-29 10:47:42 -08:00
Brad Fitzpatrick 30e491eeaa Merge "camput: delete command, to send a delete claim" 2013-11-29 18:20:45 +00:00
Brad Fitzpatrick fceebe1d43 localdisk: cleanup temp files in migration, clean dirs better 
Change-Id: I8999fafc3e87822ec87df227e7815e7cdaf397cb
 2013-11-28 21:22:00 -08:00
Brad Fitzpatrick 64f47f3a24 localdisk: move migration stuff to its own file 
Change-Id: I987e0133622658dcf3117d65119723f12a4fe74f
 2013-11-28 20:20:56 -08:00
Brad Fitzpatrick 5de1c50eda Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-28 16:40:05 -08:00
Brad Fitzpatrick 8297d9614c localdisk: change hashing structure 
Before the files were stored in directories like
sha1/012/345/sha-012345xxxxx.dat, meaning there were 4096 (16^3)
top-level directories, each with up to 4096 child directories.  We
never really did the math, and the result millions (up to 16.7
million) directories with 1 file each.

Now the hashing structure is only 256 wide (two hex digits). If we
considered 4096 files in a directory acceptable before, that means the
new scheme can go up to 256*256*4096 files (268 million), which is
about 512 times bigger than my personal Camlistore instance
now. Larger users should probably be using the diskpacked storage
backend, anyway.

On start-up, the code now migrates the old format to the new format.

Change-Id: I17f7e830c50a5b770c57ee92d51f122340a0afbb
 2013-11-28 16:33:01 -08:00
mpl 1ee5fd20c5 search: deletions are not modifications 
1) pkg/search: documented that deletions times do not
qualify as modtimes

2) pkg/index: got rid of DeletedAt, and keyDeletes

http://camlistore.org/issue/191

Change-Id: I39578913345454d36af4599e29e7053f46577846
 2013-11-29 00:29:57 +01:00
mpl d51647ae33 camput: delete command, to send a delete claim 
http://camlistore.org/issue/191

Change-Id: I053ab468420cc173e6122909bf3a0b16e4d43224
 2013-11-29 00:19:42 +01:00
Brad Fitzpatrick 3fd356f457 gofmt 
Change-Id: I900a49b3f76aec7ef65419f2a5ecaf3eae95defb
 2013-11-28 11:58:47 -08:00
Brad Fitzpatrick 7635dd0602 Merge "Add diskpacked-reindex subcommand to camtool" 2013-11-28 19:58:15 +00:00
Brad Fitzpatrick cf837af913 corpus: log more stats on start-up 
Change-Id: Iabc97be94e5e3a81b4c377ed143612432f81c230
 2013-11-28 11:55:52 -08:00
Tamás Gulácsi cb6f423eeb Add diskpacked-reindex subcommand to camtool 
For checking and/or rebuilding index.kv of diskpacked packs.

Change-Id: I1ad87974b2daf58b1e767bb1df6f7b64e8b6359a
 2013-11-28 20:44:27 +01:00
Brad Fitzpatrick 1defe7b297 serverconfig: add 'memoryIndex' bool option to high-level config 
Change-Id: If5edb272858f186931eeb57579e221bde3efb07c
 2013-11-28 11:09:16 -08:00
Brad Fitzpatrick 3db0eca58c Merge "pkg/index: check for deletions in EdgesTo" 2013-11-28 18:46:07 +00:00
mpl fa32346860 pkg/index: check for deletions in EdgesTo 
Also introduced and used kvEdgeBackward

http://camlistore.org/issue/191

Change-Id: Ifdca9f2453ec0c7db55afc40cf0b5f1441b83158
 2013-11-28 19:41:40 +01:00
Brad Fitzpatrick e2efc15cb8 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-28 09:10:49 -08:00
Brad Fitzpatrick 313f8d25cb camlistored: log more version info at start 
Change-Id: I3e1e27f6a475e438bb3bf6c06ca3a0941a2944eb
 2013-11-28 08:55:53 -08:00
mpl 2095a2c913 pkg/index: another delete test case 
This test verifies that a permanode is still considered
deleted until all its deleters have been deleted.

http://camlistore.org/issue/191

Change-Id: Ifa4f17f031884b766d9ac90f1643a62ada591ae5
 2013-11-28 16:34:49 +01:00
mpl 8ee50244f3 Merge "pkg/index: use IsDeleted in some of the search methods" 2013-11-28 15:11:11 +00:00
mpl 15bb5f142e pkg/index: use IsDeleted in some of the search methods 
1) Done in AppendClaims, PermanodeOfSignerAttrValue,
SearchPermanodesWithAttr, PathsOfSignerTarget,
and PathsLookup.

2) camtypes.Path.DateClaim is now a time.Time, and used
as such in Path search methods.

3) Added kvSignerAttrValue, kvPathBackward, and kvPathForward

4) More Delete tests. We probably want more cases, but that's
a start.

http://camlistore.org/issue/191

Change-Id: I1d42485f2c6d1eea024f70378daacaaad9197768
 2013-11-28 16:08:23 +01:00
Brad Fitzpatrick 4346056b73 Merge "genconfig: use same packed-ness for cache as blobs." 2013-11-28 05:33:46 +00:00
Brad Fitzpatrick 9520219fab Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-27 21:33:10 -08:00