Rooba
/
perkeep
mirror of https://github.com/perkeep/perkeep.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,400 Commits 44 Branches 11 Tags 59 MiB
Commit Graph

3400 Commits

Author SHA1 Message Date
Tamás Gulácsi f69306cbee Finish implementing storagetest 
Add use into localdisk (diskpacked already uses it).
Add ErrNotImplemented error for blobserver and mention the possibility
for RemoveBlobs (diskpacked deficit).

Change-Id: I6a50f263a58c8d3d1611ff9a060ea9fa4aee6163
 2013-12-01 21:05:37 +01:00
Aaron Boodman 31a6685f1d When updating spinner, don't relayout entire doc 
Change-Id: I004e691e015b8a5e64f95702dc1f08ac75c91c0f
 2013-12-01 11:32:31 -08:00
Aaron Boodman bffbb0b349 Merge "Adds camlistore.Spinner, a progress meter based on the safe icon." 2013-12-01 18:09:50 +00:00
Aaron Boodman 9104eefb7c Adds camlistore.Spinner, a progress meter based on the safe icon. 
This isn't used by anything yet, but will be used in subsequent changes.

Change-Id: I0bd3174b2e16f8cab814ce72425b5fa1fed8d07e
 2013-12-01 10:08:47 -08:00
Brad Fitzpatrick a666df55c6 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-12-01 08:49:46 -08:00
Brad Fitzpatrick 91d735df4b index: start of re-indexing smartly when dependent blobs are missing 
See https://camlistore.org/issue/102

Change-Id: Ia5f69475d8f47398bc228a96e7694d59edf277bf
 2013-11-30 23:15:17 -08:00
Brad Fitzpatrick f544114844 Fix diskpacked regression, add tests, start of storagetest. 
Regressed from rev cb6f423e. Eventually pkg storagetest should test all methods of blobserver.Storage
for all storage target types.

Change-Id: I2c1c93b76fd9280a3eb429b1d71c64a693ed1ace
 2013-11-30 13:06:04 -08:00
Brad Fitzpatrick 778f5cc6c4 index: show corpus building CPU usage, disabled blob.Parse cache, size maps 
Change-Id: I089053e8de46978573e4b5fe5cdc14ccac0d54a2
 2013-11-30 12:04:04 -08:00
Brad Fitzpatrick e0e0c9fead osutil: add CPUUsage function 
Change-Id: Iba56be21be27f8786d8d7ca847ed9f2b7c3c1390
 2013-11-30 11:59:36 -08:00
Brad Fitzpatrick 6e900be6d0 serverconfig: add CAMLI_PPROF_START 
Change-Id: Ia3d3e021710f7681de7d43de87c236f8e37e992c
 2013-11-30 10:01:32 -08:00
Brad Fitzpatrick ecc5759758 kvfile: log which db was closed 
Change-Id: I932d9b143db0842afe6df55aad8b8c17f85d1254
 2013-11-30 09:32:37 -08:00
Brad Fitzpatrick f07ec21f5d localdisk: faster enumerate. 
still not as fast as possible, compared to native unix tools (find, ls, etc).
but closer.

Change-Id: I0add6bd07257f04f232332079214312297ec689b
 2013-11-29 20:53:46 -08:00
mpl e288622e01 Merge "pkg/index: do not record a keySignerAttrValue on DelAttributeClaim" 2013-11-29 23:57:59 +00:00
mpl 6c75ceb8b5 pkg/index: do not record a keySignerAttrValue on DelAttributeClaim 
Change-Id: Ib1f81fe4879de2be7d484a5a40cc6bf0449893d5
 2013-11-30 00:56:09 +01:00
Brad Fitzpatrick 8c46afe490 Shrink camtypes.BlobMeta a bit. Saves 4 bytes of memory per blob. 
For me,

Before:
Corpus stats: 148.859 MiB mem: 799465 blobs (89.487 GiB) (282303 schema (6478 permanode, 150962 file (112224 image), ...)

After:
Corpus stats: 146.559 MiB mem: 799465 blobs (89.487 GiB) (282303 schema (6478 permanode, 150962 file (112224 image), ...)

Change-Id: Id421dfc9d72c2a24c228418c01f9a3f05900c941
 2013-11-29 13:41:39 -08:00
Brad Fitzpatrick 4eaeee9ae3 Reduce in-memory size of image info dimensions. 
For my personal server,

Before:
Corpus stats: 149.884 MiB mem: 787886 blobs (88.853 GiB) (279308 schema (6073 permanode, 150557 file (111819 image), ...)

After:
Corpus stats: 145.172 MiB mem: 787886 blobs (88.853 GiB) (279308 schema (6073 permanode, 150557 file (111819 image), ...)

Change-Id: I47484e46b1eb70c8c4b5ee356cdcb0ceecde94e7
 2013-11-29 11:01:41 -08:00
Brad Fitzpatrick 2d47f93d76 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-29 10:56:26 -08:00
Brad Fitzpatrick 7e24120c5e Intern blob.Refs in the in-memory Corpus. 
For me, saves 419054 blob.Refs and 10 MB of memory.  Before/after:

Before:
Corpus stats: 159.057 MiB mem: 787886 blobs (88.853 GiB) (279308 schema (6073 permanode, 150557 file (111819 image), ...)

After:
Corpus stats: 149.884 MiB mem: 787886 blobs (88.853 GiB) (279308 schema (6073 permanode, 150557 file (111819 image), ...)

Change-Id: Idf71011c14d0d9b7457beab86dd084651fc3b122
 2013-11-29 10:47:42 -08:00
Brad Fitzpatrick 30e491eeaa Merge "camput: delete command, to send a delete claim" 2013-11-29 18:20:45 +00:00
Brad Fitzpatrick fceebe1d43 localdisk: cleanup temp files in migration, clean dirs better 
Change-Id: I8999fafc3e87822ec87df227e7815e7cdaf397cb
 2013-11-28 21:22:00 -08:00
Brad Fitzpatrick 64f47f3a24 localdisk: move migration stuff to its own file 
Change-Id: I987e0133622658dcf3117d65119723f12a4fe74f
 2013-11-28 20:20:56 -08:00
Brad Fitzpatrick 5de1c50eda Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-28 16:40:05 -08:00
Brad Fitzpatrick 8297d9614c localdisk: change hashing structure 
Before the files were stored in directories like
sha1/012/345/sha-012345xxxxx.dat, meaning there were 4096 (16^3)
top-level directories, each with up to 4096 child directories.  We
never really did the math, and the result millions (up to 16.7
million) directories with 1 file each.

Now the hashing structure is only 256 wide (two hex digits). If we
considered 4096 files in a directory acceptable before, that means the
new scheme can go up to 256*256*4096 files (268 million), which is
about 512 times bigger than my personal Camlistore instance
now. Larger users should probably be using the diskpacked storage
backend, anyway.

On start-up, the code now migrates the old format to the new format.

Change-Id: I17f7e830c50a5b770c57ee92d51f122340a0afbb
 2013-11-28 16:33:01 -08:00
mpl 1ee5fd20c5 search: deletions are not modifications 
1) pkg/search: documented that deletions times do not
qualify as modtimes

2) pkg/index: got rid of DeletedAt, and keyDeletes

http://camlistore.org/issue/191

Change-Id: I39578913345454d36af4599e29e7053f46577846
 2013-11-29 00:29:57 +01:00
mpl d51647ae33 camput: delete command, to send a delete claim 
http://camlistore.org/issue/191

Change-Id: I053ab468420cc173e6122909bf3a0b16e4d43224
 2013-11-29 00:19:42 +01:00
Brad Fitzpatrick 3fd356f457 gofmt 
Change-Id: I900a49b3f76aec7ef65419f2a5ecaf3eae95defb
 2013-11-28 11:58:47 -08:00
Brad Fitzpatrick 7635dd0602 Merge "Add diskpacked-reindex subcommand to camtool" 2013-11-28 19:58:15 +00:00
Brad Fitzpatrick cf837af913 corpus: log more stats on start-up 
Change-Id: Iabc97be94e5e3a81b4c377ed143612432f81c230
 2013-11-28 11:55:52 -08:00
Tamás Gulácsi cb6f423eeb Add diskpacked-reindex subcommand to camtool 
For checking and/or rebuilding index.kv of diskpacked packs.

Change-Id: I1ad87974b2daf58b1e767bb1df6f7b64e8b6359a
 2013-11-28 20:44:27 +01:00
Brad Fitzpatrick 1defe7b297 serverconfig: add 'memoryIndex' bool option to high-level config 
Change-Id: If5edb272858f186931eeb57579e221bde3efb07c
 2013-11-28 11:09:16 -08:00
Brad Fitzpatrick 3db0eca58c Merge "pkg/index: check for deletions in EdgesTo" 2013-11-28 18:46:07 +00:00
mpl fa32346860 pkg/index: check for deletions in EdgesTo 
Also introduced and used kvEdgeBackward

http://camlistore.org/issue/191

Change-Id: Ifdca9f2453ec0c7db55afc40cf0b5f1441b83158
 2013-11-28 19:41:40 +01:00
Brad Fitzpatrick e2efc15cb8 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-28 09:10:49 -08:00
Brad Fitzpatrick 313f8d25cb camlistored: log more version info at start 
Change-Id: I3e1e27f6a475e438bb3bf6c06ca3a0941a2944eb
 2013-11-28 08:55:53 -08:00
mpl 2095a2c913 pkg/index: another delete test case 
This test verifies that a permanode is still considered
deleted until all its deleters have been deleted.

http://camlistore.org/issue/191

Change-Id: Ifa4f17f031884b766d9ac90f1643a62ada591ae5
 2013-11-28 16:34:49 +01:00
mpl 8ee50244f3 Merge "pkg/index: use IsDeleted in some of the search methods" 2013-11-28 15:11:11 +00:00
mpl 15bb5f142e pkg/index: use IsDeleted in some of the search methods 
1) Done in AppendClaims, PermanodeOfSignerAttrValue,
SearchPermanodesWithAttr, PathsOfSignerTarget,
and PathsLookup.

2) camtypes.Path.DateClaim is now a time.Time, and used
as such in Path search methods.

3) Added kvSignerAttrValue, kvPathBackward, and kvPathForward

4) More Delete tests. We probably want more cases, but that's
a start.

http://camlistore.org/issue/191

Change-Id: I1d42485f2c6d1eea024f70378daacaaad9197768
 2013-11-28 16:08:23 +01:00
Brad Fitzpatrick 4346056b73 Merge "genconfig: use same packed-ness for cache as blobs." 2013-11-28 05:33:46 +00:00
Brad Fitzpatrick 9520219fab Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-27 21:33:10 -08:00
Brad Fitzpatrick f1558f69b2 index: don't allow schemaVersion 0 if the index is non-empty 
Change-Id: Ifaf3fb6c4f4166028fdf9041abb274f6cb8998ca
 2013-11-27 21:31:05 -08:00
Bill Thiede 270e8f54b6 genconfig: use same packed-ness for cache as blobs. 
Change-Id: I857f7a3071bfd80ed5f4e8cef8ca2307a978655a
 2013-11-27 21:26:04 -08:00
Brad Fitzpatrick fbad538fd8 gofmt 
Change-Id: Ibec8c4b1a78bd93b78e26ff73de7a386699cc6c8
 2013-11-27 20:46:58 -08:00
Brad Fitzpatrick d2cd51a6ff corpus: print out Corpus memory/size stats 
Change-Id: I3e9ed323473488a76992e5d5a9429b7fef5fa9b2
 2013-11-27 16:58:05 -08:00
Brad Fitzpatrick 1ec41b95f0 Speed up in-memory permanode attribute search. 
Finishes Corpus.AppendPermanodeAttrValues and uses it in
PermanodeConstraint.

Change-Id: I031789e0974add24886cc0060e52ba538535c4e1
 2013-11-27 12:47:04 -08:00
Brad Fitzpatrick 3730cbafbb Merge "pkg/index: update the deletes cache when receiving a delete claim" 2013-11-27 18:29:02 +00:00
mpl 5f966c2f99 Merge "pkg/index: change deletes cache structure and meaning" 2013-11-27 17:45:52 +00:00
mpl 42e37d4456 pkg/index: update the deletes cache when receiving a delete claim 
http://camlistore.org/issue/191

Change-Id: I49da2ef4e43675fba6a80db29ba96a473c159403
 2013-11-27 18:44:39 +01:00
mpl 45b630f766 pkg/index: change deletes cache structure and meaning 
1) The deletes cache now simply reflects all the keyDeleted index entries,
instead of trying to always hold the computed deletion state of the
blobs.

2) The IsDeleted and DeletedAt methods have been changed accordingly.
Moreover, while they previously were only taking into account the most
recent delete claim that affected the input blob, they now take
into account all delete claims that concern said blob.
In other words, a blob is considered "active" only if all of its past
deleters are deleted too (or it does not have any deleter).

http://camlistore.org/issue/191

Change-Id: I1b276dac0b3c6a3fb3beaf96828a45a85d27e4c6
 2013-11-27 17:02:03 +01:00
Brad Fitzpatrick c819397cd5 corpus: keep permanode claims sorted in memory 
Change-Id: Ic0387a3789c10036e21f5e2eb10101a0ae23b0b7
 2013-11-27 00:32:24 -08:00
Brad Fitzpatrick 1eff3b0a34 Merge "search & index: change 'before' logic, and tests." 2013-11-27 06:29:04 +00:00