Commit Graph

11 Commits

Author SHA1 Message Date
Brad Fitzpatrick 76171ddb3d Change sorted.KeyValue.Find to take an optional end bound; add tests.
The new package sorted/kvtest provides a generic KeyValue test for all
implementations. Memory, SQLite, and kvfile now use it.

This speeds up the index slurping start-up of my personal Camlistore
server from 30 seconds (when it was doing 17,000+ queries in small
windows) to now just 5 seconds. That 5 seconds can be improved yet
further.

Change-Id: Idd55ba9ccd3ed12a26868a41db1af676aff7b67b
2013-12-07 08:43:18 -08:00
Brad Fitzpatrick 0163059814 sqlindex: don't iterate in small 50-row batches on big prefixes
This speeds Camlistore start-up index slurping considerably. Instead of
17,000 queries in 30 seconds, it now does 6.

A future change will improve this further, giving sorted.KeyValue impls
a hint about the end range too.

Change-Id: Icb5e2ac6edd580469e907fde4e916c00aba0ffe8
2013-12-07 06:46:02 -08:00
Brad Fitzpatrick 11a1745034 Add KeyBytes and ValueBytes accessor to sorted.KeyValue.Iterator.
Goal is to iterate faster (notably: for slurping the index to the in-memory
corpus on start-up), by doing fewer copies and generating less garbage.

Change-Id: I054b0de2b994eb1f2356aa8587a466bafeb6cf82
2013-12-04 14:18:52 +01:00
Brad Fitzpatrick 1f407edd58 sorted: add Close method to KeyValure; register 'kv' impl type for cznic/kv
Change-Id: Ie4e21450dac9dad3433c65631706f52527502cbc
2013-11-23 21:07:03 -08:00
Brad Fitzpatrick 90c1e48afe Rename index.Storage to sorted.KeyValue and move it into a new package.
Having index.Index and index.Storage both in the same package led to
confusing discussions about "an index". Better names now, and smaller
packages.
2013-11-22 23:24:54 -08:00
Brad Fitzpatrick b34c6198e4 sqlite: add start of an fd leak test
Change-Id: I84857e7a57bc4d95fa1a0ddc1c7538d1924cb93a
2013-09-13 11:21:48 +01:00
Brad Fitzpatrick afb81d0f5d Add checker for iterator Close calls
Change-Id: I90ba84635dd55ee056acedb8d0c0f5fa5d2a0ef5
2013-09-12 18:13:29 +01:00
Brad Fitzpatrick 9468e5ba70 More docs. Every package is documented now.
misc.CountingReader moves into readerutil.

pkg/atomics is folded into pkg/types.

pkg/test/testdep is folded into pkg/test, with better name/docs.

Old cruft from pkg/webserver is deleted.

Change-Id: I3f72d8b29804254ef944995fb085837c878f79f5
2013-07-07 21:12:30 -07:00
Brad Fitzpatrick 2fd4a22694 Make sqlite (more) reliable. camlistore.org/issue/114
Change-Id: Ia759de784bf4363bd9f5141ddddfa2a441a53f1d
2013-02-17 22:10:28 -08:00
mpl 295cbade1b Add postgres support
Also added the -short option for make presubmit
because postgres test is too slow.

Change-Id: I6be21c4d4cd67671eb44ab65333fc32c1e261bbd
2012-11-03 19:59:22 +01:00
Brad Fitzpatrick 1186c452ba Split mysql and generic sql index code. (Prep for sqlite indexer)
Untested.

Change-Id: Iaf53abf5bea9087ab2f8f1e2a7cda53b73ae094d
2012-04-09 09:53:57 -07:00