Rooba
/
perkeep
mirror of https://github.com/perkeep/perkeep.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,954 Commits 44 Branches 11 Tags 59 MiB
Commit Graph

45 Commits

Author SHA1 Message Date
mpl df7952b9ba mysql: create database now distinct from create tables 
Because we do not want the database name to be optional with a
placeholder (/*DB*/) in the statement that creates the database, since
it is not actually optional there - as opposed to when creating tables.

Change-Id: I05351d76d95071492d763758a11454f219524510
 2014-08-08 20:06:02 +02:00
Brad Fitzpatrick 031c6199da sorted/mysql: automatically find IPs of hosts named *.cloudsql.google.internal 
Change-Id: Ib03add64c2c6b4de7fc6059f2a9c22065e2b2e7d
 2014-08-06 17:28:55 -07:00
Brad Fitzpatrick 1e7d12f1df sqlkv: remove one of the sql expansion funcs. make batch use KeyValue's 
Fixes bug with TablePrefix where batch mutations weren't getting the db prefix.

Change-Id: Ice599e607f070e374471e5ff22b426a6a5e05800
 2014-08-06 17:23:43 -07:00
Brad Fitzpatrick ac5c2fd045 sorted/mysql: create the db too if it doesn't exist. 
Change-Id: I566d49052057848e9d4af0f01243a89db367e20a
 2014-08-06 17:21:42 -07:00
Brad Fitzpatrick 6513d525c6 sorted/mysql: don't put database name in the DSN, share connections 
Change-Id: Ia4d399e72a1290f68dc2b0de71945176c6aac482
 2014-08-06 16:15:08 -07:00
Brad Fitzpatrick d6a8ecce2d sorted/sqlkv: support for adding optional 'db.' prefix before table names 
This is so configurations using the same database server for many
databases (multiple queues + the index) can all share the same
underlying *sql.DB

Change-Id: Iae7d4835fe9045008709c953e36b729729b1b6f4
 2014-08-06 16:13:18 -07:00
Brad Fitzpatrick b65868d754 serverinit: allow specifying port numbers in user@server:port:pass 
With heuristics and tests.

Change-Id: Ifacfc725db15100c447a3251c23267a5bfd313cf
 2014-08-06 15:07:43 -07:00
Brad Fitzpatrick 511c3aad12 sqlite: skip test instead of failing when not compiled in 
Change-Id: If7ef7308cdb7b6aec5490ec332d4dace7b7565a0
 2014-08-02 10:45:49 -07:00
Andrew Gerrand c15b4c05a7 pkg/sorted/buffer: flesh out, add test 
Change-Id: I18f57425ad2e9a4d692d4efe865fea81e8f143e3
 2014-07-15 08:24:54 +10:00
Brad Fitzpatrick a352572e0a sorted/buffer: start of a sorted.KeyValue buffer for reindexing 
Change-Id: I2617d98d6cb0b32548723ca45620b05eab91bca2
 2014-06-27 12:02:16 -07:00
mpl 45ce6cfdd3 mysql: replace mymysql with go-sql-driver 
from github.com/go-sql-driver/mysql at 9a7aa3606b82e2081a13a008ada88dfdb96c20fd

Context: http://camlistore.org/issue/428

Change-Id: Ia93ac35da02933bd03d2c6b7b8e5e549ba0472cb
 2014-05-01 17:42:51 +02:00
mpl a9db189cd7 docker tests: remove container when done 
http://camlistore.org/issue/421

Change-Id: Iefde3a8235db0103e8fcfac51f5dfcce5df73ef6
 2014-04-09 23:26:14 +02:00
mpl 899acb8e72 pkg/sorted: remove API for each distinct implementation 
Change-Id: I42446683189cdb00dcfbec1f221885092fdd83d6
 2014-04-08 01:37:08 +02:00
Steven L. Speek 3e50f83b1c pkg/sorted/postgres: Renamed "postgresql" -> "postgres" in init() 
Change-Id: I8f1ba7c970b57c62741e534c23958918714cee0b
Fixes: Issue 418
 2014-04-06 21:47:53 +02:00
mpl b0c4f85ee0 sorted/postgres: test with docker 
Also, NewKeyValue now automatically creates the required tables.

http://camlistore.org/issue/263

Change-Id: I02dd7660ee08c2eaeec17131622d6f581fdd6c55
 2014-03-28 19:34:09 +01:00
mpl 08f9c821f5 sorted,dockertest: added MySQL (dockerified) test 
Two important related changes:

1) sorted/mysql now takes into account the host given in the config
2) the required tables are now automatically created by NewKeyValue

http://camlistore.org/issue/263

Change-Id: I0043f36edb0630d6484148508d3a1e08c8e88a94
 2014-03-26 00:39:38 +01:00
Govert Versluis 96637ffcd3 pkg/blobserver/mongo: Add support for blob storage in MongoDB 
Related changes:
Split docker-related test-helper functions from pkg/sorted/mongo.
These helper functions are now also used in the pkg/blobserver/mongo
tests too.
Also fixed a typo in a comment in pkg/blob/fetcher.go and a missing variable in
debug output in pkg/blobserver/storagetest/storagetest.go

Addresses http://camlistore.org/issue/127

Change-Id: I8b6f57f9ced066d6f83788fdcc87be6619c65c3c

Conflicts:
	pkg/blob/fetcher.go
 2014-03-22 12:36:59 -07:00
mpl 23ef1cf24b sorted/kvfile: added test 
http://camlistore.org/issue/263

Change-Id: I4f6e8fe3cb26bedcff0a09278924364c249a3291
 2014-03-20 23:57:05 +01:00
mpl 05058091e9 kvutil: add VerifyDb*, use kvutil in sorted 
http://camlistore.org/issue/393

Change-Id: If9482aa05f2346644196e3218dcde571e4d1a1a1
 2014-03-20 01:21:27 +01:00
mpl 4ac7601315 sorted/kv: enable VerifyDb* options with env var 
http://camlistore.org/issue/393

Change-Id: I97bbe3d4ab6eb186c3960db0995cedeec3a34ddf
 2014-03-20 00:04:18 +01:00
mpl 2dd3e17129 sorted/sqlite: initialize DB when needed, added test. 
Change-Id: Ibd418e357e2bed6a5beed2dda2f947c7c999fe92
 2014-03-19 18:16:05 +01:00
Brad Fitzpatrick 3c00ed347c mongo: test using Docker 
Under a second to start and test against a real MongoDB instace in Docker:

$ go test -v camlistore.org/pkg/sorted/mongo
=== RUN TestMongoKV
--- PASS: TestMongoKV (0.79 seconds)
PASS
ok      camlistore.org/pkg/sorted/mongo 0.791s

Test is skipped in short mode, or if docker isn't available.

Change-Id: I6f3acc96357cf9c50054b3df6727661c9329e6bf
 2014-02-09 20:02:56 -08:00
Bill Thiede fe25884b71 pkg/sorted/kvfile: faster wipe. 
Delete the underlying DB file instead of iterating over the contents and
deleting item by item.

Change-Id: I1328c44b447ba9b1dc1cf81fdaf1f2437f80a071
 2013-12-24 21:27:49 -08:00
Brad Fitzpatrick 289065c730 sorted/kvfile: make it actually compile 
Change-Id: I0a3916c1c260dfac6b3444a18510a013d429ef33
 2013-12-24 15:35:29 -08:00
Bill Thiede de3a5d1941 pkg/sorted/kvfile: add sorted.Wiper. 
This allows people with kvIndexFile in their server-config.json to run:
camlistored -reindex

Change-Id: If1008297fa29ca514260529edb2125a1ea7d5246
 2013-12-24 13:15:24 -08:00
Brad Fitzpatrick a11ff22b8e camlistored: add --reindex flag; make sqlkv a sorted.Wiper 
Change-Id: I6b16c1c32187fb754d3acdbe852d02a506236078
 2013-12-23 19:07:17 -08:00
mpl 8fec95752c pkg/index: move postgresql to sorted + some fixes 
Change-Id: Ia6410256ffe8d4cbc2b9a5c056d075707105ab08
 2013-12-24 00:37:14 +01:00
mpl e687136b36 sorted: more tests 
Adding this particular case, because I suspect it demonstrates
a problem with postgresql.

Change-Id: Ia1ffe2f0a27dede579ed652b9f614cd69adc4418
 2013-12-23 15:35:05 +01:00
mpl 62f8a252ff pkg/index: move mysql to sorted 
also minor cleanups in sqlite and mongo

Change-Id: I1f371358997d929c72a8f63d2630a1c3fa4a0240
 2013-12-18 20:18:20 +01:00
Brad Fitzpatrick fb933a84cd Merge "mongo: NewKeyValue now takes a mongo.Config" 2013-12-17 16:12:04 +00:00
mpl 6102ca9a80 mongo: NewKeyValue now takes a mongo.Config 
Change-Id: Ibfb0085836030592db4373e4ac2540355e67a9c5
 2013-12-17 15:18:14 +01:00
Brad Fitzpatrick f23dc2b20d sorted: better errors when a constructor fails 
Change-Id: I9cfbd6742210c3e5efae6b30c7f287d739dff7d2
 2013-12-16 20:27:41 -08:00
mpl b62c94fdd1 pkg/index: move sqlite implementation to sorted 
Change-Id: I1e300ffa14547fabeac8d255fd694054ebcb9e53
 2013-12-16 17:35:27 +01:00
Brad Fitzpatrick 2921a320cb Merge "index: move kvfile implementation to sorted" 2013-12-13 16:33:38 +00:00
mpl b86e95caf4 Merge "camtool: dbinit mongo support" 2013-12-13 16:27:49 +00:00
mpl 8c562e9135 camtool: dbinit mongo support 
Change-Id: I289759ed9bdb34a6e108af1364c4a6bcfbb5982f
 2013-12-13 17:25:03 +01:00
mpl ab5e385113 index: move kvfile implementation to sorted 
Change-Id: Ie6e676570af088246ef028a2f003b537d85dd6ae
 2013-12-13 16:59:45 +01:00
mpl 39996424c7 sorted/mongo: close iter now that we have recent mgo 
Change-Id: If8e7f2ba611f2d3704acece56e03804bf9bd44f7
 2013-12-13 15:44:18 +01:00
Brad Fitzpatrick bf203d6ff7 Document sorted.Wiper more. 
Change-Id: Iede5cf3ec12453910594c682a3a18f8c237fbd40
 2013-12-13 11:18:27 +04:00
mpl fcbbf2a4df index: move mongo implementation to sorted 
Also devcam server -wipe wasn't wiping for mongo anymore, now fixed.

Change-Id: Iecc9d8025ddfba8d8ae9417ee170baf02be1d52f
 2013-12-13 00:25:40 +01:00
mpl 6c49c746e2 sorted: minor typo 
Change-Id: I51fc405cc0d8e43f7893b0e088f296126639cb4e
 2013-12-10 15:17:36 +01:00
Brad Fitzpatrick 76171ddb3d Change sorted.KeyValue.Find to take an optional end bound; add tests. 
The new package sorted/kvtest provides a generic KeyValue test for all
implementations. Memory, SQLite, and kvfile now use it.

This speeds up the index slurping start-up of my personal Camlistore
server from 30 seconds (when it was doing 17,000+ queries in small
windows) to now just 5 seconds. That 5 seconds can be improved yet
further.

Change-Id: Idd55ba9ccd3ed12a26868a41db1af676aff7b67b
 2013-12-07 08:43:18 -08:00
Brad Fitzpatrick 11a1745034 Add KeyBytes and ValueBytes accessor to sorted.KeyValue.Iterator. 
Goal is to iterate faster (notably: for slurping the index to the in-memory
corpus on start-up), by doing fewer copies and generating less garbage.

Change-Id: I054b0de2b994eb1f2356aa8587a466bafeb6cf82
 2013-12-04 14:18:52 +01:00
Brad Fitzpatrick 1f407edd58 sorted: add Close method to KeyValure; register 'kv' impl type for cznic/kv 
Change-Id: Ie4e21450dac9dad3433c65631706f52527502cbc
 2013-11-23 21:07:03 -08:00
Brad Fitzpatrick 90c1e48afe Rename index.Storage to sorted.KeyValue and move it into a new package. 
Having index.Index and index.Storage both in the same package led to
confusing discussions about "an index". Better names now, and smaller
packages.
 2013-11-22 23:24:54 -08:00