Rooba
/
perkeep
mirror of https://github.com/perkeep/perkeep.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,302 Commits 44 Branches 11 Tags 59 MiB
Commit Graph

1510 Commits

Author SHA1 Message Date
Brad Fitzpatrick bf2a7b60a3 Importer overhaul. Only Foursquare is currently working. 
The rest are currently stubbed out and will need updating
to the new APIs.

Change-Id: I9d70302b3ac1026192413bf9dcd3c8f1eb420349
 2014-03-31 10:56:30 -07:00
Brad Fitzpatrick bf4426e35e types: add EmptyBody 
Change-Id: Ic0cc6cae6debb20fbc15bd95692e8565171eae49
 2014-03-31 09:36:08 -07:00
Brad Fitzpatrick bfb076ab95 context: add Context.SetHTTPClient 
Change-Id: I10729215cacb5d6faf2ffaedc12c9d7449a1d6a3
 2014-03-31 09:34:45 -07:00
mpl cc1bd94c22 Merge "camput init: ring/key related fixes" 2014-03-31 13:50:29 +00:00
Brad Fitzpatrick 9f4e8fdef8 Merge "blobserver: more docs on MyPrefix" 2014-03-30 22:45:29 +00:00
Brad Fitzpatrick b63d56ac9d blobserver: more docs on MyPrefix 
Change-Id: Icf7fa540c7eb7400719e6baf74f3be746615a4f5
 2014-03-30 14:51:54 -07:00
mpl cdc4ed5ae2 camput init: ring/key related fixes 
1) Removed exec call to gpg, because it automatically looks in .gnupg/,
which we don't use anymore as a default.
2) Now taking into account global --secret-keyring flag.
This flag is now in osutil.
3) New or modified funcs in osutil
4) Made sure --gpgkey works too.
5) Cleaned up error messages and hints.

Context: http://camlistore.org/issue/364
         http://camlistore.org/issue/368

Change-Id: I2e51032ed0597da656db100d72f5588b37308e1a
 2014-03-28 19:55:18 +01:00
mpl b0c4f85ee0 sorted/postgres: test with docker 
Also, NewKeyValue now automatically creates the required tables.

http://camlistore.org/issue/263

Change-Id: I02dd7660ee08c2eaeec17131622d6f581fdd6c55
 2014-03-28 19:34:09 +01:00
Brad Fitzpatrick 4b2acfe155 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-03-27 19:42:49 -07:00
Brad Fitzpatrick bf00b4d41c context: add method to get an HTTPClient. 
For importer work.

Change-Id: Ibfb74461157c7f81cdb04f05f9fb955f95990ebf
 2014-03-27 19:41:56 -07:00
Brad Fitzpatrick 768cb6e0c2 Merge "sorted,dockertest: added MySQL (dockerified) test" 2014-03-27 00:34:30 +00:00
mpl 08f9c821f5 sorted,dockertest: added MySQL (dockerified) test 
Two important related changes:

1) sorted/mysql now takes into account the host given in the config
2) the required tables are now automatically created by NewKeyValue

http://camlistore.org/issue/263

Change-Id: I0043f36edb0630d6484148508d3a1e08c8e88a94
 2014-03-26 00:39:38 +01:00
Brad Fitzpatrick 6a610384b0 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-03-23 17:14:18 -07:00
Brad Fitzpatrick bf5740afb2 Add BaseURL method to handler Loader. 
Change-Id: Ibc61cd19798aa1b0858cbe40b10544953db67b36
 2014-03-23 17:13:40 -07:00
Brad Fitzpatrick d66e320477 Merge "Added missing check for http.StatusOK in Stat. Also made checks for http.StatusOK / 200 uniform" 2014-03-23 23:14:41 +00:00
Steven L. Speek 3442076272 search: width and height exact matches simplified 
corrected some copy paste errors in the expr_test
file

Change-Id: I36467d96c5674cddc34009d36a05c42ddd7e3cc2
 2014-03-23 21:18:00 +01:00
Govert Versluis 33d5718843 Added missing check for http.StatusOK in Stat. 
Also made checks for http.StatusOK / 200 uniform

Change-Id: I8ef608dc098cdfd221a53be20801d7dcdcdf7655
 2014-03-23 17:37:16 +01:00
Govert Versluis 96637ffcd3 pkg/blobserver/mongo: Add support for blob storage in MongoDB 
Related changes:
Split docker-related test-helper functions from pkg/sorted/mongo.
These helper functions are now also used in the pkg/blobserver/mongo
tests too.
Also fixed a typo in a comment in pkg/blob/fetcher.go and a missing variable in
debug output in pkg/blobserver/storagetest/storagetest.go

Addresses http://camlistore.org/issue/127

Change-Id: I8b6f57f9ced066d6f83788fdcc87be6619c65c3c

Conflicts:
	pkg/blob/fetcher.go
 2014-03-22 12:36:59 -07:00
Brad Fitzpatrick a90879bac0 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-03-21 13:57:07 -07:00
Brad Fitzpatrick 3dc784e557 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-03-21 13:49:53 -07:00
Aaron Boodman 2802cfb95c Minor cleanup to importer README files. 
Change-Id: Id37ed68b5b8eefe9bc0011452a0e41f70466d87b
 2014-03-21 13:41:16 -07:00
Brad Fitzpatrick bf692123bf importer: use context.Context instead of Interrupt type 
Change-Id: I79f93c6a14934320022b90500d926453e614d001
 2014-03-21 12:27:06 -07:00
Brad Fitzpatrick 9054fe84dd Merge "Add kv queue to google drive / google cloud storage sync handlers" 2014-03-21 18:47:44 +00:00
Philip Snowberger 1ba212d707 Add kv queue to google drive / google cloud storage sync handlers 
Change-Id: I978ae42dbc6ce2840071cbadaf9351498a23415f
 2014-03-21 11:46:22 -07:00
Brad Fitzpatrick b6e8238d47 Merge "Diskpacked: support deletion" 2014-03-21 18:44:53 +00:00
Nick O'Neill 2d70c95aef Tweet importing with some basic info 
Change-Id: I73455eef50dcdc495ddc042d266b3a4a8020c877
 2014-03-20 18:57:43 -07:00
mpl 23ef1cf24b sorted/kvfile: added test 
http://camlistore.org/issue/263

Change-Id: I4f6e8fe3cb26bedcff0a09278924364c249a3291
 2014-03-20 23:57:05 +01:00
Tamás Gulácsi bc39d9888c Diskpacked: support deletion 
Delete from index and zero out blob's data.
Use FALLOC_FL_PUNCH_HOLE if available (linux >= 2.6.38).

Change-Id: I0c194fd6734a35b520212398d93dfdc5cabadeb5
 2014-03-20 21:20:36 +01:00
Brad Fitzpatrick 673e82815b Merge "kvutil: add VerifyDb*, use kvutil in sorted" 2014-03-20 19:44:08 +00:00
Brad Fitzpatrick c56635c3a5 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-03-20 12:29:56 -07:00
Brad Fitzpatrick bfe51f3ad5 gofmt 
Change-Id: Ie6be315d02a439cbfdd870a0b0b7f16cba12a168
 2014-03-20 12:29:45 -07:00
mpl 05058091e9 kvutil: add VerifyDb*, use kvutil in sorted 
http://camlistore.org/issue/393

Change-Id: If9482aa05f2346644196e3218dcde571e4d1a1a1
 2014-03-20 01:21:27 +01:00
mpl 3b66e2136c Merge "sorted/kv: enable VerifyDb* options with env var" 2014-03-19 23:05:31 +00:00
mpl 4ac7601315 sorted/kv: enable VerifyDb* options with env var 
http://camlistore.org/issue/393

Change-Id: I97bbe3d4ab6eb186c3960db0995cedeec3a34ddf
 2014-03-20 00:04:18 +01:00
Brad Fitzpatrick bf041c8955 search: escape r.AttrFilter in ClaimsRequest.URLSuffix 
Change-Id: I90e5a87330dc6ae820f2f5f6e3887a26f341a2b2
 2014-03-19 14:59:13 -07:00
Brad Fitzpatrick 0cbfae59e1 Merge "Added AttrFilter to ClaimsRequest" 2014-03-19 21:57:48 +00:00
Brad Fitzpatrick 2fea26435e Merge "sorted/sqlite: initialize DB when needed, added test." 2014-03-19 17:22:57 +00:00
mpl 2dd3e17129 sorted/sqlite: initialize DB when needed, added test. 
Change-Id: Ibd418e357e2bed6a5beed2dda2f947c7c999fe92
 2014-03-19 18:16:05 +01:00
Hernan Grecco 16d6bbb065 Added AttrFilter to ClaimsRequest 
This allows server-side filtering of claims by claimtype

Change-Id: I1f5368c17f8f9f59ea0add7df67033447e37bc61
 2014-03-19 00:45:33 -03:00
Steven L. Speek 146a42cc51 search: accept 'and', 'or', and parentheses in expressions 
'and' has precendence over 'or'.
both operators are left associative
parenthesized expressions are evaluated first
Parser refactored, parseAtom split up.

Change-Id: I1f194cc75df49bad9d30d041d689d8ba833076f1
 2014-03-18 16:40:44 -07:00
mpl b43373c45c Merge "publish: use generic queries" 2014-03-18 22:35:40 +00:00
Brad Fitzpatrick bfe61b1297 storagetest: add a test of EnumerateBlobs w/ prefix of a blobref 
Change-Id: Id865da5ba6c65e0110b0eb6c6d0bd45980d33105
 2014-03-18 13:47:02 -07:00
Brad Fitzpatrick bf4116eb93 genconfig: picasaa importer doesn't have a path 
Was removed during codereview, but not removed from here.

Change-Id: I51309f46fe9747bcc0b2476813acac9120c32b28
 2014-03-18 10:10:10 -07:00
mpl 9cadbcd5bc publish: use generic queries 
Use generic queries instead of specialized index queries. This is a step
towards the publisher app, because its client will have to use generic
queries.

Context: http://camlistore.org/issue/365

Change-Id: I2781a345e024174e3bea8511b6cdc6f342d5a7c1
 2014-03-18 17:26:29 +01:00
Brad Fitzpatrick 90404a4129 Merge "Picasa: add importer." 2014-03-18 16:05:42 +00:00
Brad Fitzpatrick bf2e1fa585 sync: don't replicate a shard's missing blobs until enumeration is complete 
Prevents spurious replication of blobs on enumeration error.

Change-Id: I38db7406f6ea52137cb757b32599b18eb7fcf3da
 2014-03-17 23:21:53 -07:00
Brad Fitzpatrick bf2d86738e blobserver: give ListMissingDestinationBlobs the ability to exit earlier 
If there's an enumeration error for src or dst, sending a zero value
will shut down ListMissingDestinationBlobs.

Change-Id: I95598e9dc2607610436faa06f40485d1abd2342f
 2014-03-17 23:20:37 -07:00
Brad Fitzpatrick bf1ec32e39 sync: add paranoia around checking storage's Enumerate implementation 
Didn't find anything, but is useful to keep in, to maybe find bugs in the future
for other storage types.

Change-Id: If0fd37e03578de233be8da95ca45623c5f12156b
 2014-03-17 23:05:27 -07:00
Brad Fitzpatrick bf8f4b2423 sync: fix bug in prefix enumeration. could send one extra item. 
Depending on timing, could lead to ListMissingDestinationBlobs getting out
of sync and causing a lot of blobs to be replicated that were fine and already
on the server.

Change-Id: I3710e59088f1fe4e526f8f11bc9d1837a727e512
 2014-03-17 23:02:01 -07:00
Brad Fitzpatrick bf88f5f06c sync: missing handler return + wording change 
Change-Id: Iebd5344a0a0e1418cb48c92b91858ebf2f9486c8
 2014-03-17 23:00:18 -07:00