Rooba
/
perkeep
mirror of https://github.com/perkeep/perkeep.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,229 Commits 44 Branches 11 Tags 59 MiB
Commit Graph

3229 Commits

Author SHA1 Message Date
Brad Fitzpatrick d0916a6802 Merge "Actually import the first 100 photos from Flickr." 2013-11-18 06:29:33 +00:00
Aaron Boodman e3d71c34d1 Actually import the first 100 photos from Flickr. 
Currently this just imports the title, description, and image data.

Change-Id: I95de17cddb0ac18eec05f8ee288764b04cd7f406
 2013-11-17 21:58:06 -08:00
Brad Fitzpatrick 48dba353e9 query: make all tests test in all three modes. 
Change-Id: I945830ddb3bfe68a7223994b94d780b46503c95e
 2013-11-17 20:20:27 -08:00
Brad Fitzpatrick cb31dd58d2 In-memory search: include FileInfo. More tests pass. 
Change-Id: Ic33357b63d1d57dc174fe18360e920730016c903
 2013-11-17 19:49:19 -08:00
Brad Fitzpatrick 84077b6d34 New package: strutil 
Change-Id: I74ea770ed59ce9a5aa2fdea10b37c418fab50598
 2013-11-17 19:31:12 -08:00
Brad Fitzpatrick e8603b1293 Put claims in memory too for in-memory search. Required index schema version bump. 
Change-Id: I194d65476bddea111277cd0b1472c56b5527226b
 2013-11-17 16:52:51 -08:00
Brad Fitzpatrick bd11fd5cb9 importer: some TODO notes 
Change-Id: Ia92be507092722d5751f42ec3c21e18cfc4cb7f0
 2013-11-17 15:00:46 -08:00
Brad Fitzpatrick 27fd9e71a7 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-17 15:00:21 -08:00
Brad Fitzpatrick 0b07171c0c index: change GetOwnerClaims interface for efficiency, rename to AppendClaims 
Change-Id: Icec235961ed8a09256df87c5fe1455cbad81ae3f
 2013-11-17 14:54:30 -08:00
Aaron Boodman de9e5103e7 Grab the userID from the flickr oauth handshake. 
Remove the mutex from auth.go because I believe the
mutex in importer.go should guaranteeing only one
flickr importer is running at a time.

Change-Id: Ice971c6c041cf21fdb2bc93cb15f861dd1c8c1d7
 2013-11-17 14:26:42 -08:00
Aaron Boodman 9c671f76eb Revert "Revert "Add genconfig support for flickr importer"" 
This reverts commit 8367de6e95.

Change-Id: I1fdfa4070572b1badc0f0980348404f905fa13a3
 2013-11-17 14:15:12 -08:00
Brad Fitzpatrick 8abaddfa80 index: rearrange interface definition. 
more stable stuff at the top. stuff that needs a re-think below.

Change-Id: Id1682b393718a2e9928088d38a4ddea477698e98
 2013-11-17 13:44:49 -08:00
Brad Fitzpatrick 8367de6e95 Revert "Add genconfig support for flickr importer" 
This reverts commit b53e79614d.

It broke the tests.
 2013-11-17 13:43:06 -08:00
Brad Fitzpatrick 1d91f3a627 Merge "Add genconfig support for flickr importer" 2013-11-17 21:31:05 +00:00
Aaron Boodman b53e79614d Add genconfig support for flickr importer 
Change-Id: I8be13f81fae15acfe87cce9d65318524efc3035d
 2013-11-17 13:29:49 -08:00
Brad Fitzpatrick 04a116fed0 TODOs 
Change-Id: I6a7866d889f7446bc89c25a19c57d0c8220bf99f
 2013-11-17 11:01:09 -08:00
Brad Fitzpatrick d82f02db50 Search optimization: keep blobs arranged by camliType in memory; use when possible. 
Change-Id: I8b3ebcf7dfece21d17074da32dc20e4d600e4c50
 2013-11-17 10:52:37 -08:00
Brad Fitzpatrick 8e8ac9092a corpus: add a generation number for future query caching 
Change-Id: I07dd61756537688f4f5e68bbcaf93bff75aaddc9
 2013-11-17 10:05:40 -08:00
Brad Fitzpatrick 494c9f37e5 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-17 09:49:51 -08:00
Brad Fitzpatrick 3eb493599e in-memory search: better structure for keeping memory corpus and kv 
index in sync, both at start-up and while running and receiving blobs.
They both use the same mechanism now.

Also adds KeyId to the index and Corpus, as the next step. Plenty more
row types remain...

Change-Id: Id79955ba25dc79d5fbd94b0e5248d33dcf71d97e
 2013-11-17 09:41:45 -08:00
Brad Fitzpatrick 6fbe62f081 Merge "Bugfix: incorrect baseURL passed to InstallHandlers" 2013-11-17 16:28:07 +00:00
Aaron Boodman 3f7543853a Bugfix: incorrect baseURL passed to InstallHandlers 
Change-Id: Iaf26a806358cd82ca09c48d83ca86155e27c289d
 2013-11-17 00:41:01 -08:00
Aaron Boodman fde2936507 Remove some unnecessary code from dummy importer. 
Change-Id: Ifd1cb70f6f5e11261353e8fe16d17c348ea95f12
 2013-11-16 22:43:17 -08:00
Brad Fitzpatrick b33a785e62 Merge "Rearrange the Flickr importer to use OAuth instead of Flickr's custom importer." 2013-11-17 06:39:24 +00:00
Aaron Boodman 9ae23e52b0 Rearrange the Flickr importer to use OAuth instead of Flickr's custom importer. 
Change-Id: I8284431fa6572a08d3c7ff0d0e655297627a4eda
 2013-11-16 22:30:21 -08:00
Brad Fitzpatrick 5ce4fab0f9 Work around lack of cgo when cross-compiling to arm. 
Change-Id: Ic230362b9f19af0c99f296f14330c970bf5d010b
 2013-11-16 22:22:48 -08:00
Brad Fitzpatrick 272db6545a index: change GetBlobMIMEType to GetBlobMeta. mime types on a blob was wrong anyway. 
And some more of in-memory search, which this makes easier / less garbage-y.

Change-Id: If1fa7dd44842244ae5e5440f84e92d66dc3321fb
 2013-11-16 19:40:14 -08:00
Brad Fitzpatrick bbabc52723 test: add Diff function. 
forgot to commit earlier.

Change-Id: I9986b6bd2266d455837bc20e47b43e4c971654d7
 2013-11-16 19:06:30 -08:00
Brad Fitzpatrick f3cc3c7ed9 search: more in-memory search work. make tests verify Scan doesn't hit Storage. 
also some string interning work.

Change-Id: I7864b56eb97318bce943afdca3b1212f4729a9a8
 2013-11-16 18:50:01 -08:00
Brad Fitzpatrick e26eacfa2f index: return FileInfo and ImageInfo as values, not pointers. 
Change-Id: I6aba3a74ce233cc834550b561f597e549fb2bfab
 2013-11-16 17:49:28 -08:00
Brad Fitzpatrick 2984897ac7 search: more in-memory search work. 
keep blob metadata in memory, and start of testing all search queries in three modes:
classic index.Storage scanning, all in-memory with corpus scanned from the index.Storage,
and the in-memory corpus built up over time as blobs arrive.

Change-Id: I40536e498a63bece5bd4897cdbbd0cef78085f44
 2013-11-16 17:24:02 -08:00
Brad Fitzpatrick c573feb226 Make the search handler conditionally request the index be slurped into memory. 
Change-Id: I7fadeba41c3b1bbcfbb0fd3470ff30cdf04b1b90
 2013-11-16 15:18:16 -08:00
Brad Fitzpatrick 705107ad80 search/index: invert depedency. search now depends on index. 
creates new package types/camtypes for misc types needed by both. might eventually go away as
search matures.

Change-Id: Ib771ead7bea39936ba478b7e5d58de997060861b
 2013-11-16 15:00:30 -08:00
Brad Fitzpatrick 25eca52283 Merge "pkg/search: add modtime in WANT to fix tests" 2013-11-16 22:26:28 +00:00
Brad Fitzpatrick 2a42678f0c index: start of optional support for keeping index all tightly in-memory for search 
Change-Id: Ie7fbafa1376505d10b0c01470089d260a0c0f97f
 2013-11-16 12:55:09 -08:00
Brad Fitzpatrick 7f8e035be6 search: little TODO 
Change-Id: I38a98c9c9b77a899536cf0d7601ce2030d6f4dda
 2013-11-16 12:54:11 -08:00
Brad Fitzpatrick 8a5161ec30 search: multiple constraints at top level 
Change-Id: I964a0d394c3c03e2409c91fb1d240847ff07ddb3
 2013-11-16 11:21:30 -08:00
Brad Fitzpatrick 17a9d061a0 search: ditch BlobSizeConstraint, use IntConstraint instead 
Change-Id: I99bece643841feab777175479743b3bd1fc421b8
 2013-11-16 11:10:09 -08:00
Brad Fitzpatrick 1e627ce7c3 search: change BlobMeta to have a CamliType, not a misleading MIMEType. 
Change-Id: I1c6e111ef27aafd5a15f9b38746361093a78e9e6
 2013-11-16 10:35:18 -08:00
mpl 01a8dbb163 pkg/search: add modtime in WANT to fix tests 
Also fixed fakeindex clock to use a time.Time.
And synchronized the fakeindex and the indextests clocks to the
same origin.

Change-Id: I986abca300d61241069132619028f470cacd61da
 2013-11-15 23:31:24 +01:00
mpl 131976f165 Merge "pkg/index: use a map to populate the mutations" 2013-11-15 00:27:26 +00:00
mpl e03d923fe1 pkg/index: use a map to populate the mutations 
When indexing upon a blob reception, we first populate
all the mutations in a map instead of in a batch mutation.
Then we transfer all the mutations in a batch and commit
it immediately. This makes the window when the batch mutation
is open much shorter, and will ease future indexing because
it allows reading from the index while writing the mutations
to the map.

Change-Id: I276282388f59ca543835bfa5ec64986453b23fe1
 2013-11-15 01:23:21 +01:00
Brad Fitzpatrick 169f38ede6 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-14 15:36:50 -08:00
Brad Fitzpatrick 9587b8e62e search: rename AttributeConstraint to PermanodeConstraint, add ModTime constraint 2013-11-14 15:31:44 -08:00
Brad Fitzpatrick e0c3cf9b01 search: optimize ClaimList.Less sort. duh. 2013-11-14 15:30:59 -08:00
Brad Fitzpatrick 6e7df1c6ee search: IntConstraint 2013-11-14 15:30:40 -08:00
Brad Fitzpatrick e5eade1bd6 Merge "httputil: set the headers before writing them" 2013-11-14 21:53:40 +00:00
Brad Fitzpatrick 469b166129 C+A: add Robert Obryk 2013-11-14 13:52:59 -08:00
Brad Fitzpatrick 6a4c0ec09c TODO: use glog 
Change-Id: I94ce97b93c73bee47f90a966f5cb1e860b23233a
 2013-11-09 19:55:49 -05:00
Brad Fitzpatrick d6a10c18f2 index: doc tweak 
Change-Id: I30836b1a6967ac86f7f16118a9bf81a293b09a27
 2013-11-09 19:20:21 -05:00