Rooba
/
perkeep
mirror of https://github.com/perkeep/perkeep.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,302 Commits 44 Branches 11 Tags 59 MiB
Commit Graph

37 Commits

Author SHA1 Message Date
Brad Fitzpatrick bf2a7b60a3 Importer overhaul. Only Foursquare is currently working. 
The rest are currently stubbed out and will need updating
to the new APIs.

Change-Id: I9d70302b3ac1026192413bf9dcd3c8f1eb420349
 2014-03-31 10:56:30 -07:00
Brad Fitzpatrick a90879bac0 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-03-21 13:57:07 -07:00
Aaron Boodman 2802cfb95c Minor cleanup to importer README files. 
Change-Id: Id37ed68b5b8eefe9bc0011452a0e41f70466d87b
 2014-03-21 13:41:16 -07:00
Brad Fitzpatrick bf692123bf importer: use context.Context instead of Interrupt type 
Change-Id: I79f93c6a14934320022b90500d926453e614d001
 2014-03-21 12:27:06 -07:00
Nick O'Neill 2d70c95aef Tweet importing with some basic info 
Change-Id: I73455eef50dcdc495ddc042d266b3a4a8020c877
 2014-03-20 18:57:43 -07:00
Tamás Gulácsi 65fa5d6d3e Picasa: add importer. 
Imports all images from all albums, and puts each image in its album.
Each album is linked under "Picasa (<username>)".
Tries to skip already imported files (check based on album/filename).
See Issue #391.

Change-Id: If00751671a429891c05bbe41f59c1421b2ff2de8
 2014-03-17 10:52:51 +01:00
Nick O'Neill f4f4368a40 Added some icons for checkins and venues. 
Change-Id: I4bfc5b7a8e099643c159ee6cb2532c71b0e62159
 2014-03-11 18:23:13 -07:00
Nick O'Neill 729d5c5851 Import places and checkins from foursquare. 
Change-Id: Ida2e411821f48a88212856398f5fe94caf7acd75
 2014-03-11 17:31:17 -07:00
Brad Fitzpatrick eb27581da7 httputil: add DecodeJSON and CloseBody helpers, then use them everywhere 
httputil.CloseBody closes a http.Response.Body nicely, such that it's
more likely to be re-used.

httputil.DecodeJSON parses JSON from a Response.Body and closes the
body. It also keeps a temporary copy of the input, for error messages.

Then use them everywhere, to improve HTTP TCP connection re-use.

And misc other clean-ups.

Change-Id: I5dc44be0165b13659459ed29ce5ab44c17d9739f
 2014-01-28 22:00:52 -08:00
Brad Fitzpatrick d1f2ae227f Start of Foursquare importer. 
Change-Id: I56b0e71a3aad697b82710ebea6fc941436a3c119
 2014-01-14 20:45:23 -08:00
Aaron Boodman 88973d4871 Clarify confusing log message in Flickr importer 
Change-Id: I9201bd12e7afb31c08b34b4bc4cd7c39567de296
 2013-12-27 00:02:33 -08:00
Dan Pupius bdcf858a0c Flickr importer paginates on sets and photostream 
Change-Id: I921400958e97c97467f6ef4d3eec642b81a8e614
 2013-12-16 21:16:18 -08:00
Brad Fitzpatrick 22101b61d8 flickr: clarify in README that the flag is a devcam flag, not server flag 
Change-Id: Iaab951c326ff90f09b6751f8fe4d439743a5e430
 2013-12-12 15:39:44 +04:00
Aaron Boodman 6a5a2d6d7c Add a README for the flickr importer package 
Change-Id: I018c1574e231cae4119fb55ceb7b63d976db0074
 2013-12-10 21:35:58 -08:00
Aaron Boodman 9855ab281d Flickr importer: import sets and populate with links to photos. 
Change-Id: I6c53dade3e96373fb3fb9ea4b172c53d5e22d988
 2013-11-22 12:21:03 -08:00
Aaron Boodman 08ea4ed13e Flickr importer: Don't refetch or import unmodified photos. 
Change-Id: I2ade0aa70e3caca8c3c1153ffee448d09d819da6
 2013-11-22 10:13:21 -08:00
Aaron Boodman 376fb91747 Merge "Flickr importer: store credentials in Camlistore, not weird JSON file." 2013-11-22 17:50:49 +00:00
Aaron Boodman 93b0bbbee5 Flickr importer: store credentials in Camlistore, not weird JSON file. 
Change-Id: I07ed4e9043c9fa448425144886dc3b7047c4e85b
 2013-11-22 09:50:03 -08:00
Brad Fitzpatrick ed889d38be dummy importer: upload a file and make a node. 2013-11-21 17:51:13 -08:00
Aaron Boodman 7c3a7bd30a Ugh. Make importer.Object.SetAttr() actually ignore duplicate values. 
I'm going to spend some time writing unit tests for this next I think.

Change-Id: Id19f24882f33adea4060ff852a986626ea8de171
 2013-11-20 23:57:11 -08:00
Aaron Boodman 2a747ccd56 Add ability to specify flickr config for devcam. 
Change-Id: Ic1139730d969558e2d6375dd86c6741e3dfd2b9b
 2013-11-18 21:16:09 -08:00
Aaron Boodman e3d71c34d1 Actually import the first 100 photos from Flickr. 
Currently this just imports the title, description, and image data.

Change-Id: I95de17cddb0ac18eec05f8ee288764b04cd7f406
 2013-11-17 21:58:06 -08:00
Brad Fitzpatrick bd11fd5cb9 importer: some TODO notes 
Change-Id: Ia92be507092722d5751f42ec3c21e18cfc4cb7f0
 2013-11-17 15:00:46 -08:00
Aaron Boodman de9e5103e7 Grab the userID from the flickr oauth handshake. 
Remove the mutex from auth.go because I believe the
mutex in importer.go should guaranteeing only one
flickr importer is running at a time.

Change-Id: Ice971c6c041cf21fdb2bc93cb15f861dd1c8c1d7
 2013-11-17 14:26:42 -08:00
Aaron Boodman fde2936507 Remove some unnecessary code from dummy importer. 
Change-Id: Ifd1cb70f6f5e11261353e8fe16d17c348ea95f12
 2013-11-16 22:43:17 -08:00
Aaron Boodman 9ae23e52b0 Rearrange the Flickr importer to use OAuth instead of Flickr's custom importer. 
Change-Id: I8284431fa6572a08d3c7ff0d0e655297627a4eda
 2013-11-16 22:30:21 -08:00
Brad Fitzpatrick f38f40c453 importer: flesh out most of the Host/Object interface 
Change-Id: Ia7e6ac3aa5e86bfa1fe950ad0c3944989c1d3f8a
 2013-10-25 00:23:29 -07:00
Brad Fitzpatrick 0324143892 importer: make the schema.Signer available to the importer 
Change-Id: Iad1a2f3bfe381b31a381957edb85d6d4aa21b38c
 2013-10-24 22:45:28 -07:00
Brad Fitzpatrick 353c43c02a wip 
Change-Id: I7cdafbd53b59fc53dafa4c19aa391f51f6982061
 2013-10-21 20:17:57 -07:00
Brad Fitzpatrick 8695ac5ca4 importer: partial API sketch. 
Change-Id: I05adf61164e4aabca35a1bb0ff1f0c8c0fd34ef1
 2013-10-20 20:54:43 -07:00
Brad Fitzpatrick 8a56f3744d importer: add accessors for Target and Search handler 
Change-Id: I49cd2a1fc08547eada21cee1506f190fe8f6d6de
 2013-10-20 12:24:20 -07:00
Brad Fitzpatrick c512e3d1be Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-10-20 12:00:22 -07:00
Brad Fitzpatrick 35e77c45ad importer: initialize Host's search handler and blobserver target 
Change-Id: I482e6f192620c254648b19b3044eedbd44de288c
 2013-10-20 12:00:15 -07:00
Aaron Boodman e4f25a2444 First little bit of Flickr importer. 
Change-Id: Ia42b07a81b01904f5722d06653c4bf9de2e81c64
 2013-10-20 11:18:53 -07:00
Brad Fitzpatrick e84d5865cf importer: rudimentary manual start/stop of importers 
Change-Id: I753f3d6e4a241c803f79446889169c4a25f1bdf3
 2013-10-19 16:59:46 -07:00
Brad Fitzpatrick cf116c6da9 importer: work on infrastructure for third-party import from other sites 
Change-Id: Ib4e5e4d543a1b2a913b99fa0a4d92f1a5b94c5dc
 2013-10-19 15:48:05 -07:00
Brad Fitzpatrick 6866b68024 importer: start of a third-party importing system. 
As discussed with Ryan Barrett.

Change-Id: I15773ee9cf3566214afbce947104be1584f6e600
 2013-01-20 16:17:32 -08:00