Rooba
/
perkeep
mirror of https://github.com/perkeep/perkeep.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,954 Commits 44 Branches 11 Tags 59 MiB
Commit Graph

141 Commits

Author SHA1 Message Date
Brad Fitzpatrick f4740e77e8 Aggregate all errors with their URLS in status.json in one place. 
Change-Id: If7a540fc3d1a6caa94b5f04d44de3207f15ac4cf
 2014-08-29 16:03:44 -07:00
Brad Fitzpatrick dbc1a681bc Add importers to status. 
For http://camlistore.org/issue/483

Change-Id: I2621c5484efb121a5e5e24debeff64ffc5139ad3
 2014-08-16 11:58:16 -07:00
mpl f54a51dc82 twitter: fix format verb in error 
Change-Id: I36e11ac4b506391f8b94c22fa0d28f652ade4a4c
 2014-08-06 16:21:42 +02:00
mpl f75081072d importers: newOauthClient refactoring 
Change-Id: Ic5864ab2f7bb63bd801fe965c68b36a245edc447
 2014-08-02 01:37:43 +02:00
mpl 6d269cb37b importers: some refactoring around oauthContext 
Change-Id: I0a7e1dabb9efdaedbda138cdd1c52afcddee5210
 2014-08-02 00:50:00 +02:00
Brad Fitzpatrick 10252d1951 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-07-31 14:35:30 -07:00
Brad Fitzpatrick 8fbd54c09b picasa: set url property on photos and albums 
Change-Id: If89ee61138e4fb80e68f87130c5fe189b7d82582
 2014-07-31 14:34:52 -07:00
Brad Fitzpatrick 1ff2918602 importers: add SupportsIncremental accessor 
If an importer is efficient at running regularly, it returns true.

If it's not true, the UI won't allow turning on automatic runs.

Flickr doesn't set it (yet).
 2014-07-31 11:34:31 -07:00
Brad Fitzpatrick b3637337be Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-07-30 17:28:02 -07:00
Brad Fitzpatrick 4ea387545a picasa: ignore shard number when comparing URLs for equality. 
Change-Id: I9e31756f3f8122f7bc563ba2a2f1e19aee553f65
 2014-07-30 17:26:46 -07:00
mpl 2437ada83f foursquare: some attributes cleanup 
Change-Id: I7a1d794e58b47edeef06c3d181991c8aa1eb0534
 2014-07-31 01:50:36 +02:00
Brad Fitzpatrick d4f22274cd picasa: document and simplify Photo fields in picago. 
Be smarter about title vs description. Don't use caption as an attribute name.

Change-Id: I746fc6d88e27b6b85a8505456e6eb423443d7840
 2014-07-30 16:24:56 -07:00
Brad Fitzpatrick 95492813b1 picasa: clean up album attributes, add tests, fix keywords with spaces 
Change-Id: I32faae16334e3999731f3980502eb6a1a6d38595
 2014-07-30 15:50:34 -07:00
mpl c032a4a8e7 flickr: set primary photo as camliContentImage on album 
Change-Id: I3e805e3e64878d7f0057957c0be476a1a613cb21
 2014-07-31 23:17:44 +02:00
Brad Fitzpatrick 8f62edbc9e Merge "flickr: some attributes changes" 2014-07-29 22:37:54 +00:00
mpl 28d5307853 flickr: some attributes changes 
Also revert Total to an "int string", the doc was a lie and Aaron was
right.

Change-Id: I81ca7c8a3e3141beed1c4a3e5784a590a11f1399
 2014-07-30 23:13:57 +02:00
Brad Fitzpatrick 606acb5915 picasa: re-use existing permanode if it doesn't conflict 2014-07-29 14:54:43 -07:00
Brad Fitzpatrick 29c63cc2ab picasa, picago: support video, change struct definitions, add start of more tests 2014-07-29 11:44:44 -07:00
Brad Fitzpatrick 1763efbefe picasa: TODO about broken video support 
Change-Id: Ie44e1129b1965f379adb8599ffe7f7693fb02eab
 2014-07-28 23:15:15 -07:00
Brad Fitzpatrick b4c4aabbd4 picasa: more work towards permanode reuse 
Change-Id: Icf590e3593036c5a1a2764a800062f40eada09c1
 2014-07-28 23:14:20 -07:00
Brad Fitzpatrick e61c7b949f picasa: start of reusing existing permanode when importing photo 
Change-Id: Id43c60f2d5cfea27fb57adb0ca81b621360039b1
 2014-07-28 21:35:58 -07:00
Brad Fitzpatrick f5f02768b3 picasa: give a title to importer root permanode 
Change-Id: I0e259db52060518dfe33f31f0222b169a8c9e0ef
 2014-07-28 19:16:17 -07:00
Brad Fitzpatrick 9a3c619405 importer: add ChildPathObjectOrFunc to let importers find a fitting permanode on miss 
Change-Id: I016f30aa9e05a1024c2cdb7b1e8be0a675f73719
 2014-07-28 19:15:53 -07:00
Brad Fitzpatrick dadb9cca16 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-07-28 16:37:40 -07:00
Brad Fitzpatrick ea6785569d picasa: simplify photo import code, only redownload photo when necessary, do 3 at once. 
Change-Id: I99551ad90359ced164b47b12f121ef207b797430
 2014-07-28 16:37:19 -07:00
Brad Fitzpatrick 2695422f2e picasa: incremental album import, and key album off its ID, not name. 
Change-Id: I748e01950fa22eec024bb08ae22060b6939513dd
 2014-07-28 13:58:24 -07:00
Brad Fitzpatrick b8756b73a5 picasa: logging and TODO 
Change-Id: Icf010ad8b781dbbf66f03160d49089d8986853a0
 2014-07-27 13:13:01 -07:00
Brad Fitzpatrick e8816c072d picasa: cleanups, missing body.Close, start of incremental support, fix SummarizeAccount 
Change-Id: I497338743c7d85dea9f374215662543f4a9ad6a9
 2014-07-27 12:25:18 -07:00
Brad Fitzpatrick db09f2fe1a Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-07-25 16:03:53 -07:00
Brad Fitzpatrick 4021f812c6 importer, picasa: move ExtendedOAuth2 out of pkg importer, fix picasa 
The ExtendedOAuth2 code tried to be generic, but wasn't (had Picasa
comments?) and was only used by Picasa anyway.  We can move it later
if there are actually 2 users and it works for both.

At the same time, make Picasa actually work by writing a correct
IsAccountReady function.
 2014-07-25 16:02:43 -07:00
Brad Fitzpatrick 3a77e7bcb0 twitter: clarify purpose of getTopLevelNode and set titles better 
Change-Id: Ie4d88b927ccd63a943afbb220594b6cb0c96a2b9
 2014-07-25 14:54:48 -07:00
Brad Fitzpatrick d0fbfd9751 nodeattr: rename Content to CamliContent, since Content also exists. 
Change-Id: I305ca3cecf9b223a125fb9207966995c9dbf83b8
 2014-07-25 14:41:17 -07:00
Brad Fitzpatrick 997d8241b9 twitter: simplify getTopLevelNode 
Change-Id: Id25ad568a79d44144ac8aaf9652e31dc7de6e864
 2014-07-25 14:10:52 -07:00
Brad Fitzpatrick 0716bfdecf twitter: revert 98a73d966d; it changed its imported root structure 
I failed to catch that in review.

Change-Id: Ie88298a2f34595b27c6f98af6f31b61a29ebac1b
 2014-07-25 14:07:17 -07:00
Brad Fitzpatrick 8af06fa3ff picasa: work on making the instructions accurate & work 2014-07-25 13:37:47 -07:00
Brad Fitzpatrick fad3a60a77 picasa: some importer attribute cleanup 
Use nodeattr constants. Don't set lat/long when unknown.
 2014-07-25 13:24:46 -07:00
Brad Fitzpatrick f2b9657be6 nodeattr: fix up comment style 
And use nodeattr.Title in twitter.
 2014-07-25 13:16:10 -07:00
Brad Fitzpatrick bc2b485656 twitter: fix compilation error from 98a73d966d 2014-07-25 13:13:30 -07:00
mpl c3cc0ac681 Merge "importer/flickr: make test data for devcam" 2014-07-23 20:25:26 +00:00
mpl 409ec362e3 importer/flickr: make test data for devcam 
A few renamings in flickr.go too.

http://camlistore.org/issue/417

Change-Id: Ied28cfadc7a546f34b87f9a43462f82ee4cb71d6
 2014-07-29 00:22:06 +02:00
mpl 7e86255cac importer/flickr: set DateCreated attribute 
Also cleaned up a few vars.
Moved some attributes to nodeattr pkg.

Change-Id: Iea496d50b12e67132fe88f371a89534c2fe32761
 2014-07-25 00:57:10 +02:00
Brad Fitzpatrick 814dad41d4 Merge "twitter importer: meaningful names for permanodes" 2014-07-23 18:07:28 +00:00
mpl 2e939b0b35 importer/picasa: make test data for devcam 
http://camlistore.org/issue/417

Change-Id: If1a8771a863e1f7d0797ec5a8e50cd3573718c95
 2014-07-23 19:47:45 +02:00
mpl 83437b0f05 Merge "importer: wire run (context) HTTP client with host HTTP client" 2014-07-17 20:53:10 +00:00
mpl 748fd21e96 importer: wire run (context) HTTP client with host HTTP client 
Killing two birds with one stone: we don't need to set it in devcam
anymore now.

Also switched to using the short form (run's one) everywhere, for
consistency.

Change-Id: I1e72ab99a07392c4970b1ed19b6e460ff422a24a
 2014-07-22 15:40:06 +02:00
mpl f81b88de6e importer/foursquare: make test data for devcam 
http://camlistore.org/issue/417

Change-Id: I753ac91aae5bd876b28e4473e4cf00af460deb95
 2014-07-21 23:56:58 +02:00
mpl b82459424d devcam server --makethings 
Only creates twitter things for now.

http://camlistore.org/issue/417

Change-Id: I05d548e012586279e165c01468524ceb055ff17b
 2014-07-21 23:24:33 +02:00
mpl e68742473d importer/foursquare: minor typo 
Change-Id: Ic3c0d2f01ea0e6dc191500ed2856810e8c9581d6
 2014-07-22 00:12:51 +02:00
Tamás Gulácsi 925f9d3f86 picasa importer: remove Run-private HTTPClient 
A more complete version of Aaron Boodman's
https://camlistore-review.googlesource.com/#/c/3066/
in removing Transports. As Aaron pointed out,
(https://groups.google.com/d/msg/camlistore/9OwQKH6z7rY/840oQrRgsGIJ) it
is illegal for oauth.Transport.Transport to ever be another instance of
oauth.Transport.

So this commit tries hard to use Context's HTTPClient
everywhere and make sure to have its Transport an *oauth.Transport, but
that's Transport is not an *oauth.Transport, but the underlying
Transport.

To eliminate "403: Token not found" errors, we set
oauth.Config.AccessType, oauth.Config.ApprovalPrompt to
"offline", "force".
This rewards us with a non-empty Token.RefreshToken,
and store the full Token in AcctAttrOAuthToken.
This enables us to import continuously without user intervention.

Change-Id: I4defb346abfcf3f6c17597fc2dc23a6e536eda37
 2014-07-17 08:12:11 +02:00
mpl aa3cda0762 importer/twitter: add TestDataMaker to generate imported tweets 
Also fix first user_timeline call (no maxId).

http://camlistore.org/issue/417

Change-Id: I1157331bd876a6400e16d0fd18712d8c459fdc20
 2014-07-14 19:26:13 +02:00