Rooba
/
perkeep
mirror of https://github.com/perkeep/perkeep.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,774 Commits 44 Branches 11 Tags 59 MiB
Commit Graph

156 Commits

Author SHA1 Message Date
mpl 718c9d8b63 pkg/importer: use go4.org/ctxutil 
side-effect: fix an unchecked type assertion, and use a conventional
key for the HTTP client associated with the context (instead of our own
string).

Change-Id: I2701de3f41f1f1c56fe3b76bbf590c0e5b39bc72
 2015-12-16 15:30:09 +01:00
Tamás Gulácsi 9f7e1df32b Remove pkg/context 
Use golang.org/x/net/context instead.

Fixes #648.

Change-Id: I676a2f25458be97610a49d6f954f2102cbd373fa
 2015-12-12 23:09:02 +01:00
mpl 40be4d8da2 vendor: go4.org/jsonconfig (and errorutil) 
Previously pkg/jsonconfig and pkg/errorutil

Copied from go4.org at rev d1b8a2fb2de6160036e4801aa5e4d855571078b8

Change-Id: I673ed55b0825baa2607289b6082f205100261d7a
 2015-12-01 17:21:49 +01:00
Brad Fitzpatrick 9868aa0f8d Remove httputil.CloseBody relic. 2015-11-23 19:02:10 -08:00
mpl 12eddf9c19 vendor: go4.org/strutil go4.org/syncutil 
They were internal packages (under pkg), which we are now moving to
go4.org, so we in turn need to vendor them in now.

Change-Id: I92224f731404d0bd4ca1c57492bed37cb3367ed4
 2015-11-21 00:21:20 +01:00
Tamás Gulácsi eedf44ef20 pkg/importer/flickr: fix photosSearchItem unmarshal 
As Adrian Tritschler reported, Flickr returns Lat/Long
with inconsistent quoting: 0 / "-32.2".

Also ints ("total", for example) can come in quoted ("6236") and
unquoted (6236) form, too.

This commit utilizes an int and a float32 with specialized Unmarshaler.

Change-Id: Ic559059272ba457860bf56a8671d5abd430eb404
 2015-07-08 06:41:11 +02:00
mpl 1c8c8087c1 importer, twitter: fix deleted account type, differentiate account and root titles 
When being deleted the importer account node was changing from an
"importerAccount" to an "importer-deleted" nodeType. I fixed it to
"importerAccount-deleted" for consistency.

Also, both the importer account node and the import root node ended up
with the "username's Twitter Account" title which was looking confusing
in the UI. I fixed the root node title to be "username's Twitter Data"
instead.

Change-Id: I335856ca92aca897a05d94a5f3e7d438b00ecc3e
 2015-06-29 23:23:15 +02:00
Brad Fitzpatrick af36d78bd7 env: add IsDev and use it everywhere 
Change-Id: I5547ce989c05f1f48774f24abee41c489326baa0
 2015-04-02 05:55:01 -07:00
Brad Fitzpatrick 9bea14733d importer: support for disabling importers at start-up 
Change-Id: Ia2fc81f496ed020669a1c6ede489197d4a52ae89
 2015-01-18 19:02:02 -08:00
Brad Fitzpatrick 5eeed2aba5 Remove some code.google.com references. 
Change-Id: I85ccf92af9389300919eed61f62aae28cc24a820
 2014-12-11 17:33:17 +11:00
Emil Hessman e8ffb9a66c pkg: fix various vet reports 
Change-Id: I0b7fa09849650fe696e59ad5c5c73a034ab80faa
 2014-11-12 23:25:53 +01:00
Gina White f9a2388cec Pinboard importer: add test, fix bug 
Add test that imports three posts.  Fix bug the test uncovered.  We need
to make a copy of the post in the importBatch loop before handing it off
to asynchronous importPost call.

Change-Id: Id8dea72b03dd3666dde97e977c921a8955e67392
 2014-10-18 08:20:45 -07:00
Eric Drechsel 4800931d55 fix importer bloblink 
ui no longer accepts ?b=blobref form

Change-Id: I40fb0aa75fe1664d5b1446c0d4670a3348935c93
 2014-09-07 19:37:19 -07:00
Brad Fitzpatrick 720ec2bd31 importer: fix start-up race 
The status handler was trying to call methods on importer before
the *importer.Host had been initialized:

==================
WARNING: DATA RACE
Read by goroutine 31:
  camlistore.org/pkg/importer.(*importer).Accounts()
      /Users/bradfitz/src/camlistore.org/pkg/importer/importer.go:827 +0x344
  camlistore.org/pkg/importer.(*Host).AccountsStatus()
      /Users/bradfitz/src/camlistore.org/pkg/importer/importer.go:360 +0x1ea
  camlistore.org/pkg/server.(*StatusHandler).currentStatus()
      /Users/bradfitz/src/camlistore.org/pkg/server/status.go:166 +0x910
  camlistore.org/pkg/server.func·007()
      /Users/bradfitz/src/camlistore.org/pkg/server/status.go:73 +0x6e
Previous write by goroutine 15:
  camlistore.org/pkg/importer.(*Host).InitHandler()
      /Users/bradfitz/src/camlistore.org/pkg/importer/importer.go:402 +0x2d3
  camlistore.org/pkg/serverinit.(*Config).InstallHandlers()
      /Users/bradfitz/src/camlistore.org/pkg/serverinit/serverinit.go:583 +0x10b7
  camlistore.org/server/camlistored.Main()
      /Users/bradfitz/src/camlistore.org/server/camlistored/camlistored.go:427 +0xa94
Goroutine 31 (running) created at:
  camlistore.org/pkg/server.(*StatusHandler).InitHandler()
      /Users/bradfitz/src/camlistore.org/pkg/server/status.go:83 +0x207
  camlistore.org/pkg/serverinit.(*Config).InstallHandlers()
      /Users/bradfitz/src/camlistore.org/pkg/serverinit/serverinit.go:583 +0x10b7
  camlistore.org/server/camlistored.Main()
      /Users/bradfitz/src/camlistore.org/server/camlistored/camlistored.go:427 +0xa94
Goroutine 15 (running) created at:
  camlistore.org/server/camlistored.TestStarts()
      /Users/bradfitz/src/camlistore.org/server/camlistored/run_test.go:70 +0xc81
  testing.tRunner()
      /Users/bradfitz/go/src/pkg/testing/testing.go:427 +0x112

Change-Id: I5f4c7aaeb15f5b99e574a419c959b01e58b431af
 2014-09-04 08:26:58 -07:00
Gina White e9276c8ac8 First version of pinboard importer. 
Change-Id: I2b06b20b2609981b03103b4a26c9c4633abe8473
 2014-09-01 13:35:06 -07:00
Brad Fitzpatrick f4740e77e8 Aggregate all errors with their URLS in status.json in one place. 
Change-Id: If7a540fc3d1a6caa94b5f04d44de3207f15ac4cf
 2014-08-29 16:03:44 -07:00
Brad Fitzpatrick dbc1a681bc Add importers to status. 
For http://camlistore.org/issue/483

Change-Id: I2621c5484efb121a5e5e24debeff64ffc5139ad3
 2014-08-16 11:58:16 -07:00
mpl f54a51dc82 twitter: fix format verb in error 
Change-Id: I36e11ac4b506391f8b94c22fa0d28f652ade4a4c
 2014-08-06 16:21:42 +02:00
mpl f75081072d importers: newOauthClient refactoring 
Change-Id: Ic5864ab2f7bb63bd801fe965c68b36a245edc447
 2014-08-02 01:37:43 +02:00
mpl 6d269cb37b importers: some refactoring around oauthContext 
Change-Id: I0a7e1dabb9efdaedbda138cdd1c52afcddee5210
 2014-08-02 00:50:00 +02:00
Brad Fitzpatrick 10252d1951 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-07-31 14:35:30 -07:00
Brad Fitzpatrick 8fbd54c09b picasa: set url property on photos and albums 
Change-Id: If89ee61138e4fb80e68f87130c5fe189b7d82582
 2014-07-31 14:34:52 -07:00
Brad Fitzpatrick 1ff2918602 importers: add SupportsIncremental accessor 
If an importer is efficient at running regularly, it returns true.

If it's not true, the UI won't allow turning on automatic runs.

Flickr doesn't set it (yet).
 2014-07-31 11:34:31 -07:00
Brad Fitzpatrick b3637337be Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-07-30 17:28:02 -07:00
Brad Fitzpatrick 4ea387545a picasa: ignore shard number when comparing URLs for equality. 
Change-Id: I9e31756f3f8122f7bc563ba2a2f1e19aee553f65
 2014-07-30 17:26:46 -07:00
mpl 2437ada83f foursquare: some attributes cleanup 
Change-Id: I7a1d794e58b47edeef06c3d181991c8aa1eb0534
 2014-07-31 01:50:36 +02:00
Brad Fitzpatrick d4f22274cd picasa: document and simplify Photo fields in picago. 
Be smarter about title vs description. Don't use caption as an attribute name.

Change-Id: I746fc6d88e27b6b85a8505456e6eb423443d7840
 2014-07-30 16:24:56 -07:00
Brad Fitzpatrick 95492813b1 picasa: clean up album attributes, add tests, fix keywords with spaces 
Change-Id: I32faae16334e3999731f3980502eb6a1a6d38595
 2014-07-30 15:50:34 -07:00
mpl c032a4a8e7 flickr: set primary photo as camliContentImage on album 
Change-Id: I3e805e3e64878d7f0057957c0be476a1a613cb21
 2014-07-31 23:17:44 +02:00
Brad Fitzpatrick 8f62edbc9e Merge "flickr: some attributes changes" 2014-07-29 22:37:54 +00:00
mpl 28d5307853 flickr: some attributes changes 
Also revert Total to an "int string", the doc was a lie and Aaron was
right.

Change-Id: I81ca7c8a3e3141beed1c4a3e5784a590a11f1399
 2014-07-30 23:13:57 +02:00
Brad Fitzpatrick 606acb5915 picasa: re-use existing permanode if it doesn't conflict 2014-07-29 14:54:43 -07:00
Brad Fitzpatrick 29c63cc2ab picasa, picago: support video, change struct definitions, add start of more tests 2014-07-29 11:44:44 -07:00
Brad Fitzpatrick 1763efbefe picasa: TODO about broken video support 
Change-Id: Ie44e1129b1965f379adb8599ffe7f7693fb02eab
 2014-07-28 23:15:15 -07:00
Brad Fitzpatrick b4c4aabbd4 picasa: more work towards permanode reuse 
Change-Id: Icf590e3593036c5a1a2764a800062f40eada09c1
 2014-07-28 23:14:20 -07:00
Brad Fitzpatrick e61c7b949f picasa: start of reusing existing permanode when importing photo 
Change-Id: Id43c60f2d5cfea27fb57adb0ca81b621360039b1
 2014-07-28 21:35:58 -07:00
Brad Fitzpatrick f5f02768b3 picasa: give a title to importer root permanode 
Change-Id: I0e259db52060518dfe33f31f0222b169a8c9e0ef
 2014-07-28 19:16:17 -07:00
Brad Fitzpatrick 9a3c619405 importer: add ChildPathObjectOrFunc to let importers find a fitting permanode on miss 
Change-Id: I016f30aa9e05a1024c2cdb7b1e8be0a675f73719
 2014-07-28 19:15:53 -07:00
Brad Fitzpatrick dadb9cca16 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-07-28 16:37:40 -07:00
Brad Fitzpatrick ea6785569d picasa: simplify photo import code, only redownload photo when necessary, do 3 at once. 
Change-Id: I99551ad90359ced164b47b12f121ef207b797430
 2014-07-28 16:37:19 -07:00
Brad Fitzpatrick 2695422f2e picasa: incremental album import, and key album off its ID, not name. 
Change-Id: I748e01950fa22eec024bb08ae22060b6939513dd
 2014-07-28 13:58:24 -07:00
Brad Fitzpatrick b8756b73a5 picasa: logging and TODO 
Change-Id: Icf010ad8b781dbbf66f03160d49089d8986853a0
 2014-07-27 13:13:01 -07:00
Brad Fitzpatrick e8816c072d picasa: cleanups, missing body.Close, start of incremental support, fix SummarizeAccount 
Change-Id: I497338743c7d85dea9f374215662543f4a9ad6a9
 2014-07-27 12:25:18 -07:00
Brad Fitzpatrick db09f2fe1a Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-07-25 16:03:53 -07:00
Brad Fitzpatrick 4021f812c6 importer, picasa: move ExtendedOAuth2 out of pkg importer, fix picasa 
The ExtendedOAuth2 code tried to be generic, but wasn't (had Picasa
comments?) and was only used by Picasa anyway.  We can move it later
if there are actually 2 users and it works for both.

At the same time, make Picasa actually work by writing a correct
IsAccountReady function.
 2014-07-25 16:02:43 -07:00
Brad Fitzpatrick 3a77e7bcb0 twitter: clarify purpose of getTopLevelNode and set titles better 
Change-Id: Ie4d88b927ccd63a943afbb220594b6cb0c96a2b9
 2014-07-25 14:54:48 -07:00
Brad Fitzpatrick d0fbfd9751 nodeattr: rename Content to CamliContent, since Content also exists. 
Change-Id: I305ca3cecf9b223a125fb9207966995c9dbf83b8
 2014-07-25 14:41:17 -07:00
Brad Fitzpatrick 997d8241b9 twitter: simplify getTopLevelNode 
Change-Id: Id25ad568a79d44144ac8aaf9652e31dc7de6e864
 2014-07-25 14:10:52 -07:00
Brad Fitzpatrick 0716bfdecf twitter: revert 98a73d966d; it changed its imported root structure 
I failed to catch that in review.

Change-Id: Ie88298a2f34595b27c6f98af6f31b61a29ebac1b
 2014-07-25 14:07:17 -07:00
Brad Fitzpatrick 8af06fa3ff picasa: work on making the instructions accurate & work 2014-07-25 13:37:47 -07:00