A working importer for the Instapaper service. The importer imports
bookmarks, full text articles, and highlights. A blob item handler for
Instapaper highlight permanodes is included to show summarized highlight
content from the server UI.
Also, this CL updates github.com/garyburd.com/go-auth to rev
bca2e7f09a178fd36b034107a00e2323bca6a82e in order to get support for
XAuth requests.
Fixes#1208
Change-Id: I72e0c40b245c7eec4a44bb475fcaa96a0ee9a1c5
And also speed up /importer/TYPE page's query by saying it's unsorted, which
gets us a better index. Hacky workaround. Should fix it in the query planner
later instead.
Fixes#1105
Change-Id: I140ffb40456ec802844584568d2f1a7a38b26e41
Foursquare was rebranded Swarm (for check-ins) some time ago, so
rename our implementation to match the upstream branding. But still
store "foursquare" as the permanode importer type, to not break old
users. That override is now added to the new Properties struct, and
all importers now return their properties.
More stuff will be added to Properties later: WIP flag, bug link,
proper title, icon?
Change-Id: I31cbe2feec3dbf9c6bdb0c866056d9c6966313e3
I had intended for this to be a small change.
I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.
And then it got a bit infectious and ended up touching everything.
I ended up doing SubFetch in the process by necessity.
At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.
Updates #733
Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
The Picasa importer was deadlocking inside the golang.org/x/oauth2
package because the Picasa code was mutating its parent's context and
accidentally creating a cycle of *oauth2.reuseTokenSource values
(double-acquisition of the reuseTokenSource's sync.Mutex was the
deadlock)
The Picasa code really just wanted to create a subcontext from the
base context, where the subcontext did the oauth2 wrapping, but then
picasa was storing that subcontext in the
camlistore.org/pkg/importer-provided mutable "RunContext" structure's
public, embedded "context.Context" field. Unfortunately, the picasa
importer code had already used that *RunContext (pre-mutation) as the
context.Context for the base HTTP client.
This change changes the camlistore.org/pkg/importer.RunContext to not
have a public embedded Context field. Instead, it only has a read-only
accessor to get at the base context, similar to Go 1.7's
net/http.Request.Context() field. This will prevent this class of
error in the future.
Fixes#698
Change-Id: I9114b64aa1bfd7ea83b36ea1ab18ed88b4a44f7e
The import path was added to the go file that included the package
documentation if one existed. Otherwise, I used what seemed to be the
primary file for the package.
Fixes#689
Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
side-effect: fix an unchecked type assertion, and use a conventional
key for the HTTP client associated with the context (instead of our own
string).
Change-Id: I2701de3f41f1f1c56fe3b76bbf590c0e5b39bc72
As Adrian Tritschler reported, Flickr returns Lat/Long
with inconsistent quoting: 0 / "-32.2".
Also ints ("total", for example) can come in quoted ("6236") and
unquoted (6236) form, too.
This commit utilizes an int and a float32 with specialized Unmarshaler.
Change-Id: Ic559059272ba457860bf56a8671d5abd430eb404
If an importer is efficient at running regularly, it returns true.
If it's not true, the UI won't allow turning on automatic runs.
Flickr doesn't set it (yet).
Works but only lightly tested.
Could use lots of refactoring with code common with twitter.
Needs a test too.
Change-Id: I07f72d29acbc0226aeeedba0afe0bf7eeeac5ad3
httputil.CloseBody closes a http.Response.Body nicely, such that it's
more likely to be re-used.
httputil.DecodeJSON parses JSON from a Response.Body and closes the
body. It also keeps a temporary copy of the input, for error messages.
Then use them everywhere, to improve HTTP TCP connection re-use.
And misc other clean-ups.
Change-Id: I5dc44be0165b13659459ed29ce5ab44c17d9739f
Remove the mutex from auth.go because I believe the
mutex in importer.go should guaranteeing only one
flickr importer is running at a time.
Change-Id: Ice971c6c041cf21fdb2bc93cb15f861dd1c8c1d7
Julien Bisconti