A working importer for the Instapaper service. The importer imports
bookmarks, full text articles, and highlights. A blob item handler for
Instapaper highlight permanodes is included to show summarized highlight
content from the server UI.
Also, this CL updates github.com/garyburd.com/go-auth to rev
bca2e7f09a178fd36b034107a00e2323bca6a82e in order to get support for
XAuth requests.
Fixes#1208
Change-Id: I72e0c40b245c7eec4a44bb475fcaa96a0ee9a1c5
The javascript code was taking into account the OmitAuthToken signal
that there should be no auth, but it hadn't been done in the gopherjs
code yet as well.
Fixes#1221
Change-Id: I3eda0cc12c99664739d6073cc1aeba1aeed8a88a
As demonstrated by the test case added in this commit, a
search panics if the DirConstraint contains a LogicalConstraint
with Op set to "not". This is because the isFileOrDirConstraint
function doesn't account for unset "B".
Change-Id: I111ace62486160254ba22ab1498de78cde2a5e9b
To rev 99ff426eb706cffe92ff3d058e168b278cabf7c7
To support the new default authentication plugin in MySQL 8.
Interestingly, the update also revealed an error unseen so far.
Each of the pkg/sorted/mysql test uses a docker container. If we did not
kill the container at the end of a test, docker would use a new,
different, IP address for each mysql container (which would dodge the
error I'm about to mention). But since we do clean up and remove the
container for each test, then each test gets a container with the same
(first in the range) IP address, because docker "thinks" everything
about the previous container was properly removed.
However, it turns out that since we had forgotten to tell the client
code to close the connection to the mysql server, then the server-side
IP:port is still seen as occupied (even though the server is gone). And
since the second (or nth) test client tries connecting to the same
IP:port, the connection fails.
TL;DR: we also need to close the connection to the DB on the client
side, even though the server in container goes away after each test.
Fixes#1114
Change-Id: I76ce4ac2ee7703f92a9431116cd82ab75da541f7
vendor: add github.com/aws/aws-sdk-go at rev bc3f534c19ffdf835e524e11f0f825b3eaf541c3
This changes all uses of the s3 client to be the official aws-sdk-go. As
such, it also deletes the previous client maintained in internal.
There are a few notable changes that come with this:
1. An 'aws_region' key is added to the low-level config. The
'get-bucket-location' call is used to identify the correct region
rather than using the redirect returned to determine it.
2. Deleting multiple objects at once is much faster (batches of up to
1000 vs 1 at a time).
3. Retriable errors are retried automatically (see #1184).
4. Newer APIs are used for some operations (v2 / batch apis).
An unfortunate result of 4 is that this is technically a backwards
incompatible change. An IAM policy which used to be sufficient for
perkeep may no longer permit some of the new api calls (even if they're
not doing anything different really).
Fixes#1184, #911
Change-Id: I077fe88def18b7e9a14267820773245e7003634c
Calling 'GetPermanodesWithAttr', as currently written, will return an
empty list for any unindexed parameters.
Rather than always returning an empty list if the user asks for a
unindexed attribute, error out.
Also add documentation of this potential foot-gun for users of the
client.
Change-Id: Iadc1b5a00aa709584affb89a1397e82aaffb692f
to c126467f60eb25f8f27e5a981f32a87e3965053f
For acme tls-alpn-01 support.
And enable it in perkeepd.
Remove http-01 challenge support.
Fixes#1078 (by making it obsolote)
Change-Id: I45220b6269024a2ff7a181f69cf708869705dca7
The Mastodon importer uses the Mastodon API (implemented by Mastodon and
Pleroma) to import posts into Perkeep. The webui also now recognizes
Mastodon posts, and displays their text.
The importer does some things differently than most other Perkeep
importers, since it needs to be able to import from arbitrary URLs. It
does not share any credentials globally between all accounts, to enable
use of multiple accounts on multiple servers.
This also introduces new dependencies of go-mastodon and linkheader
(both under the MIT license).
Resolves#1183
Change-Id: I9a532bbd5720992482fc758f14286c11d5bf6ef0
The s3 blobserver's endpoint handling is complex enough it makes sense
to record some of its constraints in a unit test.
Change-Id: Ife1637a33ba5c3265b3fdfd75307d7784e7c9524
-busy/munching pudgy is replaced with a "dancing" keepy
-sad pudgy replaced by a sad keepy
-everywhere else is normal keepy (which is one sprite of the dancing
keepy animation)
-we still need an "interested" keepy for when the user is about to drop
something on the d&d area
-glitch embedy directory has not been removed yet, to keep the pudgy
animations technique as a reference for future work.
We also need to update the animation in the GCE launcher.
Updates #1177
Change-Id: I95fe53cb51f93b8f498c8f026eddc07285c20db7
In 877eafa49b , pkg/server started
importing pkg/client , since the server has to be able to fetch blobs
from a share claim of another server. This introduced a circular
dependency in tests, as pkg/client/upload_test.go instantiates a server.
As the test uses both a client and a server, and tests uploading between
the two, it seems fitting to move it to pkg/test/integration, which
breaks the circular dep.
Change-Id: I2458ffd72912ff6e9c1a0f67e187d54cc895c2e9
This change:
-on startup, makes perkeepd log instruction about the Google geocoding
key, if it is not found in Perkeep configuration directory.
-in the web UI, makes search query errors from the search session show
up in a dialog, instead of just being logged in the javascript console.
Among other things, it will allow users to be more readily informed
that their loc: queries are failing because they don't have a geocode
key setup.
According to the Google geocoding team, we should not share a geocoding
API key (too risky/expensive), and using an API key is the only way to
make geocoding requests. So we have to let users deal with it themselves
unfortunately.
Updates #1115Fixes#1116
Change-Id: I461e7f06a04ffd874678ab8233185999024cb3d8
This change adds an 'Import share' entry in the main menu bar. This
entry creates a dialog that allows the user to enter the URL of a share
claim (the "shareURL" in the following).
That shareURL is sent to the server, which takes care of importing all
the blobs transitively reachable (directory children, and files
contents) through the share claim.
The web UI also keeps on polling the server to update the dialog on the
state of the importing process.
To handle the posted shareURL, and the polling, the server now has a new
HTTP handler: the shareImporter, installed at /ui/importshare.
In a future CL, another approach would be to integrate this handler as
yet another importer. But it would probably be better to do so once we have
a way for importers to report on their progress at any time. For now this
importer has its own dialog to do so, similar to the one for uploads.
Change-Id: I94cd03f834d7d1e9f4ec5c016ffdf16d911856c5
The storage test for s3 have been broken since
https://perkeep-review.googlesource.com/c/perkeep/+/15846.
The introduction of a proxycache made a type assertion in the test
panic, and thus the tests reliably failed when run.
The simplest fix for now is to get them back to their previous passing
state by disabling the proxycache for tests.
Change-Id: Ib75b5eabd36decebd8785ba50d4b979a7a0db653
In order to facilitate manual recovery in severe scenarios, the
--keep-going flag will make the server come back online even when there
are index or recovery errors. This will allow further querying on the
unhealthy server to identify missing data or to extract data for
migration to a new/healthy instance.
Fixes#1166
Change-Id: I4153905757d9e7d8014780dd5660a862cb8ba1ab
To support composition, all optional interfaces in Go need a way to
say, at runtime, that "oh, nevermind, I defensively implemented this
interface but it turns out the thing I'm wrapping doesn't support it".
We learned that lesson only in the past few years, but SubFetcher
predates that.
So do that here and make proxycache implemented SubFetcher and make
everything respect the fallback oh-nevermind error value.
Fixes#1170
Change-Id: I3c0958cf6692a324e98f526571d10a352a8e18b4
I have tested that, starting with an empty $GOPATH/bin, after running
devcam test, $GOPATH/bin still has none of the Perkeep related binaries.
Fixes#1139
Change-Id: I60b63ff917cf91c8d424b448f874f40a18e1c23b
Mathieu Lonjaret