A working importer for the Instapaper service. The importer imports
bookmarks, full text articles, and highlights. A blob item handler for
Instapaper highlight permanodes is included to show summarized highlight
content from the server UI.
Also, this CL updates github.com/garyburd.com/go-auth to rev
bca2e7f09a178fd36b034107a00e2323bca6a82e in order to get support for
XAuth requests.
Fixes#1208
Change-Id: I72e0c40b245c7eec4a44bb475fcaa96a0ee9a1c5
Wich also means updating github.com/gopherjs/gopherjs to
rev 0210a2f0f73c96103378b0b935f39868e5731809
Fixes#1211
Change-Id: I9ae580bffda523237d471942f39ae157ff4bd303
To rev 99ff426eb706cffe92ff3d058e168b278cabf7c7
To support the new default authentication plugin in MySQL 8.
Interestingly, the update also revealed an error unseen so far.
Each of the pkg/sorted/mysql test uses a docker container. If we did not
kill the container at the end of a test, docker would use a new,
different, IP address for each mysql container (which would dodge the
error I'm about to mention). But since we do clean up and remove the
container for each test, then each test gets a container with the same
(first in the range) IP address, because docker "thinks" everything
about the previous container was properly removed.
However, it turns out that since we had forgotten to tell the client
code to close the connection to the mysql server, then the server-side
IP:port is still seen as occupied (even though the server is gone). And
since the second (or nth) test client tries connecting to the same
IP:port, the connection fails.
TL;DR: we also need to close the connection to the DB on the client
side, even though the server in container goes away after each test.
Fixes#1114
Change-Id: I76ce4ac2ee7703f92a9431116cd82ab75da541f7
vendor: add github.com/aws/aws-sdk-go at rev bc3f534c19ffdf835e524e11f0f825b3eaf541c3
This changes all uses of the s3 client to be the official aws-sdk-go. As
such, it also deletes the previous client maintained in internal.
There are a few notable changes that come with this:
1. An 'aws_region' key is added to the low-level config. The
'get-bucket-location' call is used to identify the correct region
rather than using the redirect returned to determine it.
2. Deleting multiple objects at once is much faster (batches of up to
1000 vs 1 at a time).
3. Retriable errors are retried automatically (see #1184).
4. Newer APIs are used for some operations (v2 / batch apis).
An unfortunate result of 4 is that this is technically a backwards
incompatible change. An IAM policy which used to be sufficient for
perkeep may no longer permit some of the new api calls (even if they're
not doing anything different really).
Fixes#1184, #911
Change-Id: I077fe88def18b7e9a14267820773245e7003634c
to c126467f60eb25f8f27e5a981f32a87e3965053f
For acme tls-alpn-01 support.
And enable it in perkeepd.
Remove http-01 challenge support.
Fixes#1078 (by making it obsolote)
Change-Id: I45220b6269024a2ff7a181f69cf708869705dca7
to bf2c84553c7827a1cb6cecf19424fdb93b887d5b.
This is required per the change mentioned in
https://github.com/myitcv/react/issues/116#issuecomment-405631245.
Without this change, 'dep ensure' can't be run, unless the upstream
myitcv.io html is reverted/updated or dep fixes a few issues allowing
project-root calculations to be overridden.
Change-Id: I63ab98b8701dcea753d4e67828f47227801638be
The Mastodon importer uses the Mastodon API (implemented by Mastodon and
Pleroma) to import posts into Perkeep. The webui also now recognizes
Mastodon posts, and displays their text.
The importer does some things differently than most other Perkeep
importers, since it needs to be able to import from arbitrary URLs. It
does not share any credentials globally between all accounts, to enable
use of multiple accounts on multiple servers.
This also introduces new dependencies of go-mastodon and linkheader
(both under the MIT license).
Resolves#1183
Change-Id: I9a532bbd5720992482fc758f14286c11d5bf6ef0
This change:
-on startup, makes perkeepd log instruction about the Google geocoding
key, if it is not found in Perkeep configuration directory.
-in the web UI, makes search query errors from the search session show
up in a dialog, instead of just being logged in the javascript console.
Among other things, it will allow users to be more readily informed
that their loc: queries are failing because they don't have a geocode
key setup.
According to the Google geocoding team, we should not share a geocoding
API key (too risky/expensive), and using an API key is the only way to
make geocoding requests. So we have to let users deal with it themselves
unfortunately.
Updates #1115Fixes#1116
Change-Id: I461e7f06a04ffd874678ab8233185999024cb3d8
to db3ee9ee8931e51af7158114b9d02bf93d1a7811
and also update golang.org/x/crytpo to b47b1587369238182299fe4dad77d05b8b461e06
The golang.org/x/crypto change is to correct a typo in 'Gopkg.toml'
which resulted in the 'Gopkg.lock' not containing an actually locked
down revision.
The leveldb change fixes#1185
Change-Id: If14a0c4d6fc011716c2371351f2572b0ca2b588a
Revert back to using master of https://github.com/gopherjs/gopherjs.
Includes:
* recently-merged support for using a vendored GopherJS
* removes dependency on legacy golang.org/x/tools/go/gcimporter15,
switching to golang.org/x/tools/go/gcexportdata
Change-Id: I1f8b4c829bcfa8562614d9f8368c1d58edd10fab
dep had computed it was needed, but apparently if I have it in my GOPATH
(and not in vendor), then building works (which seems weird?), which is
what made me miss it previously.
Change-Id: I979a107aea735e4f505882e4230a20c74eb5df04
To rev fb35d3c3290d09f3524b684b5d42d0063c227158
Gopkg.lock update withheld on purpose, as something weird is going on
with myitcv.io, and I I want to ask the maintainer about it first.
Change-Id: I4fc5e3bf022371b3114210f30be1d16d8d89e741
Remove the following transitive packages with constraints. dep
does not enforce these:
- github.com/cznic/fileutil
- github.com/cznic/lldb
- github.com/cznic/mathutil
- github.com/edsrzf/mmap-go
- github.com/golang/protobuf
- github.com/gopherjs/jsbuiltin
- github.com/pkg/errors
Rerun dep ensure -update and resync packages.
Change-Id: I3a6613459628dc963516e840a453d2bc2d25835a
It is being automatically removed by dep ensure since
3dfd895def but its removal breaks the web
UI.
This change alone is therefore probably not enough, we need to teach dep not
to remove it.
Change-Id: I61d50e703e786fdebc10446784dc84d43a0c1620
Use the now renamed libsqlite3 build pragma. This requires
having sqlite3 installed on the system instead of using the
c code bundled in vendor (which is removed by the dep tool
by default)
1.6.0 corresponds with rev
github.com/mattn/go-sqlite3@6c771bb9887719704b210e87e934f08be014bdb1
Changelog: See https://github.com/mattn/go-sqlite3/releases
Change-Id: I48fc0bc005715b83805837f544e7fa7875e6d56a
As the priority is to fix GCE instances, the port for the http-01
challenge is not configurable for now (80) even not on GCE, but it will
be in a follow-up change.
update golang.org/x/crypto/* (for acme) to rev
13931e22f9e72ea58bb73048bc752b48c6d4d4ac
update golang.org/x/sys/* to rev
fff93fa7cd278d84afc205751523809c464168ab (because unix is a dep of
crypto)
remove warning about Let's Encrypt security issue from pkg/deploy/gce
I had to manually exclude vendor/golang.org/x/crypto/acme/jws_test.go
for now because it contains a private key, and git whines about it, and
i could not override it.
Fixes#1033
Change-Id: Ie4f2049e97892dee9ab513300a5f12e64976aec8
This CL is about levelDB as the HaveCache for camput, and there are
several aspects to it. To describe it, I'll take the particular example
where you want to add many permanodes (~33k) to a given set, with
camput. Something like:
for _, blob := range blobs {
do("camput attr -add sha1-foobar camliMember " + blob)
}
In a "normal" levelDB use case, everytime the number of level-0 .ldb
files goes over 4 (by default), a background compaction task is
started to transform these SST into level-1 ones, and remove the level-0
ones.
However, since our particular camput call is very short lived
(especially on a local Perkeep), not only might there be not enough time
for the compaction to be triggered, but even if it is, when the DB is
flushed (on a Close call), any ongoing compactions are cancelled. This
makes level-0 compactions very unlikely to happen on short-lived camput
calls. As a result, the number of level-0 files keeps growing until
levelDB fails while trying to open them all, because it hits the current
process ulimit.
Now, in this CL, what we propose is to systematically force a compaction
as soon as the HaveCache is opened. It is not scheduled concurrently, so
we are sure that the compaction happens before the DB actually gets used
by camput. This seems to make sure that the number of level-0 tables
never grows too much. With this change, I was able to run the above
example on 33K blobs without hitting the ulimit error.
However, it should be noted that potential problems might remain. The
compaction for levels above 0 is triggered based only on the total size
of the level (e.g. at 100MB by default for level-1), and not on the
number of files. Since we're creating many tiny tables (basically 1
entry per table), the number of files grows very fast while the total
size does not, and the compaction does not get triggered, even if forced
with CompactRange. This does not seem to be a problem for our use case,
as levelDB does not seem to need to open many of the level-1 files at
the same time, so we're not hitting the ulimit problem because of that.
If needed, there's at least one way this problem (if it is one?) could
be fixed: make the compaction trigger on other conditions, such as
number of files per level. I've experimented with it (forcing the
level-1 compaction to trigger at the 100 files limit), and it seems to
be working. But I had to do change the goleveldb code itself, and I
don't think levelDB implementations are supposed to do that.
For information, at the end of the run on the 33K blobs:
$ du -sch *.ldb
...
83M total
$ ll | wc -l
20988
And indeed, when asking for leveldb.stats on the table:
Level | Tables | Size(MB) |
-------+------------+---------------+
0 | 1 | 0.00015 |
1 | 20981 | 3.47307 |
Also, update github.com/syndtr/goleveldb to
34011bf325bce385408353a30b101fe5e923eb6e
And remove github.com/syndtr/gosnappy as goleveldb does not use it
anymore.
Also apply this change to StatCache.
Fixes#1008
Change-Id: If9f790a003e67f3c075881470e52e5f2174afa73
This upgrades the markdown conversion library to v2 and
switches camweb to use the new, simplified API.
Change-Id: I710110431fdfa32104c591abdf9b01c74a214f48
Upgrades mysql driver to a versioned release, fixes a minor issue
in dbinit that caused SHOW DATABASES queries to fail and improves
error messages to help diagnose such problems.
Changelog: vendor/github.com/go-sql-driver/mysql/CHANGELOG.md
Revision: go-sql-driver/mysql@a0583e0143
Change-Id: I4f5fdee60b30eca85564c452f72774e9c15828f3
Gopkg.toml / Gopkg.lock contain the full set of revisions used by
Perkeep now. You can remove your entire vendor tree and
dep ensure && dep prune
will restore you to a known good state.
This final change contains various files that were missing (like
LICENSE/README.md/tests) and also prunes a number of packages that
are transitively unused.
Fixes#889
Change-Id: I6ee6c4a4e14e2de6b6f33620eb7e2c44e97ce91b
Uses revisions as close to current usage as possible.
Addresses issues with repos that have different per-directory
revisions.
- cloud.google.com/go
rev: b70ccc799b9d019708c3eb9395acef6e3f6b7bc8
- github.com/golang/protobuf
rev: 1e59b77b52bf8e4b449a57e6f79f21226d571845
- google.golang.org/api
rev: 48e49d1645e228d1c50c3d54fb476b2224477303
- google.golang.org/appengine
rev: 150dc57a1b433e64154302bdc40b6bb8aefa313a
version: v1.0.0
- google.golang.org/genproto
rev: 08f135d1a31b6ba454287638a3ce23a55adace6f
Addresses #889
Change-Id: I41610d2f409d7a558a59e3018094a124efc100c5
Remove goexif local changes (a TODO, README and gofmt)
Update to rev: 709fab3d192d7c62f86043caff1e7e3fb0f42bd8
Use dep to prune unused files
Change-Id: Icce8885917b798b25eaeed4c98ff393ce0db66b8
This pulls in a clean upstream version of go-sqlite3 at revision
919cf4144a09abfd877330bbe735f3a243a84537 using the dep tool.
The current method of using the -sqlite flag in make.go appears to
enable conditional builds with/without sqlite3, obsoleting the
need for local modifications.
We also manually remove sqlite
Addresses: #889
Change-Id: I4d2c107cd12608ac0bcb7c0607629ecb2d7f880f
The mongo integration was using a very old package. It's using
a new namespace now. Upgrade and adjust all call points
Removes labix.org/v2/mgo
Introduces gopkg.in/mgo.v2 from branch v2 with revision
3f83fa5005286a7fe593b055f0d7771a7dce4655
Change-Id: I2784fca941998460f58e0ac8d3d51286401590b5
Mathieu Lonjaret