This is in relation to #580; we will need something like it to hook
Bleve up to the index. Currently only the SQL backends are implemented;
the rest is still TODO.
This also updates the version of the postgres library that we're using;
the old one didn't support setting the isolation level in TxOptions.
Change-Id: I14fdf74832e088d164b757417bfbb500a752d216
This change:
1) makes perkeep.org/server/perkeepd/ui/goui a module on its own, so that it
gets ignored by go mod with respect to perkeep.org. Because there's no reason it
should appear in perkeep.org/go.mod, as it is only used by gopherjs to rebuild
the web UI code.
2) updates a few of our dependencies to keep up with upstream changes (go get
-u, go mod tidy, go mod vendor).
3) readds manually (since go mod vendor removes them)
vendor/github.com/gopherjs/gopherjs (and its deps) as well as vendor/myitcv.io
because they are needed when one wants to rebuild the web UI.
As a result I think we can now drop dep, so it is done in this change as
well.
Change-Id: I155819163e15e214a6ba44bab2c19ae171d83d1b
To rev 99ff426eb706cffe92ff3d058e168b278cabf7c7
To support the new default authentication plugin in MySQL 8.
Interestingly, the update also revealed an error unseen so far.
Each of the pkg/sorted/mysql test uses a docker container. If we did not
kill the container at the end of a test, docker would use a new,
different, IP address for each mysql container (which would dodge the
error I'm about to mention). But since we do clean up and remove the
container for each test, then each test gets a container with the same
(first in the range) IP address, because docker "thinks" everything
about the previous container was properly removed.
However, it turns out that since we had forgotten to tell the client
code to close the connection to the mysql server, then the server-side
IP:port is still seen as occupied (even though the server is gone). And
since the second (or nth) test client tries connecting to the same
IP:port, the connection fails.
TL;DR: we also need to close the connection to the DB on the client
side, even though the server in container goes away after each test.
Fixes#1114
Change-Id: I76ce4ac2ee7703f92a9431116cd82ab75da541f7
Remove explicit version checking using pkg-config and just check if
setting journal_mode=WAL succeeds. Remove -wal command line
flag from pk tool and also remove duplicated logic for db
handling.
Fixes#840
Change-Id: I72874a5bb9c5bb3a7be552a65e5ddb0c12fd726e
Previously, `create table ...` sql migrations were being made without the
database existing. This resulted in a panic and error like:
pq: database "pk_3a94488d_blobpacked" does not exist
There seems to be an upstream issue with our postgres library in which
`CREATE DATABASE ...` queries are not prepared so we have to build the
sql manually. For now I've added a regex to make sure we don't allow
anything too crazy in.
Fixes#1022
Change-Id: I0da16759e9219347bb11713b92337021546f9d57
The mongo integration was using a very old package. It's using
a new namespace now. Upgrade and adjust all call points
Removes labix.org/v2/mgo
Introduces gopkg.in/mgo.v2 from branch v2 with revision
3f83fa5005286a7fe593b055f0d7771a7dce4655
Change-Id: I2784fca941998460f58e0ac8d3d51286401590b5
Notably: pkg/misc all moves.
And pkg/googlestorage is deleted, since it's not used. Only the
x/net/http2/h2demo code used to use it, but that ended in
https://go-review.googlesource.com/33230 (our vendored code is old).
So just nuke that dir for now. When it's refreshed, it'll either be
gone (dep prune) or new enough to not need googlestorage.
Also move pkg/pools, pkg/leak, and pkg/geocode to internal.
More remains.
Change-Id: I2640c4d18424062fdb8461ba451f1ce26719ae9d
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
This switches most usages of the pre-1.7 context library to use the
standard library. Remaining usages are in:
app/publisher/main.go
pkg/fs/...
Change-Id: Ia74acc39499dcb39892342a2c9a2776537cf49f1
When reindexing on a (My)SQL based sorted.KeyValue, we should recreate
the database schema from scratch, which means dropping the tables.
However, index.Reindex just calls Wipe on the newly created
sorted.KeyValue, which only deletes the rows, and does not drop the
tables.
Therefore, this CL changes the implementation of Wipe in the MySQL case,
so that it takes care of dropping the tables, and doing everything that
needs to be done afterwards to set up the sorted.KeyValue.
In addition, with the introduction of the sorted.NeedWipeError, we detect
upon initialization of a sorted.KeyValue if it failed because it needed
a schema update. If that is the case, and we're in reindex mode, we can
fix the sorted.KeyValue with a Wipe and carry on.
Finally, we introduce the new sorted.NewKeyValueMaybeWipe function that
automatically wipes a KeyValue when a NeedWipeError was returned upon
its creation.
Next, do the same with other sorted SQLs.
Fixes#806
Change-Id: I2032781cbf453a364880bd3e2e8b3c09aac7aed9
rm google.golang.org/cloud
add cloud.google.com/go at a47b182e769f5e75f5fc927ff6ee2678f7f552cf
update google.golang.org/api to 63cb68f1e3834e44683ca062ddf06cb9a889380a
update google.golang.org/grpc to
0e6ec3a4501ee9ee2d023abe92e436fd04ed4081
update go4.org to f5283521d7365fb2875408726e9cbf349f173767
fix in cmd/ pkg/ server/
TODO(mpl): fix misc/docker tools as well. next CL.
Fixes#832
Change-Id: I842b968a0afea8a5822913bd614d67cdbe50ee63
So far we were returning an error, which appears to be a little too
strict, so we now just ignore when a key or value is too large, log it,
and go on.
Fixes#849
Change-Id: Iadd4eaab7459643e22ab3043d1f45e3eab662b30
Regardless of whether the database tables are newly created, we were
_always_ stamping the database schema version in the meta table BEFORE
doing the database version check. Which means said check was actually
useless.
Also add a small sanity check of the database name.
Issue #806
Change-Id: I85e19ef7583ebd5ef1043a6deb0fe61abaa4d190
Follow-up from the findings of
https://camlistore-review.googlesource.com/6227 which hinted that the
iter "err" field was not needed.
-Added Error check on iterator Closing.
-Removed Error call in Next, because it.it.Next already does it. See
func (i *dbIter) Next(), that checks i.err, which is the same as calling
Error(). (in github.com/syndtr/goleveldb/leveldb/db_iter.go)
-the closed field, and related check in Next are not strictly necessary,
because that's part of what Release does, in conjunction with func (i
*dbIter) Next() which is checking if we already released (look for i.dir
== dirReleased). But we're keeping it and the panic for the benefit of
detecting programmer's errors.
Also added the missing leveldb tests in index pkg.
Because of a potential infinite loop error (likely related to issue #756,
and not introduced by this CL), I've added a timer triggered panic to
break that loop when it happens.
Change-Id: I26e0815f1d85279f0ead7bf90daae2ae03f1af63
Let double-delete test pass.
Make leveldb and memdb Deletes go on for ErrNotFound.
Add test for batch deleting nonexisting key, and partial batch delete
errors - fix it for mongo.
Fixes#760.
Change-Id: I0a0e28836a723d245564f3dabaf328bf73bf463a
Use github.com/syndtr/goleveldb/leveldb/memdb in pkg/sorted memKey.
This was the last usage of code.googe.com/p/leveldb-go.
Hopefully tackles issue #600.
Change-Id: Iadcf7b69557cf4834cb2a4353ed0d2159379f423
The import path was added to the go file that included the package
documentation if one existed. Otherwise, I used what seemed to be the
primary file for the package.
Fixes#689
Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
Arbitrary limit, but prevents things from exploding during request spikes.
Fixescamlistore/camlistore#683
Change-Id: I3699d25416fb6bbc1ea4f3b984b0ce05e6abf31b
This change allows (for example) to hook a blobserver and an indexer
together in a way similar as they would be in a default Camlistore setup.
This is particularly useful to setup tests that require a realistic
indexer, without having to create a heavier test with e.g. test.World.
This actual goal of this change is described below. I kept these changes
together because the one described below illustrates the need for the
synchandler changes.
pkg/index/stress/: stress test the indexer with more data
The goal of these benchmkarks is to help us compare the various sorted
implementations, so we can pick the most efficient one as a default.
A second, harder, goal would be to add tests that help us estimate the
reliability of the various kv stores, again so we can rule out the buggy
ones, or help fix them.
Change-Id: I8a1fe50398a4ba09c03786502b68b6c2599e5984
Otherwise when openOrCachedDB is called for the second time,
it would return an actually closed DB.
Change-Id: I820834e508fed1a34ab7bd19a8a91e398badff9c
Previously pkg/jsonconfig and pkg/errorutil
Copied from go4.org at rev d1b8a2fb2de6160036e4801aa5e4d855571078b8
Change-Id: I673ed55b0825baa2607289b6082f205100261d7a
See https://golang.org/s/go15vendor
This commit moves the google.golang.org/cloud/... packages. The rest
of third_party will move later.
This commit is a dependency for some cloud logging work which we'll
develop in Camlistore's tree.
In a few places, cznic/kv use was hardcoded, instead of picking the user
set kv type, if any. This change makes sure we use the user-defined one,
if set, syndtr/leveldb otherwise.
Relatedly, the low-level config generated for diskpacked now uses that
same principle for its internal index. Also, if a diskpacked is
configured without an index, it now defaults to syndtr/leveldb.
Please note that this patch DOES NOT change the default kv type in the
automatically generated high-level config.
Related: #632
Change-Id: I55dbd385230878e92776e6b83d34c36fd1fa9382
Store the strings, as the TODO said.
This memoization saves 9/10 of running time:
BenchmarkSql-4 (before) 300000 5429 ns/op
BenchmarkSql-4 (after) 3000000 430 ns/op
Change-Id: I4edabff96043620b5af3f5b507ddd75d65b22ff1
Also, delete my old gce package from third_party and only use the
google metadata package (which my gce package became, and which was
also already vendored into third_party)
Fixes#596
Change-Id: I64fd6f1e9dc6f433466f91f81efd2ecbf039334f
Ian Denhardt