The publisher app required GopherJS which has been behind upstream Go
for many years. This just removes all GopherJS stuff, breaking the publisher
app in the process. We'll have to rewrite it.
Signed-off-by: Brad Fitzpatrick <brad@danga.com>
The directory created by `T.TempDir()` and is automatically removed when
the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
- Looking up missing blobrefs would cause fs errors. Instead, simply
return fuse.ENOENT.
- Add tests for:
* validating that the mountpoint/<blobref> lookups work
* validating that the mountpount/<blobref> lookups don't fail when
the blob does not exist.
- GopherJS is no longer required to build the perkeepd ui.
- Rename the `-buildWebUI` flag to `-buildPublisherUI`. Only the
publisher needs gopherjs.
- Adapt `make.go` so that it no longer tries to generate the perkeepd
web interface with GopherJS.
- Adapt `make.go` so that it no longer tries to download goui.js from
perkeep.org as it isn't needed anymore.
os.Symlink returns a nil error in some cases (when permissions allow), but
still does not behave as expected. Skip Windows altogether from some of these
tests for the time being.
To rev 99ff426eb706cffe92ff3d058e168b278cabf7c7
To support the new default authentication plugin in MySQL 8.
Interestingly, the update also revealed an error unseen so far.
Each of the pkg/sorted/mysql test uses a docker container. If we did not
kill the container at the end of a test, docker would use a new,
different, IP address for each mysql container (which would dodge the
error I'm about to mention). But since we do clean up and remove the
container for each test, then each test gets a container with the same
(first in the range) IP address, because docker "thinks" everything
about the previous container was properly removed.
However, it turns out that since we had forgotten to tell the client
code to close the connection to the mysql server, then the server-side
IP:port is still seen as occupied (even though the server is gone). And
since the second (or nth) test client tries connecting to the same
IP:port, the connection fails.
TL;DR: we also need to close the connection to the DB on the client
side, even though the server in container goes away after each test.
Fixes#1114
Change-Id: I76ce4ac2ee7703f92a9431116cd82ab75da541f7
vendor: add github.com/aws/aws-sdk-go at rev bc3f534c19ffdf835e524e11f0f825b3eaf541c3
This changes all uses of the s3 client to be the official aws-sdk-go. As
such, it also deletes the previous client maintained in internal.
There are a few notable changes that come with this:
1. An 'aws_region' key is added to the low-level config. The
'get-bucket-location' call is used to identify the correct region
rather than using the redirect returned to determine it.
2. Deleting multiple objects at once is much faster (batches of up to
1000 vs 1 at a time).
3. Retriable errors are retried automatically (see #1184).
4. Newer APIs are used for some operations (v2 / batch apis).
An unfortunate result of 4 is that this is technically a backwards
incompatible change. An IAM policy which used to be sufficient for
perkeep may no longer permit some of the new api calls (even if they're
not doing anything different really).
Fixes#1184, #911
Change-Id: I077fe88def18b7e9a14267820773245e7003634c
In 877eafa49b , pkg/server started
importing pkg/client , since the server has to be able to fetch blobs
from a share claim of another server. This introduced a circular
dependency in tests, as pkg/client/upload_test.go instantiates a server.
As the test uses both a client and a server, and tests uploading between
the two, it seems fitting to move it to pkg/test/integration, which
breaks the circular dep.
Change-Id: I2458ffd72912ff6e9c1a0f67e187d54cc895c2e9
I have tested that, starting with an empty $GOPATH/bin, after running
devcam test, $GOPATH/bin still has none of the Perkeep related binaries.
Fixes#1139
Change-Id: I60b63ff917cf91c8d424b448f874f40a18e1c23b
test/integration: don't listen on file descriptors.
make.go: unrelated, but options to make it much faster.
internal/images: t.Skip on HEIC dependency failures
Fixes#1140
Updates golang/go#25210
Change-Id: I8092155411826d6ed1f8d85230b753d1369044af
In f8bc4ac6e5 I made the root handler start returning 404s for
unknown URLs, which broke this test. Update this tests.
Fixes#1132
Change-Id: I6e14811dda00e741d80e1c5603f849ffcb26f361
Since we don't support the latest MySQL version, our tests in Travis CI
are failing since they pull the latest docker image.
We should fix the issue
(https://github.com/perkeep/perkeep/issues/1114), but in the meantime,
we might as well restrict the tests to MySQL 5, so that CI tests keep on
being useful to alert us about other errors.
Change-Id: I32a2cc1dc5a4194c891183d96498dee14c759baf
As make.go now does not install binaries in ./bin, but rather let them be
installed wherever go install does, the test World should now rely on
os/exec's LookPath to find the binaries it needs.
Fixes related failing tests
Change-Id: I256e79673ca994001e831aed9e7a3ab5ba485ead
A new "put" mode is added to the pk command, so that the "pk put"
command can be used to create and upload blobs.
What this command does is actually just call the previously named
"camput" executable, which is renamed to "pk-put" in this change.
This involves adding a new way to register a mode in cmdmain, when such
a mode is just meant to call an external binary. To emphasize the
distinction, the existing func (to register a sub-command, or a mode) is
renamed from RegisterCommand to RegisterMode, and RegisterCommand is now
the name of the new func/way.
Updates #981
Updates #1056
Change-Id: Ief954c17aa88a376f551df7de4b4e9fe41ad96d1
- correct logging that logged functions instead of their value
- use ID vs Id naming
- use correct function names in comments
Change-Id: I61562cef7ebac7337ec6c85312cdf7915cb1a84b
I had intended for this to be a small change.
I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.
And then it got a bit infectious and ended up touching everything.
I ended up doing SubFetch in the process by necessity.
At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.
Updates #733
Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
In rev: 837fe8ac46 a new
version of websocket was imported and changes were made to use
origin checking. This integration test was missed during those
changes.
This change fixes this test and adds the origin data to the
request, uses the correct 'ws' scheme and uses the new Dialer
API to make requests.
Change-Id: I93e8228794665012f15370532cdeda3cb702ea00
Remove the blob.SHA{1,224}From{Bytes,String} constructors too. No
longer used. This adds blob.RefFromBytes which was missing. We had
blob.RefFromString. Now everything uses blob.RefFrom* instead of
specifying a hash function.
Some tests set a flag to force use of SHA-1 because there was too much
golden data to update. We can remove those one-by-one over time as we
fix up tests.
Updates #537
Change-Id: Ibe6428089a6221594c2b751f53f98b03b5a28dc2
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
The search handler can store and retrieve search aliases.
Keyword namedSearch handles these new atoms of the form
named:foo.
Creating an alias has been implemented using a client in
the camtool subcommands named-search-get and named-search-set.
Change-Id: I7960f83bad464eb1a971c07f33631744a5eea814
improving proxycache
- added fuller sample config to the package documentation
- switched the stats caching from sorted.kv to the stats blobserver
- added a cleaning mechanism to evict the least recently used blobs
- implemented StatBlobs to actually inspect the local cache. It still
always consults the origin, but only for the blobs necessary after
giving the cache a 50ms headstart.
- logging a few errors that were previously ignored
- added tests modeled after the tests for the localdisk blobstore
- added a method to verify the cache, and call it on initialization
- added a strictStats option to always get stats from the origin
- filling in cacheBytes on initialization
improving stats blobserver
- implemented a few more of the blobserver interfaces, Enumerator and
Remover
- Fixed a bug(?) in ReceiveBlob that seemed to prevent it from actually
storing stats
- added a test
minor improvements include:
- blobserver/memory: allowing the memory blobserver to hold actually
infinite items, if desired
- blobserver: closing dest in the NoImpl blobserver, as required by the
BlobEnumerator interface
- storagetest: not closing dest leads to deadlock
- lru: max entries of 0 now means infinite (maybe do anything <0?)
- test: a helper function to create a random blob using a global random
source that is, by default, deterministic, to make test results more
consistent.
In the future, an improved BlobHub or similar interface could allow a
tighter feedback loop in providing cache consistency. i.e. the cache
could register with backend stores to be notified of content updates,
minimizing the time between backend changes and cache correction.
The proxycache will verify itself at startup, reporting an error if
any of its blobs do not exist in the backend storage or if the backend
storage has a different size for the content than the cache.
Fixes#443
Change-Id: I9ee1efd8c1d0eed49bb82930c2489a64122d3e00
Brad Fitzpatrick