Introduce magic.IsVideoFileName, add test for it.
Fixes this error:
--- FAIL: TestIsVideo (0.08s)
search_test.go:41: IsVideo failed video=true filename=movie.mkv mimetype=application/octet-stream
FAIL
FAIL perkeep.org/pkg/types/camtypes 0.399s
Discussion about this problem is here:
https://groups.google.com/forum/#!topic/perkeep/Wdb_0Os6ZkY
Change-Id: I14f54bd60c9495d6aa665036b83c813673608ad7
The bounds response from google geo api is optional and for many searches not
returned at all. Use viewport in those cases.
Change-Id: Iaf0a055720fd6cc0a2c87e93b8e22315798a0b35
vendor: add github.com/aws/aws-sdk-go at rev bc3f534c19ffdf835e524e11f0f825b3eaf541c3
This changes all uses of the s3 client to be the official aws-sdk-go. As
such, it also deletes the previous client maintained in internal.
There are a few notable changes that come with this:
1. An 'aws_region' key is added to the low-level config. The
'get-bucket-location' call is used to identify the correct region
rather than using the redirect returned to determine it.
2. Deleting multiple objects at once is much faster (batches of up to
1000 vs 1 at a time).
3. Retriable errors are retried automatically (see #1184).
4. Newer APIs are used for some operations (v2 / batch apis).
An unfortunate result of 4 is that this is technically a backwards
incompatible change. An IAM policy which used to be sufficient for
perkeep may no longer permit some of the new api calls (even if they're
not doing anything different really).
Fixes#1184, #911
Change-Id: I077fe88def18b7e9a14267820773245e7003634c
This change:
-on startup, makes perkeepd log instruction about the Google geocoding
key, if it is not found in Perkeep configuration directory.
-in the web UI, makes search query errors from the search session show
up in a dialog, instead of just being logged in the javascript console.
Among other things, it will allow users to be more readily informed
that their loc: queries are failing because they don't have a geocode
key setup.
According to the Google geocoding team, we should not share a geocoding
API key (too risky/expensive), and using an API key is the only way to
make geocoding requests. So we have to let users deal with it themselves
unfortunately.
Updates #1115Fixes#1116
Change-Id: I461e7f06a04ffd874678ab8233185999024cb3d8
This change has two major parts, which were interwoven enough to do
them in one change:
1) make serverinit.Config fully opaque, in prep for TOML configs #1134
2) shrink the massive server/perkeepd/camlistored.go file. It was out
of control and had a bunch of code that better belonged
elsewhere. This change moves a few hundred lines of code from
camlistored.go into more logical places: internal/osutil/gce for
GCE stuff, serverinit for config stuff (KeyRingAndId), etc.
I also added a TODO to make it possible to compile perkeepd without
any GCE stuff, which I saw as a possible and worthy goal only after
moving everything away.
Updates #1134
Change-Id: Iea6f84c5aca9c70b97806f4a201ec35e0f630e3b
test/integration: don't listen on file descriptors.
make.go: unrelated, but options to make it much faster.
internal/images: t.Skip on HEIC dependency failures
Fixes#1140
Updates golang/go#25210
Change-Id: I8092155411826d6ed1f8d85230b753d1369044af
Still slow, but no more ffmpeg child processes, and I'm confident
ImageMagick will improve over time. There are actually ImageMagick
commits in the past few hours that look promising. We have a demo
tomorrow, so we're not going to bump our git rev & Docker image to the
latest ImageMagick master yet.
Updates #969
Change-Id: I7a8545821cae4214ea521d7cb46aa235ae3f0ed3
A bunch of mine had a larger initial ftyp box, which broke the second
part of the pattern. But the second part of the pattern doesn't matter
anyway. This only needs to casually recognize them. A later full
parse will determine what they really are.
This also adds some new debugging when CAMLI_DEBUG is true.
Change-Id: Ib4adc9b5447a64ba4682624e42b55f1d65779ef7
Even when AltLookupFun was defined, Lookup used to fallthrough after the
AltLookupFn, depending on what AltLookupFn returned.
This means in some cases, our tests in pkg/search did end up calling the
Google API (which was not intended). We found out because with the new
dependency on an API key for the Google service, we now rely on reading
the key from the config dir, which is wired to panic if it is accessed
during tests.
This change therefore simplifies Lookup, by making it return the results
of the AltLookupFn call, if AltLookupFn exists.
Change-Id: Ie196f892f36d309c725a6e869094364d49cffdc8
Also removed misc/buildbot while at it (which contained camput
references) since we don't use it anymore at all.
TODO: the OSX app seems to be relying on finding a binary in ../bin,
which we do not use anymore. This will probably need fixing.
Updates #981
Change-Id: I14220fbad2e81181330fca4bb2d2e5fe170e1bd6
the changes to DecodeConfig in ce955adf48
moved the call to image.DecodeConfig to before the call to exif.Decode,
because we need to do that for the HEIF case.
After the call to exif.Decode, if err is nil and is not a short read
error, we ignore the error and the code goes on. To mark that intent,
we should have explicitly set the err to nil at this point. However,
that omission did not matter before
ce955adf48 because image.DecodeConfig
was the last setter of the error.
Now that it has moved up, we do have to explicitly reset the err to nil.
While we were at it, we rewrote the whole logic to be easier to read.
This seems to fix all the tests in pkg/index and pkg/search.
Change-Id: Id0b711179ccb1365b851d3e86ecb70d47ac358b4
Just enough to index an image's dimensions, and display its thumbnails
in the web UI.
Works with docker too.
Udates #969
Change-Id: Ie443cced4f785c332ca98f09eb647f00e53a532e
Use it in pkg/server for thumbnailing, but it will actually not be
called until we add basic indexation/description of HEIF too.
Updates #969
Change-Id: I3e4e40534feaff70f404d96d4e87c8dd057f880d
- correct logging that logged functions instead of their value
- use ID vs Id naming
- use correct function names in comments
Change-Id: I61562cef7ebac7337ec6c85312cdf7915cb1a84b
when its wholeref index lines contain any "sha1-" wholeref lines.
Then, have the client use that info when uploading files to also
compute the sha-1 (in addition to the sha-224) and ask the server for
both.
By client, we mean changes to cmd/camput, to pkg/client functions (for
camput and other Go clients, such as the apps), and to the web UI.
Also, add a -sha1 mode to devcam server, as well as the
CAMLI_SHA1_ENABLED env var, to facilitate testing/debugging of the
legacy sha1 mode. Fix/document unrelated options in devcam server
while we're at it.
Change-Id: I369679d2a4719f6f8b409f78dd4d065e13a3f62f
I had intended for this to be a small change.
I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.
And then it got a bit infectious and ended up touching everything.
I ended up doing SubFetch in the process by necessity.
At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.
Updates #733
Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
Remove the blob.SHA{1,224}From{Bytes,String} constructors too. No
longer used. This adds blob.RefFromBytes which was missing. We had
blob.RefFromString. Now everything uses blob.RefFrom* instead of
specifying a hash function.
Some tests set a flag to force use of SHA-1 because there was too much
golden data to update. We can remove those one-by-one over time as we
fix up tests.
Updates #537
Change-Id: Ibe6428089a6221594c2b751f53f98b03b5a28dc2
This addresses a long-standing TODO in the BlobStatter interface to
clean it up. Just like all new Go programmers, I misused channels in
APIs. I should've cleaned this up years ago.
While here, I also added a context.
The rest should get contexts later.
This also cleans up a few things here & there.
The pkg/client statting no longer does batching, which added a lot of
complexity. There was a comment saying something like "once we have
SPDY, we can delete this". Well, we have HTTP/2 now, so seems
deletable.
All tests pass.
Change-Id: I034ce07d9b70e5cc9e5482213368993e638d4bc8
Notably: pkg/misc all moves.
And pkg/googlestorage is deleted, since it's not used. Only the
x/net/http2/h2demo code used to use it, but that ended in
https://go-review.googlesource.com/33230 (our vendored code is old).
So just nuke that dir for now. When it's refreshed, it'll either be
gone (dep prune) or new enough to not need googlestorage.
Also move pkg/pools, pkg/leak, and pkg/geocode to internal.
More remains.
Change-Id: I2640c4d18424062fdb8461ba451f1ce26719ae9d
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
The import path was added to the go file that included the package
documentation if one existed. Otherwise, I used what seemed to be the
primary file for the package.
Fixes#689
Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
U-GINA-PC\gina