This change is in anticipation of moving pkg/images to go4.org, where it
should not depend on packages in third_party.
So:
third_party/github.com/nf/cr2 -> vendor/github.com/nf/cr2
third_party/github.com/rwcarlsen/goexif -> vendor/github.com/rwcarlsen/goexif
third_party/golang.org/x/image/tiff -> vendor/golang.org/x/image/tiff
Note that third_party/go/pkg/image/jpeg was also a dependency of
pkg/images. We had vendored image/jpeg from tip at the time because it
offered advantages over the version from Go1.3
(https://github.com/camlistore/camlistore/issues/463).
Since we now depend on Go1.5, we can go back to depend on the stdlib
version, so we simply remove third_party/go/pkg/image/jpeg and adjust
the imports accordingly.
Change-Id: Ifc8ffae0551102e644a0a0c67f3ff89e04df15c7
It's more important for now to be able to monitor for other failures in
the Camlistore repo than leaving this test on.
After having looked into the issue, I know a little more but not enough
for a definitive conclusion or fix.
It's all added as comments so we don't start from scratch again later.
Also, to make up for the loss of this test, I'm adding two other tests,
TestCompareOriginalToHalveInPlace, and TestCompareOriginalToResized,
that are similar enough to catch other problems in the same context I
believe.
Context: issue #635
Change-Id: I80a30f544fb9ab2e7c69118ef8509d3d687c2d27
Setting the filename from the type, after resizing, does not work anymore
after fc890a65cd , because resized images
are now always RGBA, since we're using xdraw.
Change-Id: I561f43a3ebe9cb3e1d7404295635899ab6b62e7a
Later, we should look into using it for more cases. First, to see if we
can phase "halving in place" out, and also to see the gains with other
image types (non YCbCr).
Change-Id: I4b95e2039407f1a91e04cb502674819f17680e02
While investigating issue #635 I remembered that golang.org/x/image/draw
might offer some interesting features. It indeed happens to have some
rescaling routines that seem to work better in the YCbCr case than our
custom code, which is illustrated by these benchmarks:
% go test -v -bench Benchmark.*YCrCb.* -run Bench* -benchmem
% ./pkg/images/resize/
PASS
BenchmarkResizeYCrCb-4 20 79213653 ns/op
9437475 B/op 5 allocs/op
BenchmarkNewResizeYCrCb-4 100 15711294 ns/op
1048668 B/op 2 allocs/op
BenchmarkHalveYCrCb-4 100 17328467 ns/op
128 B/op 1 allocs/op
ok camlistore.org/pkg/images/resize 5.889s
One can see that not only the new resize is faster, and uses less memory
than the old one, but it is also about as fast as "resizing by halving
in place". And this is with the ApproxBiLinear algorithm; some quick
additional tests seemed to indicate that NearestNeighbor was twice as
fast, with about the same memory usage. I chose ApproxBiLinear as it
seemed the most conservative choice for now.
I think it is also worth investigating whether we can be rid of "halving
in place", since it would make for simpler code overall, and since
ApproxBiLinear might be faster in most cases. It would have the nice
side-effect of solving issue #635 too. However, "halving in place" will
always win when it comes to memory usage, so it depends on our
priorities.
Finally, these other new benchmarks show that using the new resize does
not make the #635 mystery much worse (going from ~16% to ~18%), and that
interestingly the results of the old resize and the new resize are
pretty different by themselves.
% go test -v -run TestCompare ./pkg/images/resize/
=== RUN TestCompareOldResizeToHalveInplace
--- FAIL: TestCompareOldResizeToHalveInplace (2.49s)
resize_test.go:377: *image.Gray PSNR 63.7793
resize_test.go:377: *image.Gray16 PSNR 59.9547
resize_test.go:377: *image.NRGBA PSNR 62.8252
resize_test.go:377: *image.NRGBA64 PSNR 62.8252
resize_test.go:377: *image.Paletted PSNR 51.4949
resize_test.go:377: *image.RGBA PSNR 62.8252
resize_test.go:377: *image.RGBA64 PSNR 62.8252
resize_test.go:377: *image.YCbCr PSNR 61.9664
resize_test.go:382: *image.YCbCr not the same 18077 pixels
different 16.35%
resize_test.go:377: *image.YCbCr PSNR 52.4121
resize_test.go:382: *image.YCbCr not the same 18139 pixels
different 16.40%
resize_test.go:377: *image.YCbCr PSNR 51.4972
resize_test.go:382: *image.YCbCr not the same 17932 pixels
different 16.21%
resize_test.go:377: *image.YCbCr PSNR 51.6399
resize_test.go:382: *image.YCbCr not the same 17881 pixels
different 16.17%
resize_test.go:377: *image.YCbCr PSNR 50.7736
resize_test.go:382: *image.YCbCr not the same 17976 pixels
different 16.25%
resize_test.go:377: *image.YCbCr PSNR 52.4536
resize_test.go:382: *image.YCbCr not the same 18180 pixels
different 16.44%
=== RUN TestCompareNewResizeToHalveInplace
--- FAIL: TestCompareNewResizeToHalveInplace (2.27s)
resize_test.go:377: *image.Gray PSNR 63.7793
resize_test.go:377: *image.Gray16 PSNR 59.9547
resize_test.go:377: *image.NRGBA PSNR 62.8252
resize_test.go:377: *image.NRGBA64 PSNR 62.8252
resize_test.go:377: *image.Paletted PSNR 51.4949
resize_test.go:377: *image.RGBA PSNR 62.8252
resize_test.go:377: *image.RGBA64 PSNR 62.8252
resize_test.go:377: *image.YCbCr PSNR 59.5902
resize_test.go:382: *image.YCbCr not the same 20757 pixels
different 18.77%
resize_test.go:377: *image.YCbCr PSNR 52.0962
resize_test.go:382: *image.YCbCr not the same 20811 pixels
different 18.82%
resize_test.go:377: *image.YCbCr PSNR 51.3374
resize_test.go:382: *image.YCbCr not the same 20671 pixels
different 18.69%
resize_test.go:377: *image.YCbCr PSNR 51.3586
resize_test.go:382: *image.YCbCr not the same 20618 pixels
different 18.64%
resize_test.go:377: *image.YCbCr PSNR 57.2346
resize_test.go:382: *image.YCbCr not the same 20705 pixels
different 18.72%
resize_test.go:377: *image.YCbCr PSNR 50.7504
resize_test.go:382: *image.YCbCr not the same 20847 pixels
different 18.85%
=== RUN TestCompareOldResizeToNewResize
--- FAIL: TestCompareOldResizeToNewResize (2.54s)
resize_test.go:377: *image.Gray PSNR +Inf
resize_test.go:377: *image.Gray16 PSNR +Inf
resize_test.go:377: *image.NRGBA PSNR +Inf
resize_test.go:377: *image.NRGBA64 PSNR +Inf
resize_test.go:377: *image.Paletted PSNR +Inf
resize_test.go:377: *image.RGBA PSNR +Inf
resize_test.go:377: *image.RGBA64 PSNR +Inf
resize_test.go:377: *image.YCbCr PSNR 59.1024
resize_test.go:382: *image.YCbCr not the same 16350 pixels
different 14.78%
resize_test.go:377: *image.YCbCr PSNR 62.6523
resize_test.go:377: *image.YCbCr PSNR 62.7010
resize_test.go:377: *image.YCbCr PSNR 59.1396
resize_test.go:382: *image.YCbCr not the same 16225 pixels
different 14.67%
resize_test.go:377: *image.YCbCr PSNR 59.0844
resize_test.go:382: *image.YCbCr not the same 16294 pixels
different 14.73%
resize_test.go:377: *image.YCbCr PSNR 59.0534
resize_test.go:382: *image.YCbCr not the same 16413 pixels
different 14.84%
Change-Id: Ia8c578a53416ec0f1cac1477a1a68acfede84b24
https://camlistore.org/r/4936 broke cr2 decoding. If the tiff library
registers with the standard library's image package before the cr2 library
does, the tiff library will fail to decode cr2 images properly.
Fixes error message:
tiff: unsupported feature: compression value 6
Seen when generating thumbnails for CR2 files.
Change-Id: I6e5f77fbbf2b5b6615ece46525e33e0bfab9b0cd
When reading EXIF data from larger TIFF files, we might fail
to read the EXIF data when we only pass in the in-memory
prefix. This change identifies when the third-party library
encounters a short read on a tag/EXIF data and triggers a
retry with the whole file by returning an ErrUnexpectedEOF.
Change-Id: Ie5cdc1613db6ccac49d91a69827f11ca3406a74b
This pulls the changes from the current HEAD of
https://github.com/rwcarlsen/goexif
(eb2943811adc24a1a40d6dc0525995d4f8563d08)
Notable changes:
- Removed explicit panics in favor of error returns
- renamed TypeCategory to Format and made format calculated upon
decoding rather than repeatedly for every format call
- Merged contributions from Camlistore (exif.LatLong(), exif.DateTime()
etc.)
- Change String method to just return the string value - and don't have
square brackets if only a single value
- add separate Int and Int64 retrieval methods
- Doc updates
Minor changes in camlistore.org/pkg/* were neccessary to reflect
changes in the API (handling of returned errors) and in names of
exported fields and methods.
Change-Id: I50412b5e68d2c9ca766ff2ad1a4ac26926baccab
In the advent of github.com/camlistore/goexif to be closed, this
commit renames the goexif folder in third_party to match the
upstream on GitHub.
The affected import paths have been rewritten accordingly.
Change-Id: I5a8871efd01987944b7f5e93979307857ae16fe7
Reseting checkAvailability each run uncovered the fact that
CAMLI_DISABLE_DEBUG wasn't being properly reset, which affected later
tests.
Change-Id: I19b790ef8182b1bc2d54cbf33fa9dfe84dbe3903
Return error of type ErrDjpegFailed when djpeg or parsing the PNM returned
fails.
Attempt to decode image again with standard library.
Change-Id: I2a0cb7b52885732b7cbbffb8e34993d232781bc0
The new package pkg/images/fastjpeg uses djpeg(1) to quickly
down-sample images at load time as described here:
http://jpegclub.org/djpeg/
Add benchmark comparing std lib jpeg.Decode and fastjpeg.DecodeDownsample
with a factor of 1, 2, 4, and 8.
Benchmark resizes a 4000x4000 pixel JPEG to 128x128.
Refactor in pkg/images to make images.Decode a little more readable.
Change-Id: I571db1f3c3068f99da4a01ec84fd246ef098a18c
And quiet noisy logging on normal write failures.
We can now stress test the thumbnail generation by setting
CAMLI_DISABLE_THUMB_CACHE=1 which will make all the thumbnails in the
browsers be unique, and not write them to cache on the server.
Then, when we're happy with the thumbnails, we just increment the
thumbnailVersion string and that busts all the browser- and
server-side caches.
Change-Id: I3cda8e85ab8b1b0b2c9113f6dff613dfbf736028
Adds more tests to cover rotations with resize when used with
MaxWidth/MaxHeight, previously only ScaledWidth/ScaledHeight were
tested.
Improve tests to compare bounds when determining equality, otherwise
an image sized 0x0 is equal to all other images.
Sort test image filenames so test order is stable and obvious.
Keep more data in memory when indexing images upon receive. Some
largish CR2 files need more data or the EXIF parsing will fail.
Should address some or all of https://camlistore.org/issue/274
Change-Id: I80d90c33538c9d62ce4480ccb58c003e18ee6629
Makes the assumption that our old image logic was 'If the image is
huge (>2400px) shrink it to twice the target thumbnail size with
resample (which is quick) then box filter the image in half (which is
slow)'. There was a bug there that was causing large intermediary
images to be allocated when the source image was just the right size
and in portrait orientation (mainly trigger by photos from my Nexus 4).
Generalizes resize code to always resample to 2 times the thumbnail
size and then smooth down. Throws more pixels away faster.
Add downsizing routines that operate inplace. This greatly reduces
our peak memory usage when generating thumbnails.
Add version optimized for our particular resize case (halving);
providing a speed boost over the generalized versions.
Add tests and benchmarks comparing new resize to old.
Test for pkg/misc/resize can be run with --output=output_dir/ to see
the images generated by the old and new resize routines along with
mask of their differences.
Addresses some of the concerns in https://camlistore.org/issue/237
Change-Id: I6464fa637da9db371f15761bb699c045604b6cb8
Running 'go run dev/envvardoc/envvardoc.go' now shows:
'All environment variables are documented'
I also took the liberty of cleaning-up our mishmash of logic for handling
boolean environment variables, and cleaned up a couple other spots that didn't
seem right.
This change adds docmentation for all variables starting with (CAM|DEV|AWS).
This leaves some variables still undocumented. If there are variables worth
documenting in the following list, maybe we should rename them to have a
CAM{LI} prefix for consistency's sake:
APPDATA pkg/osutil/paths.go:86
APPDATA pkg/osutil/paths.go:102
DISPLAY pkg/misc/gpgagent/gpgagent.go:126
GOPATH pkg/fileembed/genfileembed/genfileembed.go:321
GOPATH pkg/osutil/paths.go:168
GOPATH pkg/test/world.go:54
GOPATH server/appengine/build_test.go:77
GPGKEY cmd/camput/init.go:77
GPG_AGENT_INFO cmd/camput/init.go:153
GPG_AGENT_INFO pkg/misc/gpgagent/gpgagent.go:50
HOME pkg/jsonsign/keys.go:79
HOME pkg/jsonsign/signhandler/sig.go:64
HOME pkg/osutil/paths.go:36
HOMEPATH pkg/osutil/paths.go:34
PKG_CONFIG_PATH pkg/index/sqlite/dbschema.go:59
RUN_BROKEN_TESTS pkg/fs/fs_test.go:67
SKIP_DEP_TESTS pkg/test/testdep.go:29
TERM pkg/misc/gpgagent/gpgagent.go:133
TERM pkg/misc/pinentry/pinentry.go:99
TESTING_PORT_WRITE_FD pkg/webserver/webserver.go:135
TEST_GPGAGENT_LIB pkg/misc/gpgagent/gpgagent_test.go:27
USER pkg/netutil/ident.go:135
USER pkg/osutil/paths.go:45
USERNAME pkg/jsonconfig/eval.go:228
USERNAME pkg/osutil/paths.go:43
VERBOSE_FUSE pkg/fs/fs_test.go:133
VERBOSE_FUSE_STDERR pkg/fs/fs_test.go:137
XDG_CONFIG_HOME pkg/osutil/paths.go:104
Change-Id: Ief28710d3deefd1e65247cb5d3b1d8dde73e1f2d
images: DecodeConfig to get the predicted width
and height after EXIF correction
search&index: add GetImageInfo and use it in search
to predict the thumbnail dimensions
http://camlistore.org/issue/115
Change-Id: I358136a2ab03ea09c8f8fd2fa0dc574921c819c5
This allows to know after a call to images.Decode if the image
was actually rotated or flipped.
Without knowing this, when making a thumbnail, it could happen
(if there was no rescaling required) that an image that needed
being rotated/flipped would not get rotated, because
useBytesUnchanged would still be true.
Change-Id: Ifc5e1e1f5a8543e6754102e3b3a685b736ae8673
This change implements rotation and flipping on images.
It can be done automatically, using the EXIF Orientation as a hint,
or optionally forced.
Change-Id: I97e887599d6a191964344e81cf8e90922313d958
mpl