Previously pkg/wkfs
Copied from go4.org at rev 3970dd22c0e7e2397304ad626a20b72f8db41b58
Note that pkg/wkfs/gcs has not been moved yet (for dependencies
reasons).
Change-Id: I737e88a360bccfc643e1fa43541f65f2384c3157
It's more important for now to be able to monitor for other failures in
the Camlistore repo than leaving this test on.
After having looked into the issue, I know a little more but not enough
for a definitive conclusion or fix.
It's all added as comments so we don't start from scratch again later.
Also, to make up for the loss of this test, I'm adding two other tests,
TestCompareOriginalToHalveInPlace, and TestCompareOriginalToResized,
that are similar enough to catch other problems in the same context I
believe.
Context: issue #635
Change-Id: I80a30f544fb9ab2e7c69118ef8509d3d687c2d27
Import go4.org/cloud/cloudlaunch and go4.org/cloud/gceutil, which were
previously in pkg/cloudlaunch, at rev
2c497a8c195996d6c48402c150da4919c1d99f33
Because the version on go4.org was updated to use the latest
google.golang.org/api/* , we also update our
vendor/google.golang.org/api/* to rev
030d584ade5f79aa2ed0ce067e8f7da50c9a10d5
This in turns requires to add vendor/golang.org/x/net/context/ctxhttp
(and update vendor/golang.org/x/net/context/) to rev
4f2fc6c1e69d41baf187332ee08fbd2b296f21ed
Also adding vendor/google.golang.org/api/gensupport for the same reason.
Change-Id: I9d6c3059626f7ecc0fbf2df0415043a464d27caa
Previously pkg/singleflight
Imported from github.com/camlistore/go4/ at rev
039bcc4ca7e407e5611cbdf15a1f208be0bceda0
Also reimport vendor/go4.org/syncutil because some of it got split into
github.com/camlistore/go4/syncutil/syncdebug
Change-Id: Iaf48de71928d17e0410442ea6f5bef7262ba60d2
Previously known as pkg/throttle
Imported from github.com/camlistore/go4 at rev
4d87fc99168459b13e7c78086d210cf11474ade2
Change-Id: I21298db0790b2cf125efb612befd7dfb24780861
They were internal packages (under pkg), which we are now moving to
go4.org, so we in turn need to vendor them in now.
Change-Id: I92224f731404d0bd4ca1c57492bed37cb3367ed4
Setting the filename from the type, after resizing, does not work anymore
after fc890a65cd , because resized images
are now always RGBA, since we're using xdraw.
Change-Id: I561f43a3ebe9cb3e1d7404295635899ab6b62e7a
I broke it recently by removing gce.DevHandler and I don't think it's of
any use anymore nowadays.
Change-Id: I8d9ff85c980999bc5fb61ea08e1fce2cd4b7bea9
When logging to Google Cloud Logging, make sure the deploy handler logs
are written there too.
We tag them with the from:camli-gce-launcher label.
Change-Id: I4dab010009f98efd2aeba7c57dc4ab3ca6a566b3
Replaced github.com/bradfitz/http2 with the latest golang.org/x/net/http2
and updated the following dependencies:
golang.org/x/net @ a179abb31d787abdb406d922dff79635a8d784a4
golang.org/x/oauth2 @ 038cb4adce85ed41e285c2e7cc6221a92bfa44aa
google.golang.org/grpc @ 1684e4a49ba8e6c88a8628b26395f310a1e3d99f
Change-Id: Ic8f294f22a4ec7f7ee513fd1eb29a0b72ac41886
This makes diskpacked index type/location be set the same way it is done for
blobpacked, that is, its type will be the same as the camlistore index's
is. e.g. if the camli index is in MySQL, then the diskpacked index will
be another database in MySQL too.
Additionally, for added clarity wrt to the changes in
https://camlistore-review.googlesource.com/5496 , if we use a file based
indexer, the diskpacked index filename will now be
blobPath/diskpacked-index.kvtype (instead of blobPath/index.kvtype).
Change-Id: I7a9eb5c9caf07f62732ba2771308d351ec37b814
Because benchmarks from ./pkg/index/stress seem to show that levelDB
is more efficient than kvfile for us.
Also changed the filenames to index.kvtype, because the "camli-" prefix
is redundant, and because that's how we name the index files in the rest
of genconfig.go anyway.
Fixes issue #632
Change-Id: If663e94f2c24d646b994c39044eedc9b86327bf2
That way I don't have to constantly edit deploy.go (and fix it before
committing a CL) when debugging for GCE.
Change-Id: Ie4f75443e4356c5ce1a59931f3cb9e34344350be
When baseURL or listen is something like ":3179" or "0.0.0.0:3179", the
resulting server hostname in the client config would be useless.
Therefore, in that case we use req.Host to set the hostname instead.
Fixes issue #641
Change-Id: I33d65776fbac945a411f4328ebbbc5763dec8eb6
We redefine a MultipartReader(*http.Request) function to use instead of
req.MultipartReader, so we can get a (bugfixed) multipart Reader from
our vendored mime/multipart, instead of the buggy one from the stdlib's.
Fixes issue #642
Change-Id: I6a205bff915632d4ee77547e6e26bc0af99665e9
We can now use DialTLS when we need a custom TLS setup, i.e. with
android, or when using self-signed certificates, or when totally
skipping certificates verification.
This allows us to get rid of the scheme rewriting hack, which simplifies
things, in particular because we have less tricky URL/host parsing to
do.
For the sake of tests and not affecting "real" code, I introduced the
fake_android build tag, so the tests can use custom functions to
simulate testing whether we're on android.
Fixes#566
Change-Id: I72ac2bb69ad2365e98dd6ca2e7016ce9c2d7c57e
I just wanted to update the Google Cloud Logging code (still in review
at: https://code-review.googlesource.com/#/c/2650/) but that required
updating tons of things. For instance, gocloud now always depends on
grpc, which depends on http2, which we already had in third_party, so
that now moves into vendor.
I was unable to run the tests because of some error. The error message
was useless.
But "go run make.go" compiles everything at least.
Later, we should look into using it for more cases. First, to see if we
can phase "halving in place" out, and also to see the gains with other
image types (non YCbCr).
Change-Id: I4b95e2039407f1a91e04cb502674819f17680e02
While investigating issue #635 I remembered that golang.org/x/image/draw
might offer some interesting features. It indeed happens to have some
rescaling routines that seem to work better in the YCbCr case than our
custom code, which is illustrated by these benchmarks:
% go test -v -bench Benchmark.*YCrCb.* -run Bench* -benchmem
% ./pkg/images/resize/
PASS
BenchmarkResizeYCrCb-4 20 79213653 ns/op
9437475 B/op 5 allocs/op
BenchmarkNewResizeYCrCb-4 100 15711294 ns/op
1048668 B/op 2 allocs/op
BenchmarkHalveYCrCb-4 100 17328467 ns/op
128 B/op 1 allocs/op
ok camlistore.org/pkg/images/resize 5.889s
One can see that not only the new resize is faster, and uses less memory
than the old one, but it is also about as fast as "resizing by halving
in place". And this is with the ApproxBiLinear algorithm; some quick
additional tests seemed to indicate that NearestNeighbor was twice as
fast, with about the same memory usage. I chose ApproxBiLinear as it
seemed the most conservative choice for now.
I think it is also worth investigating whether we can be rid of "halving
in place", since it would make for simpler code overall, and since
ApproxBiLinear might be faster in most cases. It would have the nice
side-effect of solving issue #635 too. However, "halving in place" will
always win when it comes to memory usage, so it depends on our
priorities.
Finally, these other new benchmarks show that using the new resize does
not make the #635 mystery much worse (going from ~16% to ~18%), and that
interestingly the results of the old resize and the new resize are
pretty different by themselves.
% go test -v -run TestCompare ./pkg/images/resize/
=== RUN TestCompareOldResizeToHalveInplace
--- FAIL: TestCompareOldResizeToHalveInplace (2.49s)
resize_test.go:377: *image.Gray PSNR 63.7793
resize_test.go:377: *image.Gray16 PSNR 59.9547
resize_test.go:377: *image.NRGBA PSNR 62.8252
resize_test.go:377: *image.NRGBA64 PSNR 62.8252
resize_test.go:377: *image.Paletted PSNR 51.4949
resize_test.go:377: *image.RGBA PSNR 62.8252
resize_test.go:377: *image.RGBA64 PSNR 62.8252
resize_test.go:377: *image.YCbCr PSNR 61.9664
resize_test.go:382: *image.YCbCr not the same 18077 pixels
different 16.35%
resize_test.go:377: *image.YCbCr PSNR 52.4121
resize_test.go:382: *image.YCbCr not the same 18139 pixels
different 16.40%
resize_test.go:377: *image.YCbCr PSNR 51.4972
resize_test.go:382: *image.YCbCr not the same 17932 pixels
different 16.21%
resize_test.go:377: *image.YCbCr PSNR 51.6399
resize_test.go:382: *image.YCbCr not the same 17881 pixels
different 16.17%
resize_test.go:377: *image.YCbCr PSNR 50.7736
resize_test.go:382: *image.YCbCr not the same 17976 pixels
different 16.25%
resize_test.go:377: *image.YCbCr PSNR 52.4536
resize_test.go:382: *image.YCbCr not the same 18180 pixels
different 16.44%
=== RUN TestCompareNewResizeToHalveInplace
--- FAIL: TestCompareNewResizeToHalveInplace (2.27s)
resize_test.go:377: *image.Gray PSNR 63.7793
resize_test.go:377: *image.Gray16 PSNR 59.9547
resize_test.go:377: *image.NRGBA PSNR 62.8252
resize_test.go:377: *image.NRGBA64 PSNR 62.8252
resize_test.go:377: *image.Paletted PSNR 51.4949
resize_test.go:377: *image.RGBA PSNR 62.8252
resize_test.go:377: *image.RGBA64 PSNR 62.8252
resize_test.go:377: *image.YCbCr PSNR 59.5902
resize_test.go:382: *image.YCbCr not the same 20757 pixels
different 18.77%
resize_test.go:377: *image.YCbCr PSNR 52.0962
resize_test.go:382: *image.YCbCr not the same 20811 pixels
different 18.82%
resize_test.go:377: *image.YCbCr PSNR 51.3374
resize_test.go:382: *image.YCbCr not the same 20671 pixels
different 18.69%
resize_test.go:377: *image.YCbCr PSNR 51.3586
resize_test.go:382: *image.YCbCr not the same 20618 pixels
different 18.64%
resize_test.go:377: *image.YCbCr PSNR 57.2346
resize_test.go:382: *image.YCbCr not the same 20705 pixels
different 18.72%
resize_test.go:377: *image.YCbCr PSNR 50.7504
resize_test.go:382: *image.YCbCr not the same 20847 pixels
different 18.85%
=== RUN TestCompareOldResizeToNewResize
--- FAIL: TestCompareOldResizeToNewResize (2.54s)
resize_test.go:377: *image.Gray PSNR +Inf
resize_test.go:377: *image.Gray16 PSNR +Inf
resize_test.go:377: *image.NRGBA PSNR +Inf
resize_test.go:377: *image.NRGBA64 PSNR +Inf
resize_test.go:377: *image.Paletted PSNR +Inf
resize_test.go:377: *image.RGBA PSNR +Inf
resize_test.go:377: *image.RGBA64 PSNR +Inf
resize_test.go:377: *image.YCbCr PSNR 59.1024
resize_test.go:382: *image.YCbCr not the same 16350 pixels
different 14.78%
resize_test.go:377: *image.YCbCr PSNR 62.6523
resize_test.go:377: *image.YCbCr PSNR 62.7010
resize_test.go:377: *image.YCbCr PSNR 59.1396
resize_test.go:382: *image.YCbCr not the same 16225 pixels
different 14.67%
resize_test.go:377: *image.YCbCr PSNR 59.0844
resize_test.go:382: *image.YCbCr not the same 16294 pixels
different 14.73%
resize_test.go:377: *image.YCbCr PSNR 59.0534
resize_test.go:382: *image.YCbCr not the same 16413 pixels
different 14.84%
Change-Id: Ia8c578a53416ec0f1cac1477a1a68acfede84b24
I also wonder if we want to take a more comprehensive approach to
environment variables, perhaps populating a struct at startup that other
code can consult later. But it might be too soon for that kind of thing
in this project.
Change-Id: I65a34622bf906c1256ceb357ba983bc5acd6b887
mpl