Commit Graph

29 Commits

Author SHA1 Message Date
Tamás Gulácsi 7402cc0efd Delete misc unused objects
Using honnef.co/go/unused/cmd/unused

Change-Id: I672b3cb77f09e9bd80dcdc149cde4f7f2939e451
2016-04-06 17:59:51 +02:00
Will Norris 77ed42edf8 add canonical import paths
The import path was added to the go file that included the package
documentation if one existed.  Otherwise, I used what seemed to be the
primary file for the package.

Fixes #689

Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
2016-03-13 19:57:14 -07:00
mpl c38f2a8ea2 pkg/misc/amazon/s3: fix test for Client
Because omitted in b9aec12f31

Change-Id: I2ca82360b3ff90d65be6f84eabdb785efcee99a9
2016-01-27 15:57:14 +01:00
mpl b9aec12f31 blobserver/s3: limit concurrent PutObject calls
Fixes issue #675

Since camput does not seem affected by the above issue, and that camput
(pkg/client really) main difference in how it uploads is that it does
gated requests, it seemed to follow that gating the requests might fix
the problem.

With this change, I don't see the problem anymore either in
TestS3WriteFiles or by manually uploading with the web UI.

I suppose this indicates that S3 throttles incoming connections pretty
strictly, but I have no formal evidence of it.

Change-Id: I48f10923a710f4a21e44bc251e648db37ae504ec
2016-01-25 21:39:57 +01:00
mpl 92d5c1d045 pkg/blobserver/s3: directory & blobpacked support
Fixes issue #529

Change-Id: Iea7aea015b58c0325887fa07ae3a004b7822511e
2015-12-16 23:58:27 +01:00
Brad Fitzpatrick 9868aa0f8d Remove httputil.CloseBody relic. 2015-11-23 19:02:10 -08:00
mpl b6eb85631c blob SubFetcher: explicitely states with errors the testSubFetcher constraints
testSubFetcher in blobserver/storagetest was already checking that we'd
get specific error messages in the case of negative input parameters or
an out of range offset.

This change rationalizes these constraints with named errors
(ErrNegativeSubFetch and ErrOutOfRangeOffsetSubFetch) specified
in the SubFetcher interface.

It also fixes the googlestorage and s3 implementations so that they pass
the aforementioned test.

Change-Id: I25b72b842855b90ee3cab44c90654581dccf4b8e
2015-02-26 15:18:27 +01:00
Emil Hessman e8ffb9a66c pkg: fix various vet reports
Change-Id: I0b7fa09849650fe696e59ad5c5c73a034ab80faa
2014-11-12 23:25:53 +01:00
mpl 917a09ca6d Merge "blobserver/s3: implement SubFetcher" 2014-10-19 14:13:02 +00:00
mpl b24580d523 blobserver/s3: implement SubFetcher
Change-Id: I42ac364e268463998598990a8bf0df7437f4d05d
2014-10-19 16:11:54 +02:00
Brad Fitzpatrick 1f75759371 s3: style fixes
Change-Id: I0c8dacb00f57d374c3148fe5c8017a9f146d36f4
2014-10-19 13:35:42 +02:00
tnt 4ac9938c8b Fixed some formatting functions that were called with invalid number/type of arguments.
Change-Id: I261c6fb732a6cf1e54e01fa3e5388c39c6deb720
2014-07-01 12:48:33 +02:00
Brian Gitonga Marete 630ba8c68f pkg/misc/amazon/s3: Fix test broken by commit bf8d6bdb.
Commit bf8d6bdb (s3: auto-select correct region, handle buckets with
dots in hostname) changed the definition of s3.Client replacing the
second field, which was of type *http.Client, with a field of type
http.RoundTripper. But the test utility function getTestClient() was
not modified accordingly. Fix this to make `go test' in
pkg/misc/amazon/s3 succeed again (without this fix the test fails to
build).

Change-Id: Ida383e2050e215cc2446303c0ce59aa7ab6e7981
2014-05-13 14:23:04 +03:00
Brad Fitzpatrick bf8d6bdb4f s3: auto-select correct region, handle buckets with dots in hostname
Fixes camlistore.org/issue/404

Change-Id: Ia6b3f51f1fd5beba585fa8dd8a918d30905fb9bb
2014-05-08 15:31:48 -07:00
Govert Versluis cef0aa0d16 Switch to different S3 endpoint
Switch to a URL in the form of
https://s3.amazonaws.com/mybucket.mydomain.com/myObjectKey
This adds support for buckets with periods in their name and S3 clones

Addresses https://code.google.com/p/camlistore/issues/detail?id=404

Change-Id: Id5c744c88916dbac5a0ab01330929546984ef2d2
2014-04-14 10:48:03 +02:00
Govert Versluis 33d5718843 Added missing check for http.StatusOK in Stat.
Also made checks for http.StatusOK / 200 uniform

Change-Id: I8ef608dc098cdfd221a53be20801d7dcdcdf7655
2014-03-23 17:37:16 +01:00
Brad Fitzpatrick bfde349851 s3: do some retries in ListBucket, to work around Amazon-vs-Go's HTTP/TLS bug
Change-Id: Idfc8187d07f558f2e7c590dc6c14c66791f46e57
2014-03-17 22:59:02 -07:00
Antti Rasinen f837fcb9d8 Convert all S3 requests to HTTPS
Also includes fixes for two missing data items
for errors.

Change-Id: Ib174b4bf83f3f264f463b4c203c1f42213b18f5e
2014-03-09 21:00:13 +02:00
Brad Fitzpatrick ea15158f7c Enumerate changes for future sync changes
Add blobserver.EnumerateAllFrom, make S3 client API match the underlying S3 protocol
with marker instead of after, and then push after->marker conversion logic up
into the s3 enumerate code that's using the S3 client.

Change-Id: I034a7c1c8af441881ebba74bcb523bd690cd16d3
2014-03-04 12:11:59 -08:00
Brad Fitzpatrick eb27581da7 httputil: add DecodeJSON and CloseBody helpers, then use them everywhere
httputil.CloseBody closes a http.Response.Body nicely, such that it's
more likely to be re-used.

httputil.DecodeJSON parses JSON from a Response.Body and closes the
body. It also keeps a temporary copy of the input, for error messages.

Then use them everywhere, to improve HTTP TCP connection re-use.

And misc other clean-ups.

Change-Id: I5dc44be0165b13659459ed29ce5ab44c17d9739f
2014-01-28 22:00:52 -08:00
Brad Fitzpatrick f3f48eb70b Support alternate S3 hostnames
Change-Id: I9a27eb5fb9c4e5ef3b516ce9cf5a4b5bb4e1df61
2013-09-01 09:50:35 -07:00
Brad Fitzpatrick 91cdda8475 Match common Go case
Change-Id: I14606ee5ba422cf43f68dc65af2b40a096c7ca35
2013-09-01 09:13:13 -07:00
Brad Fitzpatrick 9468e5ba70 More docs. Every package is documented now.
misc.CountingReader moves into readerutil.

pkg/atomics is folded into pkg/types.

pkg/test/testdep is folded into pkg/test, with better name/docs.

Old cruft from pkg/webserver is deleted.

Change-Id: I3f72d8b29804254ef944995fb085837c878f79f5
2013-07-07 21:12:30 -07:00
Brad Fitzpatrick ca58d8e2e0 Remove noisy var _ = log.Printf lines.
Change-Id: Ia58b8ef5f271f542ae4fe61c7fb1497322770322
2013-06-14 12:55:55 -07:00
Brad Fitzpatrick 37bf028633 s3: verify configured bucket exists at start-up.
Change-Id: I43d0795641fe9dbc345a1f0916b828ede76cf4a6
2013-01-20 11:33:41 -08:00
Brad Fitzpatrick 2fe555aeb9 s3: remove debug logging
Change-Id: Idd4c3538362dd14b59ae6b150992d806ae7ff6b8
2012-12-22 18:45:27 -08:00
Brad Fitzpatrick c83e29cfde s3: improve bucket enumeration
Change-Id: I65ed4a6575250bb2005215df461ada64ac20e41e
2012-12-22 13:50:18 -08:00
Brad Fitzpatrick 21d142e538 s3: some work on Buckets, add TODO
Change-Id: I183a85a0b9639790adec025830dae2f276006290
2012-12-21 16:23:19 -08:00
Brad Fitzpatrick 0714a463c9 Update from r60 to [almost] Go 1.
A lot is still broken, but most stuff at least compiles now.

The directory tree has been rearranged now too.  Go libraries are now
under "pkg".  Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".

The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.

For now, put the camlistore root under $GOPATH/src.  Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.

Then you can:

$ go build ./server/camlistored

... etc

The build.pl script is currently disabled.  It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).

Many things are certainly broken.

Many things are disabled.  (MySQL, all indexing, etc).

Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.

The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.

Help wanted.

Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
2012-02-18 21:53:06 -08:00