Commit Graph

14 Commits

Author SHA1 Message Date
Euan Kemp 90042fe296 pkg/blobserver/s3: use official aws client
vendor: add github.com/aws/aws-sdk-go at rev bc3f534c19ffdf835e524e11f0f825b3eaf541c3

This changes all uses of the s3 client to be the official aws-sdk-go. As
such, it also deletes the previous client maintained in internal.

There are a few notable changes that come with this:

1. An 'aws_region' key is added to the low-level config. The
   'get-bucket-location' call is used to identify the correct region
   rather than using the redirect returned to determine it.
2. Deleting multiple objects at once is much faster (batches of up to
   1000 vs 1 at a time).
3. Retriable errors are retried automatically (see #1184).
4. Newer APIs are used for some operations (v2 / batch apis).

An unfortunate result of 4 is that this is technically a backwards
incompatible change. An IAM policy which used to be sufficient for
perkeep may no longer permit some of the new api calls (even if they're
not doing anything different really).

Fixes #1184, #911

Change-Id: I077fe88def18b7e9a14267820773245e7003634c
2018-07-31 15:51:31 -07:00
Daniel Heath af4ecfb5ea pkg/blobserver/s3: Replace s3-specific memory cache with a proxycache.
The s3-specific cache did not populate the cache on fetch (only on
receive) and did not cache stat() calls - proxycache does both.

Fixes #898

Change-Id: Ic0bfe684c9db79d16cb3c8c86ce87cc4676c63f0
2018-05-03 08:55:23 +10:00
Brad Fitzpatrick 194d4f9443 blobserver, all: add contexts to ReceiveBlob, Fetch & million resulting deps
I had intended for this to be a small change.

I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.

And then it got a bit infectious and ended up touching everything.

I ended up doing SubFetch in the process by necessity.

At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.

Updates #733

Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
2018-01-18 16:22:16 -08:00
Brad Fitzpatrick 57648c6b83 all: update copyright holder from Google Inc to The Perkeep Authors
The AUTHORS file is the list of copyright holders.
2018-01-03 16:52:49 -08:00
Brad Fitzpatrick d6a0b05df0 Rename import paths from camlistore.org to perkeep.org.
Part of the project renaming, issue #981.

After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.

This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.

Also, this only moves the lru package to internal. More will move to
internal later.

Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.

This updates some docs, but not all.

devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).

Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
2018-01-01 16:03:34 -08:00
mpl 92d5c1d045 pkg/blobserver/s3: directory & blobpacked support
Fixes issue #529

Change-Id: Iea7aea015b58c0325887fa07ae3a004b7822511e
2015-12-16 23:58:27 +01:00
mpl 53be9298a5 blobserver/s3: add optional in-memory Fetch cache
Change-Id: I77e27555d28002ba01f1501e8a77eb4adbf701fe
2014-10-18 19:34:25 +02:00
Brad Fitzpatrick bfb0d1e8de s3: more fault injection, and simplify receive, removing spilling to disk
Change-Id: Id7fb01c40dc863bee483887bbf14934161eb03a0
2014-03-16 19:44:52 -07:00
Brad Fitzpatrick 658472577d s3: be a Stringer, make default host more explicit, add failure injection
Change-Id: I1c24e97ef4b901e3be62fbb7e34c3f53787eedb1
2014-03-05 08:24:30 -08:00
Tamás Gulácsi 97520583b8 Use 'uint32' instead of 'int64' for blob sizes everywhere.
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.

Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
2014-02-08 17:58:12 +01:00
Brad Fitzpatrick afac3e1f5b s3: no need to verify hash in ReceiveBlob; handled elsewhere now
Change-Id: I91e85201555f769a8f37a00fe4c0e250a291f298
2013-09-01 09:12:17 -07:00
Brad Fitzpatrick 0bdf20884b all: delete pkg/blobref; convert all from *blobref.BlobRef to new blob.Ref
Change-Id: Id2dfb7f19452bedf4f3c9310b36227fd8117b225
2013-08-03 19:54:30 -07:00
Brad Fitzpatrick ca58d8e2e0 Remove noisy var _ = log.Printf lines.
Change-Id: Ia58b8ef5f271f542ae4fe61c7fb1497322770322
2013-06-14 12:55:55 -07:00
Brad Fitzpatrick 0714a463c9 Update from r60 to [almost] Go 1.
A lot is still broken, but most stuff at least compiles now.

The directory tree has been rearranged now too.  Go libraries are now
under "pkg".  Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".

The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.

For now, put the camlistore root under $GOPATH/src.  Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.

Then you can:

$ go build ./server/camlistored

... etc

The build.pl script is currently disabled.  It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).

Many things are certainly broken.

Many things are disabled.  (MySQL, all indexing, etc).

Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.

The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.

Help wanted.

Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
2012-02-18 21:53:06 -08:00