Rooba
/
perkeep
mirror of https://github.com/perkeep/perkeep.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
perkeep/pkg/blobserver
History
Euan Kemp 90042fe296 pkg/blobserver/s3: use official aws client 
vendor: add github.com/aws/aws-sdk-go at rev bc3f534c19ffdf835e524e11f0f825b3eaf541c3

This changes all uses of the s3 client to be the official aws-sdk-go. As
such, it also deletes the previous client maintained in internal.

There are a few notable changes that come with this:

1. An 'aws_region' key is added to the low-level config. The
   'get-bucket-location' call is used to identify the correct region
   rather than using the redirect returned to determine it.
2. Deleting multiple objects at once is much faster (batches of up to
   1000 vs 1 at a time).
3. Retriable errors are retried automatically (see #1184).
4. Newer APIs are used for some operations (v2 / batch apis).

An unfortunate result of 4 is that this is technically a backwards
incompatible change. An IAM policy which used to be sufficient for
perkeep may no longer permit some of the new api calls (even if they're
not doing anything different really).

Fixes #1184, #911

Change-Id: I077fe88def18b7e9a14267820773245e7003634c
 		2018-07-31 15:51:31 -07:00
..
archiver blobserver: add context to BlobRemover 2018-01-19 09:54:46 -08:00
azure blobserver: add context to BlobRemover 2018-01-19 09:54:46 -08:00
b2 all: more renaming of Camlistore to Perkeep 2018-01-30 03:02:56 -08:00
blobpacked perkeepd: added --keep-going flag to start after index/recovery errors 2018-05-18 22:29:12 -04:00
cond blobserver: add context to BlobRemover 2018-01-19 09:54:46 -08:00
dir Rename import paths from camlistore.org to perkeep.org. 2018-01-01 16:03:34 -08:00
diskpacked all: Windows fixes (don't listen on file descriptors in test.World, etc) 2018-05-01 21:38:19 -07:00
encrypt blobserver: add context to BlobRemover 2018-01-19 09:54:46 -08:00
files pkg/blobserver/files: remove use of 'asserts' pkg 2018-06-18 22:42:10 -07:00
gethandler Stop using the legacy variable name "conn" for http.ResponseWriter values. 2018-05-13 18:48:12 -07:00
google all: various lint fixes 2018-02-05 11:59:00 -08:00
handlers pkg/blobserver: print RemoveBlobs errors 2018-06-11 19:32:35 +02:00
local all: more renaming of Camlistore to Perkeep 2018-01-30 03:02:56 -08:00
localdisk devcam, localhost, storagetest: fixes for Windows 2018-05-01 10:46:19 -07:00
memory blobserver: add context to BlobRemover 2018-01-19 09:54:46 -08:00
mongo blobserver: add context to BlobRemover 2018-01-19 09:54:46 -08:00
namespace blobserver: add context to BlobRemover 2018-01-19 09:54:46 -08:00
overlay pkg/blobserver/overlay: use better names for underlying storage 2018-02-27 17:11:59 +01:00
protocol all: more renaming of Camlistore to Perkeep 2018-01-30 03:02:56 -08:00
proxycache blob: specify "oh, nevermind" error value for optional SubFetcher interface 2018-05-17 19:49:02 -07:00
remote Rename camlistored to perkeepd. 2018-04-21 11:06:09 -07:00
replica blobserver: add context to BlobRemover 2018-01-19 09:54:46 -08:00
s3 pkg/blobserver/s3: use official aws client 2018-07-31 15:51:31 -07:00
sftp blobserver/sftp: add SFTP storage target 2018-05-11 20:53:50 -07:00
shard blobserver: add context to BlobRemover 2018-01-19 09:54:46 -08:00
stats all: rename remaining occurrences of camput 2018-04-21 16:20:24 -07:00
storagetest blob: specify "oh, nevermind" error value for optional SubFetcher interface 2018-05-17 19:49:02 -07:00
union blobserver: add context to BlobRemover 2018-01-19 09:54:46 -08:00
blobhub.go all: update copyright holder from Google Inc to The Perkeep Authors 2018-01-03 16:52:49 -08:00
blobhub_appengine.go all: update copyright holder from Google Inc to The Perkeep Authors 2018-01-03 16:52:49 -08:00
blobhub_test.go all: update copyright holder from Google Inc to The Perkeep Authors 2018-01-03 16:52:49 -08:00
doc.go all: update copyright holder from Google Inc to The Perkeep Authors 2018-01-03 16:52:49 -08:00
enumerate.go all: update copyright holder from Google Inc to The Perkeep Authors 2018-01-03 16:52:49 -08:00
interface.go all: various lint fixes 2018-02-05 11:59:00 -08:00
mergedenum.go all: update copyright holder from Google Inc to The Perkeep Authors 2018-01-03 16:52:49 -08:00
mergedenum_test.go Rename import paths from camlistore.org to perkeep.org. 2018-01-01 16:03:34 -08:00
multistream.go Rename import paths from camlistore.org to perkeep.org. 2018-01-01 16:03:34 -08:00
multistream_test.go Rename import paths from camlistore.org to perkeep.org. 2018-01-01 16:03:34 -08:00
noimpl.go blobserver: add context to BlobRemover 2018-01-19 09:54:46 -08:00
receive.go blobserver, all: add contexts to ReceiveBlob, Fetch & million resulting deps 2018-01-18 16:22:16 -08:00
receive_test.go blobserver, all: add contexts to ReceiveBlob, Fetch & million resulting deps 2018-01-18 16:22:16 -08:00
registry.go all: update copyright holder from Google Inc to The Perkeep Authors 2018-01-03 16:52:49 -08:00
stat.go blobserver, all: add contexts to ReceiveBlob, Fetch & million resulting deps 2018-01-18 16:22:16 -08:00
sync.go Rename import paths from camlistore.org to perkeep.org. 2018-01-01 16:03:34 -08:00
sync_test.go all: update copyright holder from Google Inc to The Perkeep Authors 2018-01-03 16:52:49 -08:00