Commit Graph

10 Commits

Author SHA1 Message Date
Brad Fitzpatrick d87911b39a cloudstorage: fix StatBlobs
It was saying things existed (with size 0) when they didn't.
Also fix the TODO while I'm here and do multiple things in parallel.

Change-Id: Iff8b1e6470c41af9d3094dfb6ed7cc9d8f21b281
2014-08-17 13:53:36 -07:00
Brad Fitzpatrick f2f7b01160 google cloud storage: support blobserver.Generationer
Change-Id: I9dd5bbaeb99e809bd4fff6f135d8141f4e1b051a
2014-08-02 21:55:57 -07:00
Brad Fitzpatrick 43a50b9fbf google cloudstorage: stat bucket at start. WIP.
Change-Id: Ia5c0981a1f31e347218d334a0748af19867b1adf
2014-08-02 21:00:43 -07:00
Brad Fitzpatrick 935bc81fe9 google cloud storage: add some sanity checks when using service accounts
Change-Id: I9fe5efcb75816d5c7770e589e2d102884e2e6131
2014-08-02 19:51:41 -07:00
Brad Fitzpatrick c3a3883395 googlecloudstorage: whren running on GCE, get access tokens automatically from the VM
Change-Id: Ibfcca0c52ff01a8f82bb0d3081119e0ac6e16ba9
2014-08-02 19:29:57 -07:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
Tamás Gulácsi 97520583b8 Use 'uint32' instead of 'int64' for blob sizes everywhere.
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.

Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
2014-02-08 17:58:12 +01:00
Brad Fitzpatrick b82b8efe4c Start of new context package and *context.Context type.
Will eventually be plumbed through lots of APIs, especially those requiring or benefiting from
cancelation notification and/or those needing access to the HTTP context (e.g. App Engine).

Change-Id: I591496725d620126e09d49eb07cade7707c7fc64
2013-12-02 13:20:51 -08:00
Brad Fitzpatrick b24cad68dd Cleanup: remove BlobHub and time.Duration waits from storage interface
Move up a layer to the HTTP.  Also, start to remove ContextWrapper
stuff.  We've done it differently for App Engine instead, and will do
it differently yet moving forward.

Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.

Bunch of other cleanup and TODO fixing too.

Much simpler and cleaner.

Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
2013-08-21 13:57:28 -07:00
Burcu Dogan 6e6e910af1 Adding Google Drive as a storage backend experimentally. No cheap-enough way
of enumerating Drive files, needs to be fixed once Drive supports
range queries. Allows you to sync to /sto-googledrive/, but not from.

Change-Id: I6ac5ef02c5f51a6abdab5c207a3a5693c6bb8581
2013-08-17 15:30:02 +02:00