They were internal packages (under pkg), which we are now moving to
go4.org, so we in turn need to vendor them in now.
Change-Id: I92224f731404d0bd4ca1c57492bed37cb3367ed4
testSubFetcher in blobserver/storagetest was already checking that we'd
get specific error messages in the case of negative input parameters or
an out of range offset.
This change rationalizes these constraints with named errors
(ErrNegativeSubFetch and ErrOutOfRangeOffsetSubFetch) specified
in the SubFetcher interface.
It also fixes the googlestorage and s3 implementations so that they pass
the aforementioned test.
Change-Id: I25b72b842855b90ee3cab44c90654581dccf4b8e
Also then fix up diskpacked to work under these stricter rules.
Noticed this while working on blobpacked's tests and noticed them
surprisingly pass at one point. Now they no longer pass (as they
shouldn't yet, since parts are still TODO).
Updates #532
Change-Id: Ie05d19823453594486caf921cabbd149c43df221
This is round two of simplifying the BlobStreamer interface. The
continuation tokens are now sent per-item. This permits the following
item (NewMultiBlobStreamer) but also simplifies the return value (only
returns an error now) and permits the use of buffered channels as the
destination without getting out of sync.
The new NewMultiBlobStreamer is like io.MultiReader, but for
BlobStreamers, letting them be stitched together. This will be used by
the blobpacked storage layer to simplify its code, so it doesn't need
to deal with the handoff between loose and packed blobs itself. This
MultiBlobStreamer needs a 1 element buffer, which wasn't compatible
with the old BlobStreamer interface, hence the change to include the
continuation tokens with the blob on the channel.
Finally, add a new storagetest function for testing any blobstreamer
and use it for NewMultiBlobStreamer, diskpacked, and later in
blobpacked.
Updates #532
Change-Id: Iccffed289adec93ca5100c7ef8b0a8d57e05833c
1) Added fix in Upsert to allow for new file creation
2) Changed some methods argument s/id/title/ because it's actually the
file title that was meant to be worked with.
3) Fixed Trash to operate on a file title. Otherwise RemoveBlobs is
broken since blobRefs are used as the title when receiving, not as the
id.
4) Added tests.
5) Added option to storagetest to allow for skipping enumerate tests,
since enumerate not supported on gdrive yet.
Change-Id: I1f13f10214d816df53ff30e27e4e72085a31ba20
Currently unused. Added a Retried option, thinking I'd need it. I might later, or we might
add more options later.
Change-Id: I6b2f63c22685fd964012adbc8dfa06f3f6344313
Delete from index and zero out blob's data.
Use FALLOC_FL_PUNCH_HOLE if available (linux >= 2.6.38).
Change-Id: I0c194fd6734a35b520212398d93dfdc5cabadeb5
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.
SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.
Lot of code got simpler as a result.
Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.
Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
Will eventually be plumbed through lots of APIs, especially those requiring or benefiting from
cancelation notification and/or those needing access to the HTTP context (e.g. App Engine).
Change-Id: I591496725d620126e09d49eb07cade7707c7fc64
Add use into localdisk (diskpacked already uses it).
Add ErrNotImplemented error for blobserver and mention the possibility
for RemoveBlobs (diskpacked deficit).
Change-Id: I6a50f263a58c8d3d1611ff9a060ea9fa4aee6163
Regressed from rev cb6f423e. Eventually pkg storagetest should test all methods of blobserver.Storage
for all storage target types.
Change-Id: I2c1c93b76fd9280a3eb429b1d71c64a693ed1ace
mpl