To support composition, all optional interfaces in Go need a way to
say, at runtime, that "oh, nevermind, I defensively implemented this
interface but it turns out the thing I'm wrapping doesn't support it".
We learned that lesson only in the past few years, but SubFetcher
predates that.
So do that here and make proxycache implemented SubFetcher and make
everything respect the fallback oh-nevermind error value.
Fixes#1170
Change-Id: I3c0958cf6692a324e98f526571d10a352a8e18b4
Unset CGO_ENABLED if no sign of gcc.
The localdisk renaming stuff was fixed in Go ages ago in
golang/go#13673 and https://golang.org/cl/6140
And a defer in storagetest meant Windows couldn't delete files
because a file was still open.
Change-Id: I57aef85f24653b19ce10e3d1e18c778cee2d48f6
I had intended for this to be a small change.
I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.
And then it got a bit infectious and ended up touching everything.
I ended up doing SubFetch in the process by necessity.
At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.
Updates #733
Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
This addresses a long-standing TODO in the BlobStatter interface to
clean it up. Just like all new Go programmers, I misused channels in
APIs. I should've cleaned this up years ago.
While here, I also added a context.
The rest should get contexts later.
This also cleans up a few things here & there.
The pkg/client statting no longer does batching, which added a lot of
complexity. There was a comment saying something like "once we have
SPDY, we can delete this". Well, we have HTTP/2 now, so seems
deletable.
All tests pass.
Change-Id: I034ce07d9b70e5cc9e5482213368993e638d4bc8
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
improving proxycache
- added fuller sample config to the package documentation
- switched the stats caching from sorted.kv to the stats blobserver
- added a cleaning mechanism to evict the least recently used blobs
- implemented StatBlobs to actually inspect the local cache. It still
always consults the origin, but only for the blobs necessary after
giving the cache a 50ms headstart.
- logging a few errors that were previously ignored
- added tests modeled after the tests for the localdisk blobstore
- added a method to verify the cache, and call it on initialization
- added a strictStats option to always get stats from the origin
- filling in cacheBytes on initialization
improving stats blobserver
- implemented a few more of the blobserver interfaces, Enumerator and
Remover
- Fixed a bug(?) in ReceiveBlob that seemed to prevent it from actually
storing stats
- added a test
minor improvements include:
- blobserver/memory: allowing the memory blobserver to hold actually
infinite items, if desired
- blobserver: closing dest in the NoImpl blobserver, as required by the
BlobEnumerator interface
- storagetest: not closing dest leads to deadlock
- lru: max entries of 0 now means infinite (maybe do anything <0?)
- test: a helper function to create a random blob using a global random
source that is, by default, deterministic, to make test results more
consistent.
In the future, an improved BlobHub or similar interface could allow a
tighter feedback loop in providing cache consistency. i.e. the cache
could register with backend stores to be notified of content updates,
minimizing the time between backend changes and cache correction.
The proxycache will verify itself at startup, reporting an error if
any of its blobs do not exist in the backend storage or if the backend
storage has a different size for the content than the cache.
Fixes#443
Change-Id: I9ee1efd8c1d0eed49bb82930c2489a64122d3e00
This switches most usages of the pre-1.7 context library to use the
standard library. Remaining usages are in:
app/publisher/main.go
pkg/fs/...
Change-Id: Ia74acc39499dcb39892342a2c9a2776537cf49f1
Before it would still call testEnumerate(0) even if it detected
RemoveBlobs as not implemented, which of course would fail.
Change-Id: Ic555f33fbe2e001fc6f52528e22b6faed5c7ec66
Using go4.org/cloud/google/gcsutil
for the things we can't yet do efficiently with the cloud/storage
Client.
Fixes issue #652
Change-Id: I8a43a754c4e0762629ce2a21bb95d9991f6f4771
The import path was added to the go file that included the package
documentation if one existed. Otherwise, I used what seemed to be the
primary file for the package.
Fixes#689
Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
They were internal packages (under pkg), which we are now moving to
go4.org, so we in turn need to vendor them in now.
Change-Id: I92224f731404d0bd4ca1c57492bed37cb3367ed4
testSubFetcher in blobserver/storagetest was already checking that we'd
get specific error messages in the case of negative input parameters or
an out of range offset.
This change rationalizes these constraints with named errors
(ErrNegativeSubFetch and ErrOutOfRangeOffsetSubFetch) specified
in the SubFetcher interface.
It also fixes the googlestorage and s3 implementations so that they pass
the aforementioned test.
Change-Id: I25b72b842855b90ee3cab44c90654581dccf4b8e
Also then fix up diskpacked to work under these stricter rules.
Noticed this while working on blobpacked's tests and noticed them
surprisingly pass at one point. Now they no longer pass (as they
shouldn't yet, since parts are still TODO).
Updates #532
Change-Id: Ie05d19823453594486caf921cabbd149c43df221
This is round two of simplifying the BlobStreamer interface. The
continuation tokens are now sent per-item. This permits the following
item (NewMultiBlobStreamer) but also simplifies the return value (only
returns an error now) and permits the use of buffered channels as the
destination without getting out of sync.
The new NewMultiBlobStreamer is like io.MultiReader, but for
BlobStreamers, letting them be stitched together. This will be used by
the blobpacked storage layer to simplify its code, so it doesn't need
to deal with the handoff between loose and packed blobs itself. This
MultiBlobStreamer needs a 1 element buffer, which wasn't compatible
with the old BlobStreamer interface, hence the change to include the
continuation tokens with the blob on the channel.
Finally, add a new storagetest function for testing any blobstreamer
and use it for NewMultiBlobStreamer, diskpacked, and later in
blobpacked.
Updates #532
Change-Id: Iccffed289adec93ca5100c7ef8b0a8d57e05833c
1) Added fix in Upsert to allow for new file creation
2) Changed some methods argument s/id/title/ because it's actually the
file title that was meant to be worked with.
3) Fixed Trash to operate on a file title. Otherwise RemoveBlobs is
broken since blobRefs are used as the title when receiving, not as the
id.
4) Added tests.
5) Added option to storagetest to allow for skipping enumerate tests,
since enumerate not supported on gdrive yet.
Change-Id: I1f13f10214d816df53ff30e27e4e72085a31ba20
Currently unused. Added a Retried option, thinking I'd need it. I might later, or we might
add more options later.
Change-Id: I6b2f63c22685fd964012adbc8dfa06f3f6344313
Delete from index and zero out blob's data.
Use FALLOC_FL_PUNCH_HOLE if available (linux >= 2.6.38).
Change-Id: I0c194fd6734a35b520212398d93dfdc5cabadeb5
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.
SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.
Lot of code got simpler as a result.
Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.
Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
Will eventually be plumbed through lots of APIs, especially those requiring or benefiting from
cancelation notification and/or those needing access to the HTTP context (e.g. App Engine).
Change-Id: I591496725d620126e09d49eb07cade7707c7fc64
Add use into localdisk (diskpacked already uses it).
Add ErrNotImplemented error for blobserver and mention the possibility
for RemoveBlobs (diskpacked deficit).
Change-Id: I6a50f263a58c8d3d1611ff9a060ea9fa4aee6163
Regressed from rev cb6f423e. Eventually pkg storagetest should test all methods of blobserver.Storage
for all storage target types.
Change-Id: I2c1c93b76fd9280a3eb429b1d71c64a693ed1ace
Brad Fitzpatrick