Commit Graph

22 Commits

Author SHA1 Message Date
Brad Fitzpatrick 66db09453f blobserver: add context to BlobRemover
Updates #733

Change-Id: I2fffb5cad59aa994441ee82ac5d940270113ee5a
2018-01-19 09:54:46 -08:00
Brad Fitzpatrick 194d4f9443 blobserver, all: add contexts to ReceiveBlob, Fetch & million resulting deps
I had intended for this to be a small change.

I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.

And then it got a bit infectious and ended up touching everything.

I ended up doing SubFetch in the process by necessity.

At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.

Updates #733

Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
2018-01-18 16:22:16 -08:00
Brad Fitzpatrick d6a0b05df0 Rename import paths from camlistore.org to perkeep.org.
Part of the project renaming, issue #981.

After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.

This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.

Also, this only moves the lru package to internal. More will move to
internal later.

Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.

This updates some docs, but not all.

devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).

Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
2018-01-01 16:03:34 -08:00
Brad Fitzpatrick bb11d07720 blobserver/memory: add BlobStreamer support to memory.Storage
And thus also to test.Fetcher.

Change-Id: I726dc0ddeae1d6ff22e2b645e05eb41f011c381a
2014-12-23 19:40:02 -08:00
mpl 5036d6b456 blobserver: add memory storage, mainly for tests.
Mostly code moved from pkg/test/fetcher.go

Also got rid of blob.MemoryStorage

http://camlistore.org/issue/416

Change-Id: I8d17b80942e2fde1c869f71738bfb8a67ed8ab31
2014-08-31 01:17:52 +02:00
Brad Fitzpatrick 31d6f81d12 localdisk, diskpacked, test.Fetcher: implement SubFetch
Change-Id: If9d09ab59a5bb73650c2668a8e0236d461287ef7
2014-08-25 16:47:05 -07:00
Brad Fitzpatrick 3d058eca20 test fetcher: add more stats
Change-Id: Iad880544174c812a49640939b8fb2922da738b28
2014-08-11 15:38:53 -07:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
Tamás Gulácsi 97520583b8 Use 'uint32' instead of 'int64' for blob sizes everywhere.
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.

Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
2014-02-08 17:58:12 +01:00
Brad Fitzpatrick c137cfc32d test: make Fetcher use a RWMutex so fetch doesn't deadlock during enumerate
Change-Id: Iac1f18725231a298c2c636875647c3e6f3f081a4
2014-02-07 14:28:56 -08:00
Brad Fitzpatrick 31125295a4 test: fix enumerate bug in in-memory storage when double-uploaded
Change-Id: I7e6abb1c9dfc61f39fa9d315857bf7aeb13b6b59
2014-01-20 13:07:20 -08:00
Brad Fitzpatrick e29c43bfde replica: test minWritesForSuccess
Change-Id: I5c845e99b1c440c35bfb7a5a7865176fd72ae031
2014-01-18 22:53:03 -08:00
Brad Fitzpatrick b82b8efe4c Start of new context package and *context.Context type.
Will eventually be plumbed through lots of APIs, especially those requiring or benefiting from
cancelation notification and/or those needing access to the HTTP context (e.g. App Engine).

Change-Id: I591496725d620126e09d49eb07cade7707c7fc64
2013-12-02 13:20:51 -08:00
Brad Fitzpatrick 99ea128752 schema: fix a race on errors in FileReader.GetChunkOffsets, add tests
Change-Id: I51635d01c3c88c5f97ac15172c5b565f07dde350
2013-09-22 13:10:16 +01:00
Brad Fitzpatrick b24cad68dd Cleanup: remove BlobHub and time.Duration waits from storage interface
Move up a layer to the HTTP.  Also, start to remove ContextWrapper
stuff.  We've done it differently for App Engine instead, and will do
it differently yet moving forward.

Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.

Bunch of other cleanup and TODO fixing too.

Much simpler and cleaner.

Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
2013-08-21 13:57:28 -07:00
Brad Fitzpatrick 0bdf20884b all: delete pkg/blobref; convert all from *blobref.BlobRef to new blob.Ref
Change-Id: Id2dfb7f19452bedf4f3c9310b36227fd8117b225
2013-08-03 19:54:30 -07:00
Brad Fitzpatrick b0bff61d30 types: move ReadSeekCloser types from pkg/blobref to pkg/types
Change-Id: I24e2b2a068d9b09479b145409a3b7e5693dd8ac5
2013-07-28 20:08:55 -07:00
Brad Fitzpatrick 31c37b6368 Make test.Fetcher a full in-memory storage implementation
Change-Id: I7551c22998526804f525f65609f7539f674802cb
2013-07-07 13:02:42 -07:00
Brad Fitzpatrick 08f09ebf2b schema: fix FileReader bugs, simplify, add exhaustive tests
Change-Id: I5d098d7b12892d278190391dd5abd7272ed997c2
2012-12-25 17:27:52 -08:00
Brad Fitzpatrick 0a0cbe72c1 schema: start of more stressful FileReader tests, chasing a bug
Change-Id: I2aa2b815c785d37674c309a9d6b7c26c67ec3b6e
2012-12-25 12:17:45 -08:00
mpl 50df48d519 first pass with mgo as indexer
fixed indexer so that it builds as a package
added mgo as a third party
fixed mgoindexer
fixed (hackishly) tests so that they pass
I ignored everything memindex for now

Change-Id: I3c03ecfa859cf92b088a759239278b1c0edaf8aa
2012-02-20 14:40:24 +01:00
Brad Fitzpatrick 0714a463c9 Update from r60 to [almost] Go 1.
A lot is still broken, but most stuff at least compiles now.

The directory tree has been rearranged now too.  Go libraries are now
under "pkg".  Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".

The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.

For now, put the camlistore root under $GOPATH/src.  Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.

Then you can:

$ go build ./server/camlistored

... etc

The build.pl script is currently disabled.  It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).

Many things are certainly broken.

Many things are disabled.  (MySQL, all indexing, etc).

Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.

The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.

Help wanted.

Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
2012-02-18 21:53:06 -08:00