I had intended for this to be a small change.
I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.
And then it got a bit infectious and ended up touching everything.
I ended up doing SubFetch in the process by necessity.
At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.
Updates #733
Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
The DownloadHandler only accepted file schemas as input for building a
zip archive so far.
It can now zip directories and their contents as well.
Non-regular files (socket, fifo, symlink) are now handled too.
As previously, no compression is applied when zipping.
When the DownloadHandler's Fetcher is a caching fetcher, all the files
that are supposed to be included in the archive are read, so we can
report reading errors even before starting to create the archive. We now
also take advantage of this optimization to build a
blobRef->filepath mapping when checking the files. Then, when the zip
archive is actually being built, the file path can be looked up in the
map, instead of having to assemble it again with recursive concatenation
of directory names.
Change-Id: I853c495798a9a43e12f3386603a70560ff46a237
Most of it replaced with vendor/go4.org/types and
vendor/go4.org/readerutil
u32 went where needed in pkg/blobserver/*
invertedBool went in pkg/types/serverconfig
atomics64 went in pkg/fs
Change-Id: I230426cda35be4b45ed67e869f14e6fdae89be22
Previously pkg/singleflight
Imported from github.com/camlistore/go4/ at rev
039bcc4ca7e407e5611cbdf15a1f208be0bceda0
Also reimport vendor/go4.org/syncutil because some of it got split into
github.com/camlistore/go4/syncutil/syncdebug
Change-Id: Iaf48de71928d17e0410442ea6f5bef7262ba60d2
They were internal packages (under pkg), which we are now moving to
go4.org, so we in turn need to vendor them in now.
Change-Id: I92224f731404d0bd4ca1c57492bed37cb3367ed4
I also wonder if we want to take a more comprehensive approach to
environment variables, perhaps populating a struct at startup that other
code can consult later. But it might be too soon for that kind of thing
in this project.
Change-Id: I65a34622bf906c1256ceb357ba983bc5acd6b887
See package docs in blobpacked.go for background & rationale.
Will minimize disk seeks, regardless of the underlying storage
system. I always knew this day would come, but we've been putting it
off. It's finally time to separate out logical (tons of small chunks)
storage from physical (big contiguous layout for how they'll be
accessed).
First 80% is done. Last 80% remains: mostly lots of tests and more
paranoia, and fsck things, for when we inevitably lose indexes. (but
each zip file is self-describing, so the index is redundant).
Change-Id: I13f9aa79c4d9d87b5fdcece0d99d97eba031d608
LoadAllChunks is now written in terms of ForeachChunk and is bounded.
GetChunkOffsets was never meant to be exported and was unused elsewhere.
It's now unused entirely.
Change-Id: I0653385fe805fda721e0b45ea0a17d9839c3ba6e
Will be used by the blobpacked storage server.
Also, we can then rewrite much of the legacy crufty methods on
FileReader in terms of this.
Change-Id: Ia371001f1ed502fa1da2dbc4a8e1c29e5f0c9e8c
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.
SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.
Lot of code got simpler as a result.
Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
A lot is still broken, but most stuff at least compiles now.
The directory tree has been rearranged now too. Go libraries are now
under "pkg". Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".
The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.
For now, put the camlistore root under $GOPATH/src. Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.
Then you can:
$ go build ./server/camlistored
... etc
The build.pl script is currently disabled. It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).
Many things are certainly broken.
Many things are disabled. (MySQL, all indexing, etc).
Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.
The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.
Help wanted.
Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
Brad Fitzpatrick