Didn't find anything, but is useful to keep in, to maybe find bugs in the future
for other storage types.
Change-Id: If0fd37e03578de233be8da95ca45623c5f12156b
Depending on timing, could lead to ListMissingDestinationBlobs getting out
of sync and causing a lot of blobs to be replicated that were fine and already
on the server.
Change-Id: I3710e59088f1fe4e526f8f11bc9d1837a727e512
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.
SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.
Lot of code got simpler as a result.
Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
Currently controlled by an environment variable, but will become
a config option + on-demand button in UI in later commits.
Change-Id: I25fa878c9b30cdd713e2859585210eb722092f7b
Fix deadlock, much better status page, show per-blob status & errors,
clear errors when they've resolved themselves, fix known data race.
Change-Id: I968de0de4f308ff0a410adceb181a0712800d401
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.
Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
The new package sorted/kvtest provides a generic KeyValue test for all
implementations. Memory, SQLite, and kvfile now use it.
This speeds up the index slurping start-up of my personal Camlistore
server from 30 seconds (when it was doing 17,000+ queries in small
windows) to now just 5 seconds. That 5 seconds can be improved yet
further.
Change-Id: Idd55ba9ccd3ed12a26868a41db1af676aff7b67b
Will eventually be plumbed through lots of APIs, especially those requiring or benefiting from
cancelation notification and/or those needing access to the HTTP context (e.g. App Engine).
Change-Id: I591496725d620126e09d49eb07cade7707c7fc64
Previous TODO entry was:
-- Get rid of QueueCreator entirely. Plan:
-- sync handler still has a source and dest (one pair) but
instead of calling CreateQueue on the source, it instead
has an index.Storage (configured via a RequiredObject
so it can be a kvfile, leveldb, mysql, postgres etc)
-- make all the index.Storage types be instantiable
from a jsonconfig Object, perhaps with constructors keyed
on a "type" field.
-- make sync handler support blobserver.Receiver (or StatReceiver)
like indexes, so it can receive blobs. but all it needs to
do to acknowledge the ReceiveBlob is write and flush to its
index.Storage. the syncing is async by default. (otherwise callers
could just use "replica" if they wanted sync replication).
But maybe for ease of configuration switching, we could also
support a sync mode. when it needs to replicate a blob,
it uses the source.
-- future option: sync mirror to an alternate path on ReceiveBlob
that can delete. e.g. you're uploading to s3 and google,
but don't want to upload to both at once, so you use the localdisk
as a buffer to spread out your upstream bandwidth.
-- end result: no more hardlinks or queue creator.
Change-Id: I6244fc4f3a655f08470ae3160502659399f468ed
Move up a layer to the HTTP. Also, start to remove ContextWrapper
stuff. We've done it differently for App Engine instead, and will do
it differently yet moving forward.
Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.
Bunch of other cleanup and TODO fixing too.
Much simpler and cleaner.
Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
Because no localdisk means either s3 or google is the primary,
and none of them support efficient replication.
1) Added a dummy synchandler constructor for when config has "idle"
2) Set "idle" for synchandler config when no localdisk
3) fixed corresponding tests
Also:
- added error (and test) when no localdisk and both s3 and google
in config
- added s3 + mysql test
http://camlistore.org/issue/201
Change-Id: I861fdca0c203bc0181ab6d548adab501ed98d2f0
misc.CountingReader moves into readerutil.
pkg/atomics is folded into pkg/types.
pkg/test/testdep is folded into pkg/test, with better name/docs.
Old cruft from pkg/webserver is deleted.
Change-Id: I3f72d8b29804254ef944995fb085837c878f79f5
Required some sync work (full syncs on start, blocking full syncs on
start, and also adding a dev-only hack to force a depedency from
search -> sync, to control the handler initialization order, otherwise
publish handlers would race with the sync handler and they'd create
new "blog" and "pics" permanodes and we'd end up with duplicates).
A lot is still broken, but most stuff at least compiles now.
The directory tree has been rearranged now too. Go libraries are now
under "pkg". Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".
The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.
For now, put the camlistore root under $GOPATH/src. Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.
Then you can:
$ go build ./server/camlistored
... etc
The build.pl script is currently disabled. It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).
Many things are certainly broken.
Many things are disabled. (MySQL, all indexing, etc).
Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.
The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.
Help wanted.
Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
Brad Fitzpatrick