Also, upon server --reindex, check that no out-of-order blobs are
pending. From a quick reading, they shouldn't be, but I'm curious to
see. Will do a full reindex of my data later.
Change-Id: Idebf93cc264e55512afcfb99e47320dd0ae745d1
Keep track of missing dependencies both in memory and in the index's
underlying sorted.KeyValue. When we see a dependent blob arrive, see
if we can reindex things.
Fixes camlistore.org/issue/102
Change-Id: I3d8cfc463e4b8c9d158be8f9656e772839b093b9
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.
SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.
Lot of code got simpler as a result.
Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.
Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
Add disc and mediaref (a hash of the audio portion of the
file).
Also relocate taglib code to
third_party/github.com/hjfreyer/taglib-go.
Change-Id: I58364f525b787484af894663125163095256d7c6
With the sync handler + indexer in same process subscribing to all
incoming blobs, we were indexing everything twice.
Fixes camlistore.org/issue/306
Change-Id: I7da54a0e18ac613eeae36d6db29b6cdb73a37196
Adds more tests to cover rotations with resize when used with
MaxWidth/MaxHeight, previously only ScaledWidth/ScaledHeight were
tested.
Improve tests to compare bounds when determining equality, otherwise
an image sized 0x0 is equal to all other images.
Sort test image filenames so test order is stable and obvious.
Keep more data in memory when indexing images upon receive. Some
largish CR2 files need more data or the EXIF parsing will fail.
Should address some or all of https://camlistore.org/issue/274
Change-Id: I80d90c33538c9d62ce4480ccb58c003e18ee6629
1) pkg/search: documented that deletions times do not
qualify as modtimes
2) pkg/index: got rid of DeletedAt, and keyDeletes
http://camlistore.org/issue/191
Change-Id: I39578913345454d36af4599e29e7053f46577846
This change:
1) Checks if the incoming claim is a delete claim with the use
of GetBlobMeta.
2) write the keyDeleted and keyDeletes keys when it's a delete
claim, plus the usual keys when the target is a permanode.
Yet to be done in the next CLs:
1) update the index deletes cache upon reception of a delete claim
2) update most of the search functions so they use deletedAt properly
3) add new keys necessary for GetRecentPermanodes to give a fully
correct result.
I also made indextest.DumpIndex public because it turned to be useful
to debug within pkg/search/ as well.
http://camlistore.org/issue/191
Change-Id: I8d8b9d12a535b8b1de0018b4a0e359241f14d52a
index in sync, both at start-up and while running and receiving blobs.
They both use the same mechanism now.
Also adds KeyId to the index and Corpus, as the next step. Plenty more
row types remain...
Change-Id: Id79955ba25dc79d5fbd94b0e5248d33dcf71d97e
keep blob metadata in memory, and start of testing all search queries in three modes:
classic index.Storage scanning, all in-memory with corpus scanned from the index.Storage,
and the in-memory corpus built up over time as blobs arrive.
Change-Id: I40536e498a63bece5bd4897cdbbd0cef78085f44
creates new package types/camtypes for misc types needed by both. might eventually go away as
search matures.
Change-Id: Ib771ead7bea39936ba478b7e5d58de997060861b
When indexing upon a blob reception, we first populate
all the mutations in a map instead of in a batch mutation.
Then we transfer all the mutations in a batch and commit
it immediately. This makes the window when the batch mutation
is open much shorter, and will ease future indexing because
it allows reading from the index while writing the mutations
to the map.
Change-Id: I276282388f59ca543835bfa5ec64986453b23fe1
The index entry prefixed by "claim" had no keyType and
was always built "by hand" with pipes concatenation.
This change adds the documented keyPermanodeClaim to fix
that.
Change-Id: Ic59f7dbcccc6b223b155d5bffbf8e636209800cb
This change introduces a new index entry
to help with finding the children of a static directory.
It also fixes ResolvePrefixHop so that it takes
into account static directories, and not only collections.
This is the first step to support publishing static directories.
http://camlistore.org/issue/179
Change-Id: I5666e5caa6c782004054ae4c19a6b6119d4fda8b
Does a few things:
1) Adds gotaglib to third_party. If you'd like to review that, feel
free, though there's a bit of organization I'd like to do first.
2) Adds an "audioTag" key type.
3) Indexes wholerefs by various audio tags. Doesn't yet add a map from
wholeref to tags, but I can add that next.
Change-Id: I8e2a5bc27260086bad3351ac57973d1ac23cff44
Move up a layer to the HTTP. Also, start to remove ContextWrapper
stuff. We've done it differently for App Engine instead, and will do
it differently yet moving forward.
Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.
Bunch of other cleanup and TODO fixing too.
Much simpler and cleaner.
Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
misc.CountingReader moves into readerutil.
pkg/atomics is folded into pkg/types.
pkg/test/testdep is folded into pkg/test, with better name/docs.
Old cruft from pkg/webserver is deleted.
Change-Id: I3f72d8b29804254ef944995fb085837c878f79f5
images: DecodeConfig to get the predicted width
and height after EXIF correction
search&index: add GetImageInfo and use it in search
to predict the thumbnail dimensions
http://camlistore.org/issue/115
Change-Id: I358136a2ab03ea09c8f8fd2fa0dc574921c819c5
A lot is still broken, but most stuff at least compiles now.
The directory tree has been rearranged now too. Go libraries are now
under "pkg". Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".
The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.
For now, put the camlistore root under $GOPATH/src. Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.
Then you can:
$ go build ./server/camlistored
... etc
The build.pl script is currently disabled. It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).
Many things are certainly broken.
Many things are disabled. (MySQL, all indexing, etc).
Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.
The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.
Help wanted.
Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
Brad Fitzpatrick