index.New was starting outOfOrderIndexerLoop in a goroutine. And
outOfOrderIndexerLoop had an if index.BlobSource == nil check, on which
it relied to go on. However, since BlobSource was public and unguarded,
the following sequence was possible:
ix, _ := index.New()
ix.BlobSource = bs
which is racy because the BlobSource assignment may or may not happen
before the check within outOfOrderIndexerLoop.
TestOutOfOrderIndexing was relying on the fact that apparently most
of the time the assignment seems to be happening before the check.
This patch:
-makes BlobSource (now blobSource) private, rendering the race impossible
out of the index package.
-moves the initialization of blobSource, as well as the execution of
outOfOrderIndexerLoop at a unique point, in InitBlobSource (new method).
-makes sure all accesses to blobSource are guarded with the index mutex
(now a RWMutex).
Context: while working on tests for http://camlistore.org/issue/454
Change-Id: I9605f26b41abd62b42880be0620b06ce143761bc
Index "MusicBrainz Album ID" ID3v2 frames as
"musicbrainzalbumid" media tags to facilitate downloading
cover art from coverartarchive.org.
Change-Id: Ie81017dd6f76ec355ee0d1daedfb7180cb70ad59
Add disc and mediaref (a hash of the audio portion of the
file).
Also relocate taglib code to
third_party/github.com/hjfreyer/taglib-go.
Change-Id: I58364f525b787484af894663125163095256d7c6
Search expressions (e.g. "tag:funny location:Portland") can now be
evaluated server-side (expr.go and expr_test.go, still nascent). Next
step is to remove this parsing and construction from the javascript UI
code.
In addition, the search result now contains a continuation token to
continue interating through the result sets in a subsequent query.
The Javascript UI should use this now instead of parsing out
modification times of things and altering the search. If there's no
continuation token, that either means the end has been reached, or
that search type doesn't [yet] support iteration. Currently only
permanode-based queries are supported at many layers, including
continuation, but will be implemented later.
For now, the web UI should send a search query "expression" value of
non-zero length but just whitespace (like a single space: " ") to get
the home page (recently-modified permanodes") because setting both to
the zero value is an input validation error. We should probably make
an expression operator to be more clear here (like "want:permanode
sort:recent"). But the empty space hack works for now.
Change-Id: I3ea92eb8c776159f53c49db1a7439a91c507940a
The new package sorted/kvtest provides a generic KeyValue test for all
implementations. Memory, SQLite, and kvfile now use it.
This speeds up the index slurping start-up of my personal Camlistore
server from 30 seconds (when it was doing 17,000+ queries in small
windows) to now just 5 seconds. That 5 seconds can be improved yet
further.
Change-Id: Idd55ba9ccd3ed12a26868a41db1af676aff7b67b
1) pkg/search: documented that deletions times do not
qualify as modtimes
2) pkg/index: got rid of DeletedAt, and keyDeletes
http://camlistore.org/issue/191
Change-Id: I39578913345454d36af4599e29e7053f46577846
This test verifies that a permanode is still considered
deleted until all its deleters have been deleted.
http://camlistore.org/issue/191
Change-Id: Ifa4f17f031884b766d9ac90f1643a62ada591ae5
1) Done in AppendClaims, PermanodeOfSignerAttrValue,
SearchPermanodesWithAttr, PathsOfSignerTarget,
and PathsLookup.
2) camtypes.Path.DateClaim is now a time.Time, and used
as such in Path search methods.
3) Added kvSignerAttrValue, kvPathBackward, and kvPathForward
4) More Delete tests. We probably want more cases, but that's
a start.
http://camlistore.org/issue/191
Change-Id: I1d42485f2c6d1eea024f70378daacaaad9197768
1) The deletes cache now simply reflects all the keyDeleted index entries,
instead of trying to always hold the computed deletion state of the
blobs.
2) The IsDeleted and DeletedAt methods have been changed accordingly.
Moreover, while they previously were only taking into account the most
recent delete claim that affected the input blob, they now take
into account all delete claims that concern said blob.
In other words, a blob is considered "active" only if all of its past
deleters are deleted too (or it does not have any deleter).
http://camlistore.org/issue/191
Change-Id: I1b276dac0b3c6a3fb3beaf96828a45a85d27e4c6
This change:
1) Checks if the incoming claim is a delete claim with the use
of GetBlobMeta.
2) write the keyDeleted and keyDeletes keys when it's a delete
claim, plus the usual keys when the target is a permanode.
Yet to be done in the next CLs:
1) update the index deletes cache upon reception of a delete claim
2) update most of the search functions so they use deletedAt properly
3) add new keys necessary for GetRecentPermanodes to give a fully
correct result.
I also made indextest.DumpIndex public because it turned to be useful
to debug within pkg/search/ as well.
http://camlistore.org/issue/191
Change-Id: I8d8b9d12a535b8b1de0018b4a0e359241f14d52a
creates new package types/camtypes for misc types needed by both. might eventually go away as
search matures.
Change-Id: Ib771ead7bea39936ba478b7e5d58de997060861b
Also fixed fakeindex clock to use a time.Time.
And synchronized the fakeindex and the indextests clocks to the
same origin.
Change-Id: I986abca300d61241069132619028f470cacd61da
This method will be used to find out when a permanode or claim
was (un)deleted for the last time.
This change also introduces the keyDeletes index entry which
allows to find the permanodes/claims that a delete claim deletes.
Finally, since some index keys have changed, this CL introduces
a check for an index schema version, to force users to reindex
when needed.
http://camlistore.org/issue/191
Change-Id: I6f8e5d3030a6ddc8b21ff5dc5db311f2ea16f613
This change introduces a new index entry
to help with finding the children of a static directory.
It also fixes ResolvePrefixHop so that it takes
into account static directories, and not only collections.
This is the first step to support publishing static directories.
http://camlistore.org/issue/179
Change-Id: I5666e5caa6c782004054ae4c19a6b6119d4fda8b
Also added -port flag, and untied the port to the
password in devcam server, always 'pass3179' now.
http://camlistore.org/issue/212
Change-Id: Iadf693951763d47907be17c53d4807a4706eb150
misc.CountingReader moves into readerutil.
pkg/atomics is folded into pkg/types.
pkg/test/testdep is folded into pkg/test, with better name/docs.
Old cruft from pkg/webserver is deleted.
Change-Id: I3f72d8b29804254ef944995fb085837c878f79f5
Brad Fitzpatrick