Just in case two permanodes were modified in the same nanosecond (or
more likely: the client's clock resolution sucks when they created the
mutation).
Fixes http://camlistore.org/issue/261
Change-Id: I293fc0f771bccc0473fefdcc2178284066b7a05c
The WebSockets live queries exposed a latent locking bug: the search
code was acquiring the Corpus read lock multiple times. If a write
came in during the middle of a search, between two read locks, the
second RLock would hang (since a writer lock was pending) and then the
search and write would deadlock.
Instead, just acquire the read lock once at the beginning of the query and
never re-acquire it. For now: new "Locked" versions of all the Corpus APIs.
In the future: will probably just add the locked-or-not into the context, and make
sure all Corpus methods require a *context.Context.
Also fix two potential data races: one with LogicalCondition (which
might've raced on the scratch ss buffer, depending on the query), and
a test-only race that didn't matter in production, but was annoying in
tsan output regardless, so fixed.
More tests to follow later. Unblocking Aaron for now.
Change-Id: Ie1edcbd061235f1ef04f9b4c68b86814c84361af
This works well enough for MySQL at least. We'll see if it hurts the other sorted.KeyValue
implementations. If so, we might have to fix those.
Change-Id: I71e9f56f2dee9e7334c7a35a4960f55399ca306f
The new package sorted/kvtest provides a generic KeyValue test for all
implementations. Memory, SQLite, and kvfile now use it.
This speeds up the index slurping start-up of my personal Camlistore
server from 30 seconds (when it was doing 17,000+ queries in small
windows) to now just 5 seconds. That 5 seconds can be improved yet
further.
Change-Id: Idd55ba9ccd3ed12a26868a41db1af676aff7b67b
This speeds Camlistore start-up index slurping considerably. Instead of
17,000 queries in 30 seconds, it now does 6.
A future change will improve this further, giving sorted.KeyValue impls
a hint about the end range too.
Change-Id: Icb5e2ac6edd580469e907fde4e916c00aba0ffe8
Goal is to iterate faster (notably: for slurping the index to the in-memory
corpus on start-up), by doing fewer copies and generating less garbage.
Change-Id: I054b0de2b994eb1f2356aa8587a466bafeb6cf82
Will eventually be plumbed through lots of APIs, especially those requiring or benefiting from
cancelation notification and/or those needing access to the HTTP context (e.g. App Engine).
Change-Id: I591496725d620126e09d49eb07cade7707c7fc64
1) pkg/search: documented that deletions times do not
qualify as modtimes
2) pkg/index: got rid of DeletedAt, and keyDeletes
http://camlistore.org/issue/191
Change-Id: I39578913345454d36af4599e29e7053f46577846
This test verifies that a permanode is still considered
deleted until all its deleters have been deleted.
http://camlistore.org/issue/191
Change-Id: Ifa4f17f031884b766d9ac90f1643a62ada591ae5
1) Done in AppendClaims, PermanodeOfSignerAttrValue,
SearchPermanodesWithAttr, PathsOfSignerTarget,
and PathsLookup.
2) camtypes.Path.DateClaim is now a time.Time, and used
as such in Path search methods.
3) Added kvSignerAttrValue, kvPathBackward, and kvPathForward
4) More Delete tests. We probably want more cases, but that's
a start.
http://camlistore.org/issue/191
Change-Id: I1d42485f2c6d1eea024f70378daacaaad9197768
1) The deletes cache now simply reflects all the keyDeleted index entries,
instead of trying to always hold the computed deletion state of the
blobs.
2) The IsDeleted and DeletedAt methods have been changed accordingly.
Moreover, while they previously were only taking into account the most
recent delete claim that affected the input blob, they now take
into account all delete claims that concern said blob.
In other words, a blob is considered "active" only if all of its past
deleters are deleted too (or it does not have any deleter).
http://camlistore.org/issue/191
Change-Id: I1b276dac0b3c6a3fb3beaf96828a45a85d27e4c6
This change:
1) Checks if the incoming claim is a delete claim with the use
of GetBlobMeta.
2) write the keyDeleted and keyDeletes keys when it's a delete
claim, plus the usual keys when the target is a permanode.
Yet to be done in the next CLs:
1) update the index deletes cache upon reception of a delete claim
2) update most of the search functions so they use deletedAt properly
3) add new keys necessary for GetRecentPermanodes to give a fully
correct result.
I also made indextest.DumpIndex public because it turned to be useful
to debug within pkg/search/ as well.
http://camlistore.org/issue/191
Change-Id: I8d8b9d12a535b8b1de0018b4a0e359241f14d52a
index in sync, both at start-up and while running and receiving blobs.
They both use the same mechanism now.
Also adds KeyId to the index and Corpus, as the next step. Plenty more
row types remain...
Change-Id: Id79955ba25dc79d5fbd94b0e5248d33dcf71d97e
keep blob metadata in memory, and start of testing all search queries in three modes:
classic index.Storage scanning, all in-memory with corpus scanned from the index.Storage,
and the in-memory corpus built up over time as blobs arrive.
Change-Id: I40536e498a63bece5bd4897cdbbd0cef78085f44
creates new package types/camtypes for misc types needed by both. might eventually go away as
search matures.
Change-Id: Ib771ead7bea39936ba478b7e5d58de997060861b
Also fixed fakeindex clock to use a time.Time.
And synchronized the fakeindex and the indextests clocks to the
same origin.
Change-Id: I986abca300d61241069132619028f470cacd61da
When indexing upon a blob reception, we first populate
all the mutations in a map instead of in a batch mutation.
Then we transfer all the mutations in a batch and commit
it immediately. This makes the window when the batch mutation
is open much shorter, and will ease future indexing because
it allows reading from the index while writing the mutations
to the map.
Change-Id: I276282388f59ca543835bfa5ec64986453b23fe1
The index entry prefixed by "claim" had no keyType and
was always built "by hand" with pipes concatenation.
This change adds the documented keyPermanodeClaim to fix
that.
Change-Id: Ic59f7dbcccc6b223b155d5bffbf8e636209800cb
This method will be used to find out when a permanode or claim
was (un)deleted for the last time.
This change also introduces the keyDeletes index entry which
allows to find the permanodes/claims that a delete claim deletes.
Finally, since some index keys have changed, this CL introduces
a check for an index schema version, to force users to reindex
when needed.
http://camlistore.org/issue/191
Change-Id: I6f8e5d3030a6ddc8b21ff5dc5db311f2ea16f613
Start of something that builds and 'works', but barely fleshed out at all.
Likely to change drastically yet.
Change-Id: I7fba25528a341ff2dbbc4dbb579f2450d7889320
This change introduces a new index entry
to help with finding the children of a static directory.
It also fixes ResolvePrefixHop so that it takes
into account static directories, and not only collections.
This is the first step to support publishing static directories.
http://camlistore.org/issue/179
Change-Id: I5666e5caa6c782004054ae4c19a6b6119d4fda8b
Running 'go run dev/envvardoc/envvardoc.go' now shows:
'All environment variables are documented'
I also took the liberty of cleaning-up our mishmash of logic for handling
boolean environment variables, and cleaned up a couple other spots that didn't
seem right.
This change adds docmentation for all variables starting with (CAM|DEV|AWS).
This leaves some variables still undocumented. If there are variables worth
documenting in the following list, maybe we should rename them to have a
CAM{LI} prefix for consistency's sake:
APPDATA pkg/osutil/paths.go:86
APPDATA pkg/osutil/paths.go:102
DISPLAY pkg/misc/gpgagent/gpgagent.go:126
GOPATH pkg/fileembed/genfileembed/genfileembed.go:321
GOPATH pkg/osutil/paths.go:168
GOPATH pkg/test/world.go:54
GOPATH server/appengine/build_test.go:77
GPGKEY cmd/camput/init.go:77
GPG_AGENT_INFO cmd/camput/init.go:153
GPG_AGENT_INFO pkg/misc/gpgagent/gpgagent.go:50
HOME pkg/jsonsign/keys.go:79
HOME pkg/jsonsign/signhandler/sig.go:64
HOME pkg/osutil/paths.go:36
HOMEPATH pkg/osutil/paths.go:34
PKG_CONFIG_PATH pkg/index/sqlite/dbschema.go:59
RUN_BROKEN_TESTS pkg/fs/fs_test.go:67
SKIP_DEP_TESTS pkg/test/testdep.go:29
TERM pkg/misc/gpgagent/gpgagent.go:133
TERM pkg/misc/pinentry/pinentry.go:99
TESTING_PORT_WRITE_FD pkg/webserver/webserver.go:135
TEST_GPGAGENT_LIB pkg/misc/gpgagent/gpgagent_test.go:27
USER pkg/netutil/ident.go:135
USER pkg/osutil/paths.go:45
USERNAME pkg/jsonconfig/eval.go:228
USERNAME pkg/osutil/paths.go:43
VERBOSE_FUSE pkg/fs/fs_test.go:133
VERBOSE_FUSE_STDERR pkg/fs/fs_test.go:137
XDG_CONFIG_HOME pkg/osutil/paths.go:104
Change-Id: Ief28710d3deefd1e65247cb5d3b1d8dde73e1f2d
Does a few things:
1) Adds gotaglib to third_party. If you'd like to review that, feel
free, though there's a bit of organization I'd like to do first.
2) Adds an "audioTag" key type.
3) Indexes wholerefs by various audio tags. Doesn't yet add a map from
wholeref to tags, but I can add that next.
Change-Id: I8e2a5bc27260086bad3351ac57973d1ac23cff44
Also added -port flag, and untied the port to the
password in devcam server, always 'pass3179' now.
http://camlistore.org/issue/212
Change-Id: Iadf693951763d47907be17c53d4807a4706eb150
Move up a layer to the HTTP. Also, start to remove ContextWrapper
stuff. We've done it differently for App Engine instead, and will do
it differently yet moving forward.
Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.
Bunch of other cleanup and TODO fixing too.
Much simpler and cleaner.
Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
misc.CountingReader moves into readerutil.
pkg/atomics is folded into pkg/types.
pkg/test/testdep is folded into pkg/test, with better name/docs.
Old cruft from pkg/webserver is deleted.
Change-Id: I3f72d8b29804254ef944995fb085837c878f79f5
* Fix dbinit command in dev server
* Make sslmode configurable in dbinit
* Don't fail on index schema setup when plpsql language already exists
Change-Id: Ic70461963f6266172d21cda8e6f071e2442d7651
The immediate concern was to make the index case sensitive,
but the values should be stored in binary anyway as this
layer is just obeying the IndexStorage interface and should
not be concerned with character sets.
http://camlistore.org/issue/130
Change-Id: Iee913b0bd9e5c0c32a5c5570310c2ac5d0cfddf8
images: DecodeConfig to get the predicted width
and height after EXIF correction
search&index: add GetImageInfo and use it in search
to predict the thumbnail dimensions
http://camlistore.org/issue/115
Change-Id: I358136a2ab03ea09c8f8fd2fa0dc574921c819c5
This change allows to find all roots (permanodes with the
"camliRoot" attribute) from the search.html ui page.
To achieve that, func (x *Index) SearchPermanodesWithAttr
now uses a prefix string without the query part if the
query is "".
http://code.google.com/p/camlistore/issues/detail?id=35
Change-Id: I396fba683e6e7b2296d1f0df8009c07e3c2cd09d
The mongo indexer didn't have any authentication
scheme, so this change adds the user:pass@host
authentication.
The mongo indexer tests now use the root:root
credentials.
Change-Id: I7867f46e3b5f0a4a1a50f86f26d76c0bfd69e9ac
Now all tests pass like this:
bradmac-2:camlistore.org $ SKIP_DEP_TESTS=1 go test ./pkg/...
? camlistore.org/pkg/auth [no test files]
ok camlistore.org/pkg/blobref 0.023s
ok camlistore.org/pkg/blobserver 0.032s
? camlistore.org/pkg/blobserver/cond [no test files]
? camlistore.org/pkg/blobserver/google [no test files]
ok camlistore.org/pkg/blobserver/handlers 0.031s
ok camlistore.org/pkg/blobserver/localdisk 1.693s
? camlistore.org/pkg/blobserver/remote [no test files]
? camlistore.org/pkg/blobserver/replica [no test files]
? camlistore.org/pkg/blobserver/s3 [no test files]
? camlistore.org/pkg/blobserver/shard [no test files]
? camlistore.org/pkg/cacher [no test files]
ok camlistore.org/pkg/client 0.029s
? camlistore.org/pkg/errorutil [no test files]
? camlistore.org/pkg/fileembed [no test files]
? camlistore.org/pkg/fileembed/genfileembed [no test files]
? camlistore.org/pkg/fs [no test files]
ok camlistore.org/pkg/googlestorage 0.034s
? camlistore.org/pkg/httputil [no test files]
ok camlistore.org/pkg/index 0.298s
? camlistore.org/pkg/index/indextest [no test files]
ok camlistore.org/pkg/index/mongo 0.524s
ok camlistore.org/pkg/index/mysql 0.020s
ok camlistore.org/pkg/jsonconfig 0.022s
ok camlistore.org/pkg/jsonsign 0.155s
ok camlistore.org/pkg/lru 0.015s
ok camlistore.org/pkg/magic 0.030s
? camlistore.org/pkg/misc [no test files]
ok camlistore.org/pkg/misc/amazon/s3 0.028s
ok camlistore.org/pkg/misc/gpgagent 0.018s
ok camlistore.org/pkg/misc/httprange 0.026s
? camlistore.org/pkg/misc/pinentry [no test files]
? camlistore.org/pkg/misc/resize [no test files]
ok camlistore.org/pkg/netutil 0.088s
ok camlistore.org/pkg/osutil 0.016s
ok camlistore.org/pkg/rollsum 0.027s
ok camlistore.org/pkg/schema 0.017s
ok camlistore.org/pkg/search 0.016s
ok camlistore.org/pkg/server 0.039s
ok camlistore.org/pkg/serverconfig 0.031s
? camlistore.org/pkg/test [no test files]
? camlistore.org/pkg/test/asserts [no test files]
? camlistore.org/pkg/test/testdep [no test files]
? camlistore.org/pkg/webserver [no test files]
Change-Id: I9c9b0ee4f51cbcf785aed665bbb494339dd84261
It now uses the generic indexing layer, rather than its own schema.
Also, move it under pkg/index/, like mongo.
Also, link in mongo to the main binary.
Not using the tests yet (leaving for mpl :))
Change-Id: I47a2203ca6c0b5244cd458aedec46227c4363c62
fixed indexer so that it builds as a package
added mgo as a third party
fixed mgoindexer
fixed (hackishly) tests so that they pass
I ignored everything memindex for now
Change-Id: I3c03ecfa859cf92b088a759239278b1c0edaf8aa
A lot is still broken, but most stuff at least compiles now.
The directory tree has been rearranged now too. Go libraries are now
under "pkg". Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".
The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.
For now, put the camlistore root under $GOPATH/src. Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.
Then you can:
$ go build ./server/camlistored
... etc
The build.pl script is currently disabled. It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).
Many things are certainly broken.
Many things are disabled. (MySQL, all indexing, etc).
Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.
The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.
Help wanted.
Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
mpl