Commit Graph

173 Commits

Author SHA1 Message Date
Brad Fitzpatrick 2b27dd53c0 Handle Photoshop images better.
We still can't find their width/height or thumbnail the, but at least
we don't wedge the UI now.

Fixes camlistore.org/issue/389

Change-Id: I8af6cefa5d84e7f6e26f3920e4dd6e1d5eb8b192
2014-03-04 10:58:51 -08:00
Aaron Boodman 5a1e831a7b Bump is:pano threshold to 2.0.
Previous value of 1.6 was too small because iPhone 5's aspect ratio
is 1.78, meaning all landscape iPhone 5 photos were included in
is:pano. Same with Moto-X and probably other newer phones.

I tried 1.8, but it still "felt" too small. I think the correct
value is atleast 2.0, but perhaps larger. I don't have enough test
data in that range right now to get a good feel for it.

Change-Id: I6a3e054290d6f434afac8142a9abc68e53008229
2014-02-08 15:48:29 -08:00
Brad Fitzpatrick b15cde9bce Merge "pkg/search: Add durationms to MediaTagConstraint comment." 2014-01-26 18:45:33 +00:00
Daniel Erat ef6e0bc696 pkg/search: Add durationms to MediaTagConstraint comment.
Change-Id: I8c097c2c88afaf01b9eb0f62072f5e68bddf5ca0
2014-01-25 17:22:33 -08:00
Brad Fitzpatrick 92b23e391d search: basic lexer to supported quoted tokens in search expressions
Change-Id: I6c8c37e3b27ab947623bbe7431352a14aa029839
2014-01-25 11:04:02 -08:00
Daniel Erat 404548d31a pkg/index: Index more music-related properties.
Add disc and mediaref (a hash of the audio portion of the
file).

Also relocate taglib code to
third_party/github.com/hjfreyer/taglib-go.

Change-Id: I58364f525b787484af894663125163095256d7c6
2014-01-22 21:25:05 -08:00
Brad Fitzpatrick 5b03c3f8fb search, index: let media tags be searchable too.
git push from Dolores Park. Sorry, no tests. Dan Erat will tell me if
this doesn't work.

Change-Id: I557cc3d07983390b8a15b7756ee0825fced2f503
2014-01-20 15:47:36 -08:00
Brad Fitzpatrick 94ff6ac141 pkg/search, pkg/client, cmd/camtool: add 'camtool claims' subcommand
Change-Id: Id9d8381facd97a946c3582b226d433ab59b18da7
2014-01-14 11:48:49 -08:00
mpl 5a58142c2c search: title search
Add the title:foo operator to search, to search permanodes which have a
title containing foo, case insensitively.

http://camlistore.org/issue/329

Change-Id: I661df159a4057a889a7ed110b2e724fc6e43b32b
2014-01-10 17:35:30 -08:00
Daniel Erat 396c8a6211 search: Add ValueMatches{Int,Float} to PermanodeConstraint.
Adds constraints for int- and float-valued attributes.

Change-Id: Ie8354f8ea12f604b45cf2fa6a7fb45170bab7e46
2014-01-04 20:43:32 -08:00
Daniel Erat 0ea90ebec3 search: Add StringConstraint.CaseInsensitive.
Provides support for case-insensitive string comparisons.
The implementation needs work (e.g. substring searches do
potentially-large allocations).

Change-Id: I30f068094b4617b81aee2e8981f87f6ec2bc0000
2014-01-04 18:29:52 -08:00
Brad Fitzpatrick 2002359f73 Merge "search: Fix ZeroMax ValueNum permanode queries." 2014-01-05 00:07:55 +00:00
Daniel Erat 945073baf9 search: Fix ZeroMax ValueNum permanode queries.
Searching for permanodes with zero values for an attribute
returned no results due to a check that at least one value
matched. Only perform that check when a value constraint was
provided.

Change-Id: Ia25a595e4598fe682f2a85babca0b8438681a210
2014-01-04 16:05:54 -08:00
Bill Thiede 9aeb0aa2c7 pkg/client: Search fix zero-value Describe queries.
Adds support for zero value blob.Ref {Unm,M}arshalJSON.

Fix for:
https://camlistore.org/issue/308

Change-Id: I910fc4f05015b0c3ddc57eba9d1d8fe1bfe992bf
2014-01-04 08:11:23 -08:00
Brad Fitzpatrick a576379cb5 search: PermanodeConstraint.Time time constraint, 'before:' and 'after:' operators
Work in progress, but works enough to commit now.  Determing the time
of things has many TODOs, and there's some performance work to be done
(although it still appears to be instant... it just uses more CPU than
it should)

Change-Id: I4b04b5805353dfbde0b841a3a557fd0b7c297780
2013-12-31 18:31:02 -08:00
Brad Fitzpatrick dbe9b83cfa search: some refactor cleanup in prep for sorting work
Change-Id: I71d422774bb68a93c9f9d648de8e4e5bc2ef1d87
2013-12-30 21:03:29 -08:00
Brad Fitzpatrick 8a05c520f8 search: make sort type encode as a string in JSON
Change-Id: I50c5663e850f3e8710381cc59daf246c8a600482
2013-12-29 16:28:06 -08:00
Brad Fitzpatrick 7d4c86d0de client: add a Search method
Change-Id: Iec4c22bf6e4bbe40645ec178b5c2d23392ad270d
2013-12-27 16:22:06 -08:00
Bill Thiede 63096a3133 pkg/search: fix is:pano test.
Fix breakage in
https://camlistore.org/gw/7b6790f56ab3f634fb42908731b211ec2236b6cc

Change-Id: Iba1084944cec54c11702911a88c584ee0efe5fec
2013-12-24 21:25:28 -08:00
Brad Fitzpatrick 7b6790f56a search: make is:pano be a bit wider.
Change-Id: I73a27b296a62d77bd1f8967a690a361f7aa4edde
2013-12-24 17:55:46 -08:00
Brad Fitzpatrick bc15a1d7a6 search: fix two double-RLock deadlocks
Change-Id: I5d083150cb1dc4e43c6a64146dee8a3672b7ea40
2013-12-24 13:46:18 -08:00
Brad Fitzpatrick f8d9e6663d geocode: move logging
Change-Id: I42d901510f9d8085c29792fd6b7ea8e1dc082c06
2013-12-24 13:43:25 -08:00
Brad Fitzpatrick eb3407e3e1 search: add format:foo operator
Change-Id: Ia12dcc43bba019f7d6511cec36e4091c692113fb
2013-12-23 21:07:10 -08:00
Brad Fitzpatrick 04bed02906 search: add negative expressions and has:location
e.g. [is:image -has:location]

Change-Id: I03df0a1d5fea8394a33e3079d131224c116d4c0d
2013-12-23 20:39:06 -08:00
Brad Fitzpatrick 7238bd1652 search: GPS location search
Like loc:hawaii or loc:USA or loc:94128, etc.

Change-Id: I11f47bf464a812f0b62e7799752811144bb7454e
2013-12-23 19:11:55 -08:00
Brad Fitzpatrick 4a53f4d687 search: add width:[min]-[max] and height:[min]-[max] operators
Change-Id: I18a82d8f1d2ba519d78b234a240ff854f36a8c10
2013-12-22 19:26:49 -08:00
Brad Fitzpatrick 06a334c39f search: add is:portrait, is:landscape, and is:pano
Also support Width and Height constraints on raw search (not yet
exposed to the expression syntax, but will be like width:<640 or
height:100-200 probably)

Change-Id: I082e3d27b5ef5c238a8e65a6b836943d9f9ff8a6
2013-12-22 18:30:27 -08:00
Brad Fitzpatrick 82a356ab77 search: add FloatConstraint
Change-Id: Ic367e87cb239258a76a99c9d07f0956d67e242ea
2013-12-22 18:11:44 -08:00
Brad Fitzpatrick f27c12ee6a search: add is:image operator
Change-Id: Iba693538b4e7b3d0cf08d5e8601d3b5819000ca1
2013-12-22 15:11:31 -08:00
Dustin Sallings 84ff81d563 Test search describe with time travel
Change-Id: I6bd8dbfa64a27f4fa491655d52371487f6ea9bb5
2013-12-22 01:21:02 -08:00
Aaron Boodman e15052302b Implement the new "expression" support in the web ui
Change-Id: Ic0521f770ffd45c24505ff386b2c044dc307e284
2013-12-21 22:10:17 -08:00
Brad Fitzpatrick 130be113fd Merge "Include camliPath attributes as members from DescribedBlob.Members" 2013-12-22 01:38:45 +00:00
Brad Fitzpatrick 516751574e Merge "Support requesting claims as of a particular time" 2013-12-22 00:31:04 +00:00
Dustin Sallings f805ed2ba6 Support requesting claims as of a particular time
Change-Id: I44763bc7ffc59d08e577d7bbb74a00b3c798206d
2013-12-21 16:27:37 -08:00
Brad Fitzpatrick 79fb299ff6 search: server-side search expression eval + result set continuation tokens
Search expressions (e.g. "tag:funny location:Portland") can now be
evaluated server-side (expr.go and expr_test.go, still nascent). Next
step is to remove this parsing and construction from the javascript UI
code.

In addition, the search result now contains a continuation token to
continue interating through the result sets in a subsequent query.
The Javascript UI should use this now instead of parsing out
modification times of things and altering the search.  If there's no
continuation token, that either means the end has been reached, or
that search type doesn't [yet] support iteration. Currently only
permanode-based queries are supported at many layers, including
continuation, but will be implemented later.

For now, the web UI should send a search query "expression" value of
non-zero length but just whitespace (like a single space: " ") to get
the home page (recently-modified permanodes") because setting both to
the zero value is an input validation error.  We should probably make
an expression operator to be more clear here (like "want:permanode
sort:recent").  But the empty space hack works for now.

Change-Id: I3ea92eb8c776159f53c49db1a7439a91c507940a
2013-12-21 15:59:11 -08:00
Ulf Holm Nielsen daa4379bd3 Include camliPath attributes as members from DescribedBlob.Members
Change-Id: I30272acd1cae805cf9477f99611275d32231b52a
2013-12-18 13:53:00 +01:00
Brad Fitzpatrick 30c7d8859a thumbnails: add a cache-busting URL component, support ETag
And quiet noisy logging on normal write failures.

We can now stress test the thumbnail generation by setting
CAMLI_DISABLE_THUMB_CACHE=1 which will make all the thumbnails in the
browsers be unique, and not write them to cache on the server.

Then, when we're happy with the thumbnails, we just increment the
thumbnailVersion string and that busts all the browser- and
server-side caches.

Change-Id: I3cda8e85ab8b1b0b2c9113f6dff613dfbf736028
2013-12-16 20:27:49 -08:00
Brad Fitzpatrick a078ce9406 search: generate an optimized matcher and use it within the matching loop
benchmark                         old ns/op    new ns/op    delta
BenchmarkQueryRecentPermanodes        27987        27512   -1.70%

benchmark                        old allocs   new allocs    delta
BenchmarkQueryRecentPermanodes           99           87  -12.12%

benchmark                         old bytes    new bytes    delta
BenchmarkQueryRecentPermanodes         6433         6207   -3.51%

But will be better in practice with more blobs.

Change-Id: I4673702231c52864249329818145aad11da2b4fc
2013-12-14 12:07:35 +04:00
Brad Fitzpatrick d6691c16de search: add a basic query benchmark
Change-Id: I1da18ef34a5d1f83bb5be2f9684a64bc43312bd7
2013-12-14 12:06:10 +04:00
Brad Fitzpatrick b406cb3475 query_test: remove some comments, now that they're on RelationConstraint
Change-Id: Id38dcbb2cfce4ab15ec55c08005356fb2e018ee6
2013-12-14 10:33:15 +04:00
Brad Fitzpatrick 9af5a970de Mark boring files generated by FUSE as "hidden", then hide in the UI.
Makes for prettier FUSE demos w/ files showing up live in the browser
with websockets.

Change-Id: I41cc9ae0d33db39b4dbbf5b60714cee9a79b7248
2013-12-13 13:27:48 +04:00
Brad Fitzpatrick a0576f2c0d Fix corpus/search locking bugs.
The WebSockets live queries exposed a latent locking bug: the search
code was acquiring the Corpus read lock multiple times.  If a write
came in during the middle of a search, between two read locks, the
second RLock would hang (since a writer lock was pending) and then the
search and write would deadlock.

Instead, just acquire the read lock once at the beginning of the query and
never re-acquire it.  For now: new "Locked" versions of all the Corpus APIs.
In the future: will probably just add the locked-or-not into the context, and make
sure all Corpus methods require a *context.Context.

Also fix two potential data races: one with LogicalCondition (which
might've raced on the scratch ss buffer, depending on the query), and
a test-only race that didn't matter in production, but was annoying in
tsan output regardless, so fixed.

More tests to follow later. Unblocking Aaron for now.

Change-Id: Ie1edcbd061235f1ef04f9b4c68b86814c84361af
2013-12-12 11:34:07 +04:00
Brad Fitzpatrick 3da18bc2da typo: use Errorf instead of Error in test
Change-Id: Idaa1b8870975d50d22789ed6fd6787f213d0d0d3
2013-12-12 11:14:09 +04:00
Brad Fitzpatrick bbaf755995 search: WIP notes on RelationConstraint. Not implemented yet.
Change-Id: Ic179a046da146706b57d7e6e031e1665deedd036
2013-12-11 13:46:19 +04:00
Brad Fitzpatrick a833a2d376 search: validate queries
Change-Id: Icc2840852a878b1afdfb7be97ddd50008422f0f7
2013-12-09 12:10:43 +04:00
Brad Fitzpatrick 3ea697da55 Redefine Permanode attribute matching constraints.
Removes a bunch of TODOs about single- vs multi-valued attributes.

Change-Id: I512e4afb5e0f5508fbb390ead3ce5162f47c5d28
2013-12-09 11:27:21 +04:00
Aaron Boodman 933e16ebf1 Replace special-case 'recent' query in web ui with new search.
Realized there was a way more elegant way to implement infinite
scroll. Now it is general to any kind of query (though not yet
to any kind of sorting, as there isn't support for continuing with
arbitrary sorts server-side).

Change-Id: I98dc627e1292524ddd8af8960c134f49ac41f337
2013-12-07 23:34:27 -08:00
Brad Fitzpatrick ffc6c39d40 Live search results via WebSocket.
Works in the debug page now, but not wired up to the main UI.
Leaving that for Boodman.

Change-Id: I0d0acf78cbcd228e95e391d724ea43571d174511
2013-12-07 22:22:45 -08:00
Brad Fitzpatrick b2d221af44 Start of Camlistore WebSocket support, for showing live search queries in the UI.
If the JavaScript UI is open and the query you're looking at changes
due to new data arriving, the server should push the update search
results to you live, without needing to poll.

Change-Id: I00faf7737f9a459de39b21fb5226a8000feeee2b
2013-12-07 20:19:20 -08:00
Brad Fitzpatrick b020f60390 search: add test for earlier fix in commit 1e1183a8bf
Change-Id: I95c0f885079de17cd1c52b9ad40861c96eb5308c
2013-12-07 10:53:30 -08:00