Commit Graph

55 Commits

Author SHA1 Message Date
Daniel Erat ef6e0bc696 pkg/search: Add durationms to MediaTagConstraint comment.
Change-Id: I8c097c2c88afaf01b9eb0f62072f5e68bddf5ca0
2014-01-25 17:22:33 -08:00
Daniel Erat 404548d31a pkg/index: Index more music-related properties.
Add disc and mediaref (a hash of the audio portion of the
file).

Also relocate taglib code to
third_party/github.com/hjfreyer/taglib-go.

Change-Id: I58364f525b787484af894663125163095256d7c6
2014-01-22 21:25:05 -08:00
Brad Fitzpatrick 5b03c3f8fb search, index: let media tags be searchable too.
git push from Dolores Park. Sorry, no tests. Dan Erat will tell me if
this doesn't work.

Change-Id: I557cc3d07983390b8a15b7756ee0825fced2f503
2014-01-20 15:47:36 -08:00
Daniel Erat 396c8a6211 search: Add ValueMatches{Int,Float} to PermanodeConstraint.
Adds constraints for int- and float-valued attributes.

Change-Id: Ie8354f8ea12f604b45cf2fa6a7fb45170bab7e46
2014-01-04 20:43:32 -08:00
Daniel Erat 0ea90ebec3 search: Add StringConstraint.CaseInsensitive.
Provides support for case-insensitive string comparisons.
The implementation needs work (e.g. substring searches do
potentially-large allocations).

Change-Id: I30f068094b4617b81aee2e8981f87f6ec2bc0000
2014-01-04 18:29:52 -08:00
Daniel Erat 945073baf9 search: Fix ZeroMax ValueNum permanode queries.
Searching for permanodes with zero values for an attribute
returned no results due to a check that at least one value
matched. Only perform that check when a value constraint was
provided.

Change-Id: Ia25a595e4598fe682f2a85babca0b8438681a210
2014-01-04 16:05:54 -08:00
Brad Fitzpatrick a576379cb5 search: PermanodeConstraint.Time time constraint, 'before:' and 'after:' operators
Work in progress, but works enough to commit now.  Determing the time
of things has many TODOs, and there's some performance work to be done
(although it still appears to be instant... it just uses more CPU than
it should)

Change-Id: I4b04b5805353dfbde0b841a3a557fd0b7c297780
2013-12-31 18:31:02 -08:00
Brad Fitzpatrick dbe9b83cfa search: some refactor cleanup in prep for sorting work
Change-Id: I71d422774bb68a93c9f9d648de8e4e5bc2ef1d87
2013-12-30 21:03:29 -08:00
Brad Fitzpatrick 8a05c520f8 search: make sort type encode as a string in JSON
Change-Id: I50c5663e850f3e8710381cc59daf246c8a600482
2013-12-29 16:28:06 -08:00
Brad Fitzpatrick 7d4c86d0de client: add a Search method
Change-Id: Iec4c22bf6e4bbe40645ec178b5c2d23392ad270d
2013-12-27 16:22:06 -08:00
Brad Fitzpatrick bc15a1d7a6 search: fix two double-RLock deadlocks
Change-Id: I5d083150cb1dc4e43c6a64146dee8a3672b7ea40
2013-12-24 13:46:18 -08:00
Brad Fitzpatrick 04bed02906 search: add negative expressions and has:location
e.g. [is:image -has:location]

Change-Id: I03df0a1d5fea8394a33e3079d131224c116d4c0d
2013-12-23 20:39:06 -08:00
Brad Fitzpatrick 7238bd1652 search: GPS location search
Like loc:hawaii or loc:USA or loc:94128, etc.

Change-Id: I11f47bf464a812f0b62e7799752811144bb7454e
2013-12-23 19:11:55 -08:00
Brad Fitzpatrick 06a334c39f search: add is:portrait, is:landscape, and is:pano
Also support Width and Height constraints on raw search (not yet
exposed to the expression syntax, but will be like width:<640 or
height:100-200 probably)

Change-Id: I082e3d27b5ef5c238a8e65a6b836943d9f9ff8a6
2013-12-22 18:30:27 -08:00
Brad Fitzpatrick 82a356ab77 search: add FloatConstraint
Change-Id: Ic367e87cb239258a76a99c9d07f0956d67e242ea
2013-12-22 18:11:44 -08:00
Aaron Boodman e15052302b Implement the new "expression" support in the web ui
Change-Id: Ic0521f770ffd45c24505ff386b2c044dc307e284
2013-12-21 22:10:17 -08:00
Brad Fitzpatrick 79fb299ff6 search: server-side search expression eval + result set continuation tokens
Search expressions (e.g. "tag:funny location:Portland") can now be
evaluated server-side (expr.go and expr_test.go, still nascent). Next
step is to remove this parsing and construction from the javascript UI
code.

In addition, the search result now contains a continuation token to
continue interating through the result sets in a subsequent query.
The Javascript UI should use this now instead of parsing out
modification times of things and altering the search.  If there's no
continuation token, that either means the end has been reached, or
that search type doesn't [yet] support iteration. Currently only
permanode-based queries are supported at many layers, including
continuation, but will be implemented later.

For now, the web UI should send a search query "expression" value of
non-zero length but just whitespace (like a single space: " ") to get
the home page (recently-modified permanodes") because setting both to
the zero value is an input validation error.  We should probably make
an expression operator to be more clear here (like "want:permanode
sort:recent").  But the empty space hack works for now.

Change-Id: I3ea92eb8c776159f53c49db1a7439a91c507940a
2013-12-21 15:59:11 -08:00
Brad Fitzpatrick a078ce9406 search: generate an optimized matcher and use it within the matching loop
benchmark                         old ns/op    new ns/op    delta
BenchmarkQueryRecentPermanodes        27987        27512   -1.70%

benchmark                        old allocs   new allocs    delta
BenchmarkQueryRecentPermanodes           99           87  -12.12%

benchmark                         old bytes    new bytes    delta
BenchmarkQueryRecentPermanodes         6433         6207   -3.51%

But will be better in practice with more blobs.

Change-Id: I4673702231c52864249329818145aad11da2b4fc
2013-12-14 12:07:35 +04:00
Brad Fitzpatrick 9af5a970de Mark boring files generated by FUSE as "hidden", then hide in the UI.
Makes for prettier FUSE demos w/ files showing up live in the browser
with websockets.

Change-Id: I41cc9ae0d33db39b4dbbf5b60714cee9a79b7248
2013-12-13 13:27:48 +04:00
Brad Fitzpatrick a0576f2c0d Fix corpus/search locking bugs.
The WebSockets live queries exposed a latent locking bug: the search
code was acquiring the Corpus read lock multiple times.  If a write
came in during the middle of a search, between two read locks, the
second RLock would hang (since a writer lock was pending) and then the
search and write would deadlock.

Instead, just acquire the read lock once at the beginning of the query and
never re-acquire it.  For now: new "Locked" versions of all the Corpus APIs.
In the future: will probably just add the locked-or-not into the context, and make
sure all Corpus methods require a *context.Context.

Also fix two potential data races: one with LogicalCondition (which
might've raced on the scratch ss buffer, depending on the query), and
a test-only race that didn't matter in production, but was annoying in
tsan output regardless, so fixed.

More tests to follow later. Unblocking Aaron for now.

Change-Id: Ie1edcbd061235f1ef04f9b4c68b86814c84361af
2013-12-12 11:34:07 +04:00
Brad Fitzpatrick bbaf755995 search: WIP notes on RelationConstraint. Not implemented yet.
Change-Id: Ic179a046da146706b57d7e6e031e1665deedd036
2013-12-11 13:46:19 +04:00
Brad Fitzpatrick a833a2d376 search: validate queries
Change-Id: Icc2840852a878b1afdfb7be97ddd50008422f0f7
2013-12-09 12:10:43 +04:00
Brad Fitzpatrick 3ea697da55 Redefine Permanode attribute matching constraints.
Removes a bunch of TODOs about single- vs multi-valued attributes.

Change-Id: I512e4afb5e0f5508fbb390ead3ce5162f47c5d28
2013-12-09 11:27:21 +04:00
Aaron Boodman 933e16ebf1 Replace special-case 'recent' query in web ui with new search.
Realized there was a way more elegant way to implement infinite
scroll. Now it is general to any kind of query (though not yet
to any kind of sorting, as there isn't support for continuing with
arbitrary sorts server-side).

Change-Id: I98dc627e1292524ddd8af8960c134f49ac41f337
2013-12-07 23:34:27 -08:00
Brad Fitzpatrick 1e1183a8bf search: add a forgotten Cancel.
Funny, because this one line is why I finally went and added the Context type
and started plumbing it through lots of stuff.

Change-Id: Ie419362048a485629e79725de9297fe485430d4a
2013-12-07 08:49:30 -08:00
Aaron Boodman d593b6f98a LogicalConstraint: typo in comment
Change-Id: I62345763e42832e2575f492a9678dbb0cfb65ec6
2013-12-07 07:11:16 -08:00
Brad Fitzpatrick cad54aa249 search: fix a panic in Query w/ describe, add tests
Change-Id: I4a2288298fbab8a51c795b9eaed5fc3488024132
2013-12-03 18:24:02 -08:00
Brad Fitzpatrick a7c7e717f6 search: optionally also describe matched blobs in same request
Change-Id: Id9c16c19fa309d0780c6107b1dd29880d777fd05
2013-12-02 20:29:50 -08:00
Brad Fitzpatrick bf4d2721ba search: efficient permanode queries, sorted, and with a limit.
Change-Id: I7f8edfe3636ff3f1344ebb41e27d23073c630ff5
2013-12-02 20:01:37 -08:00
Brad Fitzpatrick 1ec41b95f0 Speed up in-memory permanode attribute search.
Finishes Corpus.AppendPermanodeAttrValues and uses it in
PermanodeConstraint.

Change-Id: I031789e0974add24886cc0060e52ba538535c4e1
2013-11-27 12:47:04 -08:00
Brad Fitzpatrick 9a749ce73b search: efficient in-memory scans of permanode modtimes
Change-Id: I82d34544a86cd9f3a52c2b7140dd89eac32fbfa9
2013-11-26 19:47:00 -08:00
Brad Fitzpatrick 1455a14e72 search: some refactoring to ease future work speeding PermanodeConstraint queries
Change-Id: Iac98a632bd40c723481f8e635a2c34780e87a408
2013-11-26 19:05:43 -08:00
Brad Fitzpatrick d82f02db50 Search optimization: keep blobs arranged by camliType in memory; use when possible.
Change-Id: I8b3ebcf7dfece21d17074da32dc20e4d600e4c50
2013-11-17 10:52:37 -08:00
Brad Fitzpatrick 272db6545a index: change GetBlobMIMEType to GetBlobMeta. mime types on a blob was wrong anyway.
And some more of in-memory search, which this makes easier / less garbage-y.

Change-Id: If1fa7dd44842244ae5e5440f84e92d66dc3321fb
2013-11-16 19:40:14 -08:00
Brad Fitzpatrick 705107ad80 search/index: invert depedency. search now depends on index.
creates new package types/camtypes for misc types needed by both. might eventually go away as
search matures.

Change-Id: Ib771ead7bea39936ba478b7e5d58de997060861b
2013-11-16 15:00:30 -08:00
Brad Fitzpatrick 7f8e035be6 search: little TODO
Change-Id: I38a98c9c9b77a899536cf0d7601ce2030d6f4dda
2013-11-16 12:54:11 -08:00
Brad Fitzpatrick 8a5161ec30 search: multiple constraints at top level
Change-Id: I964a0d394c3c03e2409c91fb1d240847ff07ddb3
2013-11-16 11:21:30 -08:00
Brad Fitzpatrick 17a9d061a0 search: ditch BlobSizeConstraint, use IntConstraint instead
Change-Id: I99bece643841feab777175479743b3bd1fc421b8
2013-11-16 11:10:09 -08:00
Brad Fitzpatrick 1e627ce7c3 search: change BlobMeta to have a CamliType, not a misleading MIMEType.
Change-Id: I1c6e111ef27aafd5a15f9b38746361093a78e9e6
2013-11-16 10:35:18 -08:00
Brad Fitzpatrick 9587b8e62e search: rename AttributeConstraint to PermanodeConstraint, add ModTime constraint 2013-11-14 15:31:44 -08:00
Brad Fitzpatrick 6e7df1c6ee search: IntConstraint 2013-11-14 15:30:40 -08:00
Brad Fitzpatrick f718e64289 search: start of DirConstraint. mostly notes. not implementation.
Change-Id: I6403b4f9be2aa06778fb1ef96edde2896557c3c9
2013-11-09 18:56:52 -05:00
Brad Fitzpatrick c9443d7c7b search: TimeConstraint
Change-Id: Ic50290109043c1ad906f25935e5fad4c2170ea77
2013-11-09 17:08:46 -05:00
Brad Fitzpatrick 4c70e14310 search: add LimitReader around JSON decoding. add some TODOs
Change-Id: I98069ef0aeb8c96cdc997c138222e0344981ead8
2013-11-08 14:49:23 -05:00
Brad Fitzpatrick 4d620d2c60 Merge.
Change-Id: I15452c93c287eacf42e50dac59549a23d5e219c2
2013-11-08 14:47:51 -05:00
Brad Fitzpatrick c477ef43bb search: FileConstraint, StringConstraint, start of TimeConstraint, EXIFConstraint
Change-Id: I8464a392f2671e75898b64295b0c2e2261292ff9
2013-11-08 14:46:00 -05:00
Hunter Freyer 3fb2a0be4f Adds server-side search query handler.
Change-Id: Id1bdaa0f5f5a208ccf41ce25e5c50970f66f9469
2013-11-08 13:39:20 -05:00
Brad Fitzpatrick 59f20db9b2 search: delete unimplemented and now-undesired BlobTypeConstraint
The CamliType and AnyCamliType are better.

Change-Id: I0fbe9e645fa0ca13d107f7012bcb0008dfd5d43d
2013-11-08 13:17:37 -05:00
Brad Fitzpatrick 948bc3ba48 search: query support for AttributeConstraint.ValueMatches (subqueries)
e.g. show me permanodes with a camliContent pointing to a jpeg over 2.5 MB

Change-Id: I8b9c5c4b3595b64538281f63db7852461fd0367c
2013-11-08 13:11:16 -05:00
Brad Fitzpatrick 9e1948be25 search: ValueSet
Change-Id: Ie8b1b2c24ab9dcefb4f5e5d2db885568646ca228
2013-11-08 11:35:21 -05:00