Commit Graph

233 Commits

Author SHA1 Message Date
Brad Fitzpatrick da86ded96a Merge "index: Index MusicBrainz album IDs from music files." 2014-05-14 00:02:06 +00:00
mpl 2088ce739b search: sort by creation time
Change-Id: I3695d91dde0d42419c62a23f20ba19416c53e5ef
2014-05-08 18:55:53 +02:00
Daniel Erat aa391ecdd1 index: Index MusicBrainz album IDs from music files.
Index "MusicBrainz Album ID" ID3v2 frames as
"musicbrainzalbumid" media tags to facilitate downloading
cover art from coverartarchive.org.

Change-Id: Ie81017dd6f76ec355ee0d1daedfb7180cb70ad59
2014-05-05 20:45:58 -07:00
Brad Fitzpatrick 8ac8437e91 index: add temporary hack for foursquare.com checkin times
Change-Id: Ia90097998ccb702b14c4634bb42be1e6387d61e8
2014-04-23 11:32:57 -07:00
Brad Fitzpatrick bf9f69bcfe search: allow searching by permanode location, not just file location.
Allows searching foursquare checkins by location.

Change-Id: Ib33d0b435a9bbc17cb860549b41d119f727197f0
2014-04-23 11:13:41 -07:00
Daniel Erat 82429eeedb index,search: Return media tags for describe requests.
Change-Id: Ie983465554e1dc9f5475753a074e48b363f25013
2014-04-15 21:11:33 -07:00
mpl a9db189cd7 docker tests: remove container when done
http://camlistore.org/issue/421

Change-Id: Iefde3a8235db0103e8fcfac51f5dfcce5df73ef6
2014-04-09 23:26:14 +02:00
mpl 899acb8e72 pkg/sorted: remove API for each distinct implementation
Change-Id: I42446683189cdb00dcfbec1f221885092fdd83d6
2014-04-08 01:37:08 +02:00
Brad Fitzpatrick bf32345778 index: make Postgres test use the JSON constructor.
Would caught camlistore.org/issue/418

Change-Id: Ic323607df355e664d3b76709a7ad56dc40637fcf
2014-04-06 14:29:07 -07:00
Brad Fitzpatrick bf2764cdfe index: rename the reindex method to indexBlob, to be less confusing.
Also, upon server --reindex, check that no out-of-order blobs are
pending.  From a quick reading, they shouldn't be, but I'm curious to
see.  Will do a full reindex of my data later.

Change-Id: Idebf93cc264e55512afcfb99e47320dd0ae745d1
2014-04-06 14:03:38 -07:00
mpl 0cc8f6122d Merge "pkg/index: cleanup, refactor sub pkgs" 2014-04-05 12:45:22 +00:00
mpl 291320d451 pkg/index: cleanup, refactor sub pkgs
http://camlistore.org/issue/263

Change-Id: I319bb097f0ce30b2bd5271b5c3bbff92b8dcc318
2014-04-05 01:22:05 +02:00
Brad Fitzpatrick bf2f09cab3 index: reschedule indexing a claim blob if public key blob isn't yet available
Change-Id: Ie0174bf830eb4790080b2b5e7cdc4ea0af25406f
2014-04-02 13:39:36 -07:00
mpl b0c4f85ee0 sorted/postgres: test with docker
Also, NewKeyValue now automatically creates the required tables.

http://camlistore.org/issue/263

Change-Id: I02dd7660ee08c2eaeec17131622d6f581fdd6c55
2014-03-28 19:34:09 +01:00
mpl 08f9c821f5 sorted,dockertest: added MySQL (dockerified) test
Two important related changes:

1) sorted/mysql now takes into account the host given in the config
2) the required tables are now automatically created by NewKeyValue

http://camlistore.org/issue/263

Change-Id: I0043f36edb0630d6484148508d3a1e08c8e88a94
2014-03-26 00:39:38 +01:00
mpl 9cadbcd5bc publish: use generic queries
Use generic queries instead of specialized index queries. This is a step
towards the publisher app, because its client will have to use generic
queries.

Context: http://camlistore.org/issue/365

Change-Id: I2781a345e024174e3bea8511b6cdc6f342d5a7c1
2014-03-18 17:26:29 +01:00
Brad Fitzpatrick bfc607fee7 index: reindex blobs when dependent blobs arrive out-of-order
Keep track of missing dependencies both in memory and in the index's
underlying sorted.KeyValue. When we see a dependent blob arrive, see
if we can reindex things.

Fixes camlistore.org/issue/102

Change-Id: I3d8cfc463e4b8c9d158be8f9656e772839b093b9
2014-03-15 08:44:09 -07:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
Brad Fitzpatrick bf01b14961 index: move seekFetcherMissTracker up a layer
In prep for missing blob dependency rescheduling in indexer.

Change-Id: I1d492e6aa64cfb658daec17e4621d1453c6d3607
2014-03-14 09:14:46 -07:00
Brad Fitzpatrick bfeb745882 corpus: add PermanodeHasAttrValueLocked convenience accessor
Change-Id: Ie73270ce54c3125a76987daa9f50895c5acf42db
2014-03-13 18:48:00 -07:00
Brad Fitzpatrick bf6ae30368 corpus: reverse-index claims with values that look like blobrefs
Allows efficiently finding parents of permanodes, or permanodes of
file contents, etc.

Adds Corpus.ForeachClaimBackLocked.

Change-Id: I6dffb32eab4a959cd3c5922a7f47d11fdcea5f3d
2014-03-13 18:47:25 -07:00
Brad Fitzpatrick d13bb4aaed Merge "corpus: deleted check when enumerating permanodes" 2014-03-10 17:56:53 +00:00
mpl 0fdca769e0 corpus: deleted check when enumerating permanodes
http://camlistore.org/issue/354

Change-Id: I7d76c654eb3b875ad8017a14a08a5c909bd696a0
2014-03-10 18:43:57 +01:00
Brad Fitzpatrick 260a33aa2a Make various popular storage targets be Stringers. Optional, for upcoming sync handler rewrite.
Change-Id: I8cc961ace8b6093b1dbd82bc39ba3fc334f1d5c7
2014-03-05 08:23:07 -08:00
Tamás Gulácsi 97520583b8 Use 'uint32' instead of 'int64' for blob sizes everywhere.
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.

Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
2014-02-08 17:58:12 +01:00
Daniel Erat 5603ea8e0d pkg/index: Index audio duration.
Add pkg/media with code to calculate MPEG audio duration.
Index it in a "durationms" property.

Change-Id: Ifb6251657cadc365ef3f5667a0512fde17575560
2014-01-25 10:40:06 -08:00
Daniel Erat 404548d31a pkg/index: Index more music-related properties.
Add disc and mediaref (a hash of the audio portion of the
file).

Also relocate taglib code to
third_party/github.com/hjfreyer/taglib-go.

Change-Id: I58364f525b787484af894663125163095256d7c6
2014-01-22 21:25:05 -08:00
Daniel Erat 704d3c6bfc pkg/index: Rename audiotag to mediatag.
Also fix up keys and values and add tests.

Change-Id: I7e6c5c4315705442e3517456f2ba16419af49f2f
2014-01-20 21:46:39 -08:00
Brad Fitzpatrick 5b03c3f8fb search, index: let media tags be searchable too.
git push from Dolores Park. Sorry, no tests. Dan Erat will tell me if
this doesn't work.

Change-Id: I557cc3d07983390b8a15b7756ee0825fced2f503
2014-01-20 15:47:36 -08:00
Brad Fitzpatrick 4365f69330 Fix data race in reindexing.
See https://camlistore.org/issue/331

Change-Id: I210ae4e1779b7fe248196152a6acbc5728051249
2014-01-08 20:36:17 -08:00
Brad Fitzpatrick a576379cb5 search: PermanodeConstraint.Time time constraint, 'before:' and 'after:' operators
Work in progress, but works enough to commit now.  Determing the time
of things has many TODOs, and there's some performance work to be done
(although it still appears to be instant... it just uses more CPU than
it should)

Change-Id: I4b04b5805353dfbde0b841a3a557fd0b7c297780
2013-12-31 18:31:02 -08:00
Brad Fitzpatrick 14b950496f index, corpus: prevent indexing dup blobs
With the sync handler + indexer in same process subscribing to all
incoming blobs, we were indexing everything twice.

Fixes camlistore.org/issue/306

Change-Id: I7da54a0e18ac613eeae36d6db29b6cdb73a37196
2013-12-30 20:17:47 -08:00
Brad Fitzpatrick 38237bcec3 index: remove a spammy, harmless log message
Change-Id: Ibae1d6631ed1c6a086cf454e3d5e6fd9320936e3
2013-12-29 16:27:32 -08:00
Brad Fitzpatrick d5cf14d977 Make the reindex flag actually work.
Don't log.Fatal first saying the schema version is wrong.

Change-Id: I86272515cd62e5db13a5f31cffb91316285fb3fe
2013-12-25 09:49:51 -08:00
Brad Fitzpatrick e6496b45d8 index: update index-out-of-sync error message to suggest camlistored --reindex
Change-Id: I17dcd758e49a25c05ab5519a49ed3832610d2ba0
2013-12-24 15:46:17 -08:00
Brad Fitzpatrick 7728d7e2f8 index: bump schema version for EXIF changes
Change-Id: If91e3429d8ffb6a9f1eb7d6c5ab86d99bcc0a120
2013-12-24 15:42:45 -08:00
Brad Fitzpatrick bc15a1d7a6 search: fix two double-RLock deadlocks
Change-Id: I5d083150cb1dc4e43c6a64146dee8a3672b7ea40
2013-12-24 13:46:18 -08:00
Brad Fitzpatrick c9b0249c94 index: let reindexer optionally start at CAMLI_REINDEX_START
Change-Id: I3a3018c5ff72e0662bbe285eef299c7092366c5f
2013-12-23 22:02:30 -08:00
Brad Fitzpatrick 88c3709e29 corpus: ignore exiftag rows for now
Change-Id: I5ccbb8517c8576bc0f554db73f2abd4b827b8388
2013-12-23 20:38:35 -08:00
Brad Fitzpatrick a11ff22b8e camlistored: add --reindex flag; make sqlkv a sorted.Wiper
Change-Id: I6b16c1c32187fb754d3acdbe852d02a506236078
2013-12-23 19:07:17 -08:00
Brad Fitzpatrick a7b3f4ee01 index: index all photo EXIF tags
Change-Id: I00b2eebfc75de38eed5c212ac6d52e0da07297bc
2013-12-23 16:21:19 -08:00
mpl 8fec95752c pkg/index: move postgresql to sorted + some fixes
Change-Id: Ia6410256ffe8d4cbc2b9a5c056d075707105ab08
2013-12-24 00:37:14 +01:00
Brad Fitzpatrick 06a334c39f search: add is:portrait, is:landscape, and is:pano
Also support Width and Height constraints on raw search (not yet
exposed to the expression syntax, but will be like width:<640 or
height:100-200 probably)

Change-Id: I082e3d27b5ef5c238a8e65a6b836943d9f9ff8a6
2013-12-22 18:30:27 -08:00
Brad Fitzpatrick 79fb299ff6 search: server-side search expression eval + result set continuation tokens
Search expressions (e.g. "tag:funny location:Portland") can now be
evaluated server-side (expr.go and expr_test.go, still nascent). Next
step is to remove this parsing and construction from the javascript UI
code.

In addition, the search result now contains a continuation token to
continue interating through the result sets in a subsequent query.
The Javascript UI should use this now instead of parsing out
modification times of things and altering the search.  If there's no
continuation token, that either means the end has been reached, or
that search type doesn't [yet] support iteration. Currently only
permanode-based queries are supported at many layers, including
continuation, but will be implemented later.

For now, the web UI should send a search query "expression" value of
non-zero length but just whitespace (like a single space: " ") to get
the home page (recently-modified permanodes") because setting both to
the zero value is an input validation error.  We should probably make
an expression operator to be more clear here (like "want:permanode
sort:recent").  But the empty space hack works for now.

Change-Id: I3ea92eb8c776159f53c49db1a7439a91c507940a
2013-12-21 15:59:11 -08:00
mpl 62f8a252ff pkg/index: move mysql to sorted
also minor cleanups in sqlite and mongo

Change-Id: I1f371358997d929c72a8f63d2630a1c3fa4a0240
2013-12-18 20:18:20 +01:00
mpl 6102ca9a80 mongo: NewKeyValue now takes a mongo.Config
Change-Id: Ibfb0085836030592db4373e4ac2540355e67a9c5
2013-12-17 15:18:14 +01:00
Brad Fitzpatrick 67341654ad Merge "images: fix Decode when resize + rotate + max W/H." 2013-12-16 22:21:42 +00:00
Brad Fitzpatrick 599fb8c6b3 Merge "pkg/index: move sqlite implementation to sorted" 2013-12-16 18:34:40 +00:00
Bill Thiede 2d4fb25c34 images: fix Decode when resize + rotate + max W/H.
Adds more tests to cover rotations with resize when used with
MaxWidth/MaxHeight, previously only ScaledWidth/ScaledHeight were
tested.

Improve tests to compare bounds when determining equality, otherwise
an image sized 0x0 is equal to all other images.

Sort test image filenames so test order is stable and obvious.

Keep more data in memory when indexing images upon receive.  Some
largish CR2 files need more data or the EXIF parsing will fail.

Should address some or all of https://camlistore.org/issue/274

Change-Id: I80d90c33538c9d62ce4480ccb58c003e18ee6629
2013-12-16 10:01:07 -08:00
mpl b62c94fdd1 pkg/index: move sqlite implementation to sorted
Change-Id: I1e300ffa14547fabeac8d255fd694054ebcb9e53
2013-12-16 17:35:27 +01:00