Commit Graph

24 Commits

Author SHA1 Message Date
Brad Fitzpatrick 20eca7aad0 search: new DescribeRule describe support
Step towards getting rid of the the integer describe depth API.

Now, after the integer describe depth, all the DescribeRules will also
run to expand things.

Change-Id: I5cfe7e6058be51328e529a2299e13a6a2ba5f869
2014-04-19 12:58:55 -07:00
Daniel Erat 991675db09 test: Fix FakeIndex breakage from 82429eee.
Change-Id: I403fe39617b3d61212b6a0e6aeaffa350c441221
2014-04-16 09:35:41 -07:00
Tamás Gulácsi 97520583b8 Use 'uint32' instead of 'int64' for blob sizes everywhere.
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.

Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
2014-02-08 17:58:12 +01:00
Brad Fitzpatrick bf4d2721ba search: efficient permanode queries, sorted, and with a limit.
Change-Id: I7f8edfe3636ff3f1344ebb41e27d23073c630ff5
2013-12-02 20:01:37 -08:00
Brad Fitzpatrick 8c46afe490 Shrink camtypes.BlobMeta a bit. Saves 4 bytes of memory per blob.
For me,

Before:
Corpus stats: 148.859 MiB mem: 799465 blobs (89.487 GiB) (282303 schema (6478 permanode, 150962 file (112224 image), ...)

After:
Corpus stats: 146.559 MiB mem: 799465 blobs (89.487 GiB) (282303 schema (6478 permanode, 150962 file (112224 image), ...)

Change-Id: Id421dfc9d72c2a24c228418c01f9a3f05900c941
2013-11-29 13:41:39 -08:00
Brad Fitzpatrick a918fac281 search: let GetRecentPermanodes take a before param.
For infinite scroll.  Will be done better later, but this works for now.

Change-Id: I88dc7793b643226a4a5e3a583c5de5d4c7fa4819
2013-11-25 20:35:59 -08:00
Brad Fitzpatrick 531b4dc32b search: remove MIME from DescribedBlob; fix Handler tests.
Change-Id: I2d92155e3446325520d91379ac24becef49ca56f
2013-11-18 16:36:08 -08:00
Brad Fitzpatrick 0b07171c0c index: change GetOwnerClaims interface for efficiency, rename to AppendClaims
Change-Id: Icec235961ed8a09256df87c5fe1455cbad81ae3f
2013-11-17 14:54:30 -08:00
Brad Fitzpatrick 3eb493599e in-memory search: better structure for keeping memory corpus and kv
index in sync, both at start-up and while running and receiving blobs.
They both use the same mechanism now.

Also adds KeyId to the index and Corpus, as the next step. Plenty more
row types remain...

Change-Id: Id79955ba25dc79d5fbd94b0e5248d33dcf71d97e
2013-11-17 09:41:45 -08:00
Brad Fitzpatrick 272db6545a index: change GetBlobMIMEType to GetBlobMeta. mime types on a blob was wrong anyway.
And some more of in-memory search, which this makes easier / less garbage-y.

Change-Id: If1fa7dd44842244ae5e5440f84e92d66dc3321fb
2013-11-16 19:40:14 -08:00
Brad Fitzpatrick e26eacfa2f index: return FileInfo and ImageInfo as values, not pointers.
Change-Id: I6aba3a74ce233cc834550b561f597e549fb2bfab
2013-11-16 17:49:28 -08:00
Brad Fitzpatrick 705107ad80 search/index: invert depedency. search now depends on index.
creates new package types/camtypes for misc types needed by both. might eventually go away as
search matures.

Change-Id: Ib771ead7bea39936ba478b7e5d58de997060861b
2013-11-16 15:00:30 -08:00
mpl 01a8dbb163 pkg/search: add modtime in WANT to fix tests
Also fixed fakeindex clock to use a time.Time.
And synchronized the fakeindex and the indextests clocks to the
same origin.

Change-Id: I986abca300d61241069132619028f470cacd61da
2013-11-15 23:31:24 +01:00
Brad Fitzpatrick 17ecf8df07 search: start of general Query support. very rough draft.
Start of something that builds and 'works', but barely fleshed out at all.
Likely to change drastically yet.

Change-Id: I7fba25528a341ff2dbbc4dbb579f2450d7889320
2013-10-18 17:17:35 -07:00
mpl d488c576fc search: support for static directory children
This change introduces a new index entry
to help with finding the children of a static directory.
It also fixes ResolvePrefixHop so that it takes
into account static directories, and not only collections.

This is the first step to support publishing static directories.

http://camlistore.org/issue/179

Change-Id: I5666e5caa6c782004054ae4c19a6b6119d4fda8b
2013-09-10 23:06:48 +02:00
Brad Fitzpatrick 0bdf20884b all: delete pkg/blobref; convert all from *blobref.BlobRef to new blob.Ref
Change-Id: Id2dfb7f19452bedf4f3c9310b36227fd8117b225
2013-08-03 19:54:30 -07:00
mpl 0dfd84a7d8 search handler: return correct thumbnail dimensions
images: DecodeConfig to get the predicted width
and height after EXIF correction
search&index: add GetImageInfo and use it in search
to predict the thumbnail dimensions

http://camlistore.org/issue/115

Change-Id: I358136a2ab03ea09c8f8fd2fa0dc574921c819c5
2013-03-25 17:06:15 +01:00
Brad Fitzpatrick 8155b3867a Fix case of MimeType to MIMEType, per Go style.
Change-Id: I6c67a61cea2c71d7fa5d52c4ae283e33e993fd0f
2013-02-18 11:16:13 -08:00
Brad Fitzpatrick ab98ebfa23 some FakeIndex docs
Change-Id: I508e5326c07c8cc7bd39068a2d7e342d0b27884c
2012-11-04 12:38:33 +01:00
Brad Fitzpatrick 0ccf9ba3f1 some TODOs
Change-Id: Ifcc2612ea323347ba78d5cb493ebe9fabd3a10aa
2012-11-03 22:27:27 +01:00
Brad Fitzpatrick 4b0d2841ee Add search.EdgesTo method to search interface. No implementation yet.
Change-Id: Ie3a9eb91e218e172b195d55cb9c17cb6c8f819ce
2012-11-03 16:08:37 +01:00
Brad Fitzpatrick c463c3a26e fix pkg/server tests
Change-Id: I66fa74f52035b3185ac6390d4e38244f4b29597f
2012-03-29 16:21:07 -07:00
mpl 50df48d519 first pass with mgo as indexer
fixed indexer so that it builds as a package
added mgo as a third party
fixed mgoindexer
fixed (hackishly) tests so that they pass
I ignored everything memindex for now

Change-Id: I3c03ecfa859cf92b088a759239278b1c0edaf8aa
2012-02-20 14:40:24 +01:00
Brad Fitzpatrick 0714a463c9 Update from r60 to [almost] Go 1.
A lot is still broken, but most stuff at least compiles now.

The directory tree has been rearranged now too.  Go libraries are now
under "pkg".  Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".

The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.

For now, put the camlistore root under $GOPATH/src.  Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.

Then you can:

$ go build ./server/camlistored

... etc

The build.pl script is currently disabled.  It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).

Many things are certainly broken.

Many things are disabled.  (MySQL, all indexing, etc).

Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.

The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.

Help wanted.

Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
2012-02-18 21:53:06 -08:00