The problem was newSorted was taking care of doing the DB
initialization. It was not only unnecessary, since newKeyValueFromConfig
from sorted/sqlite will do it by calling initDB, but also incorrect
because WAL was not getting enabled. Which is what was causing the
"database is locked" error "fixed" in commit
f6bf779cd6
Change-Id: Ib4badeda4ed4c6c4be43d5f7fc4c543a467b0cd1
So when you describe a file, you also gets its wholeref.
TODO: we'll need to migrate old indexes to this new format on
start-up.
Change-Id: I4a3fb000d68bde46474275c2070ef285a6d6ecfc
Go's image.DecodeConfig needs more than 1MiB on some images (e.g. some
Lens Blur pics taken with Google Camera). Now we first try a 512KiB header
and retry with a full FileReader if that fails.
https://camlistore.org/bugs/477
Change-Id: I286d15d86a69951737d94dd3692d4e9e1992b324
This pulls the changes from the current HEAD of
https://github.com/rwcarlsen/goexif
(eb2943811adc24a1a40d6dc0525995d4f8563d08)
Notable changes:
- Removed explicit panics in favor of error returns
- renamed TypeCategory to Format and made format calculated upon
decoding rather than repeatedly for every format call
- Merged contributions from Camlistore (exif.LatLong(), exif.DateTime()
etc.)
- Change String method to just return the string value - and don't have
square brackets if only a single value
- add separate Int and Int64 retrieval methods
- Doc updates
Minor changes in camlistore.org/pkg/* were neccessary to reflect
changes in the API (handling of returned errors) and in names of
exported fields and methods.
Change-Id: I50412b5e68d2c9ca766ff2ad1a4ac26926baccab
In the advent of github.com/camlistore/goexif to be closed, this
commit renames the goexif folder in third_party to match the
upstream on GitHub.
The affected import paths have been rewritten accordingly.
Change-Id: I5a8871efd01987944b7f5e93979307857ae16fe7
This pulls the changes from the current HEAD of
https://github.com/rwcarlsen/goexif
(rev cf045e9d6ba052fd348f82394d364cca4937589a)
Changes to goexif from upstream:
- Add support for reading Nikon and Canon maker notes
- Adds parser registration to exif package
- Renamed cmd to exifstat
- Renamed exported fields and methods in goexif/tiff
- adds support for bare tiff images. bug fix and minor cosmetics
- support pulling the thumbnail
- adds thumbnail support to exif pkg
- tiff defines DataType and constants for the datatype values
- Update covnertVals and TypeCategory to use new constants for DataType
- Renamed test data dir in exif tests
- created type+constants for raw tiff tag data types
Not merged from upstream:
- ~1 MB of test JPGs in goexif/exif/samples
Minor changes in camlistore.org/pkg/* were neccessary to reflect the
name changes in the exported fields and methods.
Change-Id: I0fdcad2d7b5e01e0d4160a5eb52b8ec750d353cf
problem: the out-of-order mechanism based on the outOfOrderIndexerLoop
was not working for some claims.
Let C be a delete claim on permanode P. If C was received before P was,
C was marked as being received with the "have" index row. However, for
the deletion to be marked in the index, some information about P is
needed (its meta row), so C could not be fully indexed upon reception.
Then, when P was finally received, the outOfOrderIndexerLoop would kick
in and retry indexing C. Which would fail, because a test based on the
"have" row would (wrongly) detect that C is already indexed and return
early.
In this patch:
-we introduce the "|indexed" suffix to the "have" - value part - row
(receive.go). If a blob is received but some of its dependencies are
missing, the have row value is written without the suffix. Upon
reception of a blob, we now test for the presence of the suffix in the
have row. If missing, the reception continues instead of returning
early. The existing mechanism that was detecting missing dependencies
for file blobs has been adapted to work with this suffix too.
-the index enumeration (enumstat.go), which relies on "have" rows, has
been adapted to work with the new "have" row format, while staying
compatible with the old format. And related tests have been added.
http://camlistore.org/issue/454
Change-Id: I2559d08a12b2a4e0f0691fc7e31f1ed1f874625e
index.New was starting outOfOrderIndexerLoop in a goroutine. And
outOfOrderIndexerLoop had an if index.BlobSource == nil check, on which
it relied to go on. However, since BlobSource was public and unguarded,
the following sequence was possible:
ix, _ := index.New()
ix.BlobSource = bs
which is racy because the BlobSource assignment may or may not happen
before the check within outOfOrderIndexerLoop.
TestOutOfOrderIndexing was relying on the fact that apparently most
of the time the assignment seems to be happening before the check.
This patch:
-makes BlobSource (now blobSource) private, rendering the race impossible
out of the index package.
-moves the initialization of blobSource, as well as the execution of
outOfOrderIndexerLoop at a unique point, in InitBlobSource (new method).
-makes sure all accesses to blobSource are guarded with the index mutex
(now a RWMutex).
Context: while working on tests for http://camlistore.org/issue/454
Change-Id: I9605f26b41abd62b42880be0620b06ce143761bc
Index "MusicBrainz Album ID" ID3v2 frames as
"musicbrainzalbumid" media tags to facilitate downloading
cover art from coverartarchive.org.
Change-Id: Ie81017dd6f76ec355ee0d1daedfb7180cb70ad59
Also, upon server --reindex, check that no out-of-order blobs are
pending. From a quick reading, they shouldn't be, but I'm curious to
see. Will do a full reindex of my data later.
Change-Id: Idebf93cc264e55512afcfb99e47320dd0ae745d1
Two important related changes:
1) sorted/mysql now takes into account the host given in the config
2) the required tables are now automatically created by NewKeyValue
http://camlistore.org/issue/263
Change-Id: I0043f36edb0630d6484148508d3a1e08c8e88a94
Use generic queries instead of specialized index queries. This is a step
towards the publisher app, because its client will have to use generic
queries.
Context: http://camlistore.org/issue/365
Change-Id: I2781a345e024174e3bea8511b6cdc6f342d5a7c1
Keep track of missing dependencies both in memory and in the index's
underlying sorted.KeyValue. When we see a dependent blob arrive, see
if we can reindex things.
Fixes camlistore.org/issue/102
Change-Id: I3d8cfc463e4b8c9d158be8f9656e772839b093b9
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.
SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.
Lot of code got simpler as a result.
Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
Allows efficiently finding parents of permanodes, or permanodes of
file contents, etc.
Adds Corpus.ForeachClaimBackLocked.
Change-Id: I6dffb32eab4a959cd3c5922a7f47d11fdcea5f3d
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.
Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
Brad Fitzpatrick