Commit Graph

165 Commits

Author SHA1 Message Date
Felix Geller 036f8b19af pkg/index: Pass whole file for finding modtime when needed
When reading EXIF data for large tiff files, our optimistic
file prefix sometimes isn't enough and we need to pass in
the whole file. We already did this in several places (image
decoding and indexing), this change adds it for finding the
modtime (for which we try to use EXIF data when
available). It pulls common functionality out into a
separate func and changes the existing uses of this pattern
to make use of the func.

Change-Id: I2b786775f168f47f46fb5ac707e3744991139a21
2015-05-26 08:16:10 +12:00
mpl 25a545da57 search: fix tests needing null around
follow-up of 491bbf525e

Change-Id: Id32e33f8bce1944510283da8e8cfa8bf96c0765e
2015-04-04 19:10:10 +02:00
Mario Russo a6fe834326 Capture the file's lastModifiedDate provided by File API when uploading from the UI
this change introduces one API to filewriter, which takes an additional param for the file's lastmodtime to store with the file blob

Issue #317

Change-Id: I8da3134306c28e6945661296aaafb7752b4bf8b5
2015-02-26 09:59:43 +01:00
Salmān Aljammāz 8bee1495ab client: log when GPG signatures mismatch
Mismatching GPG keys seems to be a very common issue for newcomers.

Change-Id: If83eee5d0ba5f0c85a4d75a31ab15a5469c2965d
2015-01-12 14:06:22 +03:00
Brad Fitzpatrick 54b1925847 Merge "DirReader.Readdir incorrectly logs the contents of err where it should really log the contents of res.err" 2014-11-24 18:36:16 +00:00
aarzilli b92df8065d DirReader.Readdir incorrectly logs the contents of err where it should really log the contents of res.err
Change-Id: Ic607d823af4b6147a54f896ffabaabf67b7b1c02
2014-11-24 18:41:50 +01:00
mpl 21a61dad40 Merge "index,camtool: try and cope better with broken exif" 2014-11-18 15:24:20 +00:00
mpl fda1399e9c index,camtool: try and cope better with broken exif
http://camlistore.org/issue/493

Change-Id: I40aebd67252cf82a3a5a143af6c258d7ed2aecda
2014-11-10 19:05:35 +01:00
Jingguo Yao 9b6a9c587a schema: fix typos in error messages
1. Error messages in dirreader should mention dirreader instead of
   filereader.
2. Remove \n at the ends of error messages.
3. Use errors.New for plain text error messages.
4. Give precise static-set information in one error message.

Change-Id: Iae0f46bca492e9dbbe41b4c1bebf5f5950d722b5
2014-11-03 20:44:57 +08:00
Jingguo Yao 963c0f78db schema: correct a typo in error message
Function Stat is invoked, but the error message mentions lstat.

Change-Id: I66b5cca3cdc2c6281ce720977d34452502a92aa7
2014-11-01 10:41:45 +08:00
Jingguo Yao 51a16a4082 schema: use correct name for function parameter
The comment for Readdir talks about n, but the parameter name of Readdir
is count.

Change-Id: I3669730eb1972077ce81219c4ce64f19e176caa1
2014-10-29 09:17:11 +08:00
mpl 2fc1c5a81c schema: set a default UnixPermission in FileMode if needed
Change-Id: I98fd229d0cbd69069a7c39a26e5c8b4444d13ed1
2014-10-20 17:52:02 +02:00
mpl 0aa0b18e5f fix SetShareSearch doc
Change-Id: I751b3b209f974830960480460e14c811ee9a5617
2014-09-30 00:38:57 +02:00
Mathieu Lonjaret 312f7a2aa6 Merge "Merge upstream goexif" 2014-09-17 13:49:35 +00:00
Fabian Wickborn 59a451c2dc Merge upstream goexif
This pulls the changes from the current HEAD of
https://github.com/rwcarlsen/goexif
(eb2943811adc24a1a40d6dc0525995d4f8563d08)

Notable changes:
- Removed explicit panics in favor of error returns
- renamed TypeCategory to Format and made format calculated upon
  decoding rather than repeatedly for every format call
- Merged contributions from Camlistore (exif.LatLong(), exif.DateTime()
  etc.)
- Change String method to just return the string value - and don't have
  square brackets if only a single value
- add separate Int and Int64 retrieval methods
- Doc updates

Minor changes in camlistore.org/pkg/* were neccessary to reflect
changes in the API (handling of returned errors) and in names of
exported fields and methods.

Change-Id: I50412b5e68d2c9ca766ff2ad1a4ac26926baccab
2014-09-17 10:40:38 +02:00
Brad Fitzpatrick 50cced8b7b schema: modify ForeachChunk to give schema blob path, not just parent
Then in blobpacked, use it, so we don't miss any intermediate 'bytes' schema blobs.

Change-Id: I8eedab562f26ad23a340aca8e24a7154a8414029
2014-09-06 20:14:27 -07:00
Brad Fitzpatrick 8f2bdd3a82 Few little fixes to 51ef6152f
Change-Id: Id319d1fa4ad725f1ce9178fd0fbd7f787880a6fa
2014-09-06 19:19:36 -07:00
Aaron Boodman 51ef6152fd Add -search argument to camput share.
Bug: https://code.google.com/p/camlistore/issues/detail?id=512

Change-Id: Ic0dd559c98589107fcdeade8784699cf6b2f1761
2014-09-06 18:50:21 -07:00
Fabian Wickborn 2aed1b8241 Renamed goexif folder to match upstream URL
In the advent of github.com/camlistore/goexif to be closed, this
commit renames the goexif folder in third_party to match the
upstream on GitHub.

The affected import paths have been rewritten accordingly.

Change-Id: I5a8871efd01987944b7f5e93979307857ae16fe7
2014-09-05 17:27:59 +02:00
Fabian Wickborn f0d9c04bc2 Merge goexif with upstream package
This pulls the changes from the current HEAD of
https://github.com/rwcarlsen/goexif
(rev cf045e9d6ba052fd348f82394d364cca4937589a)

Changes to goexif from upstream:
- Add support for reading Nikon and Canon maker notes
- Adds parser registration to exif package
- Renamed cmd to exifstat
- Renamed exported fields and methods in goexif/tiff
- adds support for bare tiff images. bug fix and minor cosmetics
- support pulling the thumbnail
- adds thumbnail support to exif pkg
- tiff defines DataType and constants for the datatype values
- Update covnertVals and TypeCategory to use new constants for DataType
- Renamed test data dir in exif tests
- created type+constants for raw tiff tag data types

Not merged from upstream:
- ~1 MB of test JPGs in goexif/exif/samples

Minor changes in camlistore.org/pkg/* were neccessary to reflect the
name changes in the exported fields and methods.

Change-Id: I0fdcad2d7b5e01e0d4160a5eb52b8ec750d353cf
2014-09-05 08:36:42 +02:00
tnt aad1a492f1 Fatal -> Fatalf
Change-Id: I8930950b89eb1dc99c1567a0ae9cb3ef848ce67f
2014-09-04 00:43:57 +02:00
Brad Fitzpatrick 37e38ad8b9 schema: add disabled failing ForeachChunk test
For blobkpacked.

Change-Id: Id133032aed21c9beeb17aff08d2500281ed3a5a2
2014-09-02 22:26:11 -07:00
Brad Fitzpatrick d63bcc04c2 blobpacked: export, document Manifest type. slight tweaks to zip format.
Change-Id: I9fbb005f0ecc6f013575255fae9e62b8397701ee
2014-09-01 16:25:37 -07:00
Brad Fitzpatrick 5b1a327916 schema: fix test I broken when removing GetChunkOffsets
Rewrite in terms of ForeachChunk instead.

Change-Id: Id018420c47ffe116246c92b06c61a9dc9bbbee62
2014-09-01 15:11:14 -07:00
Brad Fitzpatrick 8d79f59908 blobpacked: first cut at packing, creating zip files with contiguous chunks.
See package docs in blobpacked.go for background & rationale.

Will minimize disk seeks, regardless of the underlying storage
system. I always knew this day would come, but we've been putting it
off. It's finally time to separate out logical (tons of small chunks)
storage from physical (big contiguous layout for how they'll be
accessed).

First 80% is done. Last 80% remains: mostly lots of tests and more
paranoia, and fsck things, for when we inevitably lose indexes. (but
each zip file is self-describing, so the index is redundant).

Change-Id: I13f9aa79c4d9d87b5fdcece0d99d97eba031d608
2014-08-30 00:09:10 -07:00
Brad Fitzpatrick 1be26ea3a4 schema: rewrite FileReader.LoadAllChunks, delete GetChunkOffsets
LoadAllChunks is now written in terms of ForeachChunk and is bounded.

GetChunkOffsets was never meant to be exported and was unused elsewhere.
It's now unused entirely.

Change-Id: I0653385fe805fda721e0b45ea0a17d9839c3ba6e
2014-08-29 15:26:36 -07:00
Brad Fitzpatrick 3388e363b9 schema: add FileReader.ForeachChunk
Will be used by the blobpacked storage server.

Also, we can then rewrite much of the legacy crufty methods on
FileReader in terms of this.

Change-Id: Ia371001f1ed502fa1da2dbc4a8e1c29e5f0c9e8c
2014-08-29 14:51:33 -07:00
Brad Fitzpatrick 17ab1c238f Change strutil.StringFromBytes to depend on Go 1.3.
Change-Id: I83fd1369a199dba7ef41227df4d0b95d9a16aabb
2014-08-14 12:59:54 -07:00
Brad Fitzpatrick 766919ec33 schema: more minor FileReader cleanups
Change-Id: Ia37d82024faf16c262f339d8f7f807d23670edda
2014-08-11 17:32:00 -07:00
Brad Fitzpatrick 71a1a1ff8d schema: reduce FileReader blob reads 32-85x by caching the last blob read
Little reads into big blobs kept re-reading the same blob.

Change-Id: I098c9d9a9443dacc93f60f96fff1edd421ced198
2014-08-11 15:55:50 -07:00
Brad Fitzpatrick fe481e5da7 schema: some FileReader cleanups: use blob.Ref as map key, simplify zeroReader
Change-Id: I37ec27bfe9673e7d5566216269d94d109215ac99
2014-08-11 15:53:33 -07:00
Brad Fitzpatrick da3ce4a13b schema: use existing constant instead of literal
Change-Id: Ia988f4c897788be696326de9a75973161ae3d550
2014-08-07 11:04:09 -07:00
mpl 2437ada83f foursquare: some attributes cleanup
Change-Id: I7a1d794e58b47edeef06c3d181991c8aa1eb0534
2014-07-31 01:50:36 +02:00
Brad Fitzpatrick 95492813b1 picasa: clean up album attributes, add tests, fix keywords with spaces
Change-Id: I32faae16334e3999731f3980502eb6a1a6d38595
2014-07-30 15:50:34 -07:00
Brad Fitzpatrick 80e543efe2 nodeattr: add more, rearrange, clean up some comments.
Change-Id: Ifc1809ed07166c18253cf0a82e708fccb3071f08
2014-07-28 13:56:43 -07:00
Brad Fitzpatrick b531c32f11 nodeattr: moving more from importer/attrs.go
Change-Id: Iebf336c44af77ccbe655aadbe1e57ad339ba5201
2014-07-25 14:48:03 -07:00
Brad Fitzpatrick d0fbfd9751 nodeattr: rename Content to CamliContent, since Content also exists.
Change-Id: I305ca3cecf9b223a125fb9207966995c9dbf83b8
2014-07-25 14:41:17 -07:00
Brad Fitzpatrick fad3a60a77 picasa: some importer attribute cleanup
Use nodeattr constants. Don't set lat/long when unknown.
2014-07-25 13:24:46 -07:00
Brad Fitzpatrick f2b9657be6 nodeattr: fix up comment style
And use nodeattr.Title in twitter.
2014-07-25 13:16:10 -07:00
mpl 7e86255cac importer/flickr: set DateCreated attribute
Also cleaned up a few vars.
Moved some attributes to nodeattr pkg.

Change-Id: Iea496d50b12e67132fe88f371a89534c2fe32761
2014-07-25 00:57:10 +02:00
Brad Fitzpatrick c2dd9a1fce schema: add test of time.ParseInLocation with UnknownLocation
Change-Id: I23c1007b53d794e36e9eeaad03e9d11a4cd3c397
2014-07-24 13:44:31 -07:00
Brad Fitzpatrick 804f11d49e schema/nodeattr: new leaf package for permanode attributes
All the attributes from pkg/importer/attrs.go should eventually move here.

Change-Id: I252749ca32fdb0f367dfa906e728d0a679e8eb5b
2014-07-24 13:37:45 -07:00
Brad Fitzpatrick ab885a369c schema: add UnknownLocation and IsZoneKnown
Change-Id: Ia056d03a367502ea4d06e5748cc7de7072c3e12c
2014-07-24 13:35:18 -07:00
Brad Fitzpatrick 077763fcbb schema: use GPS location to find timezone from EXIF when UTC offset is unknown
Change-Id: Ia3590424db36508b491d8f19829738fe102e5c3d
2014-07-13 10:26:54 -07:00
Brad Fitzpatrick af4df6f8c5 schema: rewrite symlink test to not delete testdata, not use asserts
Change-Id: I238ee5d55203f5c53b294d82e2cc626c4ef78ef2
2014-07-13 10:02:30 -07:00
tnt 4ac9938c8b Fixed some formatting functions that were called with invalid number/type of arguments.
Change-Id: I261c6fb732a6cf1e54e01fa3e5388c39c6deb720
2014-07-01 12:48:33 +02:00
tnt 5d8eeb534b Fixed wrong calls to formatting functions.
E.g: fmt.Println("%s", "hello world") -> fmt.Printf("%s", "hello world")

Change-Id: I51c0bd54c14173e074fafd86c83a5287fcdb1520
2014-06-30 22:05:22 +02:00
Brad Fitzpatrick f3a519d3e5 schema: remove FileReader.FileSchema returning private type
Change-Id: I66af87155edfb7672294072fe18878af0811cf62
2014-06-13 15:47:14 -07:00
Brad Fitzpatrick 26bdefc4b0 osutil: remove err error stutter, add osutil.ErrNotSupported
Change-Id: I483f30d3d4db39a9ffa0f83c1e055d7a59edba4b
2014-05-31 10:42:44 +09:00
Brian Gitonga Marete 53ab352fa7 pkg/schema, cmd/camget, cmd/camput: Support sockets.
This patch:

1. Introduces a JSON schema for unix domain sockets (socket files).
2. Adds support for the new schema to pkg/schema.
3. Adds support for uploading and restoring sockets to camput and
camget respectively.

Depends on CL 2706 (https://camlistore-review.googlesource.com/2706/)

Change-Id: Ib0d6e9f9ebd348f55efee5a08cd774f3969d0760
2014-05-21 18:51:14 +03:00