Commit Graph

4984 Commits

Author SHA1 Message Date
Fabian Wickborn f0d9c04bc2 Merge goexif with upstream package
This pulls the changes from the current HEAD of
https://github.com/rwcarlsen/goexif
(rev cf045e9d6ba052fd348f82394d364cca4937589a)

Changes to goexif from upstream:
- Add support for reading Nikon and Canon maker notes
- Adds parser registration to exif package
- Renamed cmd to exifstat
- Renamed exported fields and methods in goexif/tiff
- adds support for bare tiff images. bug fix and minor cosmetics
- support pulling the thumbnail
- adds thumbnail support to exif pkg
- tiff defines DataType and constants for the datatype values
- Update covnertVals and TypeCategory to use new constants for DataType
- Renamed test data dir in exif tests
- created type+constants for raw tiff tag data types

Not merged from upstream:
- ~1 MB of test JPGs in goexif/exif/samples

Minor changes in camlistore.org/pkg/* were neccessary to reflect the
name changes in the exported fields and methods.

Change-Id: I0fdcad2d7b5e01e0d4160a5eb52b8ec750d353cf
2014-09-05 08:36:42 +02:00
Brad Fitzpatrick 108e16510f Merge "Adds less (CSS processor) v1.7.4 (Apache v2)" 2014-09-04 06:22:37 +00:00
Lindsey Simon ffb660b54d Adds less (CSS processor) v1.7.4 (Apache v2)
Change-Id: I364e46d395c0752df8c585a5aeab3aa9a289162c
2014-09-03 22:22:56 -07:00
Brad Fitzpatrick fa8edbdcd2 Merge "typo in doc for CreateHandler" 2014-09-04 01:10:28 +00:00
tnt aad1a492f1 Fatal -> Fatalf
Change-Id: I8930950b89eb1dc99c1567a0ae9cb3ef848ce67f
2014-09-04 00:43:57 +02:00
tnt 1a9aa07496 typo in doc for CreateHandler
Change-Id: I31b1f87c6ec9d24d518b2d0ebf6e0e4dd81e7ff6
2014-09-04 00:38:59 +02:00
Aaron Boodman 2acf13c702 Fix FOPA (flash of permanode aspect) when clicking permanodes with children.
We were waiting to show the container aspect until the child search session
returned. This meant that we temporarily showed the permanode aspect then
swapped to the container aspect shortly thereafter (because the container
aspect is higher priority).

The right solution to this is to change the search protocol so that clients
can know how many children a permanode has without getting the entire list
of them. But for now, it just so happens that search returns the entire
list, so we can use that instead.

Change-Id: Iaf93f689e965a1dc9ef73d834d46c57026c100cd
2014-09-03 00:30:46 -07:00
Aaron Boodman 6f91febae4 Don't trigger sad piggy when we purposely close sockets.
Change-Id: I2e3b0e894fff180a028fcffbf649ff18f69fd963
2014-09-03 00:07:08 -07:00
Aaron Boodman dbdda81230 Be defensive against getting null updates on web socket.
Change-Id: Id7a7b850ddaa57fe24d8e7ec380fb8881b27def5
2014-09-03 00:00:28 -07:00
Brad Fitzpatrick 37e38ad8b9 schema: add disabled failing ForeachChunk test
For blobkpacked.

Change-Id: Id133032aed21c9beeb17aff08d2500281ed3a5a2
2014-09-02 22:26:11 -07:00
Brad Fitzpatrick 8d7a7d9fcd Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-09-01 16:35:51 -07:00
Brad Fitzpatrick d63bcc04c2 blobpacked: export, document Manifest type. slight tweaks to zip format.
Change-Id: I9fbb005f0ecc6f013575255fae9e62b8397701ee
2014-09-01 16:25:37 -07:00
mpl b6448890dc Merge "test/integration: update TestWebsocketQuery" 2014-09-01 23:11:14 +00:00
mpl 638c660227 test/integration: update TestWebsocketQuery
Because 6517bbc392 introduced automatic
status update message upon connection.

Change-Id: I17383930d97b38d2a4abf7a646aa291bf84d7014
2014-09-02 00:57:23 +02:00
Brad Fitzpatrick d187f9b497 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-09-01 15:17:35 -07:00
Brad Fitzpatrick 59bd675e50 Add pinboard importer to unreleased 0.9 release notes
Change-Id: I29111d96f8f375a3ad3dff9c0086c96adfb0de5e
2014-09-01 15:16:24 -07:00
Brad Fitzpatrick 2368cf1c68 Merge "First version of pinboard importer." 2014-09-01 22:15:35 +00:00
Brad Fitzpatrick 76ce00022a C+A: add Gina White (Individual CLA)
Change-Id: Ie79f12a51bee2dc743b3cb9453d609c1e17fe8cd
2014-09-01 15:15:22 -07:00
Brad Fitzpatrick 5b1a327916 schema: fix test I broken when removing GetChunkOffsets
Rewrite in terms of ForeachChunk instead.

Change-Id: Id018420c47ffe116246c92b06c61a9dc9bbbee62
2014-09-01 15:11:14 -07:00
Gina White e9276c8ac8 First version of pinboard importer.
Change-Id: I2b06b20b2609981b03103b4a26c9c4633abe8473
2014-09-01 13:35:06 -07:00
mpl 9ddc3988aa Merge "serverinit: add support for memory-only indexer. Enable in devcam." 2014-09-01 18:02:53 +00:00
Brad Fitzpatrick eca8ac0a3d Merge "blobserver: add memory storage, mainly for tests." 2014-08-31 03:35:01 +00:00
mpl 5036d6b456 blobserver: add memory storage, mainly for tests.
Mostly code moved from pkg/test/fetcher.go

Also got rid of blob.MemoryStorage

http://camlistore.org/issue/416

Change-Id: I8d17b80942e2fde1c869f71738bfb8a67ed8ab31
2014-08-31 01:17:52 +02:00
Brad Fitzpatrick a3ab848451 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-08-30 11:18:35 -07:00
Brad Fitzpatrick a563ba8d1f blobpacked: populate manifest.json chunks and verify them in tests
This switches to using a fork of archive/zip for now, until we can upstream
changes and wait a release cycle.

Plenty of TODOs remain, but making progress.

Change-Id: I3d7439dc405ba7f64b8d791952f1623a3a7e2024
2014-08-30 11:17:01 -07:00
Brad Fitzpatrick 9e0065fe8d zip: add LastDataOffset field to zip.Writer.
For blobpacked.

Change-Id: I23c325d945b0eff2ec8f47259bc527f1d4f6031f
2014-08-30 11:16:09 -07:00
Brad Fitzpatrick 2fbb26f9bb Checking in an unmodified copy of Go's archive/zip package.
The import is happening at hg 47de34bbcca1 but the last modification
of the package was at:

changeset:   20518:8ab50c14cd95
user:        Brad Fitzpatrick <bradfitz@golang.org>
date:        Mon Aug 04 16:12:55 2014 -0700
summary:     archive/zip: accept bogus trailing zeros in extras

... which will go into Go 1.4.

We don't need that fix in Camlistore, but rather we need additional
data about the state of the *zip.Writer while it's writing.  So this
is a pristine import in preparation for future changes so the changes
will be easy to see in the history, in case we want to move the
changes to the Go standard library (likely). But Go goes into 1.4
freeze in two days and it's the weekend.

Change-Id: I07117b7c5a7f58bd1e8abf6227c5fb0fa8047fa3
2014-08-30 09:55:13 -07:00
Aaron Boodman 95bf8e42a3 Get continual server status updates over websocket.
Also show channel-level websocket errors in piggy menu.

Change-Id: Iad5f00e5578f28f837cd2e88cbdab3f715a9fed1
2014-08-30 09:43:27 -07:00
Aaron Boodman 16085b9a4d Move importerRoot from status.json to discovery.
It seems to make more sense there...

Change-Id: I8450f1cb30ed331cfe15ddb49f55438c5bf9fd5b
2014-08-30 09:09:05 -07:00
Brad Fitzpatrick f66dccec08 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-08-30 00:13:19 -07:00
Brad Fitzpatrick 8d79f59908 blobpacked: first cut at packing, creating zip files with contiguous chunks.
See package docs in blobpacked.go for background & rationale.

Will minimize disk seeks, regardless of the underlying storage
system. I always knew this day would come, but we've been putting it
off. It's finally time to separate out logical (tons of small chunks)
storage from physical (big contiguous layout for how they'll be
accessed).

First 80% is done. Last 80% remains: mostly lots of tests and more
paranoia, and fsck things, for when we inevitably lose indexes. (but
each zip file is self-describing, so the index is redundant).

Change-Id: I13f9aa79c4d9d87b5fdcece0d99d97eba031d608
2014-08-30 00:09:10 -07:00
Aaron Boodman 40390c651a Adjust position of error icon.
Change-Id: I2b86d8b52a3c616ca3f2408973d85e70fb27a2d7
2014-08-29 20:55:52 -07:00
Aaron Boodman ef5e0dfc82 Show server errors in the piggy menu.
When there is a server error, the piggy becomes sad and periodically
shakes his head. When you expand the piggy menu you see one or more
entries explaining why he is sad.

Change-Id: If1e28dbf02c9befaa46b62838cb704220f719bc6
2014-08-29 16:08:19 -07:00
Brad Fitzpatrick f4740e77e8 Aggregate all errors with their URLS in status.json in one place.
Change-Id: If7a540fc3d1a6caa94b5f04d44de3207f15ac4cf
2014-08-29 16:03:44 -07:00
Brad Fitzpatrick 1be26ea3a4 schema: rewrite FileReader.LoadAllChunks, delete GetChunkOffsets
LoadAllChunks is now written in terms of ForeachChunk and is bounded.

GetChunkOffsets was never meant to be exported and was unused elsewhere.
It's now unused entirely.

Change-Id: I0653385fe805fda721e0b45ea0a17d9839c3ba6e
2014-08-29 15:26:36 -07:00
Brad Fitzpatrick 3388e363b9 schema: add FileReader.ForeachChunk
Will be used by the blobpacked storage server.

Also, we can then rewrite much of the legacy crufty methods on
FileReader in terms of this.

Change-Id: Ia371001f1ed502fa1da2dbc4a8e1c29e5f0c9e8c
2014-08-29 14:51:33 -07:00
Brad Fitzpatrick 6517bbc392 Send system status JSON on websockets upon connect and when it changes.
Change-Id: Ic0f211611d63b9d519ee3c5da0dd96c6b77459d5
2014-08-29 14:12:02 -07:00
Brad Fitzpatrick ee479c55a3 blobpacked: prefix meta rows with 'b:'
Change-Id: I203a0b1dac5f137415c586059c5d5e672b6359d8
2014-08-29 11:26:23 -07:00
Tamás Gulácsi 31dcddca41 Let Camlistore build on Solaris
The biggest obstacle is that syscall lacks Mkfifo on Solaris
(I've opened issue 8609 on Go for this).
Now I provide the same implementation for it as there is on Linux:
call Mknod.

github.com/cznic/fileutil needs IsEOF, so I've copied the plan9
(hole punching is stubbed out) implementation.

github.com/lib/pq: I've added solaris as a posix build tag.

Change-Id: Id667c488606a274849dc9b0af76abf1daeab5ad7
2014-08-28 14:26:11 +02:00
mpl bb49185463 serverinit: add support for memory-only indexer. Enable in devcam.
To avoid confusion, the previously named "memoryIndex" configuration
parameter, which actually meant to slurp the index into the memory
corpus, has been renamed to "copyIndexToMemory". "memoryIndex" is now
the parameter to enable a memory-only indexer.

http://camlistore.org/issue/416

Change-Id: I7026e5463fda3a1887a88e0b07d4e3431c60c8b7
2014-08-26 22:46:13 +02:00
mpl d7cc6a39b5 Merge "client: add test with multiple ignores in the config" 2014-08-26 14:02:01 +00:00
Brad Fitzpatrick 3eea26ce6c blobpacked: initial baby steps towards blob packing
Change-Id: Ied4d40513bc08311492b955eb8ad1beb5751df13
2014-08-25 21:18:18 -07:00
Brad Fitzpatrick 31b2a3faef Add new pools package, to aggregate common sync.Pool types.
Change-Id: I69a21844f38ca3452497726d4786bd144d0218f0
2014-08-25 21:17:50 -07:00
Brad Fitzpatrick d4e470ef52 Google cloud storage: add tests, using storagetest
Change-Id: Ib4ba379e6b16ff17d253a93c272ad3ab8b971a66
2014-08-25 19:06:21 -07:00
Brad Fitzpatrick 8fd3277b06 storagetest: slight refactor, add support for options.
Currently unused. Added a Retried option, thinking I'd need it. I might later, or we might
add more options later.

Change-Id: I6b2f63c22685fd964012adbc8dfa06f3f6344313
2014-08-25 19:04:18 -07:00
Brad Fitzpatrick 8515f7b159 Google cloud storage: delete objects in a batch
Change-Id: I12117edef32656b2cb508c95aab008acd2b8591d
2014-08-25 19:03:25 -07:00
Brad Fitzpatrick 5d7f70d355 Google cloud storage: return an error if non-Camlistore object is enumerated over
Otherwise it can mess up the limit= parameter.

Change-Id: I37eb4e15bfba99479618bffc9422d1d9b7ce2f69
2014-08-25 19:03:01 -07:00
Brad Fitzpatrick df6d04cce9 oauth: fix Token.Expired
also sent off as https://codereview.appspot.com/134830043

Change-Id: Id15fd7adb77c87a012112cbfda1d02084bd8e895
2014-08-25 18:58:18 -07:00
Brad Fitzpatrick 77a5f0f375 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-08-25 16:50:42 -07:00
Brad Fitzpatrick 00e3ebcbc5 blobpacked: use SubFetch to fetch a block packed in another one
Currently untested, since the packing code isn't done.

Change-Id: Ida4b587509fa3aff82b1d070f3ef2366936e393e
2014-08-25 16:50:28 -07:00