Commit Graph

228 Commits

Author SHA1 Message Date
mpl 21dda2b4ef app/hello: dummy server application (hello world)
Change-Id: I6690b9459325af5a76d1de679d56701eefdd195e
2014-05-08 22:15:01 +02:00
Aaron Boodman 2c675b3a87 Add FontAwesome. This will be used for icons in the UI.
Change-Id: I92fecd44f12cce83f4daee88bce062d9123a48a7
2014-04-05 17:19:29 -07:00
Brad Fitzpatrick bfe51f3ad5 gofmt
Change-Id: Ie6be315d02a439cbfdd870a0b0b7f16cba12a168
2014-03-20 12:29:45 -07:00
mpl b43373c45c Merge "publish: use generic queries" 2014-03-18 22:35:40 +00:00
mpl 9cadbcd5bc publish: use generic queries
Use generic queries instead of specialized index queries. This is a step
towards the publisher app, because its client will have to use generic
queries.

Context: http://camlistore.org/issue/365

Change-Id: I2781a345e024174e3bea8511b6cdc6f342d5a7c1
2014-03-18 17:26:29 +01:00
Brad Fitzpatrick bf2e1fa585 sync: don't replicate a shard's missing blobs until enumeration is complete
Prevents spurious replication of blobs on enumeration error.

Change-Id: I38db7406f6ea52137cb757b32599b18eb7fcf3da
2014-03-17 23:21:53 -07:00
Brad Fitzpatrick bf1ec32e39 sync: add paranoia around checking storage's Enumerate implementation
Didn't find anything, but is useful to keep in, to maybe find bugs in the future
for other storage types.

Change-Id: If0fd37e03578de233be8da95ca45623c5f12156b
2014-03-17 23:05:27 -07:00
Brad Fitzpatrick bf8f4b2423 sync: fix bug in prefix enumeration. could send one extra item.
Depending on timing, could lead to ListMissingDestinationBlobs getting out
of sync and causing a lot of blobs to be replicated that were fine and already
on the server.

Change-Id: I3710e59088f1fe4e526f8f11bc9d1837a727e512
2014-03-17 23:02:01 -07:00
Brad Fitzpatrick bf88f5f06c sync: missing handler return + wording change
Change-Id: Iebd5344a0a0e1418cb48c92b91858ebf2f9486c8
2014-03-17 23:00:18 -07:00
Brad Fitzpatrick bfbe29fc4a Linkify status HTML, and link to sync handler status pages too.
Change-Id: I18467511fd34f3bc232f38c2b91075c616c220a1
2014-03-17 21:52:28 -07:00
Brad Fitzpatrick bf28dd4488 More status handler HTML+JSON, more sync status.
Change-Id: I0381853191d5b871af649d102b976e592def791f
2014-03-16 20:14:57 -07:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
Brad Fitzpatrick bfa30b3013 sync: add a button to start a full validation, even when disabled
Change-Id: I229fa70843bb4b9d206788bf150c436c56c82cb7
2014-03-07 17:32:59 -08:00
Brad Fitzpatrick bfa8efc1b9 sync: background validation of src-vs-destination in sync handler
Currently controlled by an environment variable, but will become
a config option + on-demand button in UI in later commits.

Change-Id: I25fa878c9b30cdd713e2859585210eb722092f7b
2014-03-07 10:57:41 -08:00
Brad Fitzpatrick abca567581 sync: work on full-validation-on-startup. 90% done, disabled.
Change-Id: I5bf062f3b22c2cc41329ff6b23f11198ae543c0f
2014-03-06 16:52:29 -08:00
Brad Fitzpatrick 58ac8b5469 sync: restore key part accidentally removed prior to earlier submit
Would cause accounting errors before in the face of duplicate uploads.

Change-Id: Ie7c49da1adaf2b9c98ef1015f875a4df8b66729f
2014-03-06 13:23:27 -08:00
Brad Fitzpatrick c1892b5ae5 Rewritten sync handler.
Fix deadlock, much better status page, show per-blob status & errors,
clear errors when they've resolved themselves, fix known data race.

Change-Id: I968de0de4f308ff0a410adceb181a0712800d401
2014-03-05 08:51:22 -08:00
Brad Fitzpatrick 07178b877a Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-03-04 14:18:27 -08:00
Brad Fitzpatrick 4731e3abec sync: rename lk to more conventional mu
Change-Id: I59f12da9462e4768b59eb135c10f356916191ea7
2014-03-04 13:57:33 -08:00
mpl 08923e1c00 camtool sync: warn when src and dest have different identities
http://camlistore.org/issue/281

Change-Id: I2e2d595534b14476293a031b88e312de3e0b3f4a
2014-03-01 23:47:49 +01:00
Bill Thiede 58b6497b14 ui: Add QR code for configuring mobile clients.
Creates new page @ /ui/mobile.html for generating a QR code that
embeds configuration data for setting up mobile clients.

Imports code.google.com/p/rsc/{gf256,qr} version:

parent: 149:a3cbaf2339c4 tip

This addresses the server and web components of
https://camlistore.org/issue/372

Change-Id: If9239dd138707080afd7dedb5770d04893cbf0ba
2014-02-21 19:58:54 -08:00
Andrew Gerrand 0a1fe281ca remove redundant return statement
Change-Id: I09f61e61c0aa2c0ecaec04eb7541374a3265879e
2014-02-11 10:50:30 +11:00
Tamás Gulácsi 97520583b8 Use 'uint32' instead of 'int64' for blob sizes everywhere.
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.

Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
2014-02-08 17:58:12 +01:00
Bill Thiede 5c884e52ca diskpacked: keep fd open to every pack file.
Bytes read/writen per pack file, as well as per configured diskpacked
configuration are now available as expvars.

Also add reader stat helpers to pkg/types and updated the original
user in server/image.go

Change-Id: Ifc9d76c57aab329d4b947e9a4ef9eac008bc608d
2014-01-20 10:50:42 -08:00
Aaron Boodman 8387acb5ce Merge "Trial fix for infinite piggie bug." 2014-01-18 22:45:11 +00:00
Brad Fitzpatrick 0c1a271d38 When caching resized images, don't use a file schema blob and lots of little
chunks when the size of the resized image is under 16 MB.

Also add some more expvars counters.

Change-Id: I769247edf3819ca95669bdf9a1c8d3876c54095b
2014-01-18 10:12:29 -08:00
Brad Fitzpatrick 1cbd7365b4 Fix thumbnail cache filename fmt pattern bug.
I noticed thumbnail files named like:

{"camliVersion": 1,
  "camliType": "file",
  "fileName": "scaled:sha1-da55749abffc24004177c32e8ea00687044910b8:2000x256:tv%!d(string=2)",
  "parts": [
    {
      "blobRef": "sha1-82810ad435f5354e4d8ad358a011bc221053b1bb",
      "size": 11239
    }
  ]
}

Change-Id: I3bc729917c226622e70fb662e0413eeacfda6d93
2014-01-18 09:18:13 -08:00
Aaron Boodman 5b16c9202b Trial fix for infinite piggie bug.
Change-Id: I589c1c6dd8d68552376f274c9bcb091fc6a289f3
2014-01-17 22:05:57 -08:00
Brad Fitzpatrick 48e67cab16 devcam: add --publish=false and --mini option to start a lite server with no blobs
Change-Id: Idcde77f72126800d610c632940552f04c396634d
2014-01-13 20:11:43 -08:00
Brad Fitzpatrick c63fedf263 Fix sync handler spinning on failure.
Change-Id: I4bdd568103cc6d50e208a2f3bc7523fa3abad1c8
2014-01-13 20:10:45 -08:00
Aaron Boodman d4fbfe7180 Implement generalized navigation handling, use to implement escape handling in detail view.
Change-Id: Iba5359e9d17fcaa200a9cbb586ac9d6461c179aa
2014-01-10 15:50:05 -08:00
Brad Fitzpatrick 34ed712a4a Move defaultMaxResizeBytes to pkg/constants
Change-Id: Ibb5f82b6f33a4417649f1538a996f1fe7eac885e
2014-01-06 08:08:47 -08:00
Bill Thiede 56510593ff pkg/server: set thumbnail RAM limit per-handler.
Move from one pool for all thumbnail resizing to per-ImageHandler
instace.

Uses the value of "maxResizeBytes" if specified in the "handlerArgs"
section of a "publish" or "ui" handler's low-level config.

If we end up wanting to share pools, i.e. if all published
handlers should share a pool, we'll need to do a bit more work.

Implements suggestion from https://camlistore.org/r/1803 review.
Further improves on https://camlistore.org/issue/316.

Change-Id: Ia93fad119b546064173ac3e2c7f0ab8509744628
2014-01-05 17:49:59 -08:00
Bill Thiede 1e447b0f69 pkg/server: RAM based concurrency limiter.
The RAM limit is based on the expected amount of allocations not the
measured allocation.

Issue https://camlistore.org/issue/316

Change-Id: I3ccff6ddf880f3f9a90f6c27f15f9662fad02712
2014-01-04 23:05:18 -08:00
Brad Fitzpatrick abfd09567f Merge "pkg/server: stream images not slurp to RAM." 2014-01-04 19:43:24 +00:00
Bill Thiede 862dd6229a pkg/server: properly set format with CR2 images.
Issue http://camlistore.org/issue/289 is still present with CR2
files; this corrects that.

Change-Id: I578447778f5868762e16b6ee15d352271eb9c7e3
2014-01-03 21:25:22 -08:00
Bill Thiede 79cd061838 pkg/server: stream images not slurp to RAM.
More improvement on https://camlistore.org/issue/237

Change-Id: I50076f2fc84de796026029cdab46a897354d0174
2014-01-03 20:44:09 -08:00
Aaron Boodman 07cce4a3e6 Integrate the detail view into index.html.
The detail view now loads with only two HTTP requests: one for the
piggy, and one for the big image. No other resources are loaded,
even the blob descriptions, because they are cached by SearchSession,
which is shared between the search ui and the detail ui.

Likewise, when you go back to the search results page, no additional
requests are done.

Also:
- When you press back on detail page, scroll position is now usually
  restored (if you refresh between index and detail, it still won't
  restore).
- Detail page live updates! It came for free with using SearchSession.

Change-Id: I58ada7bdd30afc2644860fbda6167fc5ce865b0d
2014-01-01 23:13:23 -08:00
Aaron Boodman e24ca1b78d Merge "Increase pigginess in new detail UI." 2014-01-01 01:52:47 +00:00
Aaron Boodman 2d70e6794f Increase pigginess in new detail UI.
Change-Id: I4eb1acf7e6cd114edf7c5f66fcb4fb6ff8842bdb
2013-12-31 17:49:42 -08:00
mpl 914176593a pkg/server: actually use the image format
http://camlistore.org/issue/289

Change-Id: Id1f344e567349ed8d540357bfb397252ac131456
2013-12-30 23:47:11 +01:00
Aaron Boodman e4daf4c4b5 Add first bit of a React-based detail page.
When complete, this will be part of index.html, but for now I am developing it separately.

Change-Id: Ibda6fb9f396f64d771919934907dbe4e5cd9d030
2013-12-29 23:50:10 -08:00
Aaron Boodman 4a3b87fe24 Serve React from both devcam server and camlistored.
In camlistored, only react.min.js is served. In devcam all files in
the read directory are served.

Change-Id: I0cb8a4c328e550b5303edef4be7571d39d123340
2013-12-28 21:49:41 -08:00
Bill Thiede 02201753ed server/ui: set Content-Type when serving .svg.
Fixes warning 'Resource interpreted as Image but transferred with MIME
type text/xml:' on Chrome, and more importantly returns the previously
missing icons to my camlistore instances.  Everyone else must be
hitting cache, or something is weird with my setup.  I see missing
icons on Safari / Chome on Mac, and Firefox / Chrome on Linux.

Change-Id: I4809d8c7948a49fe80fdf1a97ac98b2161ba0307
2013-12-27 19:12:37 -08:00
Brad Fitzpatrick 9fd329df80 Merge "cmd/camtool: (re)index command" 2013-12-26 17:29:03 +00:00
mpl 2d85e017ff cmd/camtool: (re)index command
http://camlistore.org/issue/193

Change-Id: I498f92bdc153f44dc84d4b47f03c47a8e7b54ad9
2013-12-26 18:23:15 +01:00
Brad Fitzpatrick 130be113fd Merge "Include camliPath attributes as members from DescribedBlob.Members" 2013-12-22 01:38:45 +00:00
Dustin Sallings 2d15de3933 Validate PublishHandler.ViewerIsOwner with auth.Allowed
Change-Id: Ic12926696a6f082a794577e395ff277bbb34daec
2013-12-19 14:51:12 -08:00
Ulf Holm Nielsen daa4379bd3 Include camliPath attributes as members from DescribedBlob.Members
Change-Id: I30272acd1cae805cf9477f99611275d32231b52a
2013-12-18 13:53:00 +01:00
Brad Fitzpatrick 30c7d8859a thumbnails: add a cache-busting URL component, support ETag
And quiet noisy logging on normal write failures.

We can now stress test the thumbnail generation by setting
CAMLI_DISABLE_THUMB_CACHE=1 which will make all the thumbnails in the
browsers be unique, and not write them to cache on the server.

Then, when we're happy with the thumbnails, we just increment the
thumbnailVersion string and that busts all the browser- and
server-side caches.

Change-Id: I3cda8e85ab8b1b0b2c9113f6dff613dfbf736028
2013-12-16 20:27:49 -08:00