To decide whether a search submitted to the app search proxy is allowed,
we compare its results to the domain blobs, result of the master query,
that we cache when the master query is set.
However, since the results of the master query are liable to change when
new blobs arrive (e.g. a new camliMember is added to the set that is
published), that cache may need to be invalidated. Otherwise, we might
reply with a 403 to search query that is actually allowed.
Therefore, this CL adds a refresh of the cache on two instances:
-When the app handler gets a search query that seems to be forbidden.
Before replying with a 403, we refresh the cache with the master query,
and recheck whether the search query is allowed.
-When the publisher gets a request for a "members" page, or the "file"
page, it preemptively asks the app handler to refresh. Now that a lot of
the client workflow has been moved to javascript/the browser, these
kinds of requests should not happen too often, so it seems a reasonable
place to ask for a refresh. But this might change, so we should of
course be careful not to flood the app handler with refresh requests in
the future.
In any case, the app handler is suppressing the refresh requests, so
that it does not perform refreshes at more that one per minute.
As a smarter approach, we could later imagine a way for the app handler
to be aware of when new blobs get to the blobserver (akin to the blob
hub that the sync handler uses?), so that it only ever refreshes when
needed.
Fixes#851
Change-Id: Idc14cce5018053deac01ec454e5c936ed93e5a05
Some of the publisher features have moved from the server-side app to
the client-side app (the browser) thanks to gopherjs. Some of these
features imply doing some search queries against Camlistore, which
requires authentication. The server-side app receives the necessary
credentials on creation, from Camlistore. However, we can't just
communicate them to the client-side (as we do with the web UI) since the
publisher app itself does not require any auth and is supposed to be
exposed to the world.
Therefore, we need to allow some search queries to be done without
authentication.
To this end, the app handler on Camlistore now assumes a new role: it is
also a search proxy for the app. The app sends an unauthenticated search
query to the app handler (instead of directly to the search handler),
and it is the role of the app handler to verify that this query is
allowed for the app, and if yes, to forward the search to the Camlistore's
search handler.
We introduce a new mechanism to filter the search queries in the form of
a master query. Upon startup, the publisher registers, using the new
CAMLI_APP_MASTERQUERY_URL env var, a *search.SearchQuery with the app
handler. The app handler runs that query and caches all the blob refs
included in the response to that query. In the following, all incoming
search queries are run by the app handler, which checks that none of the
response blobs are out of the set defined by the aforementioned cached
blob refs. If that check fails, the search response is not forwarded to
the app/client.
The process can be improved in a subsequent CL (or patchset), with finer
grained domains, i.e. a master search query per published camliPath,
instead of one for the whole app handler.
Change-Id: I00d91ff73e0cbe78744bfae9878077dc3a8521f4
These improvements on the server app handler should help writing
and running stand-alone apps.
The two main goals are:
1) "simple" configurations should work automatically; the parameters for
the app are derived from the Listen and BaseURL of the Camlistore
server.
2) More advanced configurations, such as being behind a proxy, should be
easily configurable through the app's Listen, BackendURL, and ApiHost
parameters.
I had worked on them while doing the scanning cabinet app, and I am
backporting them now since we haven't landed the scanning cabinet yet,
and people have been having trouble setting up the publisher.
pkg/app/app_test.go is gone because app.ListenAddress is now dumb. The
hard work is done in pkg/server/app instead.
Fixes#818
Change-Id: Ice2610d6bac611b209cc3a928e67fa6093a41d3e
It was just routing links from other pages to
https://camlistore.org/code#contributing
Let this be handled by a redirect
/doc/contributing → /doc/code#contributing
instead.
/doc/contributing.md is allowed to 404. It has only been valid since
April and was only linked from release notes, so there are probably no
external links.
Change-Id: I04809760cbdb2cf37e98d36f545d2952f96f3113
Current version of doc/release/monthly.html generated with
go run ./misc/monthly.go -rev cdadfd650b
Change-Id: Icdebf60cb64dfcc12766976572f3a34eaeb135f3
interestingly, I also discovered that GitHub will properly resolve links
that are relative to the root path (e.g. "/doc/foo")... it properly
rewrites these links to reference the named file relative to the
repository root.
also add a /doc/contributing.md file. This will never be served from
the website, since we have a redirect handler in place, but is added so
that we don't have broken links when browsing the docs on GitHub.
Change-Id: I47517a5f6303065df02ac11771dcee56cd06a4b7
GitHub treats the CONTRIBUTING file special, in that it is referenced
any time someone files a new issue or starts to create a pull request.
Also make minor updates to file to use markdown and current hacking
instructions.
Change-Id: I68f0d7a69397851ee6c65671ae692cbfb2b93946
this is now needed to have camweb wrap the file in the site template
rather than just serving it raw.
Change-Id: I1ac20030dd7bf878429fd70386e2e2814c66822c
The help page in the camlistore binary has a direct link for "Search bar
predicates" that links to this search-ui.txt on gerrit. Restore this
file so that these links will still resolve, but with a pointer to the
new docs on camlistore.org.
Also update the help link for future builds of camlistore to point to
the correct URL.
Change-Id: Ib81ff820869d86e6e6f664e8b52c5f20d4eed4d3
It looks better than the half-organization we had so far, and it will
actually make it easier for editorial changes to come. Also keeps us
more honest about doing it sooner rather than later.
Change-Id: I6303325f6d75389c7e8ca908f3740cf92af3836a
Update links to reference the website URLs rather than gerrit. There is
some more cleanup that could be done with the docs (for example, using
tables for the request and response keys), but this is focused on just
basic rendering.
Git is treating these as file deletions and additions... apparently
there were too many changes to treat them as renames, though that's what
they really are.
Refs #720
Change-Id: Ibca17b78e1412c049843236bfbc8f2a9e368bb23
Program to create the zip file containing the Camlistore source for a
release.
Issues #690 and #688
Change-Id: Ibbd3596a6c76cf70c2c29a415c3098bedf48b17f
- Available at '/help/'
- Loads and parses server config upon init and generates the necessary client config for access
- Links to 'Downloadable Tools' are not functional (task is not yet completed)
- A link was added to the default server page as well as a 'Help' option to the Piggy menu
Issue: #536
Change-Id: I10629de034c908ebc76f6feacbbfca1df7bf1c2b
This makes downloads and the thumbnailer's reading of full images use
blobpacked's WholeRefFetcher fast path, when available. The end result
is many fewer blob fetches.
Updates #532
Change-Id: I3bfd1cb0f04f68f81b80a63378c7b365bd73c974
This reverts commit 2d67328243.
I decided to do this only with heuristics, since we have to maintain
backward compat with data before this camliNodeType existed.
Change-Id: Ieba5dd1f8a4c8f5aedc1477c4fab1ffebcc2878c
To avoid confusion, the previously named "memoryIndex" configuration
parameter, which actually meant to slurp the index into the memory
corpus, has been renamed to "copyIndexToMemory". "memoryIndex" is now
the parameter to enable a memory-only indexer.
http://camlistore.org/issue/416
Change-Id: I7026e5463fda3a1887a88e0b07d4e3431c60c8b7
Problem: make.go creates an isolated temp gopath ./tmp/build-gopath. The
integration tests make use of that gopath (by running make.go) to build
the tools, and run the test world in it. Similarly, devcam test uses
make.go to setup that temp gopath, and runs the tests from the source
files in that gopath. Consequently, when the integration tests are run
through devcam test, even though they're run from the temp gopath, they
would use the make.go in it, which would create a nested temp gopath
(CAMLIROOT/tmp/build-gopath/src/camlistore.org/tmp/build-gopath) in
which to run the tests.
This patch addresses this issue by creating a new flag (-envGoPath), and
the corresponding env var (CAMLI_MAKE_USEGOPATH), which tells make.go
not to create a new temporary gopath (and hence not to mirror any
files), and to rely on the already set GOPATH env var instead.
Also refactored make.go a bit, and added a couple options and doc to
devcam test.
Change-Id: Ia8a5d7a31e6e317f05218d9e18fb886001cd19cb
The new package pkg/images/fastjpeg uses djpeg(1) to quickly
down-sample images at load time as described here:
http://jpegclub.org/djpeg/
Add benchmark comparing std lib jpeg.Decode and fastjpeg.DecodeDownsample
with a factor of 1, 2, 4, and 8.
Benchmark resizes a 4000x4000 pixel JPEG to 128x128.
Refactor in pkg/images to make images.Decode a little more readable.
Change-Id: I571db1f3c3068f99da4a01ec84fd246ef098a18c
This patch:
1. Introduces a JSON schema for unix domain sockets (socket files).
2. Adds support for the new schema to pkg/schema.
3. Adds support for uploading and restoring sockets to camput and
camget respectively.
Depends on CL 2706 (https://camlistore-review.googlesource.com/2706/)
Change-Id: Ib0d6e9f9ebd348f55efee5a08cd774f3969d0760
This patch:
1. Introduces a JSON schema for FIFOs (named pipes).
2. Adds support for uploading and restoring FIFOs to camput and camget
respectively.
It includes tests for the basic schema support in pkg/schema and
integration tests for camput and camget.
This CL does not add FIFO support to the HTTP server and fs (cammount
support) code. Those will be addressed in a subsequent patch.
Change-Id: Iab926073c36e653d25e693a559178bea9bf19b9a
And remove all mentions of CAMLI_CLIENT_IDENTITY because there isn't any
need for the both of them.
http://camlistore.org/issue/412
Change-Id: I0d6b8cff0f23738231de98171a23550d82f05fe9
The handler was adding 1 to things to see if it was at the end, but that was causing
HTTP requests to Amazon like: limit 1000, limit 1, limit 1000, limit 1, etc.
This makes it twice as fast.
Change-Id: Ibb7e3f6ae7229a21c87817c7438324d36e7b491a
Testing clients should now use CAMLI_CLIENT_IDENTITY,
CAMLI_DISABLE_CLIENT_CONFIG_FILE (both new) and existing
CAMLI_SECRET_RING, set using the new test.World accessors.
Change-Id: I9fc0571b008f033674466bbc98129ae1919a0640
For demos, when we want to show using camput/camget but select the devcam server
instead of our production servers.
Change-Id: Ie2adfd4f80df3119a705b51bb9736c3b66612d1c
No more dynamic upload URL, which trips up half our new users behind
reverse proxies when the camlistored process doesn't know its
forward-facing URL.
The original camlistore stat + upload protocol was influenced by App
Engine's limitations at the time, and some of our indecision about
where the Camlistore design is going. We understand the Camlistore
design now, and App Engine's former limitations are gone. Time to
clean things up.
More REST-y now too.
See http://camlistore.org/issue/123
Change-Id: I92c6552f830b925cef379c204a982a2213bf2f4b
This includes read-write support for extended attributes for mutable
files, and enough support on all other filesystem paths to prevent OS X
from falling back to AppleDouble files.
Change-Id: I2b275591f3bc69c25e1e9c8d59a7fa4cf0bb97ff
1) pkg/search: documented that deletions times do not
qualify as modtimes
2) pkg/index: got rid of DeletedAt, and keyDeletes
http://camlistore.org/issue/191
Change-Id: I39578913345454d36af4599e29e7053f46577846
mpl