rev b9bcb1da229a59cc1e1d168401662cb6450aae08
Also added pkg/misc/fakefsnotify just so we only have to do:
- "github.com/fsnotify/fsnotify"
+ fsnotify "camlistore.org/pkg/misc/fakefsnotify"
in vendor/github.com/gopherjs/gopherjs/build/build.go
in order to avoid vendoring fsnotify, without having to grossly
patch the gopherjs code as we previously did.
Change-Id: Ic106a6afa1117d2061415355c236433f835895f1
To decide whether a search submitted to the app search proxy is allowed,
we compare its results to the domain blobs, result of the master query,
that we cache when the master query is set.
However, since the results of the master query are liable to change when
new blobs arrive (e.g. a new camliMember is added to the set that is
published), that cache may need to be invalidated. Otherwise, we might
reply with a 403 to search query that is actually allowed.
Therefore, this CL adds a refresh of the cache on two instances:
-When the app handler gets a search query that seems to be forbidden.
Before replying with a 403, we refresh the cache with the master query,
and recheck whether the search query is allowed.
-When the publisher gets a request for a "members" page, or the "file"
page, it preemptively asks the app handler to refresh. Now that a lot of
the client workflow has been moved to javascript/the browser, these
kinds of requests should not happen too often, so it seems a reasonable
place to ask for a refresh. But this might change, so we should of
course be careful not to flood the app handler with refresh requests in
the future.
In any case, the app handler is suppressing the refresh requests, so
that it does not perform refreshes at more that one per minute.
As a smarter approach, we could later imagine a way for the app handler
to be aware of when new blobs get to the blobserver (akin to the blob
hub that the sync handler uses?), so that it only ever refreshes when
needed.
Fixes#851
Change-Id: Idc14cce5018053deac01ec454e5c936ed93e5a05
Some of the publisher features have moved from the server-side app to
the client-side app (the browser) thanks to gopherjs. Some of these
features imply doing some search queries against Camlistore, which
requires authentication. The server-side app receives the necessary
credentials on creation, from Camlistore. However, we can't just
communicate them to the client-side (as we do with the web UI) since the
publisher app itself does not require any auth and is supposed to be
exposed to the world.
Therefore, we need to allow some search queries to be done without
authentication.
To this end, the app handler on Camlistore now assumes a new role: it is
also a search proxy for the app. The app sends an unauthenticated search
query to the app handler (instead of directly to the search handler),
and it is the role of the app handler to verify that this query is
allowed for the app, and if yes, to forward the search to the Camlistore's
search handler.
We introduce a new mechanism to filter the search queries in the form of
a master query. Upon startup, the publisher registers, using the new
CAMLI_APP_MASTERQUERY_URL env var, a *search.SearchQuery with the app
handler. The app handler runs that query and caches all the blob refs
included in the response to that query. In the following, all incoming
search queries are run by the app handler, which checks that none of the
response blobs are out of the set defined by the aforementioned cached
blob refs. If that check fails, the search response is not forwarded to
the app/client.
The process can be improved in a subsequent CL (or patchset), with finer
grained domains, i.e. a master search query per published camliPath,
instead of one for the whole app handler.
Change-Id: I00d91ff73e0cbe78744bfae9878077dc3a8521f4
These improvements on the server app handler should help writing
and running stand-alone apps.
The two main goals are:
1) "simple" configurations should work automatically; the parameters for
the app are derived from the Listen and BaseURL of the Camlistore
server.
2) More advanced configurations, such as being behind a proxy, should be
easily configurable through the app's Listen, BackendURL, and ApiHost
parameters.
I had worked on them while doing the scanning cabinet app, and I am
backporting them now since we haven't landed the scanning cabinet yet,
and people have been having trouble setting up the publisher.
pkg/app/app_test.go is gone because app.ListenAddress is now dumb. The
hard work is done in pkg/server/app instead.
Fixes#818
Change-Id: Ice2610d6bac611b209cc3a928e67fa6093a41d3e
It was just routing links from other pages to
https://camlistore.org/code#contributing
Let this be handled by a redirect
/doc/contributing → /doc/code#contributing
instead.
/doc/contributing.md is allowed to 404. It has only been valid since
April and was only linked from release notes, so there are probably no
external links.
Change-Id: I04809760cbdb2cf37e98d36f545d2952f96f3113
Current version of doc/release/monthly.html generated with
go run ./misc/monthly.go -rev cdadfd650b
Change-Id: Icdebf60cb64dfcc12766976572f3a34eaeb135f3
interestingly, I also discovered that GitHub will properly resolve links
that are relative to the root path (e.g. "/doc/foo")... it properly
rewrites these links to reference the named file relative to the
repository root.
also add a /doc/contributing.md file. This will never be served from
the website, since we have a redirect handler in place, but is added so
that we don't have broken links when browsing the docs on GitHub.
Change-Id: I47517a5f6303065df02ac11771dcee56cd06a4b7
GitHub treats the CONTRIBUTING file special, in that it is referenced
any time someone files a new issue or starts to create a pull request.
Also make minor updates to file to use markdown and current hacking
instructions.
Change-Id: I68f0d7a69397851ee6c65671ae692cbfb2b93946
this is now needed to have camweb wrap the file in the site template
rather than just serving it raw.
Change-Id: I1ac20030dd7bf878429fd70386e2e2814c66822c
The help page in the camlistore binary has a direct link for "Search bar
predicates" that links to this search-ui.txt on gerrit. Restore this
file so that these links will still resolve, but with a pointer to the
new docs on camlistore.org.
Also update the help link for future builds of camlistore to point to
the correct URL.
Change-Id: Ib81ff820869d86e6e6f664e8b52c5f20d4eed4d3
It looks better than the half-organization we had so far, and it will
actually make it easier for editorial changes to come. Also keeps us
more honest about doing it sooner rather than later.
Change-Id: I6303325f6d75389c7e8ca908f3740cf92af3836a
Update links to reference the website URLs rather than gerrit. There is
some more cleanup that could be done with the docs (for example, using
tables for the request and response keys), but this is focused on just
basic rendering.
Git is treating these as file deletions and additions... apparently
there were too many changes to treat them as renames, though that's what
they really are.
Refs #720
Change-Id: Ibca17b78e1412c049843236bfbc8f2a9e368bb23
Mathieu Lonjaret