Or to be more precise, golang.org/x/crypto/acme/autocert
The default behaviour regarding HTTPS certificates changes as such:
1) If the high-level config does not specify a certificate, the
low-level config used to be generated with a default certificate path.
This is no longer the case.
2) If the low-level config does not specify a certificate, we used to
generate self-signed ones at the default path. This is no longer always
the case. We only do this if our hostname does not look like an FQDN,
otherwise we try Let's Encrypt.
3) As a result, if the high-level config does not specify a certificate,
and the hostname looks like an FQDN, it is no longer the case that we'll
generate a self-signed. Let's Encrypt will be tried instead.
To sum up, the new rules are:
If cert/key files are specified, and found, use them.
If cert/key files are specified, not found, and the default values,
generate them (self-signed CA used as a cert), and use them.
If cert/key files are not specified, use Let's Encrypt if we have an
FQDN, otherwise generate self-signed.
Regarding cert caching:
On non-GCE, store the autocert cache dir in
osutil.CamliConfigDir()/letsencrypt.cache
On GCE, store in /tmp/camli-letsencrypt.cache
Fixes#701Fixes#859
Change-Id: Id78a9c6f113fa93e38d690033c10a749d1844ea6
When receiving a file, we were only trying to guess its MIME type
through its contents (pkg/magic). We're now making a better effort at it
by guessing from the filename extension if needed.
Also:
pkg/magic: get rid of all the extra video extensions that are already
covered by mime.TypeByExtension. Because it's redundant and
confusing.
app/publisher, pkg/types/camtypes: also use mime.TypeByExtension as an
extra effort. Especially since a reindex would be necessary to benefit
from the pkg/index change.
There are other places in Camlistore that could use such an effort.
Maybe we should have a camtypes.*FileInfo.MIME() method that tries all
the ways to guess the MIME type of the file?
Change-Id: Ib9a2bc42af77c5394dac578ae415524b5111ad4e
To decide whether a search submitted to the app search proxy is allowed,
we compare its results to the domain blobs, result of the master query,
that we cache when the master query is set.
However, since the results of the master query are liable to change when
new blobs arrive (e.g. a new camliMember is added to the set that is
published), that cache may need to be invalidated. Otherwise, we might
reply with a 403 to search query that is actually allowed.
Therefore, this CL adds a refresh of the cache on two instances:
-When the app handler gets a search query that seems to be forbidden.
Before replying with a 403, we refresh the cache with the master query,
and recheck whether the search query is allowed.
-When the publisher gets a request for a "members" page, or the "file"
page, it preemptively asks the app handler to refresh. Now that a lot of
the client workflow has been moved to javascript/the browser, these
kinds of requests should not happen too often, so it seems a reasonable
place to ask for a refresh. But this might change, so we should of
course be careful not to flood the app handler with refresh requests in
the future.
In any case, the app handler is suppressing the refresh requests, so
that it does not perform refreshes at more that one per minute.
As a smarter approach, we could later imagine a way for the app handler
to be aware of when new blobs get to the blobserver (akin to the blob
hub that the sync handler uses?), so that it only ever refreshes when
needed.
Fixes#851
Change-Id: Idc14cce5018053deac01ec454e5c936ed93e5a05
So far only images were served with their MIME types set properly, so
they would display directly in the browser, instead of being served as a
file download.
Now the same is done for a subset of text types: i.e. text/plain,
text/html, text/xml, and text/json. Aside from the browsing convenience,
the obvious advantage is being able to serve HTML directly, which should
allow us to build other things on top of the publisher.
Also a bit of related refactoring: moving the extension matching to
pkg/magic
Change-Id: Id98065c7c685036a272d1d2e293bfcbca33015ee
Since the app handler should not trim the r.URL.Path of the handler's
prefix, it is now the responsibility of the app to cope with that
prefix.
Fixes#833
Change-Id: Ie1fa9801b26767c3e3b6612498380261e22cdf07
Some of the publisher features have moved from the server-side app to
the client-side app (the browser) thanks to gopherjs. Some of these
features imply doing some search queries against Camlistore, which
requires authentication. The server-side app receives the necessary
credentials on creation, from Camlistore. However, we can't just
communicate them to the client-side (as we do with the web UI) since the
publisher app itself does not require any auth and is supposed to be
exposed to the world.
Therefore, we need to allow some search queries to be done without
authentication.
To this end, the app handler on Camlistore now assumes a new role: it is
also a search proxy for the app. The app sends an unauthenticated search
query to the app handler (instead of directly to the search handler),
and it is the role of the app handler to verify that this query is
allowed for the app, and if yes, to forward the search to the Camlistore's
search handler.
We introduce a new mechanism to filter the search queries in the form of
a master query. Upon startup, the publisher registers, using the new
CAMLI_APP_MASTERQUERY_URL env var, a *search.SearchQuery with the app
handler. The app handler runs that query and caches all the blob refs
included in the response to that query. In the following, all incoming
search queries are run by the app handler, which checks that none of the
response blobs are out of the set defined by the aforementioned cached
blob refs. If that check fails, the search response is not forwarded to
the app/client.
The process can be improved in a subsequent CL (or patchset), with finer
grained domains, i.e. a master search query per published camliPath,
instead of one for the whole app handler.
Change-Id: I00d91ff73e0cbe78744bfae9878077dc3a8521f4
This change allows the publisher to use resources from a SourceRoot
directory, without having to rebuild the publisher binary, instead of
only using embedded resources.
Change-Id: Ife29e3015b8595a33f175a62d98fcf5ffa689134
Using go:generate to call a shell script with some go doc + sed hackery.
we could probably do it better with go/types later if needed.
Change-Id: Ie1cf04d418b8b498f83f7029eb736dbc779feeb5
Done with gopherjs and jquery.
Some build tagging added in pkg/schema and pkg/netutil because
gopherjs does not support cgo (so no os/user).
Issue #798
Change-Id: Ib1e1e94185f75cdf696aa2dd31c57fa9e3af84a1
Run gopherjs to generate trivial javascript code that is used by the
publisher app.
Context:
https://github.com/camlistore/camlistore/issues/798#issuecomment-226902924
github.com/gopherjs vendored in at rev
f3c437955da554f2643747a598b0cc772a749f3f
PLEASE NOTE that this copy of gopherjs has been modified to avoid
depending on fsnotify. Hence the -w flag and the gopherjs serve command
are most likely broken.
Diff for that modification:
https://gist.github.com/mpl/ac9033bb28207401b7cedc3d74e6c096
Dependencies for building gopherjs:
kardianos/osext 29ae4ffbc9a6fe9fb2bc5029050ce6996ea1d3bc
neelance/sourcemap 8c68805598ab8d5637b1a72b5f7d381ea0f39c31
spf13/cobra c678ff029ee250b65714e518f4f5c5cb934955de
spf13/pflag 7f60f83a2c81bc3c3c0d5297f61ddfa68da9d3b7
golang.org/x/crypto/ssh/terminal
c197bcf24cde29d3f73c7b4ac6fd41f4384e8af6
golang.org/x/tools/go/types/typeutil
ac02106e04bdb66a2db0413d931012bea165d7e0
github.com/gopherjs/jquery vendored in at
fbbfc4bbe29a29cb05788b66be44e0ac7f43cac7
jquery vendored in at 2.2.3
Change-Id: I7ff2d8e43e8a963f5ac1d13a2c936f263f7c53fc
These improvements on the server app handler should help writing
and running stand-alone apps.
The two main goals are:
1) "simple" configurations should work automatically; the parameters for
the app are derived from the Listen and BaseURL of the Camlistore
server.
2) More advanced configurations, such as being behind a proxy, should be
easily configurable through the app's Listen, BackendURL, and ApiHost
parameters.
I had worked on them while doing the scanning cabinet app, and I am
backporting them now since we haven't landed the scanning cabinet yet,
and people have been having trouble setting up the publisher.
pkg/app/app_test.go is gone because app.ListenAddress is now dumb. The
hard work is done in pkg/server/app instead.
Fixes#818
Change-Id: Ice2610d6bac611b209cc3a928e67fa6093a41d3e
The describe requests were launching a storm of RLocks which weren't
safe in the presence of goroutines trying to acquire write locks.
Instead, make the corpus locking the responsibility of the caller and
add Lock/Unlock/RLock/RUnlock methods to the index and move locking up
a level.
This also adds a fair bit of context.Context plumbing which was used
in earlier debugging.
Fixescamlistore/camlistore#709
Change-Id: I8d7254d1e1da541f8c080d62f5408aac807fd3b1
The import path was added to the go file that included the package
documentation if one existed. Otherwise, I used what seemed to be the
primary file for the package.
Fixes#689
Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
They were internal packages (under pkg), which we are now moving to
go4.org, so we in turn need to vendor them in now.
Change-Id: I92224f731404d0bd4ca1c57492bed37cb3367ed4
There are several places where we're using a *search.Handler, when we
only want to be able to do query and describe requests. When these
places need to be moved also out of the context of camlistored, (like
when making a server app such as the publisher), the QueryDescriber can
be used, and the adequate implementation (*search.Handler,
*client.Client, or any other) is used depending on the context.
Change-Id: Ie1c5bb6f1bbba32c24a8eb2d1175af95ffcc8335
mpl
