CAMLI_GOPHERJS_GOROOT has effect only if verifyGopherjsGoroot is called,
which only happened if go version was "devel".
This CL runs the verifyGopherjsGoroot check for go1.8rc and go1.8beta
versions too.
Change-Id: I7c80976d46ce8d69e394d226c685c8554176ea0f
In other words, when on the camliRoot node, there is an attribute such
as:
camliPath:foo = sha1-foo
Where sha1-foo is a permanode that is not a set (i.e. it does not have
camliMembers or camliPaths), and typically is a permanode with some
camliContent.
Change-Id: Ib827130bb2456c4c0d7bfb40e40a425515ee1bde
Instead of reporting "404 not found" make it clear
publisher is working, but needs an explicit path.
Change-Id: Ic686b82335ba36e0649dd563831b1221a8579e0d
pkg/osutil started depending on pkg/env in
c55c8602d3 , when we added
DefaultLetsEncryptCache(), because it behaves differently whether we're
on GCE or not.
This in turn made pkg/osutil depend on
cloud.google.com/go/compute/metadata and a couple of others. This, in
itself, was not so bad since the main user of pkg/osutil,
server/camlistored already depends on all these same dependencies
anyway.
However, pkg/types/camtypes depends on pkg/osutil (although maybe it
shouldn't) because it calls osutil.UserClientConfigPath() for an error
message.
And finally, in app/publisher/js - which is some Go code meant as input
for gopherjs, so that it gets compiled into javascript that we include
in the publisher web page - we depend on pkg/types/camtypes for some
search query results.
So as of c55c8602d3, we "leaked" a couple
of additional and unnecessary dependencies into the gopherjs generated
code. That made the minified output unnoticeably grow, and apparently
had no other adverse effect, so we didn't notice.
That is, until we landed b0b6a0a89e, which
updated a ton of vendored dependencies, most of them in
cloud.google.com/go and google.golang.org/api. And interestingly, the
update made cloud.google.com/go/compute/metadata depend on a lot of
these other dependencies. This had two noticeable consequences:
1) The (unminified) generated js code grew from 6.1M to 8.2M, which in
itself is super bad already.
2) The js code actually stopped working. For some reason (that I can't
explain, and will probably leave to the GopherJS devs if they're
interested), the members/functions that we export to the javascript
namespace with gopherjs (typically with a
js.Global.Set("ExportedNameForFoo", FooFunc)), are not visible anymore,
which of course breaks everything.
Therefore, this CL fixes this issue by simply doing what we knew we
should have done from the start:
we now let pkg/osutil/gce (which already depends on pkg/env) itself
register a LetsEncryptCacheFunc into pkg/osutil, which removes the need
for pkg/osutil to depend on pkg/env.
Change-Id: Ie8f33e9cb873919dd1728068dd8a5d0170282f88
When reindexing on a (My)SQL based sorted.KeyValue, we should recreate
the database schema from scratch, which means dropping the tables.
However, index.Reindex just calls Wipe on the newly created
sorted.KeyValue, which only deletes the rows, and does not drop the
tables.
Therefore, this CL changes the implementation of Wipe in the MySQL case,
so that it takes care of dropping the tables, and doing everything that
needs to be done afterwards to set up the sorted.KeyValue.
In addition, with the introduction of the sorted.NeedWipeError, we detect
upon initialization of a sorted.KeyValue if it failed because it needed
a schema update. If that is the case, and we're in reindex mode, we can
fix the sorted.KeyValue with a Wipe and carry on.
Finally, we introduce the new sorted.NewKeyValueMaybeWipe function that
automatically wipes a KeyValue when a NeedWipeError was returned upon
its creation.
Next, do the same with other sorted SQLs.
Fixes#806
Change-Id: I2032781cbf453a364880bd3e2e8b3c09aac7aed9
This CL changes the GCE launcher to work with the new features of
camlistored: i.e. that it can automatically get a hostname in
camlistore.net, and that it can get an HTTPS certificate from Let's
Encrypt, for said hostname.
In order for the user to easily (without having to look at the logs)
know what their hostname is, camlistored stores it as the
"camlistore-hostname" key in the custom metadata of the GCE instance.
The deployer can then query for that key, to report the hostname on the
instance creation success page.
Change-Id: Iaaef2d51f34fa5e1e0ee90097919abab7ee72a12
In order to use HTTPS, one must have a certificate, and one must have a
domain name for which the certificate is valid.
The first part is solved by the use of Let's Encrypt. For the second
part, we want to provide to any Camlistore instance a name such as
<gpgKeyId>.camlistore.net, where gpgKeyId is the fingerprint of its GPG
key. The DNS for camlistore.net agrees to add a record for that name if
and only if the Camlistore instance can prove it owns the GPG key, as
well as the IP address bound to that name in the DNS record.
A protocol such as the above is already implemented in pkg/gpgchallenge.
This CL:
- uses the client-side of the gpgchallenge protocol in camlistored, so
that it can claim a hostname in camlistore.net on startup (and then use
that hostname when requesting a certificate from Let's Encrypt).
- adds the configuration parameter "CamliNetIP" for the high-level
config. This parameter specifies the IP address that camlistored will
supply during the gpgpchallenge, so it can prove to the DNS server that
we own this address.
Fixes#722
Change-Id: I6bf4ec149b6dffd0ae93a6fa7bf208b2e8a05445
Requires updating vendor/cloud.google.com/go/storage because of the
BucketHandle.List removal, which in turns triggers a bunch of other
updates or additions.
cloud.google.com/go/internal/optional
cloud.google.com/go/storage
at rev 641b1469e744485a8e6235bffc3b7bf366758a85
github.com/googleapis/gax-go
at rev da06d194a00e19ce00d9011a13931c3f6f6887c7
google.golang.org/api/*
at rev f7e067168bd43883b2156f16b6c2fd91beec76e2
Fixes#884
Change-Id: I7e27a24fa2ffbb747e66bde0d10dabaad1ac60d6
The Client used to start its own listener and http server, so it could
receive the Server's challenge. However, that design does not work when
the Client must be used in an application that is already an http(s)
server.
Therefore, this change adds a Handler method to the Client, that returns
the Client's handler, as well as the pattern it should registered for
with an HTTPS server. This means, it is now the responsibility of the
caller to setup the listener for the Client before the Challenge can be
started.
Change-Id: I160e21c470322f7acad209ac28a15eaeed36c2c4
Store records in Google Datastore in production (on GCE), with an LRU
cache in front.
Keep dev mode that stores in sorted mem key value though.
Change-Id: I17b088b56a68a019f4e253b60c6bd42395a64984
-check claimed IP is a global unicast one
-set timeout on requests, so they don't make us create long-lived
connections for nothing
-prevent redirects
-disable keepalive connections
Change-Id: Ibdbdd9916ddc93eb58505b04b4f21de60e37c745
As the requests to the publisher are proxied through Camlistore's app
handler, there's no point in the publisher having its own autocert
Manager to request a certificate. Therefore, the publisher reuses
(readonly) camlistored's autocert CacheDir to get its certificate.
It follows that, for now, Let's Encrypt only works for the publisher if
it is running on the same host as camlistored (or more precisely, if they
share the same filesystem).
Fixes#458
Change-Id: Icf3be2913f85f9ec6f94b831ad58e1949b4d6961
A demo client and server are included, to help with testing.
TODO(mpl): restrict IP space
Issue #722
Change-Id: I6952069e8885c06adcbb8fca103af9d83f2fa9a7
Previously commands like camput, when invoked with simply just -h or
--help, they would only print the global flags defined on
flag.CommandLine of the standard library. In order to reach the richer
help text including mode information, you had to specify some mode, even
a fake one.
This change just replaces the flag.Usage function so that the global
help flag handling will call into the code that prints the nicer
message.
Fixes#890
Change-Id: Ib9ae73472148de6e92eeba92f12b5af7debe2bd1
Or to be more precise, golang.org/x/crypto/acme/autocert
The default behaviour regarding HTTPS certificates changes as such:
1) If the high-level config does not specify a certificate, the
low-level config used to be generated with a default certificate path.
This is no longer the case.
2) If the low-level config does not specify a certificate, we used to
generate self-signed ones at the default path. This is no longer always
the case. We only do this if our hostname does not look like an FQDN,
otherwise we try Let's Encrypt.
3) As a result, if the high-level config does not specify a certificate,
and the hostname looks like an FQDN, it is no longer the case that we'll
generate a self-signed. Let's Encrypt will be tried instead.
To sum up, the new rules are:
If cert/key files are specified, and found, use them.
If cert/key files are specified, not found, and the default values,
generate them (self-signed CA used as a cert), and use them.
If cert/key files are not specified, use Let's Encrypt if we have an
FQDN, otherwise generate self-signed.
Regarding cert caching:
On non-GCE, store the autocert cache dir in
osutil.CamliConfigDir()/letsencrypt.cache
On GCE, store in /tmp/camli-letsencrypt.cache
Fixes#701Fixes#859
Change-Id: Id78a9c6f113fa93e38d690033c10a749d1844ea6
In particular, specify "blobref" when search is not about permanode
results (instead of the "-created" default).
Fixes#886
Change-Id: I112288d42ea498873a5dcc3ddd37aa780620f309
And add golang.org/x/crypto/openpgp instead, at rev
ede567c8e044a5913dad1d1af3696d9da953104c
As well as golang.org/x/crypto/cast5 , as a dep.
Change-Id: I385fb0cd0eef438b43fa1dcd2a36e6a1442fd3b1
mpl