When reindexing on a (My)SQL based sorted.KeyValue, we should recreate
the database schema from scratch, which means dropping the tables.
However, index.Reindex just calls Wipe on the newly created
sorted.KeyValue, which only deletes the rows, and does not drop the
tables.
Therefore, this CL changes the implementation of Wipe in the MySQL case,
so that it takes care of dropping the tables, and doing everything that
needs to be done afterwards to set up the sorted.KeyValue.
In addition, with the introduction of the sorted.NeedWipeError, we detect
upon initialization of a sorted.KeyValue if it failed because it needed
a schema update. If that is the case, and we're in reindex mode, we can
fix the sorted.KeyValue with a Wipe and carry on.
Finally, we introduce the new sorted.NewKeyValueMaybeWipe function that
automatically wipes a KeyValue when a NeedWipeError was returned upon
its creation.
Next, do the same with other sorted SQLs.
Fixes#806
Change-Id: I2032781cbf453a364880bd3e2e8b3c09aac7aed9
This CL changes the GCE launcher to work with the new features of
camlistored: i.e. that it can automatically get a hostname in
camlistore.net, and that it can get an HTTPS certificate from Let's
Encrypt, for said hostname.
In order for the user to easily (without having to look at the logs)
know what their hostname is, camlistored stores it as the
"camlistore-hostname" key in the custom metadata of the GCE instance.
The deployer can then query for that key, to report the hostname on the
instance creation success page.
Change-Id: Iaaef2d51f34fa5e1e0ee90097919abab7ee72a12
In order to use HTTPS, one must have a certificate, and one must have a
domain name for which the certificate is valid.
The first part is solved by the use of Let's Encrypt. For the second
part, we want to provide to any Camlistore instance a name such as
<gpgKeyId>.camlistore.net, where gpgKeyId is the fingerprint of its GPG
key. The DNS for camlistore.net agrees to add a record for that name if
and only if the Camlistore instance can prove it owns the GPG key, as
well as the IP address bound to that name in the DNS record.
A protocol such as the above is already implemented in pkg/gpgchallenge.
This CL:
- uses the client-side of the gpgchallenge protocol in camlistored, so
that it can claim a hostname in camlistore.net on startup (and then use
that hostname when requesting a certificate from Let's Encrypt).
- adds the configuration parameter "CamliNetIP" for the high-level
config. This parameter specifies the IP address that camlistored will
supply during the gpgpchallenge, so it can prove to the DNS server that
we own this address.
Fixes#722
Change-Id: I6bf4ec149b6dffd0ae93a6fa7bf208b2e8a05445
Store records in Google Datastore in production (on GCE), with an LRU
cache in front.
Keep dev mode that stores in sorted mem key value though.
Change-Id: I17b088b56a68a019f4e253b60c6bd42395a64984
Or to be more precise, golang.org/x/crypto/acme/autocert
The default behaviour regarding HTTPS certificates changes as such:
1) If the high-level config does not specify a certificate, the
low-level config used to be generated with a default certificate path.
This is no longer the case.
2) If the low-level config does not specify a certificate, we used to
generate self-signed ones at the default path. This is no longer always
the case. We only do this if our hostname does not look like an FQDN,
otherwise we try Let's Encrypt.
3) As a result, if the high-level config does not specify a certificate,
and the hostname looks like an FQDN, it is no longer the case that we'll
generate a self-signed. Let's Encrypt will be tried instead.
To sum up, the new rules are:
If cert/key files are specified, and found, use them.
If cert/key files are specified, not found, and the default values,
generate them (self-signed CA used as a cert), and use them.
If cert/key files are not specified, use Let's Encrypt if we have an
FQDN, otherwise generate self-signed.
Regarding cert caching:
On non-GCE, store the autocert cache dir in
osutil.CamliConfigDir()/letsencrypt.cache
On GCE, store in /tmp/camli-letsencrypt.cache
Fixes#701Fixes#859
Change-Id: Id78a9c6f113fa93e38d690033c10a749d1844ea6
In particular, specify "blobref" when search is not about permanode
results (instead of the "-created" default).
Fixes#886
Change-Id: I112288d42ea498873a5dcc3ddd37aa780620f309
I had to add the "-help" flag to camlistored, as otherwise relying on
the automatic usage gets us an exit status == 2, and hence an error when
running the Command.
fixes#475
Change-Id: I6f90329dbdb876a77c13d016f698de5026169e4a
Naive implementation for now. I imitated how things were setup for the
left and right keys, but maybe the listener should be on the image
container instead of on the index page? It seems to work anyway.
No animation when starting to swipe though, which means users have no
clue swipe exists/works until they actually try to do it for real.
Change-Id: If0baed3c8d3ff1f5d8886fca891a8ac40980f66e
The -recovery flag from camlistored, now forces the blobpacked index to
be rebuilt, regardless of its state.
Fixes#876
Change-Id: I4e6bd5374ec68d7bb32de9fc119abbc881707625
When we get a query for a name we are authoritative about, we should
reply with NXDOMAIN when this name does not exist.
This change moves the name lookup to as early as possible to make sure
of that. This means we're now doing lookups even for cases where we
technically wouldn't have needed them, so maybe a substantial increase
in load? We'll see.
Change-Id: I5e9946dd67757856f626f484b547197c6246cccd
As Let's Encrypt DNS server (Unbound) is pretty strict, it
wouldn't resolve names camlistore.net until we implemented more of the
DNS protocol and fix various things.
Since I had no way at first to know what exactly it didn't like, I
started by fixing all errors and warnings reported at:
http://dnsviz.net/d/camlistore.net/dnssec/
Therefore, this CL adds:
-TCP support
-NS response
-SOA response
-MX (empty) response
-DNSKEY (empty) response
-TXT (empty) response
-explicit non-support of EDNS
Then I found out we also needed this:
-https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00
-CAA response. A proper response is optional here, as Let's Encrypt only
enforces the response if it finds one. But we do have to reply.
Fixes#867
Change-Id: Ib45f8a642cd83cf19c8ab36435644a2c645a70e7
Before that change, one would only get a response to a DNS query, if
directly sending the query to the dns server.
I think the main problem was that one has to copy the question section
of the query in the response message, which is part of what SetReply
does.
With this change, I seem to get responses (for e.g. www.camlistore.net
and 6401800c.camlistore.net), without having to specify a server to
query to dig (or nslookup), so by letting the resolver and the DNS
hierarchy follow its normal course.
Issue #722
Change-Id: I530529920fdaa3e5fb025c7684b563e3f05772ac
server/camnetdns: update to cloud.google.com/go imports.
vendor: add github.com/miekg/dns at rev
3f1f7c8ec9ead89493df11f2c3d8bec353a2c2c0 as a dep of server/camnetdns
Change-Id: Ief8e0ad3e09a6c3511c72a7991309808f855d3df
rm google.golang.org/cloud
add cloud.google.com/go at a47b182e769f5e75f5fc927ff6ee2678f7f552cf
update google.golang.org/api to 63cb68f1e3834e44683ca062ddf06cb9a889380a
update google.golang.org/grpc to
0e6ec3a4501ee9ee2d023abe92e436fd04ed4081
update go4.org to f5283521d7365fb2875408726e9cbf349f173767
fix in cmd/ pkg/ server/
TODO(mpl): fix misc/docker tools as well. next CL.
Fixes#832
Change-Id: I842b968a0afea8a5822913bd614d67cdbe50ee63
This allows deferring HTTP authentication and authorization to a proxy
(such as nginx) between Camlistore and the rest of the world, without
breaking the web UI.
Fixes#816
Change-Id: Ia4b5be8f2236ddac68dc0d3a09f0f24e588c4995
This change enables camlistored to log on Google Cloud Logging even when
not running on Google Compute Engine.
The main reason is to help with debugging cloud logging bugs, because
it usually is easier to do so from a local camlistored than from one
running on GCE.
Related: issue #704
Change-Id: I7de23cf9365683f8641a6bbad0deab754fc579f7
The "click to reload" menu item displayed when the websocket connection
fails was non-functional. Clicking on the menu item only produced an
error (Uncaught TypeError: Illegal invocation) in the JavaScript
console.
Change-Id: Iea44dd35c92c99562c35f87c38687f4fa433ad41
The import path was added to the go file that included the package
documentation if one existed. Otherwise, I used what seemed to be the
primary file for the package.
Fixes#689
Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
Since issue #660 showed that we could not rely on Basic Auth for some
very particular cases (of POST requests), we wanted to replace Basic
Auth with a pre-generated token based authentication mechanism for these
cases.
However, as there already is such a mechanism used to authenticate
websocket connections, we simply extended the use of that mechanism.
Therefore, the token that is initially generated for websocket
connections is also now a valid token to authenticate any other
connection. The relevant types and method names have been changed to
reflect that. The javascript code pertaining to the sensitive cases
mentioned above has been changed to use that token.
Some doc has been added to point out how security-sensitive the
OpDiscovery permission is, since it gives access to the auth token.
Fixes issue #660
Change-Id: Iafed3b6e4804364ca2559414c8d87dc4a30f6637
Mathieu Lonjaret