When the blobpacked index gets corrupted/destroyed, it needs to be
rebuilt and hence camlistored refuses to start without the -recovery
flag being set if it detects it is needed.
This is a problem on GCE, because camlistored is handled by systemd, and
so the only way to restart camlistored with -recovery is to edit the
systemd service, then issue 'systemctl daemon-reload', and 'systemctl
restart camlistored'. Then revert the change for the next time
camlistored restarts. This is not user-friendly.
This change adds a check on camlistored startup based on the presence
and value (as a boolean) of the "camlistore-recovery" key as an instance
attribute. Therefore, the user only has to add the
("camlistore-recovery", "true") attribute in the Custom metadata of
their Google Cloud instance page to switch to recovery mode. (And
restart the instance).
As an additional feature (for non-GCE users), this change also adds the
option to restart the server in recovery mode from the status page of
the web UI.
Change-Id: I44f5ca293ddd0a0033fc5d9c2edca1bac0ee9c8f
The code to find the prefix URL that is used to display the share URL on
after successfully sharing an item, worked only on the main web UI page.
This change makes it work on hopefully any page.
Change-Id: I05e085f91ef7ebe39f880104e52ca487b80c607a
The download action of the web UI uses an ephemeral form, that we do not
connect to the DOM, behind the scenes. However, according to
https://html.spec.whatwg.org/multipage/forms.html#form-submission-algorithm,
step 2., a form must be connected to the DOM in order for it to be
submitted.
This change connects the form right before submitting, and removes it
from the DOM right after submission.
Fixes#921
Change-Id: I4d993c633c60a369ce4260f84731a64e8b955c76
The default server config on GCE (as deployed by the launcher) did not
have a share handler. This CL adds one, so that users can benefit from
the new sharing feature from the web UI.
Also, the button for sharing from the web UI does not appear anymore if
the config that the web UI gets from the discovery does not have a
"shareRoot", because it's a strong hint that the server does not have a
share handler.
Change-Id: I6c444995339fda8dba864b1d6729fb7c1b6d72bd
Sharing through the web UI was limited to a selection of one (permanode)
file or directory.
This change enables sharing of all the selected items, as long as they
are (permanodes of) files or directories, by creating a new directory
blob containing the selected items. The new directory is the item
getting shared. For clarity, and hopefully helping with search later,
the new directory is named "shared-YYYYMMDDhhmmss".
Change-Id: Iabd2266c9af85f3e8fbfa56ce98f653c0ce0f1ee
To share a file or directory through Camlistore's sharing
mechanism (https://camlistore.org/doc/sharing).
Only single-item sharing supported for now.
On success, the share URL is displayed in a dialog.
If the item is a file, it can be fetched with any HTTP client through
that URL. If the item is a dir, the share URL is of the claim, so one
must follow the transitive chain of sharing to get the directory's
children, i.e. use camget -shared.
Fixes#880
Change-Id: I324a684dec10225e5b8314dda7f1b77a0de6f727
We want to add a feature for clients (the web UI), where they can select
a bunch of files and ask the server for a zip archive of all these files.
This CL modifies the DownloadHandler so it does exactly that upon
reception of a POST request with a query parameter of the form
files=sha1-foo,sha1-bar,sha1-baz
This CL also adds a new button to the contextual sidebar of the web UI,
that takes care of sending the download request to the server.
known limitations: only permanodes with file as camliContent are
accepted as a valid selection (i.e. no sets, or static-dirs, etc) for
now.
Implementation detail:
We're creating an ephemeral DOM form on the fly to send the request.
The reason is: if we sent it as a Go http request, we'd have to read
the response manually and then we'd have no way of writing it to disk.
If we did it with an xhr, we could write the response to disk by
creating a File or Blob and then using URL.createObjectURL(), but we'd
have to keep the response in memory while doing so, which is
unacceptable for large enough archives.
Fixes#899
Change-Id: I104f7c5bd10ab3369e28d33752380dd12b5b3e6b
This change adds an "About" entry to the header menu, that pops up an
"About" dialog when clicked on.
But, under this innocent sounding feature is actually a more important
change: this feature is partly added through the use of gopherjs
generated code. In addition, the element added (a menu item) is
integrated
seamlessly into the menu as a React element, thanks to the use of
github.com/myitcv/gopherjs/react.
pkg/env had to be broken with build tags, because importing
"cloud.google.com/go/compute/metadata" (or one of its deps) seems to be
breaking the generated javascript.
https://github.com/camlistore/camlistore/issues/904
github.com/myitcv/gopherjs/react at rev
c04b811da4a086defd882a94cc1901da2d2158b0
honnef.co/go/js/dom at rev 24aa052bc5c63cfb9383bf59493ee48621ca788c
Issue #904Fixes#798
Change-Id: Ic09b94014d520277f8178727293787ece468babd
React would bail out with:
Uncaught TypeError: this.refs.searchbox.getDOMNode is not a function
at Object.getSearchNode_ (http://localhost:3179/ui/header.js:321:33)
Change-Id: I26e3474ed08c9339eda2e78de87b458a4130c746
In addition to API changes, the V2 logging API uses
google.golang.org/grpc, which does not rely directly on net/http. This
is a problem for us because, among other things, when needed (on
CoreOS), we insert the system Root CAs in the TLSConfig of the
default http client through httputil.InstallCerts, which has no effect
on google.golang.org/grpc calls. Therefore, we switch to
github.com/bradfitz/grpc-go instead, which does rely on net/http.
And since GRPC requires HTTP/2, we also now need to manually configure
the default transport (since we're already modifying it) to be HTTP/2
enabled.
Related dependencies updated/added:
updated:
cloud.google.com/go/logging
a64eb5d53f434cc548fd6b2898b5a3e8a0c31b78
github.com/golang/protobuf
8ee79997227bf9b34611aee7946ae64735e6fd93
added:
github.com/bradfitz/grpc-go
188a132adcfba339f1f2d5da52498451341f9ee8
as a replacement for google.golang.org/grpc
github.com/googleapis/gax-go
da06d194a00e19ce00d9011a13931c3f6f6887c7
google.golang.org/genproto/googleapis/logging
08f135d1a31b6ba454287638a3ce23a55adace6f
google.golang.org/genproto/googleapis/datastore
08f135d1a31b6ba454287638a3ce23a55adace6f
google.golang.org/api/support/bundler
e4c04685e5d7db47ff294aa9e514b3a638c431c9
google.golang.org/genproto/googleapis/rpc/status
08f135d1a31b6ba454287638a3ce23a55adace6f
Fixes#873
Change-Id: I3ba14c94122dd9f1b88ef61c2dc26430690d28a9
The main motivation is that we want to use gopherjs +
github.com/myitcv/gopherjs/react , which requires a recent react.
This CL also fully switches to only using minified js files.
The changes in server/camlistored/ui are the result of some react
API changes, such as:
- wraps component instantiation in React.createElement
- React.classSet -> classNames
- Need tbody between table and tr in a couple places
- component array elts need unique keys (longstanding warning)
- change root elt from body to a child div (to fix warning)
- don't check isMounted in container render function
- fix a bunch of style keys using kebab-case instead of camelCase
Change-Id: Idc22d07704688ccde3e4f67d4f188a5ad32b6e32
Add the -staging flag to camweb so one can deploy on a staging
instance.
The instance name is then "camweb-staging" instead of "camweb", and its
hostname is "staging.camlistore.net", instead of "camlistore.org".
server/camnetdns: automatically and regularly discovers the IP of the
"camweb-staging" instance, so it can keep the record for
"staging.camlistore.net" up to date.
Change-Id: I7d64a774049ac01472666ee7e351fd6fb5e4c287
When reindexing on a (My)SQL based sorted.KeyValue, we should recreate
the database schema from scratch, which means dropping the tables.
However, index.Reindex just calls Wipe on the newly created
sorted.KeyValue, which only deletes the rows, and does not drop the
tables.
Therefore, this CL changes the implementation of Wipe in the MySQL case,
so that it takes care of dropping the tables, and doing everything that
needs to be done afterwards to set up the sorted.KeyValue.
In addition, with the introduction of the sorted.NeedWipeError, we detect
upon initialization of a sorted.KeyValue if it failed because it needed
a schema update. If that is the case, and we're in reindex mode, we can
fix the sorted.KeyValue with a Wipe and carry on.
Finally, we introduce the new sorted.NewKeyValueMaybeWipe function that
automatically wipes a KeyValue when a NeedWipeError was returned upon
its creation.
Next, do the same with other sorted SQLs.
Fixes#806
Change-Id: I2032781cbf453a364880bd3e2e8b3c09aac7aed9
This CL changes the GCE launcher to work with the new features of
camlistored: i.e. that it can automatically get a hostname in
camlistore.net, and that it can get an HTTPS certificate from Let's
Encrypt, for said hostname.
In order for the user to easily (without having to look at the logs)
know what their hostname is, camlistored stores it as the
"camlistore-hostname" key in the custom metadata of the GCE instance.
The deployer can then query for that key, to report the hostname on the
instance creation success page.
Change-Id: Iaaef2d51f34fa5e1e0ee90097919abab7ee72a12
In order to use HTTPS, one must have a certificate, and one must have a
domain name for which the certificate is valid.
The first part is solved by the use of Let's Encrypt. For the second
part, we want to provide to any Camlistore instance a name such as
<gpgKeyId>.camlistore.net, where gpgKeyId is the fingerprint of its GPG
key. The DNS for camlistore.net agrees to add a record for that name if
and only if the Camlistore instance can prove it owns the GPG key, as
well as the IP address bound to that name in the DNS record.
A protocol such as the above is already implemented in pkg/gpgchallenge.
This CL:
- uses the client-side of the gpgchallenge protocol in camlistored, so
that it can claim a hostname in camlistore.net on startup (and then use
that hostname when requesting a certificate from Let's Encrypt).
- adds the configuration parameter "CamliNetIP" for the high-level
config. This parameter specifies the IP address that camlistored will
supply during the gpgpchallenge, so it can prove to the DNS server that
we own this address.
Fixes#722
Change-Id: I6bf4ec149b6dffd0ae93a6fa7bf208b2e8a05445
Store records in Google Datastore in production (on GCE), with an LRU
cache in front.
Keep dev mode that stores in sorted mem key value though.
Change-Id: I17b088b56a68a019f4e253b60c6bd42395a64984
Or to be more precise, golang.org/x/crypto/acme/autocert
The default behaviour regarding HTTPS certificates changes as such:
1) If the high-level config does not specify a certificate, the
low-level config used to be generated with a default certificate path.
This is no longer the case.
2) If the low-level config does not specify a certificate, we used to
generate self-signed ones at the default path. This is no longer always
the case. We only do this if our hostname does not look like an FQDN,
otherwise we try Let's Encrypt.
3) As a result, if the high-level config does not specify a certificate,
and the hostname looks like an FQDN, it is no longer the case that we'll
generate a self-signed. Let's Encrypt will be tried instead.
To sum up, the new rules are:
If cert/key files are specified, and found, use them.
If cert/key files are specified, not found, and the default values,
generate them (self-signed CA used as a cert), and use them.
If cert/key files are not specified, use Let's Encrypt if we have an
FQDN, otherwise generate self-signed.
Regarding cert caching:
On non-GCE, store the autocert cache dir in
osutil.CamliConfigDir()/letsencrypt.cache
On GCE, store in /tmp/camli-letsencrypt.cache
Fixes#701Fixes#859
Change-Id: Id78a9c6f113fa93e38d690033c10a749d1844ea6
In particular, specify "blobref" when search is not about permanode
results (instead of the "-created" default).
Fixes#886
Change-Id: I112288d42ea498873a5dcc3ddd37aa780620f309
I had to add the "-help" flag to camlistored, as otherwise relying on
the automatic usage gets us an exit status == 2, and hence an error when
running the Command.
fixes#475
Change-Id: I6f90329dbdb876a77c13d016f698de5026169e4a
Naive implementation for now. I imitated how things were setup for the
left and right keys, but maybe the listener should be on the image
container instead of on the index page? It seems to work anyway.
No animation when starting to swipe though, which means users have no
clue swipe exists/works until they actually try to do it for real.
Change-Id: If0baed3c8d3ff1f5d8886fca891a8ac40980f66e
The -recovery flag from camlistored, now forces the blobpacked index to
be rebuilt, regardless of its state.
Fixes#876
Change-Id: I4e6bd5374ec68d7bb32de9fc119abbc881707625
When we get a query for a name we are authoritative about, we should
reply with NXDOMAIN when this name does not exist.
This change moves the name lookup to as early as possible to make sure
of that. This means we're now doing lookups even for cases where we
technically wouldn't have needed them, so maybe a substantial increase
in load? We'll see.
Change-Id: I5e9946dd67757856f626f484b547197c6246cccd
As Let's Encrypt DNS server (Unbound) is pretty strict, it
wouldn't resolve names camlistore.net until we implemented more of the
DNS protocol and fix various things.
Since I had no way at first to know what exactly it didn't like, I
started by fixing all errors and warnings reported at:
http://dnsviz.net/d/camlistore.net/dnssec/
Therefore, this CL adds:
-TCP support
-NS response
-SOA response
-MX (empty) response
-DNSKEY (empty) response
-TXT (empty) response
-explicit non-support of EDNS
Then I found out we also needed this:
-https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00
-CAA response. A proper response is optional here, as Let's Encrypt only
enforces the response if it finds one. But we do have to reply.
Fixes#867
Change-Id: Ib45f8a642cd83cf19c8ab36435644a2c645a70e7
Before that change, one would only get a response to a DNS query, if
directly sending the query to the dns server.
I think the main problem was that one has to copy the question section
of the query in the response message, which is part of what SetReply
does.
With this change, I seem to get responses (for e.g. www.camlistore.net
and 6401800c.camlistore.net), without having to specify a server to
query to dig (or nslookup), so by letting the resolver and the DNS
hierarchy follow its normal course.
Issue #722
Change-Id: I530529920fdaa3e5fb025c7684b563e3f05772ac
server/camnetdns: update to cloud.google.com/go imports.
vendor: add github.com/miekg/dns at rev
3f1f7c8ec9ead89493df11f2c3d8bec353a2c2c0 as a dep of server/camnetdns
Change-Id: Ief8e0ad3e09a6c3511c72a7991309808f855d3df
rm google.golang.org/cloud
add cloud.google.com/go at a47b182e769f5e75f5fc927ff6ee2678f7f552cf
update google.golang.org/api to 63cb68f1e3834e44683ca062ddf06cb9a889380a
update google.golang.org/grpc to
0e6ec3a4501ee9ee2d023abe92e436fd04ed4081
update go4.org to f5283521d7365fb2875408726e9cbf349f173767
fix in cmd/ pkg/ server/
TODO(mpl): fix misc/docker tools as well. next CL.
Fixes#832
Change-Id: I842b968a0afea8a5822913bd614d67cdbe50ee63
This allows deferring HTTP authentication and authorization to a proxy
(such as nginx) between Camlistore and the rest of the world, without
breaking the web UI.
Fixes#816
Change-Id: Ia4b5be8f2236ddac68dc0d3a09f0f24e588c4995
This change enables camlistored to log on Google Cloud Logging even when
not running on Google Compute Engine.
The main reason is to help with debugging cloud logging bugs, because
it usually is easier to do so from a local camlistored than from one
running on GCE.
Related: issue #704
Change-Id: I7de23cf9365683f8641a6bbad0deab754fc579f7
The "click to reload" menu item displayed when the websocket connection
fails was non-functional. Clicking on the menu item only produced an
error (Uncaught TypeError: Illegal invocation) in the JavaScript
console.
Change-Id: Iea44dd35c92c99562c35f87c38687f4fa433ad41
Mathieu Lonjaret
