When reindexing on a (My)SQL based sorted.KeyValue, we should recreate
the database schema from scratch, which means dropping the tables.
However, index.Reindex just calls Wipe on the newly created
sorted.KeyValue, which only deletes the rows, and does not drop the
tables.
Therefore, this CL changes the implementation of Wipe in the MySQL case,
so that it takes care of dropping the tables, and doing everything that
needs to be done afterwards to set up the sorted.KeyValue.
In addition, with the introduction of the sorted.NeedWipeError, we detect
upon initialization of a sorted.KeyValue if it failed because it needed
a schema update. If that is the case, and we're in reindex mode, we can
fix the sorted.KeyValue with a Wipe and carry on.
Finally, we introduce the new sorted.NewKeyValueMaybeWipe function that
automatically wipes a KeyValue when a NeedWipeError was returned upon
its creation.
Next, do the same with other sorted SQLs.
Fixes#806
Change-Id: I2032781cbf453a364880bd3e2e8b3c09aac7aed9
This CL changes the GCE launcher to work with the new features of
camlistored: i.e. that it can automatically get a hostname in
camlistore.net, and that it can get an HTTPS certificate from Let's
Encrypt, for said hostname.
In order for the user to easily (without having to look at the logs)
know what their hostname is, camlistored stores it as the
"camlistore-hostname" key in the custom metadata of the GCE instance.
The deployer can then query for that key, to report the hostname on the
instance creation success page.
Change-Id: Iaaef2d51f34fa5e1e0ee90097919abab7ee72a12
In order to use HTTPS, one must have a certificate, and one must have a
domain name for which the certificate is valid.
The first part is solved by the use of Let's Encrypt. For the second
part, we want to provide to any Camlistore instance a name such as
<gpgKeyId>.camlistore.net, where gpgKeyId is the fingerprint of its GPG
key. The DNS for camlistore.net agrees to add a record for that name if
and only if the Camlistore instance can prove it owns the GPG key, as
well as the IP address bound to that name in the DNS record.
A protocol such as the above is already implemented in pkg/gpgchallenge.
This CL:
- uses the client-side of the gpgchallenge protocol in camlistored, so
that it can claim a hostname in camlistore.net on startup (and then use
that hostname when requesting a certificate from Let's Encrypt).
- adds the configuration parameter "CamliNetIP" for the high-level
config. This parameter specifies the IP address that camlistored will
supply during the gpgpchallenge, so it can prove to the DNS server that
we own this address.
Fixes#722
Change-Id: I6bf4ec149b6dffd0ae93a6fa7bf208b2e8a05445
Store records in Google Datastore in production (on GCE), with an LRU
cache in front.
Keep dev mode that stores in sorted mem key value though.
Change-Id: I17b088b56a68a019f4e253b60c6bd42395a64984
Or to be more precise, golang.org/x/crypto/acme/autocert
The default behaviour regarding HTTPS certificates changes as such:
1) If the high-level config does not specify a certificate, the
low-level config used to be generated with a default certificate path.
This is no longer the case.
2) If the low-level config does not specify a certificate, we used to
generate self-signed ones at the default path. This is no longer always
the case. We only do this if our hostname does not look like an FQDN,
otherwise we try Let's Encrypt.
3) As a result, if the high-level config does not specify a certificate,
and the hostname looks like an FQDN, it is no longer the case that we'll
generate a self-signed. Let's Encrypt will be tried instead.
To sum up, the new rules are:
If cert/key files are specified, and found, use them.
If cert/key files are specified, not found, and the default values,
generate them (self-signed CA used as a cert), and use them.
If cert/key files are not specified, use Let's Encrypt if we have an
FQDN, otherwise generate self-signed.
Regarding cert caching:
On non-GCE, store the autocert cache dir in
osutil.CamliConfigDir()/letsencrypt.cache
On GCE, store in /tmp/camli-letsencrypt.cache
Fixes#701Fixes#859
Change-Id: Id78a9c6f113fa93e38d690033c10a749d1844ea6
In particular, specify "blobref" when search is not about permanode
results (instead of the "-created" default).
Fixes#886
Change-Id: I112288d42ea498873a5dcc3ddd37aa780620f309
I had to add the "-help" flag to camlistored, as otherwise relying on
the automatic usage gets us an exit status == 2, and hence an error when
running the Command.
fixes#475
Change-Id: I6f90329dbdb876a77c13d016f698de5026169e4a
Naive implementation for now. I imitated how things were setup for the
left and right keys, but maybe the listener should be on the image
container instead of on the index page? It seems to work anyway.
No animation when starting to swipe though, which means users have no
clue swipe exists/works until they actually try to do it for real.
Change-Id: If0baed3c8d3ff1f5d8886fca891a8ac40980f66e
The -recovery flag from camlistored, now forces the blobpacked index to
be rebuilt, regardless of its state.
Fixes#876
Change-Id: I4e6bd5374ec68d7bb32de9fc119abbc881707625
When we get a query for a name we are authoritative about, we should
reply with NXDOMAIN when this name does not exist.
This change moves the name lookup to as early as possible to make sure
of that. This means we're now doing lookups even for cases where we
technically wouldn't have needed them, so maybe a substantial increase
in load? We'll see.
Change-Id: I5e9946dd67757856f626f484b547197c6246cccd
As Let's Encrypt DNS server (Unbound) is pretty strict, it
wouldn't resolve names camlistore.net until we implemented more of the
DNS protocol and fix various things.
Since I had no way at first to know what exactly it didn't like, I
started by fixing all errors and warnings reported at:
http://dnsviz.net/d/camlistore.net/dnssec/
Therefore, this CL adds:
-TCP support
-NS response
-SOA response
-MX (empty) response
-DNSKEY (empty) response
-TXT (empty) response
-explicit non-support of EDNS
Then I found out we also needed this:
-https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00
-CAA response. A proper response is optional here, as Let's Encrypt only
enforces the response if it finds one. But we do have to reply.
Fixes#867
Change-Id: Ib45f8a642cd83cf19c8ab36435644a2c645a70e7
Before that change, one would only get a response to a DNS query, if
directly sending the query to the dns server.
I think the main problem was that one has to copy the question section
of the query in the response message, which is part of what SetReply
does.
With this change, I seem to get responses (for e.g. www.camlistore.net
and 6401800c.camlistore.net), without having to specify a server to
query to dig (or nslookup), so by letting the resolver and the DNS
hierarchy follow its normal course.
Issue #722
Change-Id: I530529920fdaa3e5fb025c7684b563e3f05772ac
server/camnetdns: update to cloud.google.com/go imports.
vendor: add github.com/miekg/dns at rev
3f1f7c8ec9ead89493df11f2c3d8bec353a2c2c0 as a dep of server/camnetdns
Change-Id: Ief8e0ad3e09a6c3511c72a7991309808f855d3df
rm google.golang.org/cloud
add cloud.google.com/go at a47b182e769f5e75f5fc927ff6ee2678f7f552cf
update google.golang.org/api to 63cb68f1e3834e44683ca062ddf06cb9a889380a
update google.golang.org/grpc to
0e6ec3a4501ee9ee2d023abe92e436fd04ed4081
update go4.org to f5283521d7365fb2875408726e9cbf349f173767
fix in cmd/ pkg/ server/
TODO(mpl): fix misc/docker tools as well. next CL.
Fixes#832
Change-Id: I842b968a0afea8a5822913bd614d67cdbe50ee63
This allows deferring HTTP authentication and authorization to a proxy
(such as nginx) between Camlistore and the rest of the world, without
breaking the web UI.
Fixes#816
Change-Id: Ia4b5be8f2236ddac68dc0d3a09f0f24e588c4995
This change enables camlistored to log on Google Cloud Logging even when
not running on Google Compute Engine.
The main reason is to help with debugging cloud logging bugs, because
it usually is easier to do so from a local camlistored than from one
running on GCE.
Related: issue #704
Change-Id: I7de23cf9365683f8641a6bbad0deab754fc579f7
The "click to reload" menu item displayed when the websocket connection
fails was non-functional. Clicking on the menu item only produced an
error (Uncaught TypeError: Illegal invocation) in the JavaScript
console.
Change-Id: Iea44dd35c92c99562c35f87c38687f4fa433ad41
The import path was added to the go file that included the package
documentation if one existed. Otherwise, I used what seemed to be the
primary file for the package.
Fixes#689
Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
Since issue #660 showed that we could not rely on Basic Auth for some
very particular cases (of POST requests), we wanted to replace Basic
Auth with a pre-generated token based authentication mechanism for these
cases.
However, as there already is such a mechanism used to authenticate
websocket connections, we simply extended the use of that mechanism.
Therefore, the token that is initially generated for websocket
connections is also now a valid token to authenticate any other
connection. The relevant types and method names have been changed to
reflect that. The javascript code pertaining to the sensitive cases
mentioned above has been changed to use that token.
Some doc has been added to point out how security-sensitive the
OpDiscovery permission is, since it gives access to the auth token.
Fixes issue #660
Change-Id: Iafed3b6e4804364ca2559414c8d87dc4a30f6637
Follow-up to 82c6feb61a and builds upon the work of Mathieu Lonjaret's Change 5345.
When viewing a set, and the dropped file is a dup for an existing permanode('s content), we still need to add the permanode to the set if it is not already a member of it. The above commit overlooked that point.
Fixes#622
Change-Id: I694e583625886a7bf164ab9fd5bf8874a7cc2b23
At rev c3c16077b235db4c6b2a2a6121c2d429707f4ed4
And use it in new package pkg/camlegal
And remove pkg/legal
Change-Id: I5e398aa5f145a02a04d33719276fd2f8b82ab12b
Previously pkg/jsonconfig and pkg/errorutil
Copied from go4.org at rev d1b8a2fb2de6160036e4801aa5e4d855571078b8
Change-Id: I673ed55b0825baa2607289b6082f205100261d7a
Previously pkg/wkfs
Copied from go4.org at rev 3970dd22c0e7e2397304ad626a20b72f8db41b58
Note that pkg/wkfs/gcs has not been moved yet (for dependencies
reasons).
Change-Id: I737e88a360bccfc643e1fa43541f65f2384c3157
When uploading a file, we were already checking if the contents and file
blob already existed, and acted smartly accordingly. However, we were
still always creating a new permanode (and camliContent claim) for that
file.
This CL addresses that last point.
Fixes#622
Change-Id: Ifb5c8846e20b6684d25a7749c64b09904e07bb6f
According to
https://developer.mozilla.org/en-US/docs/Web/Guide/API/DOM/Manipulating_the_browser_history
the second (=title) argument is ignored. However, in the chrome console
I'm getting
Uncaught TypeError: Failed to execute 'replaceState' on 'History': 2 arguments required, but only 1 present.
errors so I figure it's worth fixing.
Change-Id: I6b745503a68511c8010c41257b6790e7df9b8acb
opt_fail as an optional argument didn't work in the cases where the
func would also receive an event as argument. Because when opt_fail is
not provided, the arguments get shifted and the event value would end up
in the opt_fail variable, or vice-versa.
So in these cases, I switched to the more robust
func({success: success, fail: fail}, event) syntax
As a result, safeFail wasn't that useful anymore, and it was incorrectly
bound in some cases, so I removed it.
I also removed a few intermediate function calls which made reading
unnecessarily more complicated.
Change-Id: Ie70694a26e52f302837bbf6ffc16b0a0bb77ad7e
1) cam.object dep was needed for any of the tests to run
2) fixed the return logic of navigate() to match its doc. Should be of
no consequence though, as afaik this return value is not used anywhere.
3) added tests for 2)
Change-Id: I5c236ab95482b50437f0047d63e42603f754438d
Reintroduction of the left/right arrow and the Around query
broke saved selection state.
Fixes: #609
Change-Id: I0974fd8af48e0eb9fe9d16faecd41b17a6f3824b
- Available at '/help/'
- Loads and parses server config upon init and generates the necessary client config for access
- Links to 'Downloadable Tools' are not functional (task is not yet completed)
- A link was added to the default server page as well as a 'Help' option to the Piggy menu
Issue: #536
Change-Id: I10629de034c908ebc76f6feacbbfca1df7bf1c2b
Also refactor updateTargetSearchSession_ and updateChildSearchSession_
for query string VS target blobRef precedence clarity.
Fixes#614Fixes#615
Change-Id: Ib70f9fb582e96a6dac3b0920ce324c03501bcd32
Also, delete my old gce package from third_party and only use the
google metadata package (which my gce package became, and which was
also already vendored into third_party)
Fixes#596
Change-Id: I64fd6f1e9dc6f433466f91f81efd2ecbf039334f
this change introduces one API to filewriter, which takes an additional param for the file's lastmodtime to store with the file blob
Issue #317
Change-Id: I8da3134306c28e6945661296aaafb7752b4bf8b5
Fix some UI queries, since UI always queries with "-created".
Also add Unsorted, to override the CreatedDesc default, for when we
really do not want to sort, e.g. when we don't have a corpus and
trying to sort would make us error out.
Issue #579
Change-Id: Ife0aa816c5e8cac8dc6612d0ffd104238abc6838
We now show it when either there are existing children, or when
the permanode doesn't have a more specific type.
Change-Id: Ic90829a60bf5f6213afd53476473eed4ffee6ca8
This reverts commit 2d67328243.
I decided to do this only with heuristics, since we have to maintain
backward compat with data before this camliNodeType existed.
Change-Id: Ieba5dd1f8a4c8f5aedc1477c4fab1ffebcc2878c
-"unique key" warnings for items dynamically added via arrays
-"bind" errors when binding without providing 'null' as first param
Related Issue: https://code.google.com/p/camlistore/issues/detail?id=426
Change-Id: I9ab3d8e1097fc37bed54f1e0a6c078dd264ef579
This change makes gce/create.go generate a self-signed certificate
with the hostname from the -hostname argument and upload it to GCS
before creating a new camlistore instance.
It also makes camlistored use baseURL to figure out the hostname
when generating its self-signed certificate.
Change-Id: I64f85853dab34a7ce95e5d5997e58f2e5da43496
SearchSession::isComplete() could sometimes return the wrong answer.
Nulling out the continuation is needed to make loadMoreResults()
idempotent. So had to add more state to SearchSession specifically
for tracking completeness.
Change-Id: Ic2a90c73f3a764f358476cffa6d498ea53103421
To be able to use it (ATM only in low-level config, by specifying
metaIndex with the stores/queues/caches).
Change-Id: I7743ebffb90f59b83124063f8f488a9b80c869ca
Currently disabled but can be tested via the DEBUG_TAGS flag in index.js. Potential enhancements are documented in tags_control.js. This patch
removes one piece of unused code.
Change-Id: I751fca63fbc93c209628f03a3f3aa6e87d01e309
We were waiting to show the container aspect until the child search session
returned. This meant that we temporarily showed the permanode aspect then
swapped to the container aspect shortly thereafter (because the container
aspect is higher priority).
The right solution to this is to change the search protocol so that clients
can know how many children a permanode has without getting the entire list
of them. But for now, it just so happens that search returns the entire
list, so we can use that instead.
Change-Id: Iaf93f689e965a1dc9ef73d834d46c57026c100cd
When there is a server error, the piggy becomes sad and periodically
shakes his head. When you expand the piggy menu you see one or more
entries explaining why he is sad.
Change-Id: If1e28dbf02c9befaa46b62838cb704220f719bc6
Otherwise, we end up showing a stupid empty contents view for every
tweet, foursquare checkin, etc.
This has the downside that when you remove the last item from a
set, the contents view goes away.
Change-Id: If5f53a713aaff63d9fb294b675a5e55ca452eff1
Before we did not show the search aspect when the results were empty.
This was intended for the case where you're looking at an image or
something that doesn't have children.
But it had the side effect of meaning that we didn't show the search
aspect for empty sets, or empty search results, which isn't right
because you still want users to be able add items to empty sets.
Change-Id: Iad187b38317a5adcefbfd06f92047c03c5815583
Instead of /ui/?b=<hash>, /ui/?p=<hash>, /ui/?d=<hash>, etc...,
it's now just: /ui/<hash>
This makes constructing links easier, because you don't have to
know whether a blob is also a permanode, or a directory, or
whatever.
Change-Id: I771fa1b1699b9009269c7eee632e7e8fd5335331
Also add support for piggy menu items that are just links. Links are
nice because you can right-click 'em and so on.
Change-Id: I0cfb4ba6f58141ee396916925793a06033457028
Mathieu Lonjaret
