Commit Graph

5983 Commits

Author SHA1 Message Date
Will Norris 0b45279ce9 Merge "serverinit: remove trailing commas causing invalid JSON" 2016-04-29 19:50:19 +00:00
Will Norris d3baa6e638 serverinit: remove trailing commas causing invalid JSON
configHandler strips the key "_knownkeys" from all JSON objects in its
output using a regex on the marshaled JSON.  If this was the last key of
an object which contains multiples keys, this leaves the JSON in an
invalid state.  For example,

    {
        "foo": "bar",
        "_knownkeys": {}
    }

becomes:

    {
        "foo": "bar",
    }

The trailing comma here is illegal.  To fix this, we run one more regex
across the output removing any comma at the end of a line just before a
closing brace.  This is never valid in JSON, so is safe to remove
globally here.

Change-Id: I05f54e9f094fbe9cbc720f073b2b78ae80525106
2016-04-29 12:49:26 -07:00
mpl 6a80dd874d misc/docker/go: go 1.6.2
Change-Id: Ib1e479f02f1b103b371ca80d9f8a511f4f398e1a
2016-04-29 12:20:09 -07:00
mpl a0276d4f9d misc/docker: replace CopyObject calls
storage.Client.CopyObject does not exist anymore. Replaced with CopyTo
calls.

Also removed checks based on the presence of /.dockerinit to know
whether we're on docker because apparently they're gone or something.
I'll investigate why and a replacement later; we want a new GCE docker
image asap and I'm pressed for time.

Change-Id: I6b3ebcb048f03bdf53db94b41fe96ddcb6ed7537
2016-04-29 11:48:50 -07:00
Will Norris 0e708f0b74 Merge "serverinit: improve handling of redacted data in config handler" 2016-04-29 03:28:13 +00:00
Mathieu Lonjaret 5545134529 Merge "blobserver/cloudstorage: use cloud/storage Client" 2016-04-29 03:00:20 +00:00
Will Norris aff1874258 serverinit: improve handling of redacted data in config handler
make the "REDACTED" placeholder a valid JSON string and include trailing
comma in attempt to make the document valid JSON (primarily helpful with
browser extensions that parse and render JSON documents, even though the
content type is text/plain).

Add client_secret to the list of redacted keys (used with google cloud
storage, possibly others).

Change-Id: Ia10fc9fffbd667ea7018b750b2a98db0b05dcf82
2016-04-28 13:02:24 -07:00
Will Norris a1ec9bf60e rename HACKING to CONTRIBUTING.md
GitHub treats the CONTRIBUTING file special, in that it is referenced
any time someone files a new issue or starts to create a pull request.
Also make minor updates to file to use markdown and current hacking
instructions.

Change-Id: I68f0d7a69397851ee6c65671ae692cbfb2b93946
2016-04-27 12:16:29 -07:00
Will Norris 7effca63b7 Merge "website: add html ext to docs/releases/*" 2016-04-27 18:53:54 +00:00
Will Norris e4b61b0146 website: add html ext to docs/releases/*
this is now needed to have camweb wrap the file in the site template
rather than just serving it raw.

Change-Id: I1ac20030dd7bf878429fd70386e2e2814c66822c
2016-04-27 11:52:13 -07:00
mpl 9e9d5e0bf2 blobserver/cloudstorage: use cloud/storage Client
Using go4.org/cloud/google/gcsutil
for the things we can't yet do efficiently with the cloud/storage
Client.

Fixes issue #652

Change-Id: I8a43a754c4e0762629ce2a21bb95d9991f6f4771
2016-04-27 11:25:58 -07:00
Mathieu Lonjaret c905fe1b47 Merge "vendor: update google.golang.org/cloud" 2016-04-27 18:08:24 +00:00
mpl 219f966a95 vendor: update google.golang.org/cloud
to rev 36e3752134d445180a77cd40c922c48850b6fe9c

So we get ErrObjectNotExist on ObjectHandle.Delete

And update google.golang.org/api to rev
9737cc9e103c00d06a8f3993361dec083df3d252
for consistency.

Change-Id: I58c706d1d71a98ad01f4fef8929eac52338c1bbe
2016-04-27 11:04:47 -07:00
Will Norris 7ec2b0ac2e website: update doc URLs
s/docs/doc/ and s/http/https/ for camlistore.org doc URLs in website and
code.

Change-Id: I875e2acece1f594a304f2bdb63f756fcb315abc8
2016-04-27 08:48:50 -07:00
Will Norris 95a55498e6 website: fix wikipedia links on prior-art page
Change-Id: I6afa1cc2459fb2d372349492890fba55048c0493
2016-04-27 08:11:30 -07:00
Will Norris 29dcc70f3c website: restore search-ui.txt with link to new page
The help page in the camlistore binary has a direct link for "Search bar
predicates" that links to this search-ui.txt on gerrit.  Restore this
file so that these links will still resolve, but with a pointer to the
new docs on camlistore.org.

Also update the help link for future builds of camlistore to point to
the correct URL.

Change-Id: Ib81ff820869d86e6e6f664e8b52c5f20d4eed4d3
2016-04-26 22:28:05 -07:00
Mathieu Lonjaret 11c23bfb2b Merge "website: update misc docs to markdown" 2016-04-27 01:13:02 +00:00
Will Norris 620d837a3d website: update misc docs to markdown
- merge json-signing/json-signing.txt into json-signing/README.md
- rename publishing/README to publishing/README.md
- convert several doc/* files

Change-Id: I0bb7c03adcfac3848f97a70092de384c72f45958
2016-04-26 18:10:58 -07:00
Will Norris dfdc297a2d Merge "website: terminology: fix protocol links" 2016-04-27 01:03:38 +00:00
mpl 74b9c311e8 website: terminology: fix protocol links
Change-Id: I67b391f604175105ffc8f6ac65436a3e222d4e68
2016-04-26 17:50:36 -07:00
mpl 794ed25819 website: keep doc/schema flat
It looks better than the half-organization we had so far, and it will
actually make it easier for editorial changes to come. Also keeps us
more honest about doing it sooner rather than later.

Change-Id: I6303325f6d75389c7e8ca908f3740cf92af3836a
2016-04-26 17:34:47 -07:00
Mathieu Lonjaret bc9ac9e966 Merge "website: convert schema docs to markdown" 2016-04-26 23:49:53 +00:00
Will Norris 2f79a55baa website: convert schema docs to markdown
Refs #720

Change-Id: Iccfaf7384cffa78a51606868f83f4019e9441791
2016-04-26 16:32:38 -07:00
Will Norris 3cc1efc75e website: serve non-HTML files as raw bytes
The website has several non-HTML files that mostly serve as examples
(eg: https://camlistore.org/doc/json-signing/example/signing-before.camli).
Today these files are parsed and rendered the same as HTML files, which
is particularly bad for these json-signing examples.  Instead, they
should be served raw, without any interpretation.

Change-Id: I09f7de130e55da7b2e585f1a110bbbf391f2279f
2016-04-26 16:25:45 -07:00
Will Norris 066db0952c website: use custom markdown options
This is all so that we can enable auto creating header IDs based on the
text of the header.  For example "# Foo" is converted to:

  <h1 id="foo">Foo</h1>

I'm not really sure why this isn't a default option, but whatever.  It's
also unfortunate that we have to copy all of the flags and extensions
from MarkdownCommon, but at least this will make it easier to add more
options in the future, I gues.

Change-Id: I65800c75560ad3b9a3fd0389b722c07ba11ee098
2016-04-26 12:47:40 -07:00
Will Norris e3777187be vendor: update github.com/russross/blackfriday
rev 151efb040f183f3b968215a236c71a2614b62879

Change-Id: I2f5a2231ce07d9fd3f64d13a6965e06c60738b51
2016-04-26 10:41:00 -07:00
Brad Fitzpatrick f53b531967 Merge "camweb: redirect /doc/dir to /doc/dir/ with trailing slash" 2016-04-26 17:00:58 +00:00
Brad Fitzpatrick b62f3b399b Merge "search: remove broken websocket status optimization." 2016-04-26 16:47:52 +00:00
Brad Fitzpatrick f7e40292cb camweb: redirect /doc/dir to /doc/dir/ with trailing slash
Also fix up non-canonical links in docs after rename of /docs/ to /doc/

Updates #732

Change-Id: I7131f57cc7f8edf150cc785f42fe48917835c361
2016-04-26 09:46:07 -07:00
Will Norris 0d77ec13cb Merge "convert protocol docs to markdown" 2016-04-26 16:39:07 +00:00
Will Norris 974f810b4b convert protocol docs to markdown
Update links to reference the website URLs rather than gerrit.  There is
some more cleanup that could be done with the docs (for example, using
tables for the request and response keys), but this is focused on just
basic rendering.

Git is treating these as file deletions and additions... apparently
there were too many changes to treat them as renames, though that's what
they really are.

Refs #720

Change-Id: Ibca17b78e1412c049843236bfbc8f2a9e368bb23
2016-04-26 09:37:01 -07:00
Brad Fitzpatrick 7d326336cf search: remove broken websocket status optimization.
It was breaking the websocket integration test. We still need to
compute the last status message even if nobody is currently connected,
because when a websocket connection does arrive, we need to send the
last commputed status message.

Change-Id: Icea0fcc69941aa6249d32f0a4ef4ec03bdfb1e20
2016-04-26 09:29:39 -07:00
mpl 135f0373d2 vendor: add golang.org/x/sys/unix
Needed by bazil.org/fuse , it should have been added with
e0a9699c63

rev 8fdfb00a6a1add0f7a145480e3729a5427a74375

Fixes #730

Change-Id: I2f9c7aa21a5ead82d7043392e4a55baf3cecffe6
2016-04-26 08:57:58 -07:00
Will Norris fc619464df website: serve godocs for /app/ packages
Change-Id: If2836840aa5523e74a734051111e0ed614ea200e
2016-04-25 22:40:17 -07:00
Mathieu Lonjaret 7df8dbb2db Merge "website: always remove old demoblob3179 container" 2016-04-26 05:21:34 +00:00
Mathieu Lonjaret d422b6e942 Merge "website: serve /doc/ URLs from top-level doc dir" 2016-04-26 05:17:54 +00:00
Will Norris 7cb074bfde website: serve /doc/ URLs from top-level doc dir
- move existing content from website/content/docs to top-level doc
  directory
- redirect /docs/ URLs to /doc/

Fixes #720

Change-Id: I92e26bbc1fdbe994089080ae3a971d85c12921ab
2016-04-25 22:13:09 -07:00
Brad Fitzpatrick 858504d793 Merge "server/camnetdns: set authority flag on dns response" 2016-04-26 01:08:24 +00:00
Jeromy 444b2a81f3 server/camnetdns: set authority flag on dns response
Updates #722

Change-Id: I18277a0181234e9a9e497100a5f86613fca2dba4
2016-04-25 18:07:52 -07:00
Mathieu Lonjaret e0d422768d Merge "website: refactor mainHandler to allow diff roots" 2016-04-26 01:02:23 +00:00
Will Norris 6196e6ab98 website: refactor mainHandler to allow diff roots
This helps set up serving /docs/ requests out of different file root.
Additional, this allows the actual files on disk used to satisfy
requests to contain a .md or .html file extension (basically simulating
Apache multiviews).  Finally, requests that map to directories can be
satisfied by either a index.html or README.md file.

Ref #720

Change-Id: Id6076c242edd74b99a12699cd58c4567973d0d42
2016-04-25 18:01:27 -07:00
Brad Fitzpatrick 3c5b519c1e Merge "website: redirect to https" 2016-04-26 00:40:22 +00:00
Mathieu Lonjaret 8092071278 Merge "website: do a full cloning of camlistore.org on startup" 2016-04-26 00:35:18 +00:00
mpl d9fa1ed87e website: do a full cloning of camlistore.org on startup
We used to have --depth=1 when cloning camlistore.org on startup, for
efficiency reasons.
But not having all the commits locally caused problems when pushing to
github, because it would see the remote had commits that we didn't.
So we're now doing a full cloning.

It looks like it's now taking ~4s (as opposed to ~2.5s before), and
consuming ~50MB diskspace (as opposed to 35MB before).

Plus a few bugfixes.

Change-Id: If7dbae1d3119d8b3336fb7d735f6bd0ba7606fc6
2016-04-25 17:30:17 -07:00
Jeromy d77d4b8b51 server/camnetdns: log all incoming requests
Updates #722

Change-Id: Ief6b172d8ec4e62f22332a117f379ea4998a3f76
2016-04-25 16:53:50 -07:00
Jeromy e53f91085b server/camnetdns: default to port 53 on gce
Updates #722

Change-Id: I4d435634b6c78a4d4d1392616870d50f0c458e5c
2016-04-25 16:38:04 -07:00
Jeromy 4841ce270f server/camnetdns: add cloudlaunch config
Updates #722

Change-Id: Ia7532118183b5c38dfedc3aaa437f5284301b4c9
2016-04-25 16:26:26 -07:00
Brad Fitzpatrick 15ad9bf381 Merge "server/camnetdns: start of camlistore.net DNS server" 2016-04-25 23:08:34 +00:00
Jeromy 21c5373a0b server/camnetdns: start of camlistore.net DNS server
Updates #722

Change-Id: Ie4c2d6d036f13d7fb6f5545131397f55d1d9a91c
2016-04-25 16:05:25 -07:00
Mathieu Lonjaret 7332aad3aa Merge "server/camlistored: upload gpg key blob on startup" 2016-04-25 22:43:38 +00:00