Commit Graph

6536 Commits

Author SHA1 Message Date
mpl 281d2a16b6 clients/android: for building, make caching volume dirs on the host
The ~/.gradle and ~/.android dirs are mounted as volumes for the docker
container when building the app, because we want the gradle and android
deps to be cached on the host after they've been downloaded.

The build process was assuming that these dirs always already existed,
and this assumption makes the building process fail if they actually
don't.

Therefore this change creates the directories on the host if they don't
already exist.

Change-Id: I90b5f73db0ffcbe76adc12157e142091808bde2b
2017-11-14 19:24:56 +01:00
Mathieu Lonjaret 32fbc99a66 Merge "app/publisher/js/zsearch.go: do not track in git" 2017-10-29 15:16:19 +00:00
mpl 28bc007676 pkg/serverinit: add low-level config generation for Backblaze B2
Fixes #971

Change-Id: Iba944e3597009b18a380007b72fba5127e9a1698
2017-10-26 19:59:01 +02:00
Mathieu Lonjaret 3911a41bd5 Merge "cmd/*: always output on fatal errors" 2017-10-22 15:07:06 +00:00
mpl a6a3cac0df cmd/*: always output on fatal errors
This CL addresses issues #685 and #862.

The general problem is that some critical errors, that lead clients such
as camput to exit with failure, are not displayed when not running in
verbose mode.

The reason that happens is because of code such as:

	if *cmdmain.FlagVerbose {
		log.SetOutput(cmdmain.Stderr)
	} else {
		log.SetOutput(ioutil.Discard)
	}

which means that in non-verbose mode we discard absolutely all log
messages, even those that would be printed during a Fatal* call.

To address that problem, we introduce a logger, as well as the Printf
and Logf functions using it, in pkg/cmdmain. These two functions only
output when *cmdmain.FlagVerbose is true.

Commands such as camput or camtool should now always:
1) log.SetOutput(cmdmain.Stderr) in init().
2) use log.Printf for messages that should always be printed.
3) use cmdmain.Printf/Logf for messages that should only be printed when
*cmdmain.FlagVerbose is true.
4) use log.Fatal for critical errors.
5) optionally, set the Verbose and Logger of the client(s) they are
using.

Also, camput and camtool are now relying on the global -verbose flag
from cmdmain, instead of having to define one for each subcommand.

fixes #685
fixes #862

Change-Id: I088032fd28184a201076097bf878894b22a8a120
2017-10-19 02:22:44 +02:00
Daniel Lipovetsky 493cc283d2 website: update help page community URL
Remove the trailing slash from the community URL

Fixes #967

Change-Id: I487eaa71d6a1536293c6aa655dc280d9f2cdb276
2017-10-16 22:55:30 -07:00
Mathieu Lonjaret 39cd6029e3 Merge "server/camlistored/ui: remove useless crossorigin and integrity for leaflet" 2017-10-08 03:05:18 +00:00
mpl b95aa162d7 server/camlistored/ui: remove useless crossorigin and integrity for leaflet
I had pasted them from an example when I started to work on the map
aspect, and forgot to look up their meaning/use afterwards.

Fixes #964

Change-Id: Ic92dcd9641a7fad49707c24f114b370f2c887828
2017-10-06 01:09:56 +02:00
Mathieu Lonjaret 5ca092b8ca Merge "vendor: update Google Closure" 2017-10-05 22:57:27 +00:00
mpl c3e7fc8002 app/publisher: remove useless urlquery escaper
According to https://golang.org/pkg/html/template/#ErrorCode
(ErrPredefinedEscaper), the template engine already performs
sanitization equivalent to urlquery on any pipeline, and therefore makes
this extra urlquery call unnecessary, and even maybe harmful.

Change-Id: I7fcce395bf015b64022d1ac66b42069cdefb69eb
2017-10-04 18:41:32 +02:00
mpl 00bc12d403 misc/docker/go: update to go1.9
Change-Id: I3c0adb33918df1553ec8e935bca4d50f01816c06
2017-10-04 01:35:55 +02:00
mpl 52b139ea6c vendor: update Google Closure
To rev 37a4c36ce6286bb78bceb20579fecdfe7a759e02

Fix vendor/embed/closure/updatelibrary.go to now fetch from github.
Fix pkg/misc/closure/gendeps.go to work with new addDependency calls.

Fixes #903
Fixes #961

Change-Id: Ie555cf9bf5a8624845095fb3351482a690a2571c
2017-10-03 17:33:49 +02:00
mpl 5f0a625e76 server/camlistored: cross-browser scrollTop getter
It looks like our vendored version of goog.dom.getDocumentScrollElement().scrollTop
does not work anymore for all browsers, as infinite scrolling is broken
for some.

This change is a work-around while we're working on a cleaner fix, like
seeing if an updated closure lib works correctly.

Fixes #961

Change-Id: I713529ff796f50ff7cf9234befbf6abbfb75e3cb
2017-10-03 16:10:32 +02:00
mpl 683fa350f7 app/publisher: remove now disallowed html escaper from template
Apparently the html escaper was not being used in the right order/place,
and the auto-escaper does a better job than us at escaping?

For reference: https://go-review.googlesource.com/c/go/+/37880

I have noticed some odd results when trying other escaping combinations,
but that's a Go problem, if any, that I will investigate independently.

Fixes #960

Change-Id: I43e6eca26404d0bcbc7a48764393186f1d112dbc
2017-09-29 00:15:51 +02:00
mpl f7cbcc7812 app/publisher/js/zsearch.go: do not track in git
If we keep app/publisher/js/zsearch.go, when running make.go, it first gets mirrored into
CAMLI_ROOT/tmp/build-gopath-nosqlite/src/camlistore.org/app/publisher/js/zsearch.go
, with the same modtime as app/publisher/js/zsearch.go.

Then in genSearchTypes,
CAMLI_ROOT/tmp/build-gopath-nosqlite/src/camlistore.org/app/publisher/js/zsearch.go gets
regenerated anyway, because its modtime is older than that of
CAMLI_ROOT/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/search/describe.go

Then in a subsequent run of make.go, the mirroring overwrites the newly regenerated file,
because its modtime is different (indeed, it is newer) than that of
app/publisher/js/zsearch.go

And the cycle repeats ad vitam eternam.

I think if we added app/publisher/js/zsearch.go to mirrorIgnored, it
might fix the problem, but I don't see the point in even keeping
zearch.go in git at all, so I propose with this CL to simply remove it.

If removed, it is simply generated at the first make.go run, and never
again afterwards (except if pkg/search/describe.go changes).

Fixes #957

Change-Id: Ia6fcc50ce33513a2003809783fc323ab36a60b52
2017-09-22 00:10:51 +02:00
Mathieu Lonjaret be77d7a4cb Merge "website: add "ref" predicate in search-ui" 2017-09-21 14:39:03 +00:00
mpl f9093c6970 website: add "ref" predicate in search-ui
Also add anchor to each of the predicates

Change-Id: I9f785cdb5dbaf4c7947ee314570987c09dadc855
2017-09-21 16:37:25 +02:00
Brad Fitzpatrick 281d071a86 Merge "pkg/importer/allimporters: enable gphotos" 2017-09-21 01:08:00 +00:00
mpl a3bbb6ca3d pkg/importer/allimporters: enable gphotos
Change-Id: I4ea6d12c7d41d64e55654e9a539aaa62e9473d73
2017-09-21 01:41:19 +02:00
Sterchele Nicolas 2e46fca5cc Update camlistore and gopherjs to go1.9
Vendoring:
- update gopherjs to rev: b40cd48c38f9a18eb3db20d163bad78de12cf0b7
- as usual, replace the fsnotify import statements with our dummy
  pkg/misc/fakefsnotify, as described in
https://camlistore-review.googlesource.com/c/9409
- add vendor/golang.org/x/tools/go/gcimporter15 dependency from
  rev: e531a2a1c15f94033f6fa87666caeb19a688175f

- make.go + .travis.yml: make go1.9 the required version
- Update comments on doc/environments-vars.md for CAMLI_GOPHERJS_GOROOT

Fixes #948
Fixes #949

Change-Id: I9285aebadad1502708d164a9dc69e044a325cb3a
2017-09-20 08:23:11 +02:00
Ronald Evers 9ab1799c84 pkg/types/camtypes: fix a typo
Noticed this typo at http://localhost:3179/?camli.mode=config.

Change-Id: Ide8868fbef10dbb16f85ef53db02d4bf769cfd35
2017-09-17 17:05:43 +02:00
mpl 4a8e6f03b5 pkg/index: add simple integrity check
This change adds a check right after index initialization that
enumerates blobs for a few seconds and verifies that all of them are
indexed.

A warning is logged if any of the blobs are not found in the index.

Issue #947

Change-Id: Idc0df2121c1fb58e7560173b7753eaaddc4e653b
2017-09-15 15:52:58 +02:00
Paul Lindner fa46c3935d Correct various misspelled words
Change-Id: I236e880526e4c2b0bd318da041983d557e0aa885
2017-09-11 08:33:31 -07:00
mpl fdf50ad339 pkg/deploy/gce: update GCE API service name
Fixes #943

Change-Id: I676b76dc6b7e25bb5c418150aedf2766a1089ee8
2017-09-07 16:24:00 +02:00
Mathieu Lonjaret 4e6d89caf7 Merge "pkg/serverinit: catch typos in high-level server config" 2017-09-06 16:32:20 +00:00
mpl d982e37c4a pkg/serverinit: catch typos in high-level server config
Some errors such are missing fields, or wrong values, or conflicting
values were already caught logically when the high-level configuration
is transformed into the low-level configuration. Or later when
initializing handlers.

However, there was no mechanism to catch typos such as "httsCert"
instead of "httpsCert", or "leveldb" instead of "levelDB" in the
high-level configuration. Such errors would result in misconfiguration
(e.g. use of Let's Encrypt instead of the desired HTTPS certificate)
which can then even go unnoticed since the server still starts.

Therefore, this change generates a fake serverconfig.Config with all its
fields set, so that its JSON encoding results in a list of all the
possible configuration fields. This allows to compare the given
configuration fields with that list, and catch invalid names.

Change-Id: I4d6e61462353e52938db93ba332df2d52225e750
2017-09-06 18:19:14 +02:00
Mathieu Lonjaret 94a9d6a37c Merge "pkg/server: make DownloadHandler also handle directories when zipping" 2017-09-06 12:42:13 +00:00
mpl 5fef0785f2 pkg/server: make DownloadHandler also handle directories when zipping
The DownloadHandler only accepted file schemas as input for building a
zip archive so far.

It can now zip directories and their contents as well.

Non-regular files (socket, fifo, symlink) are now handled too.

As previously, no compression is applied when zipping.

When the DownloadHandler's Fetcher is a caching fetcher, all the files
that are supposed to be included in the archive are read, so we can
report reading errors even before starting to create the archive. We now
also take advantage of this optimization to build a
blobRef->filepath mapping when checking the files. Then, when the zip
archive is actually being built, the file path can be looked up in the
map, instead of having to assemble it again with recursive concatenation
of directory names.

Change-Id: I853c495798a9a43e12f3386603a70560ff46a237
2017-09-06 00:06:45 +02:00
mpl 1a830e5444 server/camlistored/ui: map aspect, add switch for clustering
controlled by the CAMLI_DEV_MAP_CLUSTERING env var

Change-Id: I7d790ca354e4c06b45833affb94337cb120ec7d3
2017-08-30 20:10:46 +02:00
mpl 109b101b33 pkg/types/camtypes: fix Expand isEmpty and tests
related to 93c6d682d2

Change-Id: Ib69cb6125113bafb530e5250b8cf3e20a7c2510f
2017-08-30 16:57:23 +02:00
Mathieu Lonjaret a5baf69602 Merge "vendor: upgrade to use myitcv.io/react" 2017-08-30 14:43:50 +00:00
Paul Jolly 4581abc238 vendor: upgrade to use myitcv.io/react
myitcv.io/react replaces github.com/myitcv/gopherjs/react.

This commits also includes relevant changes to reflect the new API of
myitcv.io/react:

https://github.com/myitcv/react/wiki/Changelog

myitcv.io/react
rev 0b147f16090bc5007116c494ca7ea95f78ee4a16

myitcv.io/gogenerate (required by myitcv.io/react/cmd/reactGen)
rev bd69a94c96953d20e106734856b69d71c8fa122b

github.com/gopherjs/jsbuiltin (required by myitcv.io/react)
rev 67703bfb044e3192fbcab025c3aeaeedafad1f2f

Change-Id: I1951519d30455915aa2f75fbb418057f8075e27c
2017-08-30 08:46:33 +01:00
mpl d717b58fd3 pkg/index: fix enumeration tests
related to 93c6d682d2

Change-Id: I2fa0df6da70df1297712ee0c9279a625b0ac88ca
2017-08-30 01:01:56 +02:00
mpl 8d4a554c3a pkg/search: fix bestByLocation test
related to 93c6d682d2

Change-Id: Ide7a120f501f4a140c039c0666b05f82efe510d2
2017-08-29 23:13:19 +02:00
mpl 088cdae82d server/camlistored/ui: map aspect improvements
-keep the browser URL bar in sync with the current search/zoom-level

-introduce the "map:" predicate, to be used as the current viewport in
the map aspect. This was previously achieved with the locrect predicate. We
try to keep this new predicate unknown to the server, and replaced on
the fly by an equivalent locrect predicate.

-stay on the map aspect when the search expression has a map predicate.
Same thing when loading a URL with a map predicate parameter: try to
load directly on map aspect, instead of going to search aspect first.

-make sure there's only at most one map query in flight at all times

Change-Id: Ibf928ee5c926670c221f212f14ca9799e1e7bb79
2017-08-29 18:13:07 +00:00
Brad Fitzpatrick b411c990f7 Merge "pkg/search: change corpus enumeration signatures for speed" 2017-08-29 18:12:58 +00:00
Brad Fitzpatrick 93c6d682d2 pkg/search: change corpus enumeration signatures for speed
Avoid select overhead in hot paths. Just use funcs.

Also, for sort-by-map searches, don't do a describe and pass over all
the results doing location lookups a second time. Remember the
location from the initial matching. Cache it on the search value.

Reduces some sort-by-map searches from 10 seconds to 3 seconds for
me. (still too slow, but good start)

Change-Id: I632954738df9accd802f28364ed11e48ddba0d14
2017-08-29 11:10:10 -07:00
mpl be17ff99a1 server/camlistored/ui: map aspect, initialize on world view
Change-Id: I7493a71653efe4552e90764e16d903465fa69fb1
2017-08-23 19:57:58 +02:00
mpl 13e170994d server/camlistored/ui: map aspect, remove local OSM proxy
It was accidentally submitted with
3d405cd9ac

Change-Id: Id722cc2da355af8a02824a40b2bb19d9788f83ea
2017-08-23 17:39:44 +02:00
Mathieu Lonjaret 5c9c433b5e Merge "pkg/index: ignore indexed NaN location" 2017-08-23 15:26:47 +00:00
mpl d72a8b3045 pkg/index: ignore indexed NaN location
We only started preventing NaNs from locations from being indexed at
ee13a3060b, so files indexed before
that could have introduced indexed NaNs, which we were not checking
against, until now.

Change-Id: I31fc8b9482cbd546591d553d7d8804700c7cf175
2017-08-23 17:19:28 +02:00
Mathieu Lonjaret 531b074ae4 Merge "server/camlistored/ui: map aspect, freeze marker cluster" 2017-08-23 14:35:01 +00:00
mpl 2aa72f1204 vendor/embed/leaflet: embed 2x markers
Change-Id: I93c5b462e1a9649ccc1973e7f3607c207358248d
2017-08-23 03:19:44 +02:00
mpl ccab088e61 vendor/embed/fontawesome: embed woff2 files as well
Change-Id: Idbbd099b5ee32b179d7b9c035e8477d0924bf5d9
2017-08-23 03:09:25 +02:00
mpl 3d405cd9ac server/camlistored/ui: map aspect, freeze marker cluster
Add a plugin to the markers cluster, strongly inspired (but trimmed, and
modified) from https://github.com/ghybs/Leaflet.MarkerCluster.Freezable
to freeze the markers cluster as it is on each zoom level.

Fixes #940

Change-Id: Id7f1ed8182a5bd701c6c47e02b144a67f9b69b36
2017-08-22 19:53:23 +02:00
Mathieu Lonjaret ee8960864b Merge "pkg/search: add "ref:" predicate support" 2017-08-21 16:47:00 +00:00
mpl d1d8075c00 server/camlistore/ui: map aspect, remove unused code
Change-Id: Iebcff716614796b37128be89ed220cd278ad640d
2017-08-21 16:57:52 +02:00
mpl f0d679edeb server/camlistored/ui: map aspect, actually delay redrawing
Change-Id: I303a122b5409c72f131dfc030954eec89a05d2fc
2017-08-19 01:42:47 +02:00
mpl 2d3b781103 pkg/search: add "ref:" predicate support
The "ref:prefix" search predicate is simply the equivalent of the

Constraint {
	BlobRefPrefix: prefix,
}

search constraint.

The "ref:prefix" search expression was already supported by the search
box of the web UI, but as opposed to (all, I think) other search
expressions, it was not supported server-side. Which means, it had to be
converted to a search Constraint as the above, before being sent in the
query.

This change therefore fixes this inconsistency. In addition, but
relatedly, since the map aspect relies on expressing the zoom-level as a
locrect expression, it is much simpler if the search query it uses only
has to be constructed from search expressions, and without search
constraints. So if we want to be able to support marking a single node
search with the map aspect, while dealing only with search expressions,
this change is necessary.

Fixes #939

Change-Id: Ia58f410198ecd1f7e0981321da370d687df3a120
2017-08-17 01:23:55 +02:00
Mathieu Lonjaret a7ec4646ca Merge changes I28c1184a,I0004c9ff
* changes:
  server/camlistored/ui: use marker cluster in map aspect
  server/camlistored/ui: on zooming/panning on map aspect, do new search
2017-08-16 22:56:59 +00:00