Commit Graph

1113 Commits

Author SHA1 Message Date
Brad Fitzpatrick 9ca28d644b Merge "pkg/serverinit: on GCE, reset name if instance name is in camlistore.net" 2017-12-31 02:04:05 +00:00
Govert Versluis 8548962dbe Add Azure blobserver support.
Fixes #425

Change-Id: I02bb29e6503bfef0894cbfde0c2a3304cf70c932
2017-12-29 12:39:49 -08:00
Daniel Heath 59ac14caa5 Vendor OpenSans (apache 2 licence)
Change-Id: Ie74b37efc007011b32810fd91c65b7750588a76e
2017-12-29 10:22:49 -08:00
Paul Lindner e97eae91dc all: lint fixes for "don't use leading k in Go names"
Change-Id: I8647c1dcc0535b1fc8dc0e1b1ca4623c23d6ac7c
2017-12-13 11:31:25 -08:00
Paul Lindner 15feaeb24c all: lint fixes for 'error strings should not be capitalized or end with punctuation or a newline'
Change-Id: I9c3766a51ac8be694ae76befff4b6fa9a85e34eb
2017-12-11 06:13:25 -08:00
Paul Lindner f28eb3cce8 Remove appengine server and the old python blobserver
Change-Id: I734f4621f77ce902050e9b37805e7f8912f75e97
2017-11-30 19:13:37 -08:00
Mathieu Lonjaret 7e9a0f9d42 Merge "camlistored/ui: reset new attribute row on navigation" 2017-11-29 20:44:28 +00:00
Paul Lindner 3e703921d1 vendor: update myitcv.io/react and react javascript library
Update myitcv.io/react to revision:
  bca7c66b77ed8a5b86fb77cff70914c4a7cc3ce5

Update react, react-dom to v15.6.2 based on instructions in
  https://reactjs.org/blog/2017/09/25/react-v15.6.2.html

Add vendor/embed/react/update.sh script to assist with future updates

Update mobile UI to fix problems documented here:
  https://reactjs.org/warnings/legacy-factories.html

Update mobile.html and debug_console.html to use minified js

Fixes #977

Change-Id: I15ea81822e4f85669b321d5b3eb8b169f534def7
2017-11-27 08:55:30 -08:00
Paul Lindner b09cd377d7 Switch to stdlib context from golang.org/x/net/context
This switches most usages of the pre-1.7 context library to use the
standard library.  Remaining usages are in:

  app/publisher/main.go
  pkg/fs/...

Change-Id: Ia74acc39499dcb39892342a2c9a2776537cf49f1
2017-11-26 01:12:26 -08:00
mpl 1d70d377cf camlistored/ui: reset new attribute row on navigation
When on the permanode details aspect, and navigating from one permanode
to the next, if some change had already been started on the new
attribute row, the change would "propagate" to the next permanode, which
is not desirable.

That is because the state of the new attribute row was not reset on
navigation, which this change does.

Fixes #923

Change-Id: I6dee8763b2c1fcac4b3592b21adc954eab9d0ffe
2017-11-23 17:50:15 +01:00
Mathieu Lonjaret 39cd6029e3 Merge "server/camlistored/ui: remove useless crossorigin and integrity for leaflet" 2017-10-08 03:05:18 +00:00
mpl b95aa162d7 server/camlistored/ui: remove useless crossorigin and integrity for leaflet
I had pasted them from an example when I started to work on the map
aspect, and forgot to look up their meaning/use afterwards.

Fixes #964

Change-Id: Ic92dcd9641a7fad49707c24f114b370f2c887828
2017-10-06 01:09:56 +02:00
mpl 4784b22aeb pkg/serverinit: on GCE, reset name if instance name is in camlistore.net
On GCE, on startup, we did not set the camliNetIP in the high-level
config if the camlistore-hostname instance var vas already set.

The camliNetIP presence in the config is the signal for the server
that it should be configured as part of camlistore.net (and in
particular that it should update the record for its name on the
camlistore.net DNS server). Part of this configuration is to set the
camlistore-hostname var for the instance.

Therefore, a server which had been configured once as described above,
would not, on a subsequent restart, behave as if part of camlistore.net
and would skip the related configuration code path. Unless the
camlistore-hostname var was manually wiped before restart.

As this manual step is not an obvious one, this CL changes the
initialization, so that if a camlistore-hostname var is set, but the
value is a subdomain of "camlistore.net", the var is treated as empty
and initialization proceeds as if with a new server.

Fixes #963

Change-Id: Iab70185d7b90ef7e70bb831d363ff9d525922e35
2017-10-04 17:36:02 +02:00
mpl 52b139ea6c vendor: update Google Closure
To rev 37a4c36ce6286bb78bceb20579fecdfe7a759e02

Fix vendor/embed/closure/updatelibrary.go to now fetch from github.
Fix pkg/misc/closure/gendeps.go to work with new addDependency calls.

Fixes #903
Fixes #961

Change-Id: Ie555cf9bf5a8624845095fb3351482a690a2571c
2017-10-03 17:33:49 +02:00
mpl 5f0a625e76 server/camlistored: cross-browser scrollTop getter
It looks like our vendored version of goog.dom.getDocumentScrollElement().scrollTop
does not work anymore for all browsers, as infinite scrolling is broken
for some.

This change is a work-around while we're working on a cleaner fix, like
seeing if an updated closure lib works correctly.

Fixes #961

Change-Id: I713529ff796f50ff7cf9234befbf6abbfb75e3cb
2017-10-03 16:10:32 +02:00
Mathieu Lonjaret 94a9d6a37c Merge "pkg/server: make DownloadHandler also handle directories when zipping" 2017-09-06 12:42:13 +00:00
mpl 5fef0785f2 pkg/server: make DownloadHandler also handle directories when zipping
The DownloadHandler only accepted file schemas as input for building a
zip archive so far.

It can now zip directories and their contents as well.

Non-regular files (socket, fifo, symlink) are now handled too.

As previously, no compression is applied when zipping.

When the DownloadHandler's Fetcher is a caching fetcher, all the files
that are supposed to be included in the archive are read, so we can
report reading errors even before starting to create the archive. We now
also take advantage of this optimization to build a
blobRef->filepath mapping when checking the files. Then, when the zip
archive is actually being built, the file path can be looked up in the
map, instead of having to assemble it again with recursive concatenation
of directory names.

Change-Id: I853c495798a9a43e12f3386603a70560ff46a237
2017-09-06 00:06:45 +02:00
mpl 1a830e5444 server/camlistored/ui: map aspect, add switch for clustering
controlled by the CAMLI_DEV_MAP_CLUSTERING env var

Change-Id: I7d790ca354e4c06b45833affb94337cb120ec7d3
2017-08-30 20:10:46 +02:00
Paul Jolly 4581abc238 vendor: upgrade to use myitcv.io/react
myitcv.io/react replaces github.com/myitcv/gopherjs/react.

This commits also includes relevant changes to reflect the new API of
myitcv.io/react:

https://github.com/myitcv/react/wiki/Changelog

myitcv.io/react
rev 0b147f16090bc5007116c494ca7ea95f78ee4a16

myitcv.io/gogenerate (required by myitcv.io/react/cmd/reactGen)
rev bd69a94c96953d20e106734856b69d71c8fa122b

github.com/gopherjs/jsbuiltin (required by myitcv.io/react)
rev 67703bfb044e3192fbcab025c3aeaeedafad1f2f

Change-Id: I1951519d30455915aa2f75fbb418057f8075e27c
2017-08-30 08:46:33 +01:00
mpl 088cdae82d server/camlistored/ui: map aspect improvements
-keep the browser URL bar in sync with the current search/zoom-level

-introduce the "map:" predicate, to be used as the current viewport in
the map aspect. This was previously achieved with the locrect predicate. We
try to keep this new predicate unknown to the server, and replaced on
the fly by an equivalent locrect predicate.

-stay on the map aspect when the search expression has a map predicate.
Same thing when loading a URL with a map predicate parameter: try to
load directly on map aspect, instead of going to search aspect first.

-make sure there's only at most one map query in flight at all times

Change-Id: Ibf928ee5c926670c221f212f14ca9799e1e7bb79
2017-08-29 18:13:07 +00:00
mpl be17ff99a1 server/camlistored/ui: map aspect, initialize on world view
Change-Id: I7493a71653efe4552e90764e16d903465fa69fb1
2017-08-23 19:57:58 +02:00
mpl 13e170994d server/camlistored/ui: map aspect, remove local OSM proxy
It was accidentally submitted with
3d405cd9ac

Change-Id: Id722cc2da355af8a02824a40b2bb19d9788f83ea
2017-08-23 17:39:44 +02:00
Mathieu Lonjaret 531b074ae4 Merge "server/camlistored/ui: map aspect, freeze marker cluster" 2017-08-23 14:35:01 +00:00
mpl 3d405cd9ac server/camlistored/ui: map aspect, freeze marker cluster
Add a plugin to the markers cluster, strongly inspired (but trimmed, and
modified) from https://github.com/ghybs/Leaflet.MarkerCluster.Freezable
to freeze the markers cluster as it is on each zoom level.

Fixes #940

Change-Id: Id7f1ed8182a5bd701c6c47e02b144a67f9b69b36
2017-08-22 19:53:23 +02:00
Mathieu Lonjaret ee8960864b Merge "pkg/search: add "ref:" predicate support" 2017-08-21 16:47:00 +00:00
mpl d1d8075c00 server/camlistore/ui: map aspect, remove unused code
Change-Id: Iebcff716614796b37128be89ed220cd278ad640d
2017-08-21 16:57:52 +02:00
mpl f0d679edeb server/camlistored/ui: map aspect, actually delay redrawing
Change-Id: I303a122b5409c72f131dfc030954eec89a05d2fc
2017-08-19 01:42:47 +02:00
mpl 2d3b781103 pkg/search: add "ref:" predicate support
The "ref:prefix" search predicate is simply the equivalent of the

Constraint {
	BlobRefPrefix: prefix,
}

search constraint.

The "ref:prefix" search expression was already supported by the search
box of the web UI, but as opposed to (all, I think) other search
expressions, it was not supported server-side. Which means, it had to be
converted to a search Constraint as the above, before being sent in the
query.

This change therefore fixes this inconsistency. In addition, but
relatedly, since the map aspect relies on expressing the zoom-level as a
locrect expression, it is much simpler if the search query it uses only
has to be constructed from search expressions, and without search
constraints. So if we want to be able to support marking a single node
search with the map aspect, while dealing only with search expressions,
this change is necessary.

Fixes #939

Change-Id: Ia58f410198ecd1f7e0981321da370d687df3a120
2017-08-17 01:23:55 +02:00
mpl 1bd8f09c44 server/camlistored/ui: use marker cluster in map aspect
Let the map aspect cluster markers by proximity, using the Marker
Clustering plugin for Leaflet.

Since we already do the work of spreading markers evenly server-side,
this plugin is a bit redundant.

However, it is still usefull as it allows us to request e.g. 1000 nodes,
and display a representation of all these nodes, without actually
displaying them all individually. This offers a work-around for the
bugs/glitches described in
https://github.com/camlistore/camlistore/issues/937

Even better, it allows us to re-enable the use of the awesome markers
that were introduced in e4b7db8274, but
that we had to disable because of the aforementioned glitches.

Also, added the 2x awesome marker images that the leaflet awesome markers
plugin uses for retina/mobile displays.

https://github.com/Leaflet/Leaflet.markercluster vendored in
vendor/embed/leaflet at version 1.0.6

Related: issue #947

Change-Id: I28c1184ad171af5eca953bb05f4a943d3573bfb4
2017-08-17 00:56:04 +02:00
mpl 7832801040 server/camlistored/ui: on zooming/panning on map aspect, do new search
This change makes zooming and panning on the map aspect send a new
"MapSort" search query, so that the (1000, by default) most relevant
results for the currently displayed area always appear as markers
after a zoom/pan.

This required completely "lifting up" (in React lingo) the currentSearch
state out of the Header class, and into the Index class, which should
have already been done in a56372830a. This
is necessary because we need the expression in the search box to be mutable
both by the user('s input) and by the map aspect, which represents the
zoom level as a trailing "locrect" predicate in the search expression.

Fixes #938

Change-Id: I0004c9ff09f03b4f1d95a35e54605689eebf0c1a
2017-08-16 21:39:46 +02:00
mpl a56372830a server/camlistored/ui: prepare for map aspect zoom level
This change adds the logic to:
1) once a map aspect is loaded, get the area that includes all the
displayed results (the "zoom level")
2) modify the current query search predicate so that it includes the
above zoom-level as a locrect predicate
3) propagate that predicate change to the predicate that is displayed in
the search box, through the newly introduced currentSearch parameter
(a state parameter of the Index instance).

Change-Id: I955e6195ba71b2000c68c28278a534cec87f5311
2017-08-03 17:38:16 +02:00
mpl 49b97b2244 server/camlistored/ui: use search.MapSort in map aspect
Also make use of the new "locrect" search predicate, which allows us to
drop support for raw query with coordinates.

Also revert for now to using the native leaflet markers instead of the
AwesomeMarkers icons, as something in their style seems to be interfering
badly with the UI.

Fixes #934

Change-Id: Idd85eeb42673c43a216bce81c26a6bff5ad0c08a
2017-08-01 02:04:32 +02:00
Tilman Dilo 92b866d819 server/camlistored: listen for SIGTERM
Systemd by default sends SIGTERM to child processes when shutting down
or restarting a service, which is not handled by camlistored and results
in a hard kill.

This change makes camlistored listen for SIGTERM next to SIGINT to allow
graceful shutdowns. The log message for SIGHUP is adjusted for
consistency.

Change-Id: Ia9be84b0b2284343e1776042fdfc99363110bfb1
2017-07-11 00:07:06 +02:00
mpl e4b7db8274 server/camlistored/ui: improve map aspect search and markers
Notably:

-do not load any markers on an empty search query, because that would
mean loading absolutely all of the items with a location, which seems
like a bad idea.

-use different markers for different nodes. For now, foursquare
checkins, file images, and files have their own marker.

-vendor in https://github.com/lvoogdt/Leaflet.awesome-markers to achieve
the above, which relies on Font Awesome, which we already have in.
icons available for the markers: http://fontawesome.io/icons/

-when no location can be inferred from the search query, set the view to
encompass all markers that were drawn.

-when a location search is known, draw a rectangle representing the
results zone.

-use thumber for image in marker popup

-use title, if possible, instead of blobRef for link text in marker
popup

-switch to directly using OpenStreetMap tiles, instead of MapBox ones.

https://storage.googleapis.com/camlistore-screenshots/Screenshot_20170622-232359.png

Change-Id: Ibc84fa988aea8b8d3a2588ee8790adf6d9b5ad7a
2017-07-06 01:03:03 +02:00
Mathieu Lonjaret 2c5c524412 Merge "server/camlistored/ui: add map aspect" 2017-06-20 22:23:10 +00:00
mpl d2ced13e5e server/camlistored/ui: add map aspect
To display permanodes with a location on a map graphical interface (like
Google Maps, Open Street Map, etc).

Screenshots:
https://storage.googleapis.com/camlistore-screenshots/Screenshot_20170620-230922.png
https://storage.googleapis.com/camlistore-screenshots/Screenshot_20170620-231221.png
https://storage.googleapis.com/camlistore-screenshots/Screenshot_20170620-235006.png

Lots more features possible, but I think this is a good point to add it
as an mvp. TODOs in the code.

Using leaflet.js version 1.0.3, and tiles from mapbox.com (Open Street
Map).

Change-Id: Ib31610c122de633c5268174928c22c1cc481f388
2017-06-21 00:21:02 +02:00
mpl 04afc9ae8a pkg/server: add recovery option on web UI and instance page
When the blobpacked index gets corrupted/destroyed, it needs to be
rebuilt and hence camlistored refuses to start without the -recovery
flag being set if it detects it is needed.

This is a problem on GCE, because camlistored is handled by systemd, and
so the only way to restart camlistored with -recovery is to edit the
systemd service, then issue 'systemctl daemon-reload', and 'systemctl
restart camlistored'. Then revert the change for the next time
camlistored restarts. This is not user-friendly.

This change adds a check on camlistored startup based on the presence
and value (as a boolean) of the "camlistore-recovery" key as an instance
attribute. Therefore, the user only has to add the
("camlistore-recovery", "true") attribute in the Custom metadata of
their Google Cloud instance page to switch to recovery mode. (And
restart the instance).

As an additional feature (for non-GCE users), this change also adds the
option to restart the server in recovery mode from the status page of
the web UI.

Change-Id: I44f5ca293ddd0a0033fc5d9c2edca1bac0ee9c8f
2017-06-07 23:00:50 +02:00
mpl 8fa2ec5a06 camlistored/ui: refresh blob and permanode aspects state on navigation
Fixes #922

Change-Id: I02c3c01947f7ab29681fb96e69e6b1f17fa7b067
2017-04-28 19:35:22 +02:00
mpl 86679899f5 camlistored/ui: fix share URL dialog for all pages
The code to find the prefix URL that is used to display the share URL on
after successfully sharing an item, worked only on the main web UI page.
This change makes it work on hopefully any page.

Change-Id: I05e085f91ef7ebe39f880104e52ca487b80c607a
2017-04-27 19:12:10 +02:00
mpl dace68256b camlistored/ui: connect temp form to DOM before submitting
The download action of the web UI uses an ephemeral form, that we do not
connect to the DOM, behind the scenes. However, according to
https://html.spec.whatwg.org/multipage/forms.html#form-submission-algorithm,
step 2., a form must be connected to the DOM in order for it to be
submitted.

This change connects the form right before submitting, and removes it
from the DOM right after submission.

Fixes #921

Change-Id: I4d993c633c60a369ce4260f84731a64e8b955c76
2017-04-26 17:59:29 +02:00
Mathieu Lonjaret 9095ea9a00 Merge "server/camlistored/ui: enable multi-items sharing" 2017-04-17 15:55:39 +00:00
mpl 9e34d14ef5 pkg/serverinit: enable sharing on GCE by default
The default server config on GCE (as deployed by the launcher) did not
have a share handler. This CL adds one, so that users can benefit from
the new sharing feature from the web UI.

Also, the button for sharing from the web UI does not appear anymore if
the config that the web UI gets from the discovery does not have a
"shareRoot", because it's a strong hint that the server does not have a
share handler.

Change-Id: I6c444995339fda8dba864b1d6729fb7c1b6d72bd
2017-04-05 16:17:46 +02:00
mpl 0e55a06f5c server/camlistored/ui: enable multi-items sharing
Sharing through the web UI was limited to a selection of one (permanode)
file or directory.

This change enables sharing of all the selected items, as long as they
are (permanodes of) files or directories, by creating a new directory
blob containing the selected items. The new directory is the item
getting shared. For clarity, and hopefully helping with search later,
the new directory is named "shared-YYYYMMDDhhmmss".

Change-Id: Iabd2266c9af85f3e8fbfa56ce98f653c0ce0f1ee
2017-03-31 01:22:52 +02:00
mpl 77dfe8046d server/camlistored/ui: add "Share" feature
To share a file or directory through Camlistore's sharing
mechanism (https://camlistore.org/doc/sharing).

Only single-item sharing supported for now.

On success, the share URL is displayed in a dialog.
If the item is a file, it can be fetched with any HTTP client through
that URL. If the item is a dir, the share URL is of the claim, so one
must follow the transitive chain of sharing to get the directory's
children, i.e. use camget -shared.

Fixes #880

Change-Id: I324a684dec10225e5b8314dda7f1b77a0de6f727
2017-03-23 00:44:18 +01:00
mpl 71090f7c80 pkg/server: add files "zipper" to DownloadHandler
We want to add a feature for clients (the web UI), where they can select
a bunch of files and ask the server for a zip archive of all these files.

This CL modifies the DownloadHandler so it does exactly that upon
reception of a POST request with a query parameter of the form
files=sha1-foo,sha1-bar,sha1-baz

This CL also adds a new button to the contextual sidebar of the web UI,
that takes care of sending the download request to the server.

known limitations: only permanodes with file as camliContent are
accepted as a valid selection (i.e. no sets, or static-dirs, etc) for
now.

Implementation detail:
We're creating an ephemeral DOM form on the fly to send the request.
The reason is: if we sent it as a Go http request, we'd have to read
the response manually and then we'd have no way of writing it to disk.
If we did it with an xhr, we could write the response to disk by
creating a File or Blob and then using URL.createObjectURL(), but we'd
have to keep the response in memory while doing so, which is
unacceptable for large enough archives.

Fixes #899

Change-Id: I104f7c5bd10ab3369e28d33752380dd12b5b3e6b
2017-03-17 00:59:19 +01:00
mpl ceb5ea5b10 web UI: add "About" menu entry and dialog
This change adds an "About" entry to the header menu, that pops up an
"About" dialog when clicked on.

But, under this innocent sounding feature is actually a more important
change: this feature is partly added through the use of gopherjs
generated code. In addition, the element added (a menu item) is
integrated
seamlessly into the menu as a React element, thanks to the use of
github.com/myitcv/gopherjs/react.

pkg/env had to be broken with build tags, because importing
"cloud.google.com/go/compute/metadata" (or one of its deps) seems to be
breaking the generated javascript.
https://github.com/camlistore/camlistore/issues/904

github.com/myitcv/gopherjs/react at rev
c04b811da4a086defd882a94cc1901da2d2158b0

honnef.co/go/js/dom at rev 24aa052bc5c63cfb9383bf59493ee48621ca788c

Issue #904
Fixes #798

Change-Id: Ic09b94014d520277f8178727293787ece468babd
2017-03-07 00:24:07 +01:00
Mathieu Lonjaret a3d7e6183b Merge "camlistored/ui: remove deprecated getDOMNode calls" 2017-02-14 22:48:34 +00:00
mpl f4fc8921cd camlistored/ui: remove deprecated getDOMNode calls
getDOMNode is deprecated:
https://facebook.github.io/react/blog/2015/10/07/react-v0.14.html#dom-node-refs

Also fix a couple of component instanciations that still didn't use React.createElement.

Change-Id: Ic15d353be3815796d0aa6a3591c215056805a5bf
2017-02-13 18:12:49 +01:00
Steve Armstrong b2fb893b48 ui: Fix js error on all search queries
React would bail out with:

    Uncaught TypeError: this.refs.searchbox.getDOMNode is not a function
      at Object.getSearchNode_ (http://localhost:3179/ui/header.js:321:33)

Change-Id: I26e3474ed08c9339eda2e78de87b458a4130c746
2017-02-12 22:47:53 -08:00
mpl 4d12c68989 camlistored: do not use syslog on windows
Change-Id: I351c45739c517c107a5f16fee4298af75086d81d
2017-02-08 02:16:21 +01:00