Notably: pkg/misc all moves.
And pkg/googlestorage is deleted, since it's not used. Only the
x/net/http2/h2demo code used to use it, but that ended in
https://go-review.googlesource.com/33230 (our vendored code is old).
So just nuke that dir for now. When it's refreshed, it'll either be
gone (dep prune) or new enough to not need googlestorage.
Also move pkg/pools, pkg/leak, and pkg/geocode to internal.
More remains.
Change-Id: I2640c4d18424062fdb8461ba451f1ce26719ae9d
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
This switches most usages of the pre-1.7 context library to use the
standard library. Remaining usages are in:
app/publisher/main.go
pkg/fs/...
Change-Id: Ia74acc39499dcb39892342a2c9a2776537cf49f1
to Rev 48b2ede4844e13f1a2b7ce4d2529c9af7e359fc5
The qr package has moved from code.google.com/p/rsc/qr to it's new
canonical home at rsc.io/qr
Change-Id: Ibb04ee7e83c9707ff253a91abb4f60f9b150d61c
To rev 9dfe39835686865bff950a07b394c12a98ddc811
The xsrftoken package now lives in golang.org/x/net instead of code.google.com.
Change-Id: I4d98b1e50099dc7a1e1188f5c4311cd28c79f44a
The DownloadHandler only accepted file schemas as input for building a
zip archive so far.
It can now zip directories and their contents as well.
Non-regular files (socket, fifo, symlink) are now handled too.
As previously, no compression is applied when zipping.
When the DownloadHandler's Fetcher is a caching fetcher, all the files
that are supposed to be included in the archive are read, so we can
report reading errors even before starting to create the archive. We now
also take advantage of this optimization to build a
blobRef->filepath mapping when checking the files. Then, when the zip
archive is actually being built, the file path can be looked up in the
map, instead of having to assemble it again with recursive concatenation
of directory names.
Change-Id: I853c495798a9a43e12f3386603a70560ff46a237
Notably:
-do not load any markers on an empty search query, because that would
mean loading absolutely all of the items with a location, which seems
like a bad idea.
-use different markers for different nodes. For now, foursquare
checkins, file images, and files have their own marker.
-vendor in https://github.com/lvoogdt/Leaflet.awesome-markers to achieve
the above, which relies on Font Awesome, which we already have in.
icons available for the markers: http://fontawesome.io/icons/
-when no location can be inferred from the search query, set the view to
encompass all markers that were drawn.
-when a location search is known, draw a rectangle representing the
results zone.
-use thumber for image in marker popup
-use title, if possible, instead of blobRef for link text in marker
popup
-switch to directly using OpenStreetMap tiles, instead of MapBox ones.
https://storage.googleapis.com/camlistore-screenshots/Screenshot_20170622-232359.png
Change-Id: Ibc84fa988aea8b8d3a2588ee8790adf6d9b5ad7a
When the blobpacked index gets corrupted/destroyed, it needs to be
rebuilt and hence camlistored refuses to start without the -recovery
flag being set if it detects it is needed.
This is a problem on GCE, because camlistored is handled by systemd, and
so the only way to restart camlistored with -recovery is to edit the
systemd service, then issue 'systemctl daemon-reload', and 'systemctl
restart camlistored'. Then revert the change for the next time
camlistored restarts. This is not user-friendly.
This change adds a check on camlistored startup based on the presence
and value (as a boolean) of the "camlistore-recovery" key as an instance
attribute. Therefore, the user only has to add the
("camlistore-recovery", "true") attribute in the Custom metadata of
their Google Cloud instance page to switch to recovery mode. (And
restart the instance).
As an additional feature (for non-GCE users), this change also adds the
option to restart the server in recovery mode from the status page of
the web UI.
Change-Id: I44f5ca293ddd0a0033fc5d9c2edca1bac0ee9c8f
For the DownloadHandler, this change alleviates confusion about whether
the name from the fileInfo should be cleaned out of slashes and such
before being zip archived.
Change-Id: I3ccbd760a2c08787d4ee4e22d8dc8ed066d2f82f
In DownloadHandler, change zh.SetModTime(fi.modtime.UTC()) to
zh.SetModTime(fi.modtime) because SetModTime already converts to UTC.
Change-Id: I70457b310fa44faa823e04badee91334c8e1eca4
When the DownloadHandler handles a zip archive requests, it writes the
zip archive being built directly to the network response, in order to
avoid writing any of the files to disk or to memory. As a consequence,
as soon as the archive starts being built, if any error reading one of
the files occurs, it can't properly report it to the client as the
response had already started being sent.
In this change, if the DownloadHandler's Fetcher is a caching fetcher,
we try reading all the requested files before starting the zip archive.
As a result, we can error early and properly if any of the files can't
be read. And since the blobs get cached as they are read the first time,
reading them a second time when actually building the archive should not
be too costly.
Change-Id: I6477d82149b08b1db1471ca9ad77fef254929db0
Services often have a free download quota, which we can use to fetch
some random blobs, to compare with the source.
Change-Id: I526e62580af4cae7004d756721336d01c2f020c4
Since we don't actually ever remove blobs (until we add garbage
collection, and even then), if a share claim gets deleted (with a delete
claim), the only knowledge of the deletion resides in the Index.
So when the share handler verifies the sharing chain, there is nothing
preventing it from reading a supposedly deleted share claim (from the
blobserver), and concluding that the share chain is valid.
This change adds the index handler to the share handler, so it can check
the deletion status of a share claim, and hence support "share
cancellation".
Fixes#914
Change-Id: I572fdddee30e745aa2d2a6720c83c8e8c916515d
We want to add a feature for clients (the web UI), where they can select
a bunch of files and ask the server for a zip archive of all these files.
This CL modifies the DownloadHandler so it does exactly that upon
reception of a POST request with a query parameter of the form
files=sha1-foo,sha1-bar,sha1-baz
This CL also adds a new button to the contextual sidebar of the web UI,
that takes care of sending the download request to the server.
known limitations: only permanodes with file as camliContent are
accepted as a valid selection (i.e. no sets, or static-dirs, etc) for
now.
Implementation detail:
We're creating an ephemeral DOM form on the fly to send the request.
The reason is: if we sent it as a Go http request, we'd have to read
the response manually and then we'd have no way of writing it to disk.
If we did it with an xhr, we could write the response to disk by
creating a File or Blob and then using URL.createObjectURL(), but we'd
have to keep the response in memory while doing so, which is
unacceptable for large enough archives.
Fixes#899
Change-Id: I104f7c5bd10ab3369e28d33752380dd12b5b3e6b
Minimizing redundant info when setting up an app:
* prefix - Now optional and defaults to the handler prefix the whole app is attached to
* serverListen - Defaults to the host part of serverBaseURL
Change-Id: I395e71c54b9e81849681fe406ff1b5f412e92c9c
This change adds an "About" entry to the header menu, that pops up an
"About" dialog when clicked on.
But, under this innocent sounding feature is actually a more important
change: this feature is partly added through the use of gopherjs
generated code. In addition, the element added (a menu item) is
integrated
seamlessly into the menu as a React element, thanks to the use of
github.com/myitcv/gopherjs/react.
pkg/env had to be broken with build tags, because importing
"cloud.google.com/go/compute/metadata" (or one of its deps) seems to be
breaking the generated javascript.
https://github.com/camlistore/camlistore/issues/904
github.com/myitcv/gopherjs/react at rev
c04b811da4a086defd882a94cc1901da2d2158b0
honnef.co/go/js/dom at rev 24aa052bc5c63cfb9383bf59493ee48621ca788c
Issue #904Fixes#798
Change-Id: Ic09b94014d520277f8178727293787ece468babd
WARNING: this app is still experimental, and even its data schema might
change. Do not use in production.
This change adds a Camlistore-based port of the scanning cabinet app
originally created by Brad Fitzpatrick:
https://github.com/bradfitz/scanningcabinet
Some of it is inspired from the App Engine Go port of Patrick Borgeest:
https://bitbucket.org/pborgeest/nometicland
The data schema is roughly as follows:
-a scan is a permanode, with the node type: "scanningcabinet:scan".
-a scan's camliContent attribute is set to the actual image file.
-a scan also holds the "dateCreated" attribute, as well as the
"document" attribute, which references the document this scan is a part
of (if any).
-a document is a permanode, with the node type: "scanningcabinet:doc".
-a document page, is modeled by the "camliPath:sha1-xxx" = "pageNumber"
relation, where sha1-xxx is the blobRef of a scan.
-a document can also hold the following attributes: "dateCreated",
"tag", "locationText", "title", "startDate", and "paymentDueDate".
Known caveats, in decreasing order of concern:
-the data schema might still change.
-the scancab tool, to actually create and upload the files from physical
documents, is practically untested (since I do not own a scanner).
-some parts, in particular related to searches, are probably
sub-optimized.
-the usual unavoidable bugs.
Change-Id: If6afc509e13f7c21164a3abd276fec075a3813bb
The main motivation is that we want to use gopherjs +
github.com/myitcv/gopherjs/react , which requires a recent react.
This CL also fully switches to only using minified js files.
The changes in server/camlistored/ui are the result of some react
API changes, such as:
- wraps component instantiation in React.createElement
- React.classSet -> classNames
- Need tbody between table and tr in a couple places
- component array elts need unique keys (longstanding warning)
- change root elt from body to a child div (to fix warning)
- don't check isMounted in container render function
- fix a bunch of style keys using kebab-case instead of camelCase
Change-Id: Idc22d07704688ccde3e4f67d4f188a5ad32b6e32
When reindexing on a (My)SQL based sorted.KeyValue, we should recreate
the database schema from scratch, which means dropping the tables.
However, index.Reindex just calls Wipe on the newly created
sorted.KeyValue, which only deletes the rows, and does not drop the
tables.
Therefore, this CL changes the implementation of Wipe in the MySQL case,
so that it takes care of dropping the tables, and doing everything that
needs to be done afterwards to set up the sorted.KeyValue.
In addition, with the introduction of the sorted.NeedWipeError, we detect
upon initialization of a sorted.KeyValue if it failed because it needed
a schema update. If that is the case, and we're in reindex mode, we can
fix the sorted.KeyValue with a Wipe and carry on.
Finally, we introduce the new sorted.NewKeyValueMaybeWipe function that
automatically wipes a KeyValue when a NeedWipeError was returned upon
its creation.
Next, do the same with other sorted SQLs.
Fixes#806
Change-Id: I2032781cbf453a364880bd3e2e8b3c09aac7aed9
rm google.golang.org/cloud
add cloud.google.com/go at a47b182e769f5e75f5fc927ff6ee2678f7f552cf
update google.golang.org/api to 63cb68f1e3834e44683ca062ddf06cb9a889380a
update google.golang.org/grpc to
0e6ec3a4501ee9ee2d023abe92e436fd04ed4081
update go4.org to f5283521d7365fb2875408726e9cbf349f173767
fix in cmd/ pkg/ server/
TODO(mpl): fix misc/docker tools as well. next CL.
Fixes#832
Change-Id: I842b968a0afea8a5822913bd614d67cdbe50ee63
To decide whether a search submitted to the app search proxy is allowed,
we compare its results to the domain blobs, result of the master query,
that we cache when the master query is set.
However, since the results of the master query are liable to change when
new blobs arrive (e.g. a new camliMember is added to the set that is
published), that cache may need to be invalidated. Otherwise, we might
reply with a 403 to search query that is actually allowed.
Therefore, this CL adds a refresh of the cache on two instances:
-When the app handler gets a search query that seems to be forbidden.
Before replying with a 403, we refresh the cache with the master query,
and recheck whether the search query is allowed.
-When the publisher gets a request for a "members" page, or the "file"
page, it preemptively asks the app handler to refresh. Now that a lot of
the client workflow has been moved to javascript/the browser, these
kinds of requests should not happen too often, so it seems a reasonable
place to ask for a refresh. But this might change, so we should of
course be careful not to flood the app handler with refresh requests in
the future.
In any case, the app handler is suppressing the refresh requests, so
that it does not perform refreshes at more that one per minute.
As a smarter approach, we could later imagine a way for the app handler
to be aware of when new blobs get to the blobserver (akin to the blob
hub that the sync handler uses?), so that it only ever refreshes when
needed.
Fixes#851
Change-Id: Idc14cce5018053deac01ec454e5c936ed93e5a05
Since the app handler should not trim the r.URL.Path of the handler's
prefix, it is now the responsibility of the app to cope with that
prefix.
Fixes#833
Change-Id: Ie1fa9801b26767c3e3b6612498380261e22cdf07
Some of the publisher features have moved from the server-side app to
the client-side app (the browser) thanks to gopherjs. Some of these
features imply doing some search queries against Camlistore, which
requires authentication. The server-side app receives the necessary
credentials on creation, from Camlistore. However, we can't just
communicate them to the client-side (as we do with the web UI) since the
publisher app itself does not require any auth and is supposed to be
exposed to the world.
Therefore, we need to allow some search queries to be done without
authentication.
To this end, the app handler on Camlistore now assumes a new role: it is
also a search proxy for the app. The app sends an unauthenticated search
query to the app handler (instead of directly to the search handler),
and it is the role of the app handler to verify that this query is
allowed for the app, and if yes, to forward the search to the Camlistore's
search handler.
We introduce a new mechanism to filter the search queries in the form of
a master query. Upon startup, the publisher registers, using the new
CAMLI_APP_MASTERQUERY_URL env var, a *search.SearchQuery with the app
handler. The app handler runs that query and caches all the blob refs
included in the response to that query. In the following, all incoming
search queries are run by the app handler, which checks that none of the
response blobs are out of the set defined by the aforementioned cached
blob refs. If that check fails, the search response is not forwarded to
the app/client.
The process can be improved in a subsequent CL (or patchset), with finer
grained domains, i.e. a master search query per published camliPath,
instead of one for the whole app handler.
Change-Id: I00d91ff73e0cbe78744bfae9878077dc3a8521f4
These improvements on the server app handler should help writing
and running stand-alone apps.
The two main goals are:
1) "simple" configurations should work automatically; the parameters for
the app are derived from the Listen and BaseURL of the Camlistore
server.
2) More advanced configurations, such as being behind a proxy, should be
easily configurable through the app's Listen, BackendURL, and ApiHost
parameters.
I had worked on them while doing the scanning cabinet app, and I am
backporting them now since we haven't landed the scanning cabinet yet,
and people have been having trouble setting up the publisher.
pkg/app/app_test.go is gone because app.ListenAddress is now dumb. The
hard work is done in pkg/server/app instead.
Fixes#818
Change-Id: Ice2610d6bac611b209cc3a928e67fa6093a41d3e
This allows deferring HTTP authentication and authorization to a proxy
(such as nginx) between Camlistore and the rest of the world, without
breaking the web UI.
Fixes#816
Change-Id: Ia4b5be8f2236ddac68dc0d3a09f0f24e588c4995
There were basically two bugs in bytesHaveSchemaLink:
-typo "blob" vs "bytes" camliType. this wouldn't happen if we used
constants from a pkg for camliTypes, instead of hardcoding them..
-returning early instead of checking all the parts of a file.
Added a test for each of those bugs. They look like they could be
refactored but they're subtly different enough that it might not be
worth it.
Context: working on issue #527
Change-Id: Ifd0bd311ae8a115de2a56994b89127db6f315a75
This lets the UI work (and lets people hack on the UI) by installing
camlistored the normal Go way (go install) and just running
camlistored, without using devcam server.
Change-Id: I45779640c5b2286ca7061a626c07b3edcdc9c4ad
Brad Fitzpatrick
