Services often have a free download quota, which we can use to fetch
some random blobs, to compare with the source.
Change-Id: I526e62580af4cae7004d756721336d01c2f020c4
Since we don't actually ever remove blobs (until we add garbage
collection, and even then), if a share claim gets deleted (with a delete
claim), the only knowledge of the deletion resides in the Index.
So when the share handler verifies the sharing chain, there is nothing
preventing it from reading a supposedly deleted share claim (from the
blobserver), and concluding that the share chain is valid.
This change adds the index handler to the share handler, so it can check
the deletion status of a share claim, and hence support "share
cancellation".
Fixes#914
Change-Id: I572fdddee30e745aa2d2a6720c83c8e8c916515d
We want to add a feature for clients (the web UI), where they can select
a bunch of files and ask the server for a zip archive of all these files.
This CL modifies the DownloadHandler so it does exactly that upon
reception of a POST request with a query parameter of the form
files=sha1-foo,sha1-bar,sha1-baz
This CL also adds a new button to the contextual sidebar of the web UI,
that takes care of sending the download request to the server.
known limitations: only permanodes with file as camliContent are
accepted as a valid selection (i.e. no sets, or static-dirs, etc) for
now.
Implementation detail:
We're creating an ephemeral DOM form on the fly to send the request.
The reason is: if we sent it as a Go http request, we'd have to read
the response manually and then we'd have no way of writing it to disk.
If we did it with an xhr, we could write the response to disk by
creating a File or Blob and then using URL.createObjectURL(), but we'd
have to keep the response in memory while doing so, which is
unacceptable for large enough archives.
Fixes#899
Change-Id: I104f7c5bd10ab3369e28d33752380dd12b5b3e6b
Minimizing redundant info when setting up an app:
* prefix - Now optional and defaults to the handler prefix the whole app is attached to
* serverListen - Defaults to the host part of serverBaseURL
Change-Id: I395e71c54b9e81849681fe406ff1b5f412e92c9c
This change adds an "About" entry to the header menu, that pops up an
"About" dialog when clicked on.
But, under this innocent sounding feature is actually a more important
change: this feature is partly added through the use of gopherjs
generated code. In addition, the element added (a menu item) is
integrated
seamlessly into the menu as a React element, thanks to the use of
github.com/myitcv/gopherjs/react.
pkg/env had to be broken with build tags, because importing
"cloud.google.com/go/compute/metadata" (or one of its deps) seems to be
breaking the generated javascript.
https://github.com/camlistore/camlistore/issues/904
github.com/myitcv/gopherjs/react at rev
c04b811da4a086defd882a94cc1901da2d2158b0
honnef.co/go/js/dom at rev 24aa052bc5c63cfb9383bf59493ee48621ca788c
Issue #904Fixes#798
Change-Id: Ic09b94014d520277f8178727293787ece468babd
WARNING: this app is still experimental, and even its data schema might
change. Do not use in production.
This change adds a Camlistore-based port of the scanning cabinet app
originally created by Brad Fitzpatrick:
https://github.com/bradfitz/scanningcabinet
Some of it is inspired from the App Engine Go port of Patrick Borgeest:
https://bitbucket.org/pborgeest/nometicland
The data schema is roughly as follows:
-a scan is a permanode, with the node type: "scanningcabinet:scan".
-a scan's camliContent attribute is set to the actual image file.
-a scan also holds the "dateCreated" attribute, as well as the
"document" attribute, which references the document this scan is a part
of (if any).
-a document is a permanode, with the node type: "scanningcabinet:doc".
-a document page, is modeled by the "camliPath:sha1-xxx" = "pageNumber"
relation, where sha1-xxx is the blobRef of a scan.
-a document can also hold the following attributes: "dateCreated",
"tag", "locationText", "title", "startDate", and "paymentDueDate".
Known caveats, in decreasing order of concern:
-the data schema might still change.
-the scancab tool, to actually create and upload the files from physical
documents, is practically untested (since I do not own a scanner).
-some parts, in particular related to searches, are probably
sub-optimized.
-the usual unavoidable bugs.
Change-Id: If6afc509e13f7c21164a3abd276fec075a3813bb
The main motivation is that we want to use gopherjs +
github.com/myitcv/gopherjs/react , which requires a recent react.
This CL also fully switches to only using minified js files.
The changes in server/camlistored/ui are the result of some react
API changes, such as:
- wraps component instantiation in React.createElement
- React.classSet -> classNames
- Need tbody between table and tr in a couple places
- component array elts need unique keys (longstanding warning)
- change root elt from body to a child div (to fix warning)
- don't check isMounted in container render function
- fix a bunch of style keys using kebab-case instead of camelCase
Change-Id: Idc22d07704688ccde3e4f67d4f188a5ad32b6e32
When reindexing on a (My)SQL based sorted.KeyValue, we should recreate
the database schema from scratch, which means dropping the tables.
However, index.Reindex just calls Wipe on the newly created
sorted.KeyValue, which only deletes the rows, and does not drop the
tables.
Therefore, this CL changes the implementation of Wipe in the MySQL case,
so that it takes care of dropping the tables, and doing everything that
needs to be done afterwards to set up the sorted.KeyValue.
In addition, with the introduction of the sorted.NeedWipeError, we detect
upon initialization of a sorted.KeyValue if it failed because it needed
a schema update. If that is the case, and we're in reindex mode, we can
fix the sorted.KeyValue with a Wipe and carry on.
Finally, we introduce the new sorted.NewKeyValueMaybeWipe function that
automatically wipes a KeyValue when a NeedWipeError was returned upon
its creation.
Next, do the same with other sorted SQLs.
Fixes#806
Change-Id: I2032781cbf453a364880bd3e2e8b3c09aac7aed9
rm google.golang.org/cloud
add cloud.google.com/go at a47b182e769f5e75f5fc927ff6ee2678f7f552cf
update google.golang.org/api to 63cb68f1e3834e44683ca062ddf06cb9a889380a
update google.golang.org/grpc to
0e6ec3a4501ee9ee2d023abe92e436fd04ed4081
update go4.org to f5283521d7365fb2875408726e9cbf349f173767
fix in cmd/ pkg/ server/
TODO(mpl): fix misc/docker tools as well. next CL.
Fixes#832
Change-Id: I842b968a0afea8a5822913bd614d67cdbe50ee63
To decide whether a search submitted to the app search proxy is allowed,
we compare its results to the domain blobs, result of the master query,
that we cache when the master query is set.
However, since the results of the master query are liable to change when
new blobs arrive (e.g. a new camliMember is added to the set that is
published), that cache may need to be invalidated. Otherwise, we might
reply with a 403 to search query that is actually allowed.
Therefore, this CL adds a refresh of the cache on two instances:
-When the app handler gets a search query that seems to be forbidden.
Before replying with a 403, we refresh the cache with the master query,
and recheck whether the search query is allowed.
-When the publisher gets a request for a "members" page, or the "file"
page, it preemptively asks the app handler to refresh. Now that a lot of
the client workflow has been moved to javascript/the browser, these
kinds of requests should not happen too often, so it seems a reasonable
place to ask for a refresh. But this might change, so we should of
course be careful not to flood the app handler with refresh requests in
the future.
In any case, the app handler is suppressing the refresh requests, so
that it does not perform refreshes at more that one per minute.
As a smarter approach, we could later imagine a way for the app handler
to be aware of when new blobs get to the blobserver (akin to the blob
hub that the sync handler uses?), so that it only ever refreshes when
needed.
Fixes#851
Change-Id: Idc14cce5018053deac01ec454e5c936ed93e5a05
Since the app handler should not trim the r.URL.Path of the handler's
prefix, it is now the responsibility of the app to cope with that
prefix.
Fixes#833
Change-Id: Ie1fa9801b26767c3e3b6612498380261e22cdf07
Some of the publisher features have moved from the server-side app to
the client-side app (the browser) thanks to gopherjs. Some of these
features imply doing some search queries against Camlistore, which
requires authentication. The server-side app receives the necessary
credentials on creation, from Camlistore. However, we can't just
communicate them to the client-side (as we do with the web UI) since the
publisher app itself does not require any auth and is supposed to be
exposed to the world.
Therefore, we need to allow some search queries to be done without
authentication.
To this end, the app handler on Camlistore now assumes a new role: it is
also a search proxy for the app. The app sends an unauthenticated search
query to the app handler (instead of directly to the search handler),
and it is the role of the app handler to verify that this query is
allowed for the app, and if yes, to forward the search to the Camlistore's
search handler.
We introduce a new mechanism to filter the search queries in the form of
a master query. Upon startup, the publisher registers, using the new
CAMLI_APP_MASTERQUERY_URL env var, a *search.SearchQuery with the app
handler. The app handler runs that query and caches all the blob refs
included in the response to that query. In the following, all incoming
search queries are run by the app handler, which checks that none of the
response blobs are out of the set defined by the aforementioned cached
blob refs. If that check fails, the search response is not forwarded to
the app/client.
The process can be improved in a subsequent CL (or patchset), with finer
grained domains, i.e. a master search query per published camliPath,
instead of one for the whole app handler.
Change-Id: I00d91ff73e0cbe78744bfae9878077dc3a8521f4
These improvements on the server app handler should help writing
and running stand-alone apps.
The two main goals are:
1) "simple" configurations should work automatically; the parameters for
the app are derived from the Listen and BaseURL of the Camlistore
server.
2) More advanced configurations, such as being behind a proxy, should be
easily configurable through the app's Listen, BackendURL, and ApiHost
parameters.
I had worked on them while doing the scanning cabinet app, and I am
backporting them now since we haven't landed the scanning cabinet yet,
and people have been having trouble setting up the publisher.
pkg/app/app_test.go is gone because app.ListenAddress is now dumb. The
hard work is done in pkg/server/app instead.
Fixes#818
Change-Id: Ice2610d6bac611b209cc3a928e67fa6093a41d3e
This allows deferring HTTP authentication and authorization to a proxy
(such as nginx) between Camlistore and the rest of the world, without
breaking the web UI.
Fixes#816
Change-Id: Ia4b5be8f2236ddac68dc0d3a09f0f24e588c4995
There were basically two bugs in bytesHaveSchemaLink:
-typo "blob" vs "bytes" camliType. this wouldn't happen if we used
constants from a pkg for camliTypes, instead of hardcoding them..
-returning early instead of checking all the parts of a file.
Added a test for each of those bugs. They look like they could be
refactored but they're subtly different enough that it might not be
worth it.
Context: working on issue #527
Change-Id: Ifd0bd311ae8a115de2a56994b89127db6f315a75
This lets the UI work (and lets people hack on the UI) by installing
camlistored the normal Go way (go install) and just running
camlistored, without using devcam server.
Change-Id: I45779640c5b2286ca7061a626c07b3edcdc9c4ad
The help page in the camlistore binary has a direct link for "Search bar
predicates" that links to this search-ui.txt on gerrit. Restore this
file so that these links will still resolve, but with a pointer to the
new docs on camlistore.org.
Also update the help link for future builds of camlistore to point to
the correct URL.
Change-Id: Ib81ff820869d86e6e6f664e8b52c5f20d4eed4d3
The describe requests were launching a storm of RLocks which weren't
safe in the presence of goroutines trying to acquire write locks.
Instead, make the corpus locking the responsibility of the caller and
add Lock/Unlock/RLock/RUnlock methods to the index and move locking up
a level.
This also adds a fair bit of context.Context plumbing which was used
in earlier debugging.
Fixescamlistore/camlistore#709
Change-Id: I8d7254d1e1da541f8c080d62f5408aac807fd3b1
The import path was added to the go file that included the package
documentation if one existed. Otherwise, I used what seemed to be the
primary file for the package.
Fixes#689
Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
Since issue #660 showed that we could not rely on Basic Auth for some
very particular cases (of POST requests), we wanted to replace Basic
Auth with a pre-generated token based authentication mechanism for these
cases.
However, as there already is such a mechanism used to authenticate
websocket connections, we simply extended the use of that mechanism.
Therefore, the token that is initially generated for websocket
connections is also now a valid token to authenticate any other
connection. The relevant types and method names have been changed to
reflect that. The javascript code pertaining to the sensitive cases
mentioned above has been changed to use that token.
Some doc has been added to point out how security-sensitive the
OpDiscovery permission is, since it gives access to the auth token.
Fixes issue #660
Change-Id: Iafed3b6e4804364ca2559414c8d87dc4a30f6637
Most of it replaced with vendor/go4.org/types and
vendor/go4.org/readerutil
u32 went where needed in pkg/blobserver/*
invertedBool went in pkg/types/serverconfig
atomics64 went in pkg/fs
Change-Id: I230426cda35be4b45ed67e869f14e6fdae89be22
This change allows (for example) to hook a blobserver and an indexer
together in a way similar as they would be in a default Camlistore setup.
This is particularly useful to setup tests that require a realistic
indexer, without having to create a heavier test with e.g. test.World.
This actual goal of this change is described below. I kept these changes
together because the one described below illustrates the need for the
synchandler changes.
pkg/index/stress/: stress test the indexer with more data
The goal of these benchmkarks is to help us compare the various sorted
implementations, so we can pick the most efficient one as a default.
A second, harder, goal would be to add tests that help us estimate the
reliability of the various kv stores, again so we can rule out the buggy
ones, or help fix them.
Change-Id: I8a1fe50398a4ba09c03786502b68b6c2599e5984
This change is in anticipation of moving pkg/images to go4.org, where it
should not depend on packages in third_party.
So:
third_party/github.com/nf/cr2 -> vendor/github.com/nf/cr2
third_party/github.com/rwcarlsen/goexif -> vendor/github.com/rwcarlsen/goexif
third_party/golang.org/x/image/tiff -> vendor/golang.org/x/image/tiff
Note that third_party/go/pkg/image/jpeg was also a dependency of
pkg/images. We had vendored image/jpeg from tip at the time because it
offered advantages over the version from Go1.3
(https://github.com/camlistore/camlistore/issues/463).
Since we now depend on Go1.5, we can go back to depend on the stdlib
version, so we simply remove third_party/go/pkg/image/jpeg and adjust
the imports accordingly.
Change-Id: Ifc8ffae0551102e644a0a0c67f3ff89e04df15c7
Previously pkg/jsonconfig and pkg/errorutil
Copied from go4.org at rev d1b8a2fb2de6160036e4801aa5e4d855571078b8
Change-Id: I673ed55b0825baa2607289b6082f205100261d7a
Filippo Valsorda