A new "put" mode is added to the pk command, so that the "pk put"
command can be used to create and upload blobs.
What this command does is actually just call the previously named
"camput" executable, which is renamed to "pk-put" in this change.
This involves adding a new way to register a mode in cmdmain, when such
a mode is just meant to call an external binary. To emphasize the
distinction, the existing func (to register a sub-command, or a mode) is
renamed from RegisterCommand to RegisterMode, and RegisterCommand is now
the name of the new func/way.
Updates #981
Updates #1056
Change-Id: Ief954c17aa88a376f551df7de4b4e9fe41ad96d1
Introduction of an Owner type in pkg/index.
Related changes in pkg/index and pkg/serverinit.
Fixes#1047
Change-Id: Ifc316865833349d5202b5085e8f2c1235f3f2220
- correct logging that logged functions instead of their value
- use ID vs Id naming
- use correct function names in comments
Change-Id: I61562cef7ebac7337ec6c85312cdf7915cb1a84b
when its wholeref index lines contain any "sha1-" wholeref lines.
Then, have the client use that info when uploading files to also
compute the sha-1 (in addition to the sha-224) and ask the server for
both.
By client, we mean changes to cmd/camput, to pkg/client functions (for
camput and other Go clients, such as the apps), and to the web UI.
Also, add a -sha1 mode to devcam server, as well as the
CAMLI_SHA1_ENABLED env var, to facilitate testing/debugging of the
legacy sha1 mode. Fix/document unrelated options in devcam server
while we're at it.
Change-Id: I369679d2a4719f6f8b409f78dd4d065e13a3f62f
DBNames is supposed to provide configuration for the various databases
names. However,
1) I contend that nobody needs or wants to configure them as long as we
provide sane defaults.
2) it seems the only obvious user we have for this is to set up some of
the names on GCE.
3) having another external source for names complicates the code
further, especially when we already have the distinction between
database names for DBMS and file names for file-based databases.
4) writing a correct documentation for it is awkward.
Therefore, in this CL, I propose that we remove DBNames. Instead,
genconfig.go now sets some consistent default names for the various
queues and indexes set up on a DBMS (MySQL, PostGres, Mongo). To that
end, we introduce the new, but optional, DBUnique configuration
parameter, that is used as a part of all the database names, in order to
be able to run several Perkeep instances on the same DBMS, without name
conflicts.
In addition, the queue for the bs->index synchandler is now set up on
the same DBMS that is already in use for the index itself, instead of
using a file-base database.
And i think we could proceed likewise for the other queues.
Fixes#951
Change-Id: Ib6a638f088a563d881e3957e4042e932382b44f4
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
Avoid select overhead in hot paths. Just use funcs.
Also, for sort-by-map searches, don't do a describe and pass over all
the results doing location lookups a second time. Remember the
location from the initial matching. Cache it on the search value.
Reduces some sort-by-map searches from 10 seconds to 3 seconds for
me. (still too slow, but good start)
Change-Id: I632954738df9accd802f28364ed11e48ddba0d14
This change makes zooming and panning on the map aspect send a new
"MapSort" search query, so that the (1000, by default) most relevant
results for the currently displayed area always appear as markers
after a zoom/pan.
This required completely "lifting up" (in React lingo) the currentSearch
state out of the Header class, and into the Index class, which should
have already been done in a56372830a. This
is necessary because we need the expression in the search box to be mutable
both by the user('s input) and by the map aspect, which represents the
zoom level as a trailing "locrect" predicate in the search expression.
Fixes#938
Change-Id: I0004c9ff09f03b4f1d95a35e54605689eebf0c1a
Since the map aspect was added to the web UI, it was discovered that it
currently does not scale well with the number of matching nodes.
The actual reason is that the search session is requesting an ever
increasing window of results, to get all results, instead of taking into
account the Continue token or Around mechanism.
However, this bug gives the opportunity to optimize the results for this
kind of interface. Instead of requesting the results in creation order,
until we get them all, we can request the set of nodes that looks the
best when displayed on the map. In other words, if there are more
results that the requested limit, the selected set of nodes should be
one that spreads around on the relevant area as evenly as possible.
This kind of selection is implemented in this CL, and will be used by
queries specifying the MapSort sort type.
Related: Issue #934
Change-Id: I6eb4901b40332863f17dab1ec4bfc11f3e99092a
WARNING: this app is still experimental, and even its data schema might
change. Do not use in production.
This change adds a Camlistore-based port of the scanning cabinet app
originally created by Brad Fitzpatrick:
https://github.com/bradfitz/scanningcabinet
Some of it is inspired from the App Engine Go port of Patrick Borgeest:
https://bitbucket.org/pborgeest/nometicland
The data schema is roughly as follows:
-a scan is a permanode, with the node type: "scanningcabinet:scan".
-a scan's camliContent attribute is set to the actual image file.
-a scan also holds the "dateCreated" attribute, as well as the
"document" attribute, which references the document this scan is a part
of (if any).
-a document is a permanode, with the node type: "scanningcabinet:doc".
-a document page, is modeled by the "camliPath:sha1-xxx" = "pageNumber"
relation, where sha1-xxx is the blobRef of a scan.
-a document can also hold the following attributes: "dateCreated",
"tag", "locationText", "title", "startDate", and "paymentDueDate".
Known caveats, in decreasing order of concern:
-the data schema might still change.
-the scancab tool, to actually create and upload the files from physical
documents, is practically untested (since I do not own a scanner).
-some parts, in particular related to searches, are probably
sub-optimized.
-the usual unavoidable bugs.
Change-Id: If6afc509e13f7c21164a3abd276fec075a3813bb
In order to use HTTPS, one must have a certificate, and one must have a
domain name for which the certificate is valid.
The first part is solved by the use of Let's Encrypt. For the second
part, we want to provide to any Camlistore instance a name such as
<gpgKeyId>.camlistore.net, where gpgKeyId is the fingerprint of its GPG
key. The DNS for camlistore.net agrees to add a record for that name if
and only if the Camlistore instance can prove it owns the GPG key, as
well as the IP address bound to that name in the DNS record.
A protocol such as the above is already implemented in pkg/gpgchallenge.
This CL:
- uses the client-side of the gpgchallenge protocol in camlistored, so
that it can claim a hostname in camlistore.net on startup (and then use
that hostname when requesting a certificate from Let's Encrypt).
- adds the configuration parameter "CamliNetIP" for the high-level
config. This parameter specifies the IP address that camlistored will
supply during the gpgpchallenge, so it can prove to the DNS server that
we own this address.
Fixes#722
Change-Id: I6bf4ec149b6dffd0ae93a6fa7bf208b2e8a05445
As the requests to the publisher are proxied through Camlistore's app
handler, there's no point in the publisher having its own autocert
Manager to request a certificate. Therefore, the publisher reuses
(readonly) camlistored's autocert CacheDir to get its certificate.
It follows that, for now, Let's Encrypt only works for the publisher if
it is running on the same host as camlistored (or more precisely, if they
share the same filesystem).
Fixes#458
Change-Id: Icf3be2913f85f9ec6f94b831ad58e1949b4d6961
Or to be more precise, golang.org/x/crypto/acme/autocert
The default behaviour regarding HTTPS certificates changes as such:
1) If the high-level config does not specify a certificate, the
low-level config used to be generated with a default certificate path.
This is no longer the case.
2) If the low-level config does not specify a certificate, we used to
generate self-signed ones at the default path. This is no longer always
the case. We only do this if our hostname does not look like an FQDN,
otherwise we try Let's Encrypt.
3) As a result, if the high-level config does not specify a certificate,
and the hostname looks like an FQDN, it is no longer the case that we'll
generate a self-signed. Let's Encrypt will be tried instead.
To sum up, the new rules are:
If cert/key files are specified, and found, use them.
If cert/key files are specified, not found, and the default values,
generate them (self-signed CA used as a cert), and use them.
If cert/key files are not specified, use Let's Encrypt if we have an
FQDN, otherwise generate self-signed.
Regarding cert caching:
On non-GCE, store the autocert cache dir in
osutil.CamliConfigDir()/letsencrypt.cache
On GCE, store in /tmp/camli-letsencrypt.cache
Fixes#701Fixes#859
Change-Id: Id78a9c6f113fa93e38d690033c10a749d1844ea6
When receiving a file, we were only trying to guess its MIME type
through its contents (pkg/magic). We're now making a better effort at it
by guessing from the filename extension if needed.
Also:
pkg/magic: get rid of all the extra video extensions that are already
covered by mime.TypeByExtension. Because it's redundant and
confusing.
app/publisher, pkg/types/camtypes: also use mime.TypeByExtension as an
extra effort. Especially since a reindex would be necessary to benefit
from the pkg/index change.
There are other places in Camlistore that could use such an effort.
Maybe we should have a camtypes.*FileInfo.MIME() method that tries all
the ways to guess the MIME type of the file?
Change-Id: Ib9a2bc42af77c5394dac578ae415524b5111ad4e
Add Handler.GetPermanodeLocation, based on the existing logic of loc:* and has:loc
search predicates, and that of index/Corpus.PermanodeLatLong:
1. Permanode attributes "latitude" and "longitude"
2. Referenced permanode attributes (eg. for "foursquare.com:checkin"
its "foursquareVenuePermanode")
3. Location in permanode camliContent file metadata
The sources are checked in this order, the location from
the first source yielding a valid result is returned.
camtypes: add new Location type
index: Add GetFileLocation to Index/Interface, to make
indexed location info accessible without a corpus.
This was unlike other file metadata like image info or media tags
which had accessors in both Index and Corpus.
Fixes#777
Change-Id: I63cf143d67a12732ca2c941de64b63736be5de6e
So far only images were served with their MIME types set properly, so
they would display directly in the browser, instead of being served as a
file download.
Now the same is done for a subset of text types: i.e. text/plain,
text/html, text/xml, and text/json. Aside from the browsing convenience,
the obvious advantage is being able to serve HTML directly, which should
allow us to build other things on top of the publisher.
Also a bit of related refactoring: moving the extension matching to
pkg/magic
Change-Id: Id98065c7c685036a272d1d2e293bfcbca33015ee
This change allows the publisher to use resources from a SourceRoot
directory, without having to rebuild the publisher binary, instead of
only using embedded resources.
Change-Id: Ife29e3015b8595a33f175a62d98fcf5ffa689134
Also adds more reliable check for "mounted broken" dirs on linux (that
require a fusermount -u). They were seen as not mounted (actually not
existing), because the check relied on df, which on them would just
return the "transport endpoint is not connected" error.
Fixes#826
Change-Id: I440d10a7b42c217ee85ea7a1726e581bc74e6f4a
These improvements on the server app handler should help writing
and running stand-alone apps.
The two main goals are:
1) "simple" configurations should work automatically; the parameters for
the app are derived from the Listen and BaseURL of the Camlistore
server.
2) More advanced configurations, such as being behind a proxy, should be
easily configurable through the app's Listen, BackendURL, and ApiHost
parameters.
I had worked on them while doing the scanning cabinet app, and I am
backporting them now since we haven't landed the scanning cabinet yet,
and people have been having trouble setting up the publisher.
pkg/app/app_test.go is gone because app.ListenAddress is now dumb. The
hard work is done in pkg/server/app instead.
Fixes#818
Change-Id: Ice2610d6bac611b209cc3a928e67fa6093a41d3e
The b.sortedStorageAt call had an empty filePrefix argument, which
means it would fail if the current index was not a DBMS (but a
file-based DB).
Also, this change adds a default "blobpacked_index" database name value
for the case when a user wants a DBMS index, but hasn't configured a
DBNames map.
Issue #693
Change-Id: Ie4386e789542c006530983e20bccc7a564bc4ee3
The import path was added to the go file that included the package
documentation if one existed. Otherwise, I used what seemed to be the
primary file for the package.
Fixes#689
Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
Since issue #660 showed that we could not rely on Basic Auth for some
very particular cases (of POST requests), we wanted to replace Basic
Auth with a pre-generated token based authentication mechanism for these
cases.
However, as there already is such a mechanism used to authenticate
websocket connections, we simply extended the use of that mechanism.
Therefore, the token that is initially generated for websocket
connections is also now a valid token to authenticate any other
connection. The relevant types and method names have been changed to
reflect that. The javascript code pertaining to the sensitive cases
mentioned above has been changed to use that token.
Some doc has been added to point out how security-sensitive the
OpDiscovery permission is, since it gives access to the auth token.
Fixes issue #660
Change-Id: Iafed3b6e4804364ca2559414c8d87dc4a30f6637
Most of it replaced with vendor/go4.org/types and
vendor/go4.org/readerutil
u32 went where needed in pkg/blobserver/*
invertedBool went in pkg/types/serverconfig
atomics64 went in pkg/fs
Change-Id: I230426cda35be4b45ed67e869f14e6fdae89be22
Previously pkg/jsonconfig and pkg/errorutil
Copied from go4.org at rev d1b8a2fb2de6160036e4801aa5e4d855571078b8
Change-Id: I673ed55b0825baa2607289b6082f205100261d7a
Previously pkg/wkfs
Copied from go4.org at rev 3970dd22c0e7e2397304ad626a20b72f8db41b58
Note that pkg/wkfs/gcs has not been moved yet (for dependencies
reasons).
Change-Id: I737e88a360bccfc643e1fa43541f65f2384c3157
- Available at '/help/'
- Loads and parses server config upon init and generates the necessary client config for access
- Links to 'Downloadable Tools' are not functional (task is not yet completed)
- A link was added to the default server page as well as a 'Help' option to the Piggy menu
Issue: #536
Change-Id: I10629de034c908ebc76f6feacbbfca1df7bf1c2b
The "files" field of a file search response was always non-null (empty JSON array in case of error or no result).
This behaviour was broken by df7cc0b453, so this CL is restoring it.
Fixes#619
Change-Id: I8e4762d1f5a2dba4f7cf06d8ca164392c4bf6d6a
Brad Fitzpatrick