Calling 'GetPermanodesWithAttr', as currently written, will return an
empty list for any unindexed parameters.
Rather than always returning an empty list if the user asks for a
unindexed attribute, error out.
Also add documentation of this potential foot-gun for users of the
client.
Change-Id: Iadc1b5a00aa709584affb89a1397e82aaffb692f
In 877eafa49b , pkg/server started
importing pkg/client , since the server has to be able to fetch blobs
from a share claim of another server. This introduced a circular
dependency in tests, as pkg/client/upload_test.go instantiates a server.
As the test uses both a client and a server, and tests uploading between
the two, it seems fitting to move it to pkg/test/integration, which
breaks the circular dep.
Change-Id: I2458ffd72912ff6e9c1a0f67e187d54cc895c2e9
This change has two major parts, which were interwoven enough to do
them in one change:
1) make serverinit.Config fully opaque, in prep for TOML configs #1134
2) shrink the massive server/perkeepd/camlistored.go file. It was out
of control and had a bunch of code that better belonged
elsewhere. This change moves a few hundred lines of code from
camlistored.go into more logical places: internal/osutil/gce for
GCE stuff, serverinit for config stuff (KeyRingAndId), etc.
I also added a TODO to make it possible to compile perkeepd without
any GCE stuff, which I saw as a possible and worthy goal only after
moving everything away.
Updates #1134
Change-Id: Iea6f84c5aca9c70b97806f4a201ec35e0f630e3b
test/integration: don't listen on file descriptors.
make.go: unrelated, but options to make it much faster.
internal/images: t.Skip on HEIC dependency failures
Fixes#1140
Updates golang/go#25210
Change-Id: I8092155411826d6ed1f8d85230b753d1369044af
A new "put" mode is added to the pk command, so that the "pk put"
command can be used to create and upload blobs.
What this command does is actually just call the previously named
"camput" executable, which is renamed to "pk-put" in this change.
This involves adding a new way to register a mode in cmdmain, when such
a mode is just meant to call an external binary. To emphasize the
distinction, the existing func (to register a sub-command, or a mode) is
renamed from RegisterCommand to RegisterMode, and RegisterCommand is now
the name of the new func/way.
Updates #981
Updates #1056
Change-Id: Ief954c17aa88a376f551df7de4b4e9fe41ad96d1
Since the search API changed at
2018-01-13-6e8a5930c9fee81640c6c75a9a549fec98064186, a server built
before that revision is incompatible with a client built at or after
that revision.
This change adds a check that is run when this incompatibility could
manifest by a JSON parsing error, in order to clarify the error message.
As a side note, even if there was not this search API breakage, a server
built before 27bacd3df1 would panic anyway
if a client built at or after 38f10a7bd0
tried uploading to it. Because of the wrong assumption that br.Hash()
is never nil, which can actually happen after sha224 is introduced
(client-side), and the server does not know about sha224.
Fixes#1074
Change-Id: I3dabf25c0200625e177e0f26308465cd35e8c750
The android uploader used to only accept a server TLS certificate when
its fingerprint was specified as trusted in the client configuration. As
a result, certificates obtained by Let's Encrypt (or other CAs, for that
matter) were rejected unless treated as self-signed.
With this change, any connection verified by the standard TLS client
(using the system's root CA pool) is accepted as long as no
additional certificate fingerprint is given.
Closes#974
Change-Id: I499d41e9f48df709dda60724c6f55eca888feeb7
- correct logging that logged functions instead of their value
- use ID vs Id naming
- use correct function names in comments
Change-Id: I61562cef7ebac7337ec6c85312cdf7915cb1a84b
when its wholeref index lines contain any "sha1-" wholeref lines.
Then, have the client use that info when uploading files to also
compute the sha-1 (in addition to the sha-224) and ask the server for
both.
By client, we mean changes to cmd/camput, to pkg/client functions (for
camput and other Go clients, such as the apps), and to the web UI.
Also, add a -sha1 mode to devcam server, as well as the
CAMLI_SHA1_ENABLED env var, to facilitate testing/debugging of the
legacy sha1 mode. Fix/document unrelated options in devcam server
while we're at it.
Change-Id: I369679d2a4719f6f8b409f78dd4d065e13a3f62f
This removes NewDefault and NewFromParams.
Now the default way to create a client is:
client.New() -> (*Client, error)
Specifying a server is optional and now requires
client.OptionServer(server).
If the caller really wants to log.Fatal on error, they can use
client.NewOrFail.
Also, some of the boilerplate from GopherJS callers is now promoted to
be the default behavior in the client package.
Change-Id: Icb106cf3e13cc492fe5b2f7f240e1ad4227eaf33
I had intended for this to be a small change.
I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.
And then it got a bit infectious and ended up touching everything.
I ended up doing SubFetch in the process by necessity.
At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.
Updates #733
Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
Remove the blob.SHA{1,224}From{Bytes,String} constructors too. No
longer used. This adds blob.RefFromBytes which was missing. We had
blob.RefFromString. Now everything uses blob.RefFrom* instead of
specifying a hash function.
Some tests set a flag to force use of SHA-1 because there was too much
golden data to update. We can remove those one-by-one over time as we
fix up tests.
Updates #537
Change-Id: Ibe6428089a6221594c2b751f53f98b03b5a28dc2
This addresses a long-standing TODO in the BlobStatter interface to
clean it up. Just like all new Go programmers, I misused channels in
APIs. I should've cleaned this up years ago.
While here, I also added a context.
The rest should get contexts later.
This also cleans up a few things here & there.
The pkg/client statting no longer does batching, which added a lot of
complexity. There was a comment saying something like "once we have
SPDY, we can delete this". Well, we have HTTP/2 now, so seems
deletable.
All tests pass.
Change-Id: I034ce07d9b70e5cc9e5482213368993e638d4bc8
Calls to net.Dial* are prohibited with GopherJS. This can happen if the
client's transport is set by the user.
This change forces transportForConfig to return nil when the client
package is compiled with gopherjs, in order to make sure that a call to
newClient returns a client with a nil transport.
Change-Id: I577457bd7d924d31710168086dc2b394df3d1ae0
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
Otherwise, the android app fails to connect with a server that uses
Let's Encrypt (because it relies on SNI, which requires the ServerName
to be set).
Change-Id: I9f25486bea68e83c68584a83817c98bfc84f62b9
This switches most usages of the pre-1.7 context library to use the
standard library. Remaining usages are in:
app/publisher/main.go
pkg/fs/...
Change-Id: Ia74acc39499dcb39892342a2c9a2776537cf49f1
This CL addresses issues #685 and #862.
The general problem is that some critical errors, that lead clients such
as camput to exit with failure, are not displayed when not running in
verbose mode.
The reason that happens is because of code such as:
if *cmdmain.FlagVerbose {
log.SetOutput(cmdmain.Stderr)
} else {
log.SetOutput(ioutil.Discard)
}
which means that in non-verbose mode we discard absolutely all log
messages, even those that would be printed during a Fatal* call.
To address that problem, we introduce a logger, as well as the Printf
and Logf functions using it, in pkg/cmdmain. These two functions only
output when *cmdmain.FlagVerbose is true.
Commands such as camput or camtool should now always:
1) log.SetOutput(cmdmain.Stderr) in init().
2) use log.Printf for messages that should always be printed.
3) use cmdmain.Printf/Logf for messages that should only be printed when
*cmdmain.FlagVerbose is true.
4) use log.Fatal for critical errors.
5) optionally, set the Verbose and Logger of the client(s) they are
using.
Also, camput and camtool are now relying on the global -verbose flag
from cmdmain, instead of having to define one for each subcommand.
fixes#685fixes#862
Change-Id: I088032fd28184a201076097bf878894b22a8a120
This change makes zooming and panning on the map aspect send a new
"MapSort" search query, so that the (1000, by default) most relevant
results for the currently displayed area always appear as markers
after a zoom/pan.
This required completely "lifting up" (in React lingo) the currentSearch
state out of the Header class, and into the Index class, which should
have already been done in a56372830a. This
is necessary because we need the expression in the search box to be mutable
both by the user('s input) and by the map aspect, which represents the
zoom level as a trailing "locrect" predicate in the search expression.
Fixes#938
Change-Id: I0004c9ff09f03b4f1d95a35e54605689eebf0c1a
To rev 169ea6cabe2a4888dba958edaecc9e9751adc711
Also get rid of pkg/readerutil now that it's all moved to
go4.org/readerutil
go4.org/readerutil/singlereader not vendored in as we don't actually use
it.
Fixes#678
Change-Id: I46f14acde1d3023e94c8e99e8267ce951f1bbb61
To share a file or directory through Camlistore's sharing
mechanism (https://camlistore.org/doc/sharing).
Only single-item sharing supported for now.
On success, the share URL is displayed in a dialog.
If the item is a file, it can be fetched with any HTTP client through
that URL. If the item is a dir, the share URL is of the claim, so one
must follow the transitive chain of sharing to get the directory's
children, i.e. use camget -shared.
Fixes#880
Change-Id: I324a684dec10225e5b8314dda7f1b77a0de6f727
WARNING: this app is still experimental, and even its data schema might
change. Do not use in production.
This change adds a Camlistore-based port of the scanning cabinet app
originally created by Brad Fitzpatrick:
https://github.com/bradfitz/scanningcabinet
Some of it is inspired from the App Engine Go port of Patrick Borgeest:
https://bitbucket.org/pborgeest/nometicland
The data schema is roughly as follows:
-a scan is a permanode, with the node type: "scanningcabinet:scan".
-a scan's camliContent attribute is set to the actual image file.
-a scan also holds the "dateCreated" attribute, as well as the
"document" attribute, which references the document this scan is a part
of (if any).
-a document is a permanode, with the node type: "scanningcabinet:doc".
-a document page, is modeled by the "camliPath:sha1-xxx" = "pageNumber"
relation, where sha1-xxx is the blobRef of a scan.
-a document can also hold the following attributes: "dateCreated",
"tag", "locationText", "title", "startDate", and "paymentDueDate".
Known caveats, in decreasing order of concern:
-the data schema might still change.
-the scancab tool, to actually create and upload the files from physical
documents, is practically untested (since I do not own a scanner).
-some parts, in particular related to searches, are probably
sub-optimized.
-the usual unavoidable bugs.
Change-Id: If6afc509e13f7c21164a3abd276fec075a3813bb
When fetching shared blobs, we rely on the share chain to verify if a
blob can be reached. This chain is updated whenever we fetch an
additional link of the chain, by updating the Client.via map. However,
when some blobs of the chain are already cached in camget's DiskCache,
because we get them from the cache, we don't fetch them with
Client.FetchVia, which means the Client.via map isn't updated. And thus
the chain is broken.
This change adds Client.UpdateShareChain, and sets it as a hook to be
called by the CachingFetcher in the event of a cache hit. That way, we
ensure that the share chain is updated even when we get blobs from the
cache (instead of from the Client).
We also add a mutex to guard Client.via, because it is accessed by
concurrent smartFetch calls in case of a static-set.
As FetchVia was undocumented and not used by anyone, I made it
unexported. We can always export it again later when needed.
Fixes#856
Change-Id: I767cbec4b6f382cbccc25c0b97782b2a7472deb8
When the discovery SearchRoot is empty (because e.g. the server has no
indexer), the root.Parse(disco.SearchRoot) call during doDiscovery will
not fail. This is fine, and is actually what we want, because we don't
want discovery to fail just because there's no SearchRoot.
However, we do want func (c *Client) SearchRoot() to return
ErrNoSearchRoot in that case, which is why c.searchRoot should be set to
"" empty as well.
Change-Id: I88cbfa4a1476ba1533ccd61511a72c35de6d7516
Brad Fitzpatrick