Commit Graph

141 Commits

Author SHA1 Message Date
Brad Fitzpatrick b1abccd287 devcam, localhost, storagetest: fixes for Windows
Unset CGO_ENABLED if no sign of gcc.

The localdisk renaming stuff was fixed in Go ages ago in
golang/go#13673 and https://golang.org/cl/6140

And a defer in storagetest meant Windows couldn't delete files
because a file was still open.

Change-Id: I57aef85f24653b19ce10e3d1e18c778cee2d48f6
2018-05-01 10:46:19 -07:00
mpl a6ad4d88e9 dev/devcam: do not look for pk in ./bin dir
Change-Id: I1903b389fbed1bf836d4d3c91c5fb6d2db117b84
2018-04-25 22:51:41 -07:00
mpl b75523bbea dev/devcam: fix website go code location
also do not set GOBIN when building self

Change-Id: Ibea2ab4fcc7b60ef28dfcb6620a44b29541eedef
2018-04-25 12:19:10 -07:00
mpl 7a27eb42eb cmd: rename camget to pk-get, and make it a mode of pk
A couple of related fixes and renames too.

Updates #981
Fixes #1056

Change-Id: Id47a933c77422edfb9db41d34c38ed9d0d7a1846
2018-04-22 20:50:54 -07:00
Brad Fitzpatrick e5e29b004b dev/devcam: don't invoke make.go four times for four targets
Just once is fine.

Change-Id: I74b34f34ee5a193dcf7ac7d28acc858fd6263a68
2018-04-22 09:01:40 -07:00
mpl 9035ba9f25 make.go: fix cross compilation
When installing, respect user defined GOBIN and/or GOPATH, or the
Go defaults.
But when cross-compiling, disable GOBIN, as it's not supported.

Fixes #1100

Change-Id: I69485722d6dbfcbf7721463077f828efc950d9d3
2018-04-21 21:44:05 +02:00
Brad Fitzpatrick ca76a40bbc Rename camlistored to perkeepd.
Updates #981

Change-Id: I8fe43c240c149074c23128a89ab426af9cbf94b4
2018-04-21 11:06:09 -07:00
mpl ce4658abfc cmd: rename camput to pk-put, and make "pk put" call it
A new "put" mode is added to the pk command, so that the "pk put"
command can be used to create and upload blobs.

What this command does is actually just call the previously named
"camput" executable, which is renamed to "pk-put" in this change.

This involves adding a new way to register a mode in cmdmain, when such
a mode is just meant to call an external binary. To emphasize the
distinction, the existing func (to register a sub-command, or a mode) is
renamed from RegisterCommand to RegisterMode, and RegisterCommand is now
the name of the new func/way.

Updates #981
Updates #1056

Change-Id: Ief954c17aa88a376f551df7de4b4e9fe41ad96d1
2018-04-21 10:26:55 -07:00
Brad Fitzpatrick dd3516c6ee Remove some ancient Perl scripts.
Change-Id: If98021d39eb8560913bc1ba35f4cb7f29f25974c
2018-04-17 15:11:07 -07:00
mpl f8ef262ff9 make.go: clean up, notably remove the tmp source code mirroring
Fixes #1004

Change-Id: Iea01cfc601b8dd8f2a81fe5da40cdd272b164bb1
2018-04-17 19:39:46 +02:00
mpl 6dcbe2ee9c cmd/camtool: rename from camtool to pk
Updates #981
Updates #1056

Change-Id: Ib78c47a69fa9816d5ac6cc547586a2af321005e3
2018-03-06 16:20:59 +01:00
mpl b89bf256d1 pkg/search: start to make funcs use a GPG key ID instead of blobRef
Introduction of an Owner type  in pkg/index.
Related changes in pkg/index and pkg/serverinit.

Fixes #1047

Change-Id: Ifc316865833349d5202b5085e8f2c1235f3f2220
2018-02-07 00:54:38 +01:00
Mathieu Lonjaret 6e8a5930c9 all: make server advertise when it has legacy SHA-1 indexes, use that info
when its wholeref index lines contain any "sha1-" wholeref lines.

Then, have the client use that info when uploading files to also
compute the sha-1 (in addition to the sha-224) and ask the server for
both.

By client, we mean changes to cmd/camput, to pkg/client functions (for
camput and other Go clients, such as the apps), and to the web UI.

Also, add a -sha1 mode to devcam server, as well as the
CAMLI_SHA1_ENABLED env var, to facilitate testing/debugging of the
legacy sha1 mode.  Fix/document unrelated options in devcam server
while we're at it.

Change-Id: I369679d2a4719f6f8b409f78dd4d065e13a3f62f
2018-01-31 01:10:52 +01:00
Brad Fitzpatrick f9f109ffa1 all: more Camlistore to Perkeep renamings
Updates #981

Change-Id: I7fd4958bceaa4ada63d07aacf08af11cc03f32d7
2018-01-30 13:31:59 -08:00
Paul Lindner 459c75410e all: more renaming of Camlistore to Perkeep
Change-Id: I118e3cbcf20d80afeffc84f001388c4556f21628
2018-01-30 03:02:56 -08:00
Brad Fitzpatrick d4ff75359c pkg/client: reduce the number of New constructors, return error by default
This removes NewDefault and NewFromParams.

Now the default way to create a client is:

    client.New() -> (*Client, error)

Specifying a server is optional and now requires
client.OptionServer(server).

If the caller really wants to log.Fatal on error, they can use
client.NewOrFail.

Also, some of the boilerplate from GopherJS callers is now promoted to
be the default behavior in the client package.

Change-Id: Icb106cf3e13cc492fe5b2f7f240e1ad4227eaf33
2018-01-24 07:42:04 -08:00
Brad Fitzpatrick 38f10a7bd0 all, testhooks: use sha224 by default, add hook for some tests to use sha-1
Remove the blob.SHA{1,224}From{Bytes,String} constructors too. No
longer used. This adds blob.RefFromBytes which was missing. We had
blob.RefFromString. Now everything uses blob.RefFrom* instead of
specifying a hash function.

Some tests set a flag to force use of SHA-1 because there was too much
golden data to update. We can remove those one-by-one over time as we
fix up tests.

Updates #537

Change-Id: Ibe6428089a6221594c2b751f53f98b03b5a28dc2
2018-01-09 20:03:38 -08:00
Paul Lindner 463832ed76 clients/web/embed: Move vendor/embed to clients/web/embed and adjust paths
Addresses #889
Fixes #1024

Change-Id: I9fa1dc246b0a3a1a4e4ec7abab0a7828fdfdb41e
2018-01-09 22:33:03 +00:00
Brad Fitzpatrick 617ac40512 all: rename cammount to pk-mount
Updates #981

Change-Id: Ie1d2e3c0fd7e2b7a7801ceac0fd8f93f62d56651
2018-01-08 09:54:41 -08:00
Paul Lindner 266447a7e7 dev/devcam: import vendor files as-is, do not require gofmt
Change-Id: If3afd3769dbad0a20bd8ff2f2bbb5eb9f89f345a
2018-01-07 19:14:08 -08:00
Brad Fitzpatrick c3d05cdce9 Move more packages out of pkg/ and into internal/
Moved hashutil, httputil, osutil, netutil,
images, media, magic, video, and rollsum.
2018-01-02 21:03:30 -08:00
Brad Fitzpatrick d6a0b05df0 Rename import paths from camlistore.org to perkeep.org.
Part of the project renaming, issue #981.

After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.

This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.

Also, this only moves the lru package to internal. More will move to
internal later.

Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.

This updates some docs, but not all.

devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).

Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
2018-01-01 16:03:34 -08:00
Paul Lindner ba92702834 all: lint fixes for "should omit 2nd value from range"
Change-Id: I7bb19d376f96a39ecae7dbdb4d6808f704bae5fb
2017-12-13 11:31:25 -08:00
Paul Lindner 15feaeb24c all: lint fixes for 'error strings should not be capitalized or end with punctuation or a newline'
Change-Id: I9c3766a51ac8be694ae76befff4b6fa9a85e34eb
2017-12-11 06:13:25 -08:00
Paul Lindner f28eb3cce8 Remove appengine server and the old python blobserver
Change-Id: I734f4621f77ce902050e9b37805e7f8912f75e97
2017-11-30 19:13:37 -08:00
Mathieu Lonjaret 3911a41bd5 Merge "cmd/*: always output on fatal errors" 2017-10-22 15:07:06 +00:00
mpl a6a3cac0df cmd/*: always output on fatal errors
This CL addresses issues #685 and #862.

The general problem is that some critical errors, that lead clients such
as camput to exit with failure, are not displayed when not running in
verbose mode.

The reason that happens is because of code such as:

	if *cmdmain.FlagVerbose {
		log.SetOutput(cmdmain.Stderr)
	} else {
		log.SetOutput(ioutil.Discard)
	}

which means that in non-verbose mode we discard absolutely all log
messages, even those that would be printed during a Fatal* call.

To address that problem, we introduce a logger, as well as the Printf
and Logf functions using it, in pkg/cmdmain. These two functions only
output when *cmdmain.FlagVerbose is true.

Commands such as camput or camtool should now always:
1) log.SetOutput(cmdmain.Stderr) in init().
2) use log.Printf for messages that should always be printed.
3) use cmdmain.Printf/Logf for messages that should only be printed when
*cmdmain.FlagVerbose is true.
4) use log.Fatal for critical errors.
5) optionally, set the Verbose and Logger of the client(s) they are
using.

Also, camput and camtool are now relying on the global -verbose flag
from cmdmain, instead of having to define one for each subcommand.

fixes #685
fixes #862

Change-Id: I088032fd28184a201076097bf878894b22a8a120
2017-10-19 02:22:44 +02:00
Tamás Gulácsi 0bfa352bae pkg/importer: add gphotos, a Google Photos importer
Uses the Google Drive API to import photos from the Google Photos folder
in Google Drive.

Still a work in progress, so not yet added to allimporters.

Issues #874 #896

Change-Id: I9b9bffe071434b2a55be667579c3699a3addb8ed
2017-05-08 18:41:34 +02:00
Mathieu Lonjaret e0107f74bc Merge "Initial cut at importing financial transactions using Plaid.com. No Plaid-specific UI in this first pass; just import/data model only." 2017-03-26 00:24:48 +00:00
Michael Morrissey f35578a2e2 Initial cut at importing financial transactions using Plaid.com. No Plaid-specific UI in this first pass; just import/data model only.
vendor: add github.com/plaid/plaid-go at rev 02b6af68061bf89a293eaf15dc6c955ce02dd22b

Change-Id: I1003d1d21416b9f2c7eb40085e62ec8481a0c6ed
2017-03-25 10:41:01 -04:00
Euan Kemp 23ef425564 devcam/hook: handle 'verbose' commits
See https://github.com/golang/go/issues/16376, and the related
https://go-review.googlesource.com/#/c/25342/.

Prior to this change, `git commit -v` would result in a nonsensical
commit message.

Change-Id: Ib11de27488b01fccff07b9385f7fa988bc6fe165
2017-03-04 12:42:52 -08:00
mpl 5a24ffd854 new app: scanning cabinet
WARNING: this app is still experimental, and even its data schema might
change. Do not use in production.

This change adds a Camlistore-based port of the scanning cabinet app
originally created by Brad Fitzpatrick:
https://github.com/bradfitz/scanningcabinet

Some of it is inspired from the App Engine Go port of Patrick Borgeest:
https://bitbucket.org/pborgeest/nometicland

The data schema is roughly as follows:

-a scan is a permanode, with the node type: "scanningcabinet:scan".
-a scan's camliContent attribute is set to the actual image file.
-a scan also holds the "dateCreated" attribute, as well as the
"document" attribute, which references the document this scan is a part
of (if any).

-a document is a permanode, with the node type: "scanningcabinet:doc".
-a document page, is modeled by the "camliPath:sha1-xxx" = "pageNumber"
relation, where sha1-xxx is the blobRef of a scan.
-a document can also hold the following attributes: "dateCreated",
"tag", "locationText", "title", "startDate", and "paymentDueDate".

Known caveats, in decreasing order of concern:
-the data schema might still change.
-the scancab tool, to actually create and upload the files from physical
documents, is practically untested (since I do not own a scanner).
-some parts, in particular related to searches, are probably
sub-optimized.
-the usual unavoidable bugs.

Change-Id: If6afc509e13f7c21164a3abd276fec075a3813bb
2017-02-15 17:14:45 +01:00
mpl 93a7f46720 make.go: remove --use_gopath
Because it makes full integration with gopherjs impossible (without
polluting the user's GOPATH), as long as
https://github.com/gopherjs/gopherjs/issues/415 is not fixed.

Also it is kind of antithetical with the point of make.go anyway.

We still rely on CAMLI_MAKE_USEGOPATH for the integration tests that run
make.go to know that they shouldn't recursively create another temp
GOPATH (when they're already in such a temp dir, because they're started
through devcam test).

Change-Id: Icc6af46ec5976fdf08e9b8bf4249e307a15499cf
2016-05-09 17:22:39 +02:00
mpl 63d6e384bb path fixes for third_party move
A few more left in in misc/docker that I'll do separately

Related: 75d60962f6

Change-Id: Id5f6a3729e33aca97f8664ca8ef91afc64461891
2016-04-21 12:25:17 -07:00
Will Norris 77ed42edf8 add canonical import paths
The import path was added to the go file that included the package
documentation if one existed.  Otherwise, I used what seemed to be the
primary file for the package.

Fixes #689

Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
2016-03-13 19:57:14 -07:00
mpl cde9dcb6f3 devcam hook: add commit-msg hook
To replace the misc/commit-msg.githook bash script

Code mostly copied from golang.org/x/review/git-codereview/hook.go

Fixes issue #590

Change-Id: I887cd32f9b6113d1a6b5d8b6eea6355da5706074
2016-02-25 17:50:44 +01:00
Mathieu Lonjaret d471cba23f Merge "serverinit: enable blobPacked in default config" 2015-12-24 23:28:20 +00:00
mpl cf08d789f2 website: sidestep failInTests panics from osutil
Because camweb now depends on pkg/deploy/gce, which has global vars
initialized with calls from osutil.

And test website from devcam test

Change-Id: I233c229e96ded5bfde4e2c1fa7a18bbf757ab1fb
2015-12-07 22:48:12 +01:00
mpl 3e37ca9120 devcam test: build things in ./misc/docker too
Fixes issue #651

Change-Id: I10e4b73946dea073399bdb21888202f85a33c1b7
2015-12-03 16:49:11 +01:00
mpl 8127040762 pkg/client: use Transport.DialTLS
We can now use DialTLS when we need a custom TLS setup, i.e. with
android, or when using self-signed certificates, or when totally
skipping certificates verification.

This allows us to get rid of the scheme rewriting hack, which simplifies
things, in particular because we have less tricky URL/host parsing to
do.

For the sake of tests and not affecting "real" code, I introduced the
fake_android build tag, so the tests can use custom functions to
simulate testing whether we're on android.

Fixes #566

Change-Id: I72ac2bb69ad2365e98dd6ca2e7016ce9c2d7c57e
2015-09-26 00:09:26 +02:00
mpl 974ed85e2e devcam test: run devcam hook pre-commit
Instead of the old ./misc/pre-commit.githook
Also fix devcam hook pre-commit so we don't show the override message in
that case.

Change-Id: I390016765056b9c4d3331d12bef8f2581e5621df
2015-09-22 15:57:36 +02:00
mpl b118fc23fb devcam hook: no import path checking in pre-commit because Go 1.5
We won't be rewriting the import paths of our third-parties anymore
since we use the new vendor mechanism that comes with Go 1.5. Hence, no
need to check for those rewrites anymore.

Change-Id: Ib9557b60c077190bfd5a6db95ad582f153aa4a68
2015-08-21 19:57:09 +02:00
mpl 8ea07d33d2 devcam server: preset CAMLI_LEVELDB_ENABLED to false
Change-Id: Id35fc6f965186b4d2f1a2d427d2e5506a98cce1f
2015-08-21 18:35:30 +02:00
mpl 4f0e6079bf devcam: fix self-building wrt new vendoring
Change-Id: I26c8a49bb2fe7e03dea08a653ae6533e7ef05d17
2015-08-16 15:50:33 +02:00
mpl 9a24acca6c devcam fixv,hook: docs, bugfix, cleanup.
Change-Id: Ie5f9f9ea62c13221d6b9c9913ef953ab39fb7914
2015-08-03 19:45:10 +02:00
mpl 6fa15a228e devcam: git hooks in go, 3rd party imports too
devcam hook: install git hooks and/or run them

devcam fixv: fix import paths in vendored files

More/better docs later.

Addresses #590
Addresses #591

Change-Id: I162717bc2e8a8190a6fe81086b3e14fcdf8ab92a
2015-08-01 02:16:10 +02:00
mpl afb47c7cd9 devcam server: add syndtr/goleveldb indexer. make it the default.
Change-Id: Ice835d7505404ad9eeaf040d465409bb0e4c420d
2015-07-26 17:53:36 +02:00
mpl 66a5a6176f serverinit: enable blobPacked in default config
Also, devcam server: make loose blobs location consistent with
location on production server.

Change-Id: Ib4314f4b736595c071e710792d2e8ecacebf3a14
2015-07-23 16:43:13 +02:00
mpl 3db00e5e98 devcam: move modtime check after cwd check
checkCamliSrcRoot was meant to give a nice error msg if devcam is
invoked from outside the Camlistore source tree. However, it was never
called because the modtime check (checkModtime), which itself needs to
be run from that same location, was called first. So we were always
getting the terse "open dev/devcam: no such file or directory" error
message instead.

Fixes #589

Change-Id: I13a1bb3819217ffbaf727fa6154f077dd8babd49
2015-05-07 16:53:44 +02:00
Brad Fitzpatrick 3e84782750 devcam: always use blobpacked for devcam server
Maybe it can be an option in the future, but for now it's
always on.

Change-Id: I7ec408b145870fc7dcffa42638ebb0daf595982d
2015-02-04 21:02:38 -08:00