Commit Graph

494 Commits

Author SHA1 Message Date
Brad Fitzpatrick bec657b7e2 Remove legacy +build comments; use go:build only
Signed-off-by: Brad Fitzpatrick <brad@danga.com>
2023-12-31 09:18:39 -08:00
Brad Fitzpatrick afb071614f cmd/pk-mount, pkg/fs: drop macOS FUSE support, update bazil.org/fuse
Our bazil.org/fuse FUSE library no longer supports macOS
(https://github.com/bazil/fuse/issues/224) because osxfuse is no
longer open source (https://github.com/osxfuse/osxfuse/issues/590).

Also, Apple is making it harder and harder (eventually impossible?) to
install custom kernel modules:
https://github.com/macfuse/macfuse/wiki/Getting-Started

So just give up on macOS FUSE support for now. We can resurrect it
later via WebDAV and/or NFS server support.

Signed-off-by: Brad Fitzpatrick <brad@danga.com>
2023-12-30 15:40:51 -08:00
Brad Fitzpatrick b5823a65b9 pkg/deploy/gce: remove GCE deploy support
We used to have a web-based Perkeep launcher at perkeep.org/launch
that created a GCE-based Perkeep instance for users, where they pay
Google for compute time. (One of those "one click deploy" template things)

Unfortunately, Google broke their APIs for doing the third party VM
creations and we disabled it some years ago. But the code remains. And
now, updating it again, we find that they've broken it again:

   Error: pkg/deploy/gce/deploy.go:358:4: servicemanagement.NewServicesService(s).Enable undefined (type *servicemanagement.ServicesService has no field or method Enable)

It's not worth fighting Google's API breakages. Just remove the GCE
launcher support as it's been unused for years.

We can always resurrect this code from git if really needed. But a
Digital Ocean or Fly launcher would probably be much easier.

Signed-off-by: Brad Fitzpatrick <brad@danga.com>
2023-12-30 12:47:01 -08:00
Oleksandr Redko 63c68c6c5b tests: use T.Setenv instead of os.Setenv 2023-02-18 12:53:16 +02:00
Oleksandr Redko 001c417e73
all: replace deprecated io/ioutil with io and os (#1647) 2023-01-23 10:25:14 -08:00
Brad Fitzpatrick 921894783b all: gofmt for Go 1.19
Signed-off-by: Brad Fitzpatrick <brad@danga.com>
2022-12-31 22:45:40 -08:00
Michael Hoffmann 215de258f2 fuse: misc fixes
* propagate more context cancelations from pkg/client
* convert ErrCancel to fuse.EINTR to enable clients to handle interrupts
* improve interrupt handling in fs_test.go
* fix race in mutDir between rename and first populate
2022-07-01 07:39:38 -07:00
luz paz d605f86c82 Fix various typos
Found via `codespell -q 3 -S ./clients/web/embed,./clients/chrome -L ba,everytime,impres,keypair,msdos,pres,ro,te,ue`
2022-05-02 14:31:09 -07:00
Eng Zer Jun 2977046a4c all: use `T.TempDir` to create temporary directory
The directory created by `T.TempDir()` and is automatically removed when
the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-04 08:18:24 -08:00
Tamás Gulácsi c986ee3c62
Use modernc.org/sqlite (#1581)
* Use modernc.org/sqlite

This way it can be enabled by default, as it is a cgo-free, Go-only package.
No need for build tags, conditional compilation (whether libsqlite3-dev is installed).

* make.go: Remove unused -sqlite flag

* Remove use of -sqlite flag
2021-12-27 12:18:08 -08:00
Tamás Gulácsi de8805f4fa pkg/blobserver/diskpacked: TODO, Context and dead code cleanup 2021-08-01 10:58:40 +02:00
Alexandre Viau fb961cf310
make codebase go-vet-clean (#1379)
Co-authored-by: Bob Glickstein <bobg@emphatic.com>
2021-07-26 21:19:53 -04:00
aviau ace2de8e4b delete.go: implement Examples() 2021-01-23 19:27:51 -05:00
aviau c9bb3f4df8 pk-put+perkeepd config: support fingerprints
- Key ids in the `0xXXXXXXXXXXXXXXXX` format are now accepted.

- Key ids in the `02A6 ACA0 5F16 9329 D94A  FFB8 C804 D0BA 9F70 883D`
  format are also accepted. With or without spaces.

- Help texts now use long keyids so that its clear that we support
  them.
2021-01-23 19:27:23 -05:00
aviau 231ba4233f pkg/schema: create CamliType type
Create a CamliType type in pkg/schema and use it in a couple of
packages.

It can be implemented in other packages as we go.
2021-01-19 00:47:58 -05:00
aviau 7730b62281
Merge branch 'fix-windows' of https://github.com/willnorris/perkeep into willnorris-fix-windows
- Revert the permissions verifications on Windows, we can
  implement it in some other way later.
2021-01-14 11:34:57 -05:00
aviau 39bc87d273 cmd/pk/list: filter by type 2021-01-14 09:41:02 -05:00
Will Norris 37c1998f27 add windows implementation for cleanSysStat
this allows pk-put's havecache to properly function on Windows, fixing
the camput integration test failure.
2020-09-24 16:31:39 -07:00
mpl 2a85cb7142 internal/rollsum: replace with go4.org/rollsum
Fixes #1253

Change-Id: I64aac5739a18d2e19494881dcceff9d43355fac2
2019-02-23 01:48:18 +01:00
mpl e6dd735c20 cmd/pk: strip knownKeys from dumpconfig
Fixes #1204

Change-Id: If463b84d8250ed1b0f48e6ed24b7d37843f082cb
2018-09-26 02:00:19 +02:00
mpl 03a10b5c34 cmd/pk-mount: make -version flag actually work
Fixes #1219

Change-Id: I3eb81a3f3fdd9dbaabd3510e8d9d80e5b0ef4b8e
2018-09-24 20:12:12 +02:00
mpl 5ca78c73d2 cmd/pk-mount: add schema summary to doc
Change-Id: I516d6fe5c364b6eb1d7f8977a67791ca399d214e
2018-08-28 18:55:20 +02:00
Euan Kemp 1ca3a0443e pk-deploy: set logger to fix panic
Fixes #1206

Change-Id: Ifa4ba0d0face1f4ee176cbaed483ec173e7dbbb7
2018-08-13 10:29:36 -07:00
Brad Fitzpatrick 9d0472a968 Merge "pk/fs: add Logger, set it to discard when not debugging" 2018-05-19 02:44:04 +00:00
mpl 17c1685159 pk/fs: add Logger, set it to discard when not debugging
Change-Id: I66914d436b7e3d15416a8089fac0d1c1107ce13e
2018-05-16 20:08:56 +02:00
mpl 5789ec4b8d pkg/fs: add sha224-xx...xx hint dir
And fix sha1 assumption for file name.

Fixes #1169

Change-Id: I2a907c6cf11631a4a86268a4dc393c64402b5d35
2018-05-16 18:03:18 +02:00
Brad Fitzpatrick f3f38f0c76 perkeepd, serverinit, gce: opaque-ify serverinit.Config, trim camlistored.go
This change has two major parts, which were interwoven enough to do
them in one change:

1) make serverinit.Config fully opaque, in prep for TOML configs #1134

2) shrink the massive server/perkeepd/camlistored.go file. It was out
   of control and had a bunch of code that better belonged
   elsewhere. This change moves a few hundred lines of code from
   camlistored.go into more logical places: internal/osutil/gce for
   GCE stuff, serverinit for config stuff (KeyRingAndId), etc.

I also added a TODO to make it possible to compile perkeepd without
any GCE stuff, which I saw as a possible and worthy goal only after
moving everything away.

Updates #1134

Change-Id: Iea6f84c5aca9c70b97806f4a201ec35e0f630e3b
2018-05-13 15:50:10 -07:00
mpl b76a4f6727 internal/osutil: revert to using "test safe" func for identitySecretRing
And while we're at it, rename said func (CamliConfigDir) to
PerkeepConfigDir.

Fixes #1161

Change-Id: Ia8c9b5826046437b77c8e16ee91912e31ad319a3
2018-05-12 02:12:03 +02:00
mpl a4864af2b4 pk: fix get Describe that was copied from put
Change-Id: I4e166f2a1b37bdae4a10da2366dc23cfc8dcb054
2018-05-11 20:52:13 +02:00
Brad Fitzpatrick e621fc3e01 Merge "importer/feed: Add date fields to imported items" 2018-05-07 14:57:39 +00:00
Amanda Cameron 276137d656 importer/feed: Add date fields to imported items
This also changed a few of the magic strings in the SetAttrs call to use
the nodeattr constants.

One thing I'm unsure about is if "link" should be changed to
nodeattr.URL, but that's left out of this initial revision.

This also sneaks in a change to devimport that was blocking
the testing of the feed service, as it doesn't use authentication.

Fixes #1148

Change-Id: Ic3cf85dc30c446954f3780683cba99f118b46fb6
2018-05-07 08:32:16 -04:00
Paul Lindner 19079950b0 pkg/sorted/sqlite: default to using journal_mode=WAL
Remove explicit version checking using pkg-config and just check if
setting journal_mode=WAL succeeds.  Remove -wal command line
flag from pk tool and also remove duplicated logic for db
handling.

Fixes #840

Change-Id: I72874a5bb9c5bb3a7be552a65e5ddb0c12fd726e
2018-05-06 01:47:07 -07:00
Brad Fitzpatrick 789f861964 all: rename some references to .config/camlistore to .config/perkeep
Updates #981

Change-Id: Id76fe17d80380bdb973bc85ec8f54ba9500aff40
2018-05-02 21:36:29 -07:00
Brad Fitzpatrick f483cba421 cmdmain, cmd/pk: support for demoting prominence of some subcommands
We saw users get confused about the flood of help out of useless
commands like dbinit.

Change-Id: I627bd4f0e79abfcf2274626112d31c965c43396a
2018-05-02 13:48:04 -07:00
Brad Fitzpatrick 46bb719dc5 buildinfo: track Version and Dockerfile separately
And don't hard-code VERSION info in the Dockerfile.

Change-Id: I35cb22fbb56cd634be4f1342c54ca86ce0e79901
2018-05-02 13:36:32 -07:00
Brad Fitzpatrick 576dabac82 Merge "cmd/pk-mount: in debug mode, dump stacks on SIGQUIT" 2018-04-27 18:24:38 +00:00
Brad Fitzpatrick c5c5270dd0 cmd/pk-mount: in debug mode, dump stacks on SIGQUIT
Change-Id: I70737247bf29063a7061c330960b7657213266cb
2018-04-27 11:24:04 -07:00
Brad Fitzpatrick e52593865b cmd/pk-mount: default to mounting at /pk if it exists
And recommend /pk.

Change-Id: I5255c203e46b503a1b4c59856e085d7359d84f4f
2018-04-26 15:46:24 -07:00
mpl 7a27eb42eb cmd: rename camget to pk-get, and make it a mode of pk
A couple of related fixes and renames too.

Updates #981
Fixes #1056

Change-Id: Id47a933c77422edfb9db41d34c38ed9d0d7a1846
2018-04-22 20:50:54 -07:00
mpl 2bb666ccf6 all: rename remaining occurrences of camput
Also removed misc/buildbot while at it (which contained camput
references) since we don't use it anymore at all.

TODO: the OSX app seems to be relying on finding a binary in ../bin,
which we do not use anymore. This will probably need fixing.

Updates #981

Change-Id: I14220fbad2e81181330fca4bb2d2e5fe170e1bd6
2018-04-21 16:20:24 -07:00
Brad Fitzpatrick ca76a40bbc Rename camlistored to perkeepd.
Updates #981

Change-Id: I8fe43c240c149074c23128a89ab426af9cbf94b4
2018-04-21 11:06:09 -07:00
mpl ce4658abfc cmd: rename camput to pk-put, and make "pk put" call it
A new "put" mode is added to the pk command, so that the "pk put"
command can be used to create and upload blobs.

What this command does is actually just call the previously named
"camput" executable, which is renamed to "pk-put" in this change.

This involves adding a new way to register a mode in cmdmain, when such
a mode is just meant to call an external binary. To emphasize the
distinction, the existing func (to register a sub-command, or a mode) is
renamed from RegisterCommand to RegisterMode, and RegisterCommand is now
the name of the new func/way.

Updates #981
Updates #1056

Change-Id: Ief954c17aa88a376f551df7de4b4e9fe41ad96d1
2018-04-21 10:26:55 -07:00
Brad Fitzpatrick a13abdeb8c cmd/camput: add flag to specify hash function for raw blobs
Also a bit more logging around indexing in debug mode.

Change-Id: I2eb67cfec12cff102ba64b17de0369bde38e416a
2018-04-20 21:02:43 -07:00
Brad Fitzpatrick 27abfcced5 cmd/pk: make "search" command highlight position of JSON syntax errors
Change-Id: Ie63b32f9dee403b9a985e45a646ae1ece5d2cc7c
2018-04-20 20:30:18 -07:00
mpl 6dcbe2ee9c cmd/camtool: rename from camtool to pk
Updates #981
Updates #1056

Change-Id: Ib78c47a69fa9816d5ac6cc547586a2af321005e3
2018-03-06 16:20:59 +01:00
James Hillyerd 8c18a7bfec cmd/camtool: add "-1" flag to search subcommand
search -1 will output one blob ref per line instead of JSON, suitable
for piping into xargs.

Fixes #1034

Change-Id: I7bf812767fa9859ff5f5d98858abb688906f20f6
2018-02-28 11:36:09 -08:00
mpl db2604f981 pkg/schema: break static-sets in subsets for large directories
The current maximum size for a schema blob is 1MB. For a large enough
directory (~20000 children), the resulting static-set JSON schema is
over that maximum size.

We could increase that maximum, but we would eventually hit the maximum
blob size (16MB), which would only allow for ~300000 children. Even if
that is an uncommon size, it is technically possible to have such large
directories, so I don't think it would be reasonable to restrict users
to such a limit. So it does not seems like enough of a solution.

The solution proposed in this CL is to spread the children of a
directory (when they are more numerous than a given maximum, here set to
10000) onto several static-sets, recursively if needed. These
static-sets (subsets of the whole lot of children) are stored in the new
"mergeSets" field of their parent static-set schema. The actual fileRefs
or dirRefs, are still stored in the "members" field of the subset they were
spread in. The "mergeSets" and "members" field of a static-set are therefore
mutually exclusive.

Fixes #924

Change-Id: Ibe47b50795d5288fe904d3cce0cc7f780d313408
2018-02-09 01:36:38 +01:00
Mathieu Lonjaret 6e8a5930c9 all: make server advertise when it has legacy SHA-1 indexes, use that info
when its wholeref index lines contain any "sha1-" wholeref lines.

Then, have the client use that info when uploading files to also
compute the sha-1 (in addition to the sha-224) and ask the server for
both.

By client, we mean changes to cmd/camput, to pkg/client functions (for
camput and other Go clients, such as the apps), and to the web UI.

Also, add a -sha1 mode to devcam server, as well as the
CAMLI_SHA1_ENABLED env var, to facilitate testing/debugging of the
legacy sha1 mode.  Fix/document unrelated options in devcam server
while we're at it.

Change-Id: I369679d2a4719f6f8b409f78dd4d065e13a3f62f
2018-01-31 01:10:52 +01:00
Brad Fitzpatrick f9f109ffa1 all: more Camlistore to Perkeep renamings
Updates #981

Change-Id: I7fd4958bceaa4ada63d07aacf08af11cc03f32d7
2018-01-30 13:31:59 -08:00
Paul Lindner 459c75410e all: more renaming of Camlistore to Perkeep
Change-Id: I118e3cbcf20d80afeffc84f001388c4556f21628
2018-01-30 03:02:56 -08:00