Commit Graph

7217 Commits

Author SHA1 Message Date
Brad Fitzpatrick 2606360f1d server/perkeepd: more cleanups
And move the camlistore.net code to a new file, autodns.go.

Change-Id: I7503b72c9c3f1c7a2207cab35c746c34dd90740c
2018-05-13 16:57:56 -07:00
Brad Fitzpatrick 7da4196852 server/perkeepd: rename source files from camlistore*.go to perkeepd*.go
Updates #981

Change-Id: I6c59fcd8d2ae56e102507a162c83ead85d20b039
2018-05-13 16:01:02 -07:00
Brad Fitzpatrick f3f38f0c76 perkeepd, serverinit, gce: opaque-ify serverinit.Config, trim camlistored.go
This change has two major parts, which were interwoven enough to do
them in one change:

1) make serverinit.Config fully opaque, in prep for TOML configs #1134

2) shrink the massive server/perkeepd/camlistored.go file. It was out
   of control and had a bunch of code that better belonged
   elsewhere. This change moves a few hundred lines of code from
   camlistored.go into more logical places: internal/osutil/gce for
   GCE stuff, serverinit for config stuff (KeyRingAndId), etc.

I also added a TODO to make it possible to compile perkeepd without
any GCE stuff, which I saw as a possible and worthy goal only after
moving everything away.

Updates #1134

Change-Id: Iea6f84c5aca9c70b97806f4a201ec35e0f630e3b
2018-05-13 15:50:10 -07:00
Brad Fitzpatrick 3b48afb2eb Gopkg.toml: fix typo in comment
Change-Id: I4094a026382b3ad5ec4222ac68eca5c36dbaf375
2018-05-13 15:23:07 -07:00
Brad Fitzpatrick 8bde2a9e9e vendor: add files missing from b57013cb82
Change-Id: I6945e5c53150346dee2933a95d29c5afc3d5b77e
2018-05-13 15:21:53 -07:00
Brad Fitzpatrick aef92f7e14 pkg/serverinit: start making serverinit.Config representation more opaque
Updates #1134

Change-Id: I56fa1c565c267dad0e1ca69f7d36338828ac954e
2018-05-13 11:16:18 -07:00
Brad Fitzpatrick b57013cb82 blobserver/sftp: add SFTP storage target
Fixes #1111

Change-Id: I4bedfa44241c1e6cc607658c991179ee04f9eb31
2018-05-11 20:53:50 -07:00
mpl b76a4f6727 internal/osutil: revert to using "test safe" func for identitySecretRing
And while we're at it, rename said func (CamliConfigDir) to
PerkeepConfigDir.

Fixes #1161

Change-Id: Ia8c9b5826046437b77c8e16ee91912e31ad319a3
2018-05-12 02:12:03 +02:00
Mathieu Lonjaret 5ffbaffbcd Merge "misc/release: update release tools" 2018-05-11 21:33:12 +00:00
mpl a4864af2b4 pk: fix get Describe that was copied from put
Change-Id: I4e166f2a1b37bdae4a10da2366dc23cfc8dcb054
2018-05-11 20:52:13 +02:00
mpl e27e2302a4 misc/release: update release tools
misc/docker/release moved to misc/release because it did not really have
anything to do with making docker images, it just happens to use them.

misc/monthly.go has been moved to misc/release/make-release.go, and
reworked to be more generic.

misc/docker/dock.go has been trimmed down to only deal with creating
docker images

Fixes #1153

Change-Id: I4cb566551007300aefa6cb23714b90461f0e3e51
2018-05-11 02:51:55 +02:00
Brad Fitzpatrick e22801a8ab server/perkeepd: don't ignore error from serverinit.WriteDefaultConfigFile
Fixes #1154

Change-Id: I1b35ab70ef3ad20fa8527a074485661a36b04e52
2018-05-09 20:51:42 -07:00
Brad Fitzpatrick d50667a178 blobserver/files: fix a data race in os.FileInfo stat readahead
And fix some log messages to remove "localdisk", since this code is
now shared by sftp and in the future will be shared by webdav, ftps,
etc.

Change-Id: I4c428cdd6c90bb509045d1914922326a10f48a17
2018-05-09 17:03:02 +00:00
mpl 2e052c5fe5 pkg/index: do not return error if dir is empty
When looking for the children of a directory, we used
to return an os.ErrNotExist if no children were found.
We now only return os.ErrNotExist if the directory
itself does not exist.

Fixes #1152

Change-Id: I8dd827f5597535d98c898302d0a1ca5399b03476
2018-05-08 19:51:00 +02:00
Brad Fitzpatrick f943f019cd Merge "osutil: fix Windows use old Camlistore-named APPDATA directory" 2018-05-08 03:53:34 +00:00
Brad Fitzpatrick 245bb63fff osutil: fix Windows use old Camlistore-named APPDATA directory
Fixes #1150

Change-Id: Ia2843437abbaba96e936c1c0f46bf8b171a2e28e
2018-05-07 20:52:13 -07:00
Brad Fitzpatrick e621fc3e01 Merge "importer/feed: Add date fields to imported items" 2018-05-07 14:57:39 +00:00
Amanda Cameron 276137d656 importer/feed: Add date fields to imported items
This also changed a few of the magic strings in the SetAttrs call to use
the nodeattr constants.

One thing I'm unsure about is if "link" should be changed to
nodeattr.URL, but that's left out of this initial revision.

This also sneaks in a change to devimport that was blocking
the testing of the feed service, as it doesn't use authentication.

Fixes #1148

Change-Id: Ic3cf85dc30c446954f3780683cba99f118b46fb6
2018-05-07 08:32:16 -04:00
Paul Lindner 19079950b0 pkg/sorted/sqlite: default to using journal_mode=WAL
Remove explicit version checking using pkg-config and just check if
setting journal_mode=WAL succeeds.  Remove -wal command line
flag from pk tool and also remove duplicated logic for db
handling.

Fixes #840

Change-Id: I72874a5bb9c5bb3a7be552a65e5ddb0c12fd726e
2018-05-06 01:47:07 -07:00
Brad Fitzpatrick a9db75c56d clients/osx: fix build
Updates #1147

Change-Id: Ic529ce5f4c0bbf60ae727c8dff41dfdcc84b7631
2018-05-02 21:52:02 -07:00
Brad Fitzpatrick 789f861964 all: rename some references to .config/camlistore to .config/perkeep
Updates #981

Change-Id: Id76fe17d80380bdb973bc85ec8f54ba9500aff40
2018-05-02 21:36:29 -07:00
Brad Fitzpatrick 7a5e0bd6d2 doc: fix link that doesn't link properly on GitHub
Managing our links between GitHub, Gerrit, and the website's rendering
of the same Markdown continues to be a disaster.

Fixes #1128
Updates #1146

Change-Id: I8645c6fee4bf5271ca4fc08acb69c53785a36182
2018-05-02 21:31:46 -07:00
Brad Fitzpatrick b5793c5e1e Merge "pkg/search: add some DirConstraint search functionality" 2018-05-03 03:53:55 +00:00
mpl d6fb092e69 doc/release: fix hashsums for 0.10
We made new tarballs because we fixed the version added to the generated
binaries.

Change-Id: If019532949915d7083e1f349076649d22d8a66d3
2018-05-02 19:04:40 -07:00
mpl 6a04db2f5d misc/docker: make release tools work with latest version mechanism
Change-Id: Id2081c2f0047621fa929baeba684de219eb869ff
2018-05-02 19:04:40 -07:00
Brad Fitzpatrick be5f93b8a3 doc: shout-out to Ashley for Keepy
Change-Id: I1bec4a6fe22aa07383c4a1fa0419a7c89ac419a8
2018-05-02 19:04:40 -07:00
mpl 692e370509 doc/release: add downloads to 0.10
Change-Id: I9038f90bc425c1055ba9a86ee5f9cd92f79e55d1
2018-05-02 19:04:40 -07:00
Brad Fitzpatrick 325893ac6b doc: remove download links for 0.10
So we can cut 0.10 from this commit and anybody running pk-web locally
won't have bogus download links.

Change-Id: Ibc33c2a01786e9288a67a761be6d4d8b5909a4de
2018-05-02 19:04:40 -07:00
Brad Fitzpatrick f6ad3092bc doc: more 0.10 release note docs
Change-Id: I05bef0f001eb3652a830a085b00e008b23574e0d
2018-05-02 19:04:40 -07:00
mpl 1aa358fc6f doc: start of 0.10 release notes
Change-Id: Ida9446c6d80c9b6884c1a04a77d0466e41bd2f24
2018-05-02 19:04:40 -07:00
Brad Fitzpatrick 6446c307a1 lib: delete ancient and broken Camlistore Python library
Unmaintained, unused(?) for like 8 years. I doubt it worked.

Change-Id: I519a3ffc3b9b9019d1812e143a1fd19e8248b660
2018-05-02 18:42:09 -07:00
Brad Fitzpatrick e64ed803e0 Merge "pkg/blobserver/s3: Replace s3-specific memory cache with a proxycache." 2018-05-02 23:17:36 +00:00
Daniel Heath af4ecfb5ea pkg/blobserver/s3: Replace s3-specific memory cache with a proxycache.
The s3-specific cache did not populate the cache on fetch (only on
receive) and did not cache stat() calls - proxycache does both.

Fixes #898

Change-Id: Ic0bfe684c9db79d16cb3c8c86ce87cc4676c63f0
2018-05-03 08:55:23 +10:00
mpl 94461e25ca devcam: add -branch option to review mode
Change-Id: I0c7d78bcb2ef357368db93751a8bb7a7977e528f
2018-05-02 15:25:29 -07:00
Brad Fitzpatrick ef8c2d3a40 Add a VERSION file to the master branch.
Change-Id: Id49e8daa82a3705dc32cd6f55c2b1417289f05ed
2018-05-02 13:51:47 -07:00
Brad Fitzpatrick 0c28468f65 Merge "cmdmain, cmd/pk: support for demoting prominence of some subcommands" 2018-05-02 20:51:30 +00:00
Brad Fitzpatrick 91c36ff1d3 Merge "buildinfo: track Version and Dockerfile separately" 2018-05-02 20:51:24 +00:00
Brad Fitzpatrick ceb4f4c498 Merge "Dockerfile: update for 0.10 release" 2018-05-02 20:50:32 +00:00
Brad Fitzpatrick f483cba421 cmdmain, cmd/pk: support for demoting prominence of some subcommands
We saw users get confused about the flood of help out of useless
commands like dbinit.

Change-Id: I627bd4f0e79abfcf2274626112d31c965c43396a
2018-05-02 13:48:04 -07:00
Brad Fitzpatrick 46bb719dc5 buildinfo: track Version and Dockerfile separately
And don't hard-code VERSION info in the Dockerfile.

Change-Id: I35cb22fbb56cd634be4f1342c54ca86ce0e79901
2018-05-02 13:36:32 -07:00
Brad Fitzpatrick 9f65efad5a Dockerfile: update for 0.10 release
Revive this Dockerfile, convert it from camlistore to perkeep, and
make it a multi-stage build.

We still need to clean this up later and not hard-code the VERSION
file contents inside it. But this is something.

Change-Id: I2378449ffbcc2ff005c9d441e378266694d7d430
2018-05-01 23:17:40 -07:00
mpl c39f4cabd4 misc/docker/go: bump to 1.10.2 for release
Change-Id: I98b2f85533615d3cfdc1f657d9f675f352febbd4
2018-05-01 21:57:37 -07:00
Brad Fitzpatrick fc64010f0e Merge "all: Windows fixes (don't listen on file descriptors in test.World, etc)" 2018-05-02 04:44:22 +00:00
Brad Fitzpatrick 12894d4630 all: Windows fixes (don't listen on file descriptors in test.World, etc)
test/integration: don't listen on file descriptors.
make.go: unrelated, but options to make it much faster.
internal/images: t.Skip on HEIC dependency failures

Fixes #1140
Updates golang/go#25210

Change-Id: I8092155411826d6ed1f8d85230b753d1369044af
2018-05-01 21:38:19 -07:00
mpl 03103f00fa pkg/search: add some DirConstraint search functionality
The DirConstraint type already existed but wasn't functional as none of
the matching for any of its fields was implemented.

That functionality is required by two other features:

1) Browsing directories in the publisher requires getting information
about the children of the directory. In practice that can be achieved
with a search on the directory, accompanied with some describe rules.
But that comes with limitations, such as:
-no control over the sorting of the described children in the response
-max number of children in the response (and no way to overcome it since
you can't "continue" on a describe)
hence the need for a direct search of a directory's children.
This is implemented as DirConstraint/FileConstraint.ParentDir and will
be used in https://camlistore-review.googlesource.com/8286

2) Looking for files/directories shared by transitivity.
Knowing if an item is the target of a share claim is easy enough once
enough of ClaimConstraint is implemented. But in order to find out if an
item is effectively shared because one of its ancestors (directory) is
the target of a share claim, with transitivity set, we need some sort of
search that can do directories traversal. This is implemented as
DirConstraint.RecursiveContains and it will be used as a subquery in
https://camlistore-review.googlesource.com/9866.

Now implemented:
	DirConstraint.FileName
	DirConstraint.BlobRefPrefix
	DirConstraint.ParentDir
	DirConstraint.TopFileCount
	DirConstraint.Contains
	DirConstraint.RecursiveContains

ParentDir will also allow us to quit relying on the treeHandler in the
web UI,
and to use the more generic search queries mechanism instead.

Change-Id: I022cb51732ee4271906271fd75c6f737856b6165
2018-05-01 17:55:41 -07:00
Mathieu Lonjaret 4cc45cdcc8 Merge "misc/docker: fix release tools" 2018-05-02 00:09:38 +00:00
mpl 5540ae14ac old/camwebdav: remove
Change-Id: I1bd0b64dbb39aa51b69f030233d3ccf0031b9446
2018-05-01 16:14:58 -07:00
mpl 507684d5ee misc/docker: fix release tools
Change-Id: Ie7e75d88a4fbc7df753aedbfb00d77d1899962e1
2018-05-01 16:03:27 -07:00
Brad Fitzpatrick 31857ec50b test/integration: skip more symlink tests on Windows
Change-Id: I555a4db1e048e65517471090d4a10ad16f5f0167
2018-05-01 14:40:30 -07:00
Brad Fitzpatrick e25375ca22 schema: skip symlink tests on Windows
Change-Id: I7b076d886883ac0ed19df0cc5d3797cfc04b1c75
2018-05-01 14:17:35 -07:00