Commit Graph

6693 Commits

Author SHA1 Message Date
Govert Versluis 8548962dbe Add Azure blobserver support.
Fixes #425

Change-Id: I02bb29e6503bfef0894cbfde0c2a3304cf70c932
2017-12-29 12:39:49 -08:00
Tamás Gulácsi 4754ab6c4b pkg/blobserver/diskpacked: fail earlier in StreamBlobs
As Miki Habryn suggested at
https://groups.google.com/forum/#!topic/camlistore/WmUyUWMfZx0%5B1-25%5D

Change-Id: Ib910e5bcfa7eb33360f7b5e1085bd9bb1f0e9e6a
2017-12-29 12:19:25 -08:00
Brad Fitzpatrick 815575e493 Merge "pkg/rollsum: fix uint8 overflow" 2017-12-29 19:41:36 +00:00
Paul Lindner 2c1f37be49 website: Add Google Webmaster Tools verification file
- This allows Perkeep admins to use https://google.com/webmaster/tools
- This verification file is mapped to user pmlindner@gmail.com
- Once verified he can delegate ownership to other Perkeep maintainers.
- File must remain for periodic verifications.

Fixes #996

Change-Id: I8d991964f75acf1b14dedfeadc9d4dbe671bccd5
2017-12-29 11:32:54 -08:00
Tamás Gulácsi 2329b8038c blobserver/diskpacked: fix missing Close on Seek error
Fixes #667

Change-Id: I11eaa2cc21bfbc825b14cc91208fd8ebc9e7418e
2017-12-29 11:29:09 -08:00
Brad Fitzpatrick 956a0a810b pkg/blobserver/localdisk: simplify code, limit stat concurrency
Don't create an unbounded number of stat goroutines.

Change-Id: Ie66cc9c680bd83e649966258a8e7ef09c8af5c62
2017-12-29 11:22:47 -08:00
Daniel Heath 59ac14caa5 Vendor OpenSans (apache 2 licence)
Change-Id: Ie74b37efc007011b32810fd91c65b7750588a76e
2017-12-29 10:22:49 -08:00
Stephen Searles 6b426cb10d improve proxycache and stats blobservers
improving proxycache
- added fuller sample config to the package documentation
- switched the stats caching from sorted.kv to the stats blobserver
- added a cleaning mechanism to evict the least recently used blobs
- implemented StatBlobs to actually inspect the local cache. It still
  always consults the origin, but only for the blobs necessary after
  giving the cache a 50ms headstart.
- logging a few errors that were previously ignored
- added tests modeled after the tests for the localdisk blobstore
- added a method to verify the cache, and call it on initialization
- added a strictStats option to always get stats from the origin
- filling in cacheBytes on initialization

improving stats blobserver
- implemented a few more of the blobserver interfaces, Enumerator and
  Remover
- Fixed a bug(?) in ReceiveBlob that seemed to prevent it from actually
  storing stats
- added a test

minor improvements include:
- blobserver/memory: allowing the memory blobserver to hold actually
  infinite items, if desired
- blobserver: closing dest in the NoImpl blobserver, as required by the
  BlobEnumerator interface
- storagetest: not closing dest leads to deadlock
- lru: max entries of 0 now means infinite (maybe do anything <0?)
- test: a helper function to create a random blob using a global random
  source that is, by default, deterministic, to make test results more
  consistent.

In the future, an improved BlobHub or similar interface could allow a
tighter feedback loop in providing cache consistency. i.e. the cache
could register with backend stores to be notified of content updates,
minimizing the time between backend changes and cache correction.

The proxycache will verify itself at startup, reporting an error if
any of its blobs do not exist in the backend storage or if the backend
storage has a different size for the content than the cache.

Fixes #443

Change-Id: I9ee1efd8c1d0eed49bb82930c2489a64122d3e00
2017-12-29 10:09:15 -08:00
Brad Fitzpatrick d76629024f Merge "cmd/camput: Add --permanode option to camput for blob" 2017-12-29 15:57:30 +00:00
Brad Fitzpatrick d893baa701 Merge "website: added note about MySQL permissions in server-config" 2017-12-29 15:54:10 +00:00
Mike Lloyd 223fb6e223 website: added note about MySQL permissions in server-config
This change specifies the required permissions when using a MySQL or
MariaDB index

Change-Id: Ib8852234946037da8f902bdaf4a6346f0f23ddec
2017-12-29 15:53:54 +00:00
Brad Fitzpatrick f4b62de003 Merge "pkg/client: set ServerName before DialTLS handshake for android" 2017-12-29 15:52:40 +00:00
Brad Fitzpatrick 78e9aade99 Merge "vendor: avoid retaining irrelevant vendor files and enumerate existing removals" 2017-12-29 01:18:03 +00:00
Brad Fitzpatrick 19521f9fc2 Merge "vendor: remove unused package github.com/cznic/bufs" 2017-12-29 01:08:14 +00:00
Paul Lindner 05d75586fb vendor: remove unused package github.com/cznic/bufs
Part of solving #889

Change-Id: Ie6717e52e96b12e45ae3cf0ac4cd1fdc4da7e044
2017-12-28 15:40:03 -08:00
Paul Lindner bf0ca6088a vendor: update go4.org to rev 034d17a462f7b2dcd1a4a73553ec5357ff6e6c6e
Part of solving #889

Change-Id: Iec1d8c68481da4c94bb19f9369ac0f06f5e12dcb
2017-12-28 15:39:32 -08:00
Paul Lindner 38ae5fe6db vendor: avoid retaining irrelevant vendor files and enumerate existing removals
Previous vendoring removed a number of files manually.  This enumerates the
existing removals in .gitignore and makes it easier to transition to another
dependency management system. We also take this opportunity to add travis and
gitignore files to the list and remove those from the vendor tree.

Part of solving #889

Change-Id: Ia1b1b11ea5f2065de3a069022c2ee346e2befe78
2017-12-28 15:36:01 -08:00
Brad Fitzpatrick ab66af2b3b cmd/camput: clean up a bit
Remove premature and unused http.FileSystem abstraction, remove
seeking in defer, remove unnecessary named result parameters, remove
explicit use of sha1, etc.

Change-Id: I0a9169a11e03b7e54dc39e73212e3ada6ec2f943
2017-12-27 08:31:58 -08:00
Brad Fitzpatrick 504604e22d blob: add Ref.EqualString method
Adds allocation-free way to check if a blob ref is equal to its
stringified form.

For #972 (doesn't fix it, as that bug is about a pending CL)

Change-Id: I49c6dee162698d38bb12314623b1507ee7bb246e
2017-12-26 17:37:00 -08:00
Brad Fitzpatrick 36f351ff50 Merge "pkg/webserver: remove usage of the obsolete runsit package" 2017-12-27 01:13:33 +00:00
Paul Lindner ca067082a3 vendor: remove unused go-spew package
Change-Id: I97d629e23406831ed93ea46634ff030695586a23
2017-12-26 16:11:28 -08:00
Paul Lindner b077a5916e pkg/webserver: remove usage of the obsolete runsit package
The runsit package is obsolete.  Pull the listen code directly into webserver and
remove support for the runsit specific named ports.  Update TODO.

Change-Id: I0d8ea798375d0eb4abea86ed9e6454376233e992
2017-12-26 15:03:59 -08:00
Paul Lindner 6b283519e6 vendor: remove unused gomemcache package
Change-Id: I7cafc05d758e0efe09b7701b346236f92f73ad3c
2017-12-26 01:15:07 -08:00
mpl 8b2e541731 pkg/client: set ServerName before DialTLS handshake for android
Otherwise, the android app fails to connect with a server that uses
Let's Encrypt (because it relies on SNI, which requires the ServerName
to be set).

Change-Id: I9f25486bea68e83c68584a83817c98bfc84f62b9
2017-12-22 19:02:47 +01:00
mpl bedb1d5e37 website: update twitter handle
Change-Id: Ic5c36d565424349bb3396e5a52178bf5fd2b6842
2017-12-22 00:34:33 +01:00
mpl c03615688c website: link blobserver protocol on /doc/ page
Change-Id: I72ffe096dea13242b088b303561c237dca623dcb
2017-12-21 17:37:50 +01:00
Brad Fitzpatrick 8e02a79b5b osutil: add func to report whether the secret-keyring flag is loaded
Then use it in pkg/client and remove the workaround from cmd/pk-devimport.

Change-Id: Iebd380d8c021628ac5e52219d00cfdf8e03f6b7d
2017-12-20 12:53:22 -08:00
Brad Fitzpatrick 393294643b Merge "cmd/pk-devimport: command that runs a local importer" 2017-12-20 20:03:45 +00:00
Brad Fitzpatrick e532defb67 website: emphasize opencollective, add logo
Change-Id: I4f9260ba275e55c9e4427751a70297a302d2c5b1
2017-12-20 10:22:14 -08:00
mpl d90d6c28eb website: add link to open collective page
Change-Id: I863bfa0b8b699a3754e4795ca72dcfb39c43d2c5
2017-12-20 18:32:42 +01:00
mpl 99e71732b9 cmd/pk-devimport: command that runs a local importer
Change-Id: I62e35db0040cda51f2bc5ede4560974e3f3442cc
2017-12-20 17:58:56 +01:00
Paul Lindner d81b001d24 cmd/cammount: add more documentation and examples to help people get started with PerkeepFs
Fixes #789

Change-Id: I9f0d0a8a823902ed2ae1bf83d4a5b58fadac1d7d
2017-12-19 15:32:33 -08:00
Mathieu Lonjaret 574a9311ec Merge "website: first pass of s/Camlistore/Perkeep/ on contents" 2017-12-18 15:47:03 +00:00
mpl cea35f8a40 website: first pass of s/Camlistore/Perkeep/ on contents
A second pass will be needed on the URLs once we have moved all the
repos and such.

Issue #981

Change-Id: Ibb58f4c4f16481da69e811142e8337b826c0d164
2017-12-18 16:46:08 +01:00
Mathieu Lonjaret 403ba82d14 Merge "camweb: enable default contributor name" 2017-12-18 15:17:24 +00:00
Mathieu Lonjaret ee580cd5f5 Merge "pkg/deploy: ignore Domain field from page template" 2017-12-18 15:16:53 +00:00
Brad Fitzpatrick 95e119f16f importer/gphotos: don't return channels in internal API, use foreach funcs
This code previously had methods returning channels. Such APIs are
always error-prone and difficult to use. Switch to a synchronous func
callback pattern instead, with contexts propagated.

Change-Id: Iaa1b91227c0daf4c8562fcba8d27dbcd7ab755c5
2017-12-17 17:43:23 -08:00
Brad Fitzpatrick e996c47bd1 vendor: add golang.org/x/sync
Adds golang.org/x/sync to vendor at fd80eb99c

Change-Id: I4b24acff93f0ace5f2ff0ba3c40e8b4145e1f60d
2017-12-17 14:27:26 -08:00
Brad Fitzpatrick ac517f7b75 Merge "importer/gphotos: more little cleanups" 2017-12-17 20:06:24 +00:00
Brad Fitzpatrick 77a37d01b3 importer/gphotos: more little cleanups
Change-Id: Ia97d56eb3e3f302f60c30af08a220cfa011936e2
2017-12-17 08:29:50 -08:00
mpl b66ede8500 camweb: enable default contributor name
Because I forgot to configure git one time on another machine, one of my
commits is with a lame username, and that username gets picked (instead
of the other ones) as my identity when generating the contributors page.

Therefore, this change allows to specify a default name among the already
hardcoded information for some of the contributors, so I can override
the badly configured identity that would appear otherwise.

Change-Id: I60dfe60f83d20aa33f7bf425e5b84a34bbd5a0b1
2017-12-16 04:23:27 +01:00
mpl 253e7cb5d4 pkg/deploy: ignore Domain field from page template
Fixes #985

Change-Id: If110d392885fb89d0403ab9eb3aaf8dcdc3fe249
2017-12-16 02:01:36 +01:00
mpl c45893a69b importer/gphotos: add enabling Drive API to instructions
Fixes #984

Change-Id: I12126318cf836536306fd7a861a730fc3196c7b7
2017-12-15 19:08:51 +01:00
Brad Fitzpatrick fcd5ff14e9 importer/gphotos: misc rate limit cleanups
Remove dynamic rate limit adjustment for now. It was racy.

No need to be super fast, anyway, as long as it catches up eventually.

But we can make it smarter later. I wanted to get it correct first.

Change-Id: Id5b5fc946546d8d9c0720f1c0ec2f341a17cdd01
2017-12-14 22:16:57 -08:00
Mathieu Lonjaret 286dbacc62 Merge "camweb: make syncing to github concurrent" 2017-12-15 00:08:13 +00:00
mpl 31389f8ab2 camweb: make syncing to github concurrent
Camweb is in charge of syncing the commits from the gerrit repository to
our mirror on github.

It also sends the commits as e-mails to the Camlistore mailing-list.

The former was only happening after the latter did, but there was no
reason to do it that way, as these tasks are independent. Moreover, when
sending e-mails is problematic (issue #980), it delays syncing to
github.

This change therefore makes syncing to github concurrent with
sending the e-mails.

Change-Id: I63a2b3f5b49df58ca30ec5153ce65eafb44b5b28
2017-12-15 01:05:56 +01:00
mpl 602fe8a2d4 website: serve Perkeep title
Issue #981

Change-Id: I2d323523ed29dfcb5989178c7cc0c78c462f972f
2017-12-14 16:10:43 +01:00
Mathieu Lonjaret 198d39ba5d Merge "website: s/Camlistore/Perkeep/ on index page" 2017-12-14 03:40:45 +00:00
Paul Lindner 7ae6e00368 pkg/serverinit/testdata: fix tests broken by changing error message for golint
Change-Id: Id48a2ebaed056e50f1ac6b34c4869b07590d55d3
2017-12-13 11:33:30 -08:00
Paul Lindner 04b9c0b1c4 pkg: lint fixes for "context.Context should be the first parameter of a function"
Change-Id: I0185eec5616c72d859978aec03880860421c7193
2017-12-13 11:33:21 -08:00