Commit Graph

23 Commits

Author SHA1 Message Date
Brad Fitzpatrick 57648c6b83 all: update copyright holder from Google Inc to The Perkeep Authors
The AUTHORS file is the list of copyright holders.
2018-01-03 16:52:49 -08:00
Brad Fitzpatrick 11e9c5567c Move some packages from perkeep.org/pkg to perkeep.org/internal
Notably: pkg/misc all moves.

And pkg/googlestorage is deleted, since it's not used. Only the
x/net/http2/h2demo code used to use it, but that ended in
https://go-review.googlesource.com/33230 (our vendored code is old).
So just nuke that dir for now. When it's refreshed, it'll either be
gone (dep prune) or new enough to not need googlestorage.

Also move pkg/pools, pkg/leak, and pkg/geocode to internal.

More remains.

Change-Id: I2640c4d18424062fdb8461ba451f1ce26719ae9d
2018-01-01 20:54:48 -08:00
Brad Fitzpatrick d6a0b05df0 Rename import paths from camlistore.org to perkeep.org.
Part of the project renaming, issue #981.

After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.

This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.

Also, this only moves the lru package to internal. More will move to
internal later.

Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.

This updates some docs, but not all.

devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).

Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
2018-01-01 16:03:34 -08:00
Paul Lindner 15feaeb24c all: lint fixes for 'error strings should not be capitalized or end with punctuation or a newline'
Change-Id: I9c3766a51ac8be694ae76befff4b6fa9a85e34eb
2017-12-11 06:13:25 -08:00
Will Norris 77ed42edf8 add canonical import paths
The import path was added to the go file that included the package
documentation if one existed.  Otherwise, I used what seemed to be the
primary file for the package.

Fixes #689

Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
2016-03-13 19:57:14 -07:00
mpl b9aec12f31 blobserver/s3: limit concurrent PutObject calls
Fixes issue #675

Since camput does not seem affected by the above issue, and that camput
(pkg/client really) main difference in how it uploads is that it does
gated requests, it seemed to follow that gating the requests might fix
the problem.

With this change, I don't see the problem anymore either in
TestS3WriteFiles or by manually uploading with the web UI.

I suppose this indicates that S3 throttles incoming connections pretty
strictly, but I have no formal evidence of it.

Change-Id: I48f10923a710f4a21e44bc251e648db37ae504ec
2016-01-25 21:39:57 +01:00
Mathieu Lonjaret 68f854537c Merge "pkg/blobserver/s3: directory & blobpacked support" 2015-12-18 21:07:56 +00:00
mpl 92d5c1d045 pkg/blobserver/s3: directory & blobpacked support
Fixes issue #529

Change-Id: Iea7aea015b58c0325887fa07ae3a004b7822511e
2015-12-16 23:58:27 +01:00
mpl b6520c9ebe vendor: go4.org/fault
at rev 4210238a68e472c5c5c1b0e95d89a2c6bffaebdf

And remove pkg/fault

Change-Id: Ib80cbf7ad8db20c3e47b4e321e7355d9215a4ec9
2015-12-16 23:08:58 +01:00
mpl 40be4d8da2 vendor: go4.org/jsonconfig (and errorutil)
Previously pkg/jsonconfig and pkg/errorutil

Copied from go4.org at rev d1b8a2fb2de6160036e4801aa5e4d855571078b8

Change-Id: I673ed55b0825baa2607289b6082f205100261d7a
2015-12-01 17:21:49 +01:00
mpl b24580d523 blobserver/s3: implement SubFetcher
Change-Id: I42ac364e268463998598990a8bf0df7437f4d05d
2014-10-19 16:11:54 +02:00
mpl 53be9298a5 blobserver/s3: add optional in-memory Fetch cache
Change-Id: I77e27555d28002ba01f1501e8a77eb4adbf701fe
2014-10-18 19:34:25 +02:00
Brad Fitzpatrick bf8d6bdb4f s3: auto-select correct region, handle buckets with dots in hostname
Fixes camlistore.org/issue/404

Change-Id: Ia6b3f51f1fd5beba585fa8dd8a918d30905fb9bb
2014-05-08 15:31:48 -07:00
Govert Versluis cef0aa0d16 Switch to different S3 endpoint
Switch to a URL in the form of
https://s3.amazonaws.com/mybucket.mydomain.com/myObjectKey
This adds support for buckets with periods in their name and S3 clones

Addresses https://code.google.com/p/camlistore/issues/detail?id=404

Change-Id: Id5c744c88916dbac5a0ab01330929546984ef2d2
2014-04-14 10:48:03 +02:00
Brad Fitzpatrick bfb0d1e8de s3: more fault injection, and simplify receive, removing spilling to disk
Change-Id: Id7fb01c40dc863bee483887bbf14934161eb03a0
2014-03-16 19:44:52 -07:00
Antti Rasinen f837fcb9d8 Convert all S3 requests to HTTPS
Also includes fixes for two missing data items
for errors.

Change-Id: Ib174b4bf83f3f264f463b4c203c1f42213b18f5e
2014-03-09 21:00:13 +02:00
Brad Fitzpatrick 658472577d s3: be a Stringer, make default host more explicit, add failure injection
Change-Id: I1c24e97ef4b901e3be62fbb7e34c3f53787eedb1
2014-03-05 08:24:30 -08:00
Brad Fitzpatrick f3f48eb70b Support alternate S3 hostnames
Change-Id: I9a27eb5fb9c4e5ef3b516ce9cf5a4b5bb4e1df61
2013-09-01 09:50:35 -07:00
Brad Fitzpatrick 91cdda8475 Match common Go case
Change-Id: I14606ee5ba422cf43f68dc65af2b40a096c7ca35
2013-09-01 09:13:13 -07:00
Brad Fitzpatrick b24cad68dd Cleanup: remove BlobHub and time.Duration waits from storage interface
Move up a layer to the HTTP.  Also, start to remove ContextWrapper
stuff.  We've done it differently for App Engine instead, and will do
it differently yet moving forward.

Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.

Bunch of other cleanup and TODO fixing too.

Much simpler and cleaner.

Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
2013-08-21 13:57:28 -07:00
Brad Fitzpatrick cf0d9aca6e More docs
Change-Id: I5c21f240c85bcf91fb67487cc172bf3faeb49fff
2013-07-07 18:52:14 -07:00
Brad Fitzpatrick 37bf028633 s3: verify configured bucket exists at start-up.
Change-Id: I43d0795641fe9dbc345a1f0916b828ede76cf4a6
2013-01-20 11:33:41 -08:00
Brad Fitzpatrick 0714a463c9 Update from r60 to [almost] Go 1.
A lot is still broken, but most stuff at least compiles now.

The directory tree has been rearranged now too.  Go libraries are now
under "pkg".  Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".

The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.

For now, put the camlistore root under $GOPATH/src.  Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.

Then you can:

$ go build ./server/camlistored

... etc

The build.pl script is currently disabled.  It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).

Many things are certainly broken.

Many things are disabled.  (MySQL, all indexing, etc).

Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.

The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.

Help wanted.

Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
2012-02-18 21:53:06 -08:00