Tagging images depending on the selected arch allows us to preserve
docker caching even when switching arches.
Also, since the first build stage was not tagged, it was considered as
dangling, so it was getting removed by 'docker rmi'. So we now tag it as
well to keep it cached.
Change-Id: I0c36c625e7bd459192f97052d26d76084a03bea7
vendor: add github.com/aws/aws-sdk-go at rev bc3f534c19ffdf835e524e11f0f825b3eaf541c3
This changes all uses of the s3 client to be the official aws-sdk-go. As
such, it also deletes the previous client maintained in internal.
There are a few notable changes that come with this:
1. An 'aws_region' key is added to the low-level config. The
'get-bucket-location' call is used to identify the correct region
rather than using the redirect returned to determine it.
2. Deleting multiple objects at once is much faster (batches of up to
1000 vs 1 at a time).
3. Retriable errors are retried automatically (see #1184).
4. Newer APIs are used for some operations (v2 / batch apis).
An unfortunate result of 4 is that this is technically a backwards
incompatible change. An IAM policy which used to be sufficient for
perkeep may no longer permit some of the new api calls (even if they're
not doing anything different really).
Fixes#1184, #911
Change-Id: I077fe88def18b7e9a14267820773245e7003634c
misc/docker/release moved to misc/release because it did not really have
anything to do with making docker images, it just happens to use them.
misc/monthly.go has been moved to misc/release/make-release.go, and
reworked to be more generic.
misc/docker/dock.go has been trimmed down to only deal with creating
docker images
Fixes#1153
Change-Id: I4cb566551007300aefa6cb23714b90461f0e3e51
Since we renamed Camlistore to Perkeep, we also renamed the docker image
deployed by the launcher, as well as the systemd service that this
docker image runs in.
This means existing users on GCE do not get updates when they restart
the service or reset their instance.
This change adds a check and a function that allows perkeepd to fix the
user-data metadata of its own GCE instance, which is where the names
mentioned above are defined.
Therefore, if we deploy a new camlistore docker image containing that
code, the existing users should get a server capable of self-fixing its
own instance into a perkeep one.
Change-Id: I9ff66f684f0ad647d15e5fcdd08d26c524a18329
Still slow, but no more ffmpeg child processes, and I'm confident
ImageMagick will improve over time. There are actually ImageMagick
commits in the past few hours that look promising. We have a demo
tomorrow, so we're not going to bump our git rev & Docker image to the
latest ImageMagick master yet.
Updates #969
Change-Id: I7a8545821cae4214ea521d7cb46aa235ae3f0ed3
Also removed misc/buildbot while at it (which contained camput
references) since we don't use it anymore at all.
TODO: the OSX app seems to be relying on finding a binary in ../bin,
which we do not use anymore. This will probably need fixing.
Updates #981
Change-Id: I14220fbad2e81181330fca4bb2d2e5fe170e1bd6
A new "put" mode is added to the pk command, so that the "pk put"
command can be used to create and upload blobs.
What this command does is actually just call the previously named
"camput" executable, which is renamed to "pk-put" in this change.
This involves adding a new way to register a mode in cmdmain, when such
a mode is just meant to call an external binary. To emphasize the
distinction, the existing func (to register a sub-command, or a mode) is
renamed from RegisterCommand to RegisterMode, and RegisterCommand is now
the name of the new func/way.
Updates #981
Updates #1056
Change-Id: Ief954c17aa88a376f551df7de4b4e9fe41ad96d1
multi-stage Alpine image.
static building is a pain, so we instead copy all the shared libs from
the builder stage to the final stage.
Udates #969Fixes#1087
Change-Id: I05377fdafd6db13d04d4970ec2d52b68a53fa22b
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
This switches most usages of the pre-1.7 context library to use the
standard library. Remaining usages are in:
app/publisher/main.go
pkg/fs/...
Change-Id: Ia74acc39499dcb39892342a2c9a2776537cf49f1
So far, when building camlistored docker image for CoreOS, we were not
using make.go, and we were neither running gopherjs nor embedding the
resources (but rather provide the UI resources at their default
filesystem location).
Now that we're using gopherjs for the web UI, it is a hard dependency
for the camlistore server.
We could reproduce the steps in make.go to build gopherjs, run it to
build the web ui resources, and then move the resources at the right
place, but since make.go already does the equivalent work it seems
to make more sense to use it, which is the main point of this CL.
Similarly, it seems to make more sense to now build a binary with the
resources embedded, which is the default make.go behaviour, instead of
building a "raw" camlistored, and provide the resources as additional
directories in the container image, so this CL takes that approach too.
Finally, it was necessary to add the "-static" flag to make.go, so we
can keep on building a static camlistored binary, that does not rely on
libc for DNS. Because our container image is FROM SCRATCH, with just the
necessary binaries, in order to get a container image of a reasonable
size.
One drawback of now using make.go in
misc/docker/server/build-camlistore-server.go is we're doing some
unnecessary (since we're already running in the isolation of a
container) copying to the virtual gopath, but that seems a very tiny
price to pay. Especially considering how rarely we run that code.
Change-Id: I416c86d366cd4ed2d3b8b1636a6a65a83b9f15d7
rev b9bcb1da229a59cc1e1d168401662cb6450aae08
Also added pkg/misc/fakefsnotify just so we only have to do:
- "github.com/fsnotify/fsnotify"
+ fsnotify "camlistore.org/pkg/misc/fakefsnotify"
in vendor/github.com/gopherjs/gopherjs/build/build.go
in order to avoid vendoring fsnotify, without having to grossly
patch the gopherjs code as we previously did.
Change-Id: Ic106a6afa1117d2061415355c236433f835895f1
Issue #839 suggests to not cut the rev at all, and let it be user
defined. I have no good argument against, but I'm taking the more
conservative approach suggested by others in issue #888.
Fixes#888
Change-Id: Id65b1b074e6646a756353119a12081330775e066
Requires updating vendor/cloud.google.com/go/storage because of the
BucketHandle.List removal, which in turns triggers a bunch of other
updates or additions.
cloud.google.com/go/internal/optional
cloud.google.com/go/storage
at rev 641b1469e744485a8e6235bffc3b7bf366758a85
github.com/googleapis/gax-go
at rev da06d194a00e19ce00d9011a13931c3f6f6887c7
google.golang.org/api/*
at rev f7e067168bd43883b2156f16b6c2fd91beec76e2
Fixes#884
Change-Id: I7e27a24fa2ffbb747e66bde0d10dabaad1ac60d6
mpl