Commit Graph

180 Commits

Author SHA1 Message Date
Salman Aljammaz c90ae0219c add makefile to build systemd-docker from source
The systemd-docker image on Docker Hub uses an out of date version
of the Docker client. We build our own to get the version required
by recent CoreOS releases. See #646.

Change-Id: I0b1dc6c70f44055c8f92be44cf16269df8a63f0e
2015-10-19 20:52:48 +01:00
mpl a87632f593 dock.go: do not overwrite "production" tarball with WORKINPROGRESS
Change-Id: Ib71eaf028b56bc440a24d63af2f33156cb9935c6
2015-10-16 23:24:20 +02:00
mpl dc38e21fb1 dock.go: update CopyObject
its signature changed with last update of google.golang.org

Change-Id: I318f8966d3ced374607211c7ff27abe1ed30d28d
2015-10-16 23:03:21 +02:00
Mathieu Lonjaret def28fc337 Merge "third_party: no more import path rewriting because Go 1.5" 2015-09-17 11:59:14 +00:00
mpl 328ec25f9c third_party: no more import path rewriting because Go 1.5
We don't rewrite import paths anymore, because we use the new vendor
mechanism that comes with Go 1.5.

Also remove misc/pre-commit.githook because of the above, and because
the rest is covered by devcam hook.

Change-Id: Iafb32e19b21d1df44b9625a5f58bf898e6b51fa8
2015-09-16 22:22:51 +02:00
Mathieu Lonjaret 4c7a2b4aee Merge "dock.go: build windows tarball too" 2015-09-16 19:55:40 +00:00
mpl b432bf01c7 misc/docker: Go1.5 update + related simplifications
Update our Go image to use Go1.5.

Also, with Go1.5, we can compile static binaries (with CGO_ENABLED=0 and
--tags=netgo) without having to first rebuild Go itself from source with
CGO_ENABLED=0.

Finally, with Go1.5, we can now cross-compile straight outta the (linux)
binary tarball (without having to first build the necessary runtime in
src).

For both these above reasons, we don't need to use the Go -src tarball,
and we don't need to have a dedicated Go docker image for each GOOS we
cross-compile too.

Yay, progress!

Change-Id: Ibafb542a4771b151638e796ad3df78e0c8f1a4bf
2015-09-14 16:21:40 +02:00
mpl 14c87ca673 dock.go: build windows tarball too
Change-Id: I0b0f60828949fd321f99fc12eeef5d8e22d3cbc1
2015-08-25 19:26:55 +02:00
mpl b49c8b94de buildbot: switch to Go 1.5
Change-Id: I1b97aa44157064a64baec8f2d3f22376c9438d21
2015-08-21 19:27:52 +02:00
mpl 4d5cfb81c7 deploy, docker: go 1.5 tweaks for GCE launcher
Change-Id: Id1bae51742cb3efbb580181dc3c2ae6814fcab65
2015-08-18 15:01:04 +02:00
Brad Fitzpatrick 6f58fc2426 Fix build, move more stuff from third_party to vendor. 2015-08-18 10:19:49 +02:00
Brad Fitzpatrick 06f7accf31 Require Go 1.5 (due out soon), start move from third_party to vendor.
See https://golang.org/s/go15vendor

This commit moves the google.golang.org/cloud/... packages. The rest
of third_party will move later.

This commit is a dependency for some cloud logging work which we'll
develop in Camlistore's tree.
2015-08-15 22:45:11 +02:00
mpl e12d2aa765 misc/docker/dock.go: bump default rev
Some new features (build binaries tarballs) added in commit
988fc8a43c depend on some files that were
added to the camli repo in that very commit.
So for these new features to work, the Camlistore source used must be >=
to that revision, hence that bump.

Change-Id: Ibb68c0f1a1c85277becc02ad73b902d77b5bc465
2015-07-21 16:57:39 +02:00
mpl d68657f75a misc/docker/dock.go: names and doc tweaks
Change-Id: I0e5de36d0001e5d13dda4ec166af0bb207f4694a
2015-07-18 00:06:31 +02:00
mpl 2c57aff2c1 misc/docker/dock.go: fix typo, wrong program was called
Change-Id: I627b25c604f6ae82bbb49d635664e811ba962af8
2015-07-17 22:42:11 +02:00
mpl 988fc8a43c misc/docker: build tarball of binaries for releases
This change:

1) adds misc/docker/go/darwin/Dockerfile to create a docker image with
Go setup for cross-compiling to darwin
2) adds misc/docker/release/build-binaries.go, which works similarly to
misc/docker/server/build-camlistore-server.go. The difference is it
builds all binaries with make.go instead of just camlistored, and
camlistored is hence built with all resources embedded.
3) adds the --build_binaries flag to dock.go to make use of the above.

Maybe 3) should be done in a new file (out of dock.go), but it is
convenient for now for code reuse.

Issue #538

Change-Id: I8ecd50d118a9ff5c3f53812cac4d7a81a0053b4f
2015-05-14 23:32:40 +02:00
mpl 2b1ce70c15 dock.go: upload versioned tarball
Instead of just uploading the unversioned tarball, this change first
uploads a versioned tarball, then makes a copy of it as the unversioned one.
This allows for easier reverts when overwriting the unversioned tarball
by mistake.

Change-Id: Ia8a1b9008e7a203f2fb3b640dc12da432be1baf3
2015-05-12 15:41:20 +02:00
mpl 7242a824db dock.go improvements
The (camlistored builder + UI resources bundler) was a piece of shell
run in a docker container. That piece of shell is replaced with the
misc/docker/server/build-camlistore-server.go go program. This
program is still run within a container of the camlistore/go image.
It is added "on the fly" as a resource to the camlistore/go image
as a mounted resource.

In addition, build-camlistore-server.go can take a local directory as
the camlistore source (instead of fetching a tarball at the given
revision). dock.go has been adapted to take advantage of that.

Change-Id: If32b523d850f6c932016f18231bb1b1a283e50d3
2015-04-21 22:56:03 +02:00
Brad Fitzpatrick 6e5ba41027 dock.go: set buildinfo.GitInfo in the final binary
Change-Id: Ie7db7c191d0837a43e0a27bd0b74fc31f9f26aec
2015-04-02 08:32:48 -07:00
Brad Fitzpatrick faba473e75 Make dock.go also able to upload Docker images to GCS.
Change-Id: I00b17f44972176d7cfbaa9c956b2417dd9155b10
2015-04-02 05:34:59 -07:00
Brad Fitzpatrick 322ce34fe7 Remove old stuff from misc/docker/camlistored/Makefile
Change-Id: I6eafdc65b8a3ab9046879e5b540e9635754ebfaf
2015-04-02 02:28:45 -07:00
mpl 8f244dfae2 misc/docker/dock.go: build and add djpeg
Change-Id: I20935a3f5dfc160c5fbd5680b12c1404bfc5dfe9
2015-04-01 16:21:29 +02:00
Brad Fitzpatrick 5ee33643fd More dock.go image building work, with mpl
Change-Id: I290d9364a49c95c5c5f35864f94d04780ae30843
2015-04-01 03:48:12 -07:00
mpl db4813e19f misc/docker: final Dockerfile for camlistored
Change-Id: I9dce6a2d04b342e3c5045cc460b227d1bd49b334
2015-04-01 11:51:16 +02:00
Brad Fitzpatrick 2e923925db misc/docker: start of dock command
Change-Id: I758081373b5e109a32fd3e22c0b983598c5a31d8
2015-04-01 02:24:56 -07:00
mpl c3853b5222 misc/docker/go: go without cgo
Change-Id: I662d97467e0621af3b20f3c77bc113ace2c48456
2015-04-01 11:17:30 +02:00
mpl b2cdca7f11 docker/camlistored: use djpeg-static
Issue #564

Change-Id: I4efcbadcedff8f120497869e11784f372d93188e
2015-02-16 19:00:39 +01:00
Salmān Aljammāz 1b087e719c docker image for a static djpeg
updates #564

Change-Id: Idf40aa915e20c0584ed02ecab90b304dc159a371
2015-02-13 20:20:40 +00:00
mpl 6819477c6f docker: build djpeg, squash it, use it as base for camlistored
Issue #564

Change-Id: I78e3a7a286dedc413299de7ad4f7a99c6c889fa8
2015-02-10 17:22:51 +01:00
mpl bee9fdcaea docker/camlistored/Makefile: change build tag, add target to save
Change-Id: Ie2bcad5f9349fd4e2f1006b56de4427b5eed1461
2015-02-03 18:00:27 +01:00
mpl 424ccbdeba misc/buildbot: keep it alive until rewrite
Because of the hg->git switch and the switch to Go 1.4, which requires
bootstrapping for building from source:

-master now requires an env with a working Go to build builder (instead of
building go tip from source), and master does not monitor changes to the go
tree anymore.

-builder gets Go1.4.1 from the binary tarball (instead of building from
source), and does not build go tip anymore (nor does it run tests with
it).

Keeping it alive properly was not deemed worth the trouble since we plan
on replacing it with a new builder based on the same tools used for the
Go builders.

Change-Id: Idd0864350b27b6523675006f00719fba2d6fc72f
2015-01-27 00:56:40 +01:00
mpl 00b18d99d4 misc/docker: rm spdy proxy since we have http2
issue #534

Change-Id: Ic05b60aa1c220b9df76718edc2ca222cddc427dc
2015-01-14 15:46:18 +01:00
mpl cfd784269e deploy/gce: use subdirs for config and blobs (instead of buckets)
http://camlistore.org/issue/528

Change-Id: Icb558d7de032f9367e8fb100be87ef24c2982c23
2014-12-17 16:20:58 +01:00
mpl 99fe925efc pkg/deploy/gce: lib + tool to deploy on Google Cloud
http://camlistore.org/issue/531

Change-Id: I69ffe0544341d380bb844aef1dcca8a1ae441ea7
2014-11-26 22:50:27 +01:00
Salmān Aljammāz 4ed9ebb9fd Dockerfile for an nginx SPDY proxy
We can enable this once the changes to gce/create.go are in.

https://code.google.com/p/camlistore/issues/detail?id=534

Change-Id: I7ea77bf9b09a2f6d91e4f5e228320fa97d54dfa2
2014-11-16 14:00:52 +00:00
Salmān Aljammāz 4e39af1086 Generate self-signed certs with the correct hostname.
This change makes gce/create.go generate a self-signed certificate
with the hostname from the -hostname argument and upload it to GCS
before creating a new camlistore instance.

It also makes camlistored use baseURL to figure out the hostname
when generating its self-signed certificate.

Change-Id: I64f85853dab34a7ce95e5d5997e58f2e5da43496
2014-11-13 15:53:40 +00:00
Mathieu Lonjaret 0ac70ad2a5 Merge "devcam: run git pre-commit hook as a test." 2014-10-27 21:44:55 +00:00
mpl 762d2d2974 gce: make token cache file project dependent.
Also add a few hints.

Change-Id: Id51a933be35bc73622e335a9d323696f77dd8395
2014-10-25 01:04:49 +02:00
Bill Thiede af19bf089c devcam: run git pre-commit hook as a test.
Additional check to prevent b bad imports and un-go-fmt'd code
from sneaking in.
If users don't run tests, it will break the builder.

Change-Id: I95c74724aac68d915aa5195b6d8fcb78f92349e0
2014-10-02 09:32:23 -07:00
Brad Fitzpatrick 8bbd38f2dd GCE: put the thumbnail cache on /var so CoreOS doesn't wipe it
Also, move MySQL's data dir so they're both under /var/lib/camlistore

Change-Id: Iffc35efb68a69748ecbe59a6bfc99e9886ae325b
2014-08-23 19:55:44 -07:00
Brad Fitzpatrick f9a00dce0b GCE: use our own lighter MySQL Docker image.
Use systemd/CoreOS instead of supervisor,
Less InnoDB memory,
No snapshots,
unrelated to Docker image: keep data files on CoreOS /var/lib/mysql-camli so
they persist across reboots.

Change-Id: Ib903c386fb3e2df0b7cf2d3eb466290f2b16f94b
2014-08-23 17:49:59 -07:00
Brad Fitzpatrick 1ca6a739cd gce: use systemd-docker on CoreOS
Change-Id: I4f5a169c78ca3806aa40b3716b267b539c8bfbca
2014-08-23 09:13:06 -07:00
Brad Fitzpatrick c78e9af4b6 gce: run mysql too. link it with camlistored.
I think I'm not using systemd correctly here. Things get very unhappy
if I reboot CoreOS: the Docker container named 'db' already exists, things keep
flapping up and down (not waiting for their dependencies), etc.

Maybe I need to register stop actions too.

Change-Id: I75ac3e965c03df4f7f3938ff13c66f137948bf4d
2014-08-18 23:18:45 -07:00
Brad Fitzpatrick 082876c600 gce: don't make http client twice.
doesn't really matter, but.

Change-Id: I21495be1780e74464c7016979174ba722400ef1a
2014-08-18 17:17:33 -07:00
Brad Fitzpatrick 5a95215fcc GCE: use host's /tmp dir instead of inside container.
Change-Id: I2a52249148541c97dfdd84bac57a1605e0366471
2014-08-18 17:13:46 -07:00
Brad Fitzpatrick e86d90dd42 GCE: expose host's systemd journal to camlistored; add /var/logs handler
The machine now boots with a new pair of cooperating systemd units:

1) cam-journal-gatewayd.service: a copy of the systemd-journal-gatewayd service,
   which runs an HTTP interface to the systemd journal.
2) cam-journal-gatewayd.socket: a unix socket listener listening on unix
   socket /run/camjournald.sock. Incoming connections will forward to 1).

Then the camlistored.service unit running camlistored under Docker now
passes -v /run/camjournald.sock:/run/camjournald.sock to make that unix socket
available to the Docker container.

Then in camlistored, a new handler at /debug/logs (wrapped in auth
checks) then opens that socket and makes an HTTP request to it,
copying its response (of log lines) back to the browser.

This will ease debugging, letting people only use their browser to
debug (or send logs to the Camlistore developers more likely), rather
than sshing in to CoreOS and learning CoreOS and systemd arcana.

Change-Id: Icd5967ae7e9946d36229bdbc5d37644a11ee5e9f
2014-08-18 15:20:59 -07:00
Brad Fitzpatrick b191ece806 GCE: start systemd-journal-gatewayd.socket on boot.
Change-Id: I53e679f9a5f745c9a35904cdf65ff7844420e23b
2014-08-17 14:21:59 -07:00
Brad Fitzpatrick 274f838f6f Add make.go support for building the camlistored Docker binary.
Change-Id: I2f8ad914b217843749c0aee57c24ce44c1c619f1
2014-08-16 14:56:40 -07:00
Brad Fitzpatrick 494e9e7814 docker: add push target
Change-Id: I82fd35a3f1e5c4a76c9774aff28328a7acc97e78
2014-08-16 14:27:51 -07:00
Brad Fitzpatrick 38b97e1b9a Add Dockerfile and Makefile for 'camlistore/camlistored' Docker image.
Change-Id: Ied7055689f1946ce0e6a857b4e6ddd5f8dca79b9
2014-08-15 22:20:57 -07:00