We don't rewrite import paths anymore, because we use the new vendor
mechanism that comes with Go 1.5.
Also remove misc/pre-commit.githook because of the above, and because
the rest is covered by devcam hook.
Change-Id: Iafb32e19b21d1df44b9625a5f58bf898e6b51fa8
Update our Go image to use Go1.5.
Also, with Go1.5, we can compile static binaries (with CGO_ENABLED=0 and
--tags=netgo) without having to first rebuild Go itself from source with
CGO_ENABLED=0.
Finally, with Go1.5, we can now cross-compile straight outta the (linux)
binary tarball (without having to first build the necessary runtime in
src).
For both these above reasons, we don't need to use the Go -src tarball,
and we don't need to have a dedicated Go docker image for each GOOS we
cross-compile too.
Yay, progress!
Change-Id: Ibafb542a4771b151638e796ad3df78e0c8f1a4bf
See https://golang.org/s/go15vendor
This commit moves the google.golang.org/cloud/... packages. The rest
of third_party will move later.
This commit is a dependency for some cloud logging work which we'll
develop in Camlistore's tree.
Some new features (build binaries tarballs) added in commit
988fc8a43c depend on some files that were
added to the camli repo in that very commit.
So for these new features to work, the Camlistore source used must be >=
to that revision, hence that bump.
Change-Id: Ibb68c0f1a1c85277becc02ad73b902d77b5bc465
This change:
1) adds misc/docker/go/darwin/Dockerfile to create a docker image with
Go setup for cross-compiling to darwin
2) adds misc/docker/release/build-binaries.go, which works similarly to
misc/docker/server/build-camlistore-server.go. The difference is it
builds all binaries with make.go instead of just camlistored, and
camlistored is hence built with all resources embedded.
3) adds the --build_binaries flag to dock.go to make use of the above.
Maybe 3) should be done in a new file (out of dock.go), but it is
convenient for now for code reuse.
Issue #538
Change-Id: I8ecd50d118a9ff5c3f53812cac4d7a81a0053b4f
Instead of just uploading the unversioned tarball, this change first
uploads a versioned tarball, then makes a copy of it as the unversioned one.
This allows for easier reverts when overwriting the unversioned tarball
by mistake.
Change-Id: Ia8a1b9008e7a203f2fb3b640dc12da432be1baf3
The (camlistored builder + UI resources bundler) was a piece of shell
run in a docker container. That piece of shell is replaced with the
misc/docker/server/build-camlistore-server.go go program. This
program is still run within a container of the camlistore/go image.
It is added "on the fly" as a resource to the camlistore/go image
as a mounted resource.
In addition, build-camlistore-server.go can take a local directory as
the camlistore source (instead of fetching a tarball at the given
revision). dock.go has been adapted to take advantage of that.
Change-Id: If32b523d850f6c932016f18231bb1b1a283e50d3
Because of the hg->git switch and the switch to Go 1.4, which requires
bootstrapping for building from source:
-master now requires an env with a working Go to build builder (instead of
building go tip from source), and master does not monitor changes to the go
tree anymore.
-builder gets Go1.4.1 from the binary tarball (instead of building from
source), and does not build go tip anymore (nor does it run tests with
it).
Keeping it alive properly was not deemed worth the trouble since we plan
on replacing it with a new builder based on the same tools used for the
Go builders.
Change-Id: Idd0864350b27b6523675006f00719fba2d6fc72f
This change makes gce/create.go generate a self-signed certificate
with the hostname from the -hostname argument and upload it to GCS
before creating a new camlistore instance.
It also makes camlistored use baseURL to figure out the hostname
when generating its self-signed certificate.
Change-Id: I64f85853dab34a7ce95e5d5997e58f2e5da43496
Additional check to prevent b bad imports and un-go-fmt'd code
from sneaking in.
If users don't run tests, it will break the builder.
Change-Id: I95c74724aac68d915aa5195b6d8fcb78f92349e0
Use systemd/CoreOS instead of supervisor,
Less InnoDB memory,
No snapshots,
unrelated to Docker image: keep data files on CoreOS /var/lib/mysql-camli so
they persist across reboots.
Change-Id: Ib903c386fb3e2df0b7cf2d3eb466290f2b16f94b
I think I'm not using systemd correctly here. Things get very unhappy
if I reboot CoreOS: the Docker container named 'db' already exists, things keep
flapping up and down (not waiting for their dependencies), etc.
Maybe I need to register stop actions too.
Change-Id: I75ac3e965c03df4f7f3938ff13c66f137948bf4d
The machine now boots with a new pair of cooperating systemd units:
1) cam-journal-gatewayd.service: a copy of the systemd-journal-gatewayd service,
which runs an HTTP interface to the systemd journal.
2) cam-journal-gatewayd.socket: a unix socket listener listening on unix
socket /run/camjournald.sock. Incoming connections will forward to 1).
Then the camlistored.service unit running camlistored under Docker now
passes -v /run/camjournald.sock:/run/camjournald.sock to make that unix socket
available to the Docker container.
Then in camlistored, a new handler at /debug/logs (wrapped in auth
checks) then opens that socket and makes an HTTP request to it,
copying its response (of log lines) back to the browser.
This will ease debugging, letting people only use their browser to
debug (or send logs to the Camlistore developers more likely), rather
than sshing in to CoreOS and learning CoreOS and systemd arcana.
Change-Id: Icd5967ae7e9946d36229bdbc5d37644a11ee5e9f
I'm hoping this will help with our first test suite run (Go1) which
is often failing at the test trying to hit http://localhost:3179/ui
Change-Id: I972072ebc27b7a59136a90532000ca7e9d58a471
Because if e.g. 'devcam test' is the current child process, we want it to be
able to catch the signal and kill its own child too.
Change-Id: I3e3c9c10c8d7f5d793c98b604baf8df56608003e
Moved BasicAuth parsing and localhost detection code from pkg/auth ->
pkg/httputil for use by buildbot master.
Added user config file for remote access. The file's name is
"masterbot-config.json" and is located in osutil.CamliConfigDir(),
which on Unix will resolve to $XDG_CONFIG_HOME/camlistore/, if
XDG_CONFIG_HOME set, or ~/.config/camlistore/. On Windows it will be
under %APPDATA%\Camlistore\. The expected format is a json object
with usernames as the keys and sha1 sums of the password as the
values, i.e.:
{
"user1": "1234567890abcdef12341234567890abcdef1234",
"user2": "1234abcdef12345678901234abcdef1234567890"
}
This file is polled at a 1 minute interval and reparsed if the file's
modification time is more recent then the previous parse attempt. It
is ok for the file to go missing, it will zero out the remote user
list. A malformed file will result in the master exiting.
New commandline flags, -tlsCertFile & -tlsKeyFile, added. Specifying
both will enable TLS on the listener specified by -host. The go
source contains generate_cert.go in crypto/tls that can be used to
generate self-signed cert.pem and key.pem for testing.
Added -skiptlscheck commandline option to builder. This allows the
builder to report to https:// addresses with self-signed certs as we
don't currently have a way to specify the cert chains to be used
for TLS verification. This is a stop-gap solution.
When launching a master that listens for secure connections, we
currently need tell the builders to skip certificate validation. Add
'-builderopts="-skiptlscheck"' to the master's commandline to skip
cerfication verification.
Change-Id: I0750b5c9fa8f4def67fc05a841087b50abded2f7
Adds support for specifying additional masters to send results to.
The config file, named "builderbot-config", must be located in
osutil.CamliConfigDir(); on Unix, it will be under
$XDG_CONFIG_HOME/camlistore/, if XDG_CONFIG_HOME set, or
~/.config/camlistore/. On Windows it will be under %APPDATA%\Camlistore\.
The expected format is one host per line, comments are not allowed and used
only for illustration below. Some examples:
# Post to default path on host1, no user auth.
http://host1
# Post to / on host2, no user auth.
http://host2/
# Post to /other/path on host3, no user auth.
http://host3/other/path
# Post to default path on host4, user 'user', password 'pass'.
http://user:pass@host4
# Post to / on host4 port 7070, user 'user', password 'pass'.
http://user:pass@host4:7070/
This change also explicitly sets GOPATH to the copy of camlistore.org checked
out by the master before building the builder, and starting the build run.
This ensures we're not building against a random checkout of camlistore
pointed to by the user's environment.
The directory of the checked out source is moved from ${PWD}/camlistore.org
-> ${PWD}/src/camlistore.org so GOPATH can be set to $PWD and the requirement
of packages being under src/ is met.
Change-Id: I6e121c0aae9dae0c1832f782fa32619434ce9d2c
The error message is already preformated with \n newlinews, so <pre> is a more
appropriate tag, shows error messages with proper line-breaks.
Change-Id: Ifcf63ae4c9fa857c48bd825edb8f83f1527ed57a
-hack to disallow remote bots reports, until we have some auth working
-verify that the alternative latest hash provider response looks like
a git hash before using it.
-check that the buildbot code is the most recent before building
-a few more debugging prints
-some paths fixes
-README to get started
Change-Id: Ibe8b4472586b4902d3f28f70a2cceadfb6e41c80
This change is the first major step towards more improvements to the
buildbot. It cleans up various details, but most notably:
-it now uses a task as the basic data structure, to improve readability
and maintainability
-it breaks the bot into a master bot and a builder bot.
-it allows a master to receive reports from remote bots, and the
status page now aggregates all those reports.
-it allows a builder bot to send its reports to several master bots.
-the master bot uses whatever go version is installed to build the
builder bot, or the downloaded go tip if none is found.
The master bot is in charge of
1) monitoring the changes to the Camlistore and Go repos,
2) recompiling builder bot, which allows to add some new tests to the
test suite without having to interrupt the master bot.
3) starting a builder bot instance
4) waiting for the builder bot report and/or polling the builder bot
for its progress
5) answering the requests to display the various status pages.
http://camlistore.org/issue/185
Change-Id: I46a0b8fabbebf76b0c3ed04fd2ee73362d565cf7
This adds the error returned from exec to the log message, and
provides '(empty)' placeholders when no stdout or stderr is present
from the called binary.
This has the effect of changing the silent errors I was seeing on my
buildbot from:
"""
Error:
:
"""
To:
"""
Error:
exec: "devcam": executable file not found in $PATH
Stdout:
(empty)
Stderr:
(empty)
"""
Also provides a /stderr handler linked from the status page that
allows all the logs to be viewed. This is handy when the logs from an
individual test don't provide enough info to debug a build problem.
Change-Id: I4b3ef5a5db6aac1ddf41c41bc2fff310f19a82a1
The integration tests run by devcam test require a
GPG keyblob in the default location (for camput), and
we do not track in git such a keyblob.
This change allows devcam test to create the keyblob
with devcam put init before running the tests.
Also, devcam server is now using CAMLI_KEYID instead
of hardcoding the key id in the dev server config file.
Change-Id: I030b88c9a9321016fedb62424e0e707621206922
Also added -port flag, and untied the port to the
password in devcam server, always 'pass3179' now.
http://camlistore.org/issue/212
Change-Id: Iadf693951763d47907be17c53d4807a4706eb150
This change also removes misc/get_closure.pl and updates
the Makefile accordingly.
http://camlistore.org/issue/140
Change-Id: I508271cee24eb20d8391cb3d84e1201ce3ef567e
For folks whose $USER doesn't match their
Gerrit username, editing misc/review is not a
great solution, since the changes cannot be
committed (nf notwithstanding).
Instead, let ssh config files do their thing;
these are not part of the repo, and thus can
be set once and merrily forgotten.
Change-Id: If03fe05e5779553ed1f98310e8f22b0635ada7bb
1) Use "make forcefull". Please note that this will fail
with the default Go1 version installed by ubuntu packages.
2) Use --offline with dev-server to avoid pulling the
closure stuff.
3) Waterfall display (new stuff on top, older at bottom).
4) Added a camput -vivify run, with the subsequent camgets
to check that the file has indeed been pushed.
Change-Id: Icd164932bd61d54ddbece12ef420524643e07f1b
1) The code to get the closure library has been moved from
dev-server to misc/get_closure.pl
2) The svn checkout version for the closure library was bumped
to the latest: 2459
3) The closure library checkout now goes into tmp/closure-library/,
to keep tmp/ clean for other things we might want to add there
(like the compiler). So the path to the actual libraries is now
tmp/closure-library/closure/
4) The code to get the closure compiler build was added in
misc/get_closure.pl. The compiler goes to
tmp/closure-compiler/compiler.jar
5) The version of the closure compiler is the latest build as
of this time: 20121212
Change-Id: I7e03ad7d6fe440430c5727cf56a956efed360d43
It now uses the generic indexing layer, rather than its own schema.
Also, move it under pkg/index/, like mongo.
Also, link in mongo to the main binary.
Not using the tests yet (leaving for mpl :))
Change-Id: I47a2203ca6c0b5244cd458aedec46227c4363c62
mpl