Currently disabled until it's working reliably. See comments in code.
Much of this is from Tamás Gulácsi in https://camlistore-review.googlesource.com/#/c/2128/
Change-Id: I2fe2abfdee98584c698a2ac52202f13dda9beef2
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.
Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
httputil.CloseBody closes a http.Response.Body nicely, such that it's
more likely to be re-used.
httputil.DecodeJSON parses JSON from a Response.Body and closes the
body. It also keeps a temporary copy of the input, for error messages.
Then use them everywhere, to improve HTTP TCP connection re-use.
And misc other clean-ups.
Change-Id: I5dc44be0165b13659459ed29ce5ab44c17d9739f
Trust the have cache for skipping uploads.
Reduces total number of HTTP requests.
Also, set Transport's MaxIdleConnsPerHost to be the same as
our HTTP concurrency level, to not keep re-creating TCP
connections.
import bazil.org/fuse at f6649cd46fb76b1e04ba9eeffb773cb189278d31
remove code.google.com/p/rsc/fuse.
Huge thanks to Tommi Virtanen for the fuse support.
Change-Id: I44b95ac1ea344648593c7376f83ca675a56077b3
* Fix TODO about moving camput file's havecache up a layer into
camput itself. So now it's used for all commands (camput permanode,
camput attr, etc). Reduces HTTP requests.
* Fix camlistore.org/issue/343 -- don't do stats before uploads
in cases where it's useless. Adds a field to UploadHandle.
* Fix camlistore.org/issue/344: upload the public key whenever
uploading a signed blob. Usually this is a no-op because of the
have cache.
* Use zero (or <=0 rather), not -1, to mean unknown on UploadHandle
Size.
* More docs on public stuff.
1) disallow auth from env when not in dev mode or on android
2) allow insecureTLS with flag in debug mode
3) set transport so that insecureTLS or trustedCert modes work
http://camlistore.org/issue/325
Change-Id: I6d8056bb75ffff5f002dfe9bd631ae996dc4e5ef
Changed some keys to be consistent with the server config, and removed
some others.
We're now using jsonconfig to get the keys and validate the config.
Removed some unused funcs/methods, moved some global vars as members of a client.
Also, having two possible defaults for the secret keyring
(.config/camlistore/identity-secring.gpg and .gnupg/secring.gpg)
was annoying and confusing imho, so I removed the latter.
http://camlistore.org/issue/259
Change-Id: Icf79fa8822b62b5d91843698861dd19cb553607a
Will eventually be plumbed through lots of APIs, especially those requiring or benefiting from
cancelation notification and/or those needing access to the HTTP context (e.g. App Engine).
Change-Id: I591496725d620126e09d49eb07cade7707c7fc64
This is just a start. AndroidStatsTransport and AndroidStatusReceiver and most (all?)
of pkg/client/android.go should migrate too.
Change-Id: Id4140730bbdd218fdf7ba7cedc7870ce5164e06c
I can recreate the file, set the modification time, but can't forge creation
time, thus cannot recreate the exact same uploaded file twice.
This change targets this problem.
As this is a very rarely used flag, this is enabled only if the CAMLI_DEBUG
environment variable is true - such as the other "gross" options
(statcache, sqlitecache...).
Change-Id: I99c7805898ed97883043de4e7dcec23a472e462c
The integration tests run by devcam test require a
GPG keyblob in the default location (for camput), and
we do not track in git such a keyblob.
This change allows devcam test to create the keyblob
with devcam put init before running the tests.
Also, devcam server is now using CAMLI_KEYID instead
of hardcoding the key id in the dev server config file.
Change-Id: I030b88c9a9321016fedb62424e0e707621206922
The upload was constrained to have only 5 workers goroutines at a
time, but those goroutines could have been uploading directories and
blocked waiting for their dependent children, which never got to
upload if all 5 goroutines were uploading directories.
Now, allow an unlimited number of upload goroutines (see commit
058434449), but add a gate around things that allocate file
descriptors within the upload function.
This means that now we could have a ton of goroutines existing but
blocked on the gate, wasting ~4KB of memory each rather than the
relatively small amount of memory we used for the *node on the
container/list before.
Future CLs can reduce the memory usage and/or rewrite this.
Or maybe we forget to clean this up, but get smaller goroutine memory
usage in Go 1.3 as is being discussed. It _is_ convenient to just
launch tons of goroutines and manage dependencies with channel
operations rather than managing the dependencies state and goroutine
scheduling in your own code, just to save some memory.
Change-Id: I43d9d7795e5df4d790d6086d18e4b4b99eaacbc8
Running 'go run dev/envvardoc/envvardoc.go' now shows:
'All environment variables are documented'
I also took the liberty of cleaning-up our mishmash of logic for handling
boolean environment variables, and cleaned up a couple other spots that didn't
seem right.
This change adds docmentation for all variables starting with (CAM|DEV|AWS).
This leaves some variables still undocumented. If there are variables worth
documenting in the following list, maybe we should rename them to have a
CAM{LI} prefix for consistency's sake:
APPDATA pkg/osutil/paths.go:86
APPDATA pkg/osutil/paths.go:102
DISPLAY pkg/misc/gpgagent/gpgagent.go:126
GOPATH pkg/fileembed/genfileembed/genfileembed.go:321
GOPATH pkg/osutil/paths.go:168
GOPATH pkg/test/world.go:54
GOPATH server/appengine/build_test.go:77
GPGKEY cmd/camput/init.go:77
GPG_AGENT_INFO cmd/camput/init.go:153
GPG_AGENT_INFO pkg/misc/gpgagent/gpgagent.go:50
HOME pkg/jsonsign/keys.go:79
HOME pkg/jsonsign/signhandler/sig.go:64
HOME pkg/osutil/paths.go:36
HOMEPATH pkg/osutil/paths.go:34
PKG_CONFIG_PATH pkg/index/sqlite/dbschema.go:59
RUN_BROKEN_TESTS pkg/fs/fs_test.go:67
SKIP_DEP_TESTS pkg/test/testdep.go:29
TERM pkg/misc/gpgagent/gpgagent.go:133
TERM pkg/misc/pinentry/pinentry.go:99
TESTING_PORT_WRITE_FD pkg/webserver/webserver.go:135
TEST_GPGAGENT_LIB pkg/misc/gpgagent/gpgagent_test.go:27
USER pkg/netutil/ident.go:135
USER pkg/osutil/paths.go:45
USERNAME pkg/jsonconfig/eval.go:228
USERNAME pkg/osutil/paths.go:43
VERBOSE_FUSE pkg/fs/fs_test.go:133
VERBOSE_FUSE_STDERR pkg/fs/fs_test.go:137
XDG_CONFIG_HOME pkg/osutil/paths.go:104
Change-Id: Ief28710d3deefd1e65247cb5d3b1d8dde73e1f2d
Move up a layer to the HTTP. Also, start to remove ContextWrapper
stuff. We've done it differently for App Engine instead, and will do
it differently yet moving forward.
Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.
Bunch of other cleanup and TODO fixing too.
Much simpler and cleaner.
Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
Also made initTrustedCertsOnce a field of the Client
object, as it is a similar change.
http://camlistore.org/issue/104
Change-Id: Iabbd7f06e06d31265f390a23c4bdaac956f856f7
This change adds a dependency on lsof(1) when running on FreeBSD.
lsof(1) is not part of the base system, and lsof(1) would need to be
installed (usually from ports). If lsof(1) is not installed an error
will be returned in the form of:
ident_test.go:88: exec: "lsof": executable file not found in $PATH
Change-Id: I3cb15af369dc10a8a39aeaf82cceff3c788104b8
Brad Fitzpatrick