Trust the have cache for skipping uploads.
Reduces total number of HTTP requests.
Also, set Transport's MaxIdleConnsPerHost to be the same as
our HTTP concurrency level, to not keep re-creating TCP
connections.
* Fix TODO about moving camput file's havecache up a layer into
camput itself. So now it's used for all commands (camput permanode,
camput attr, etc). Reduces HTTP requests.
* Fix camlistore.org/issue/343 -- don't do stats before uploads
in cases where it's useless. Adds a field to UploadHandle.
* Fix camlistore.org/issue/344: upload the public key whenever
uploading a signed blob. Usually this is a no-op because of the
have cache.
* Use zero (or <=0 rather), not -1, to mean unknown on UploadHandle
Size.
* More docs on public stuff.
Changed some keys to be consistent with the server config, and removed
some others.
We're now using jsonconfig to get the keys and validate the config.
Removed some unused funcs/methods, moved some global vars as members of a client.
Also, having two possible defaults for the secret keyring
(.config/camlistore/identity-secring.gpg and .gnupg/secring.gpg)
was annoying and confusing imho, so I removed the latter.
http://camlistore.org/issue/259
Change-Id: Icf79fa8822b62b5d91843698861dd19cb553607a
This is just a start. AndroidStatsTransport and AndroidStatusReceiver and most (all?)
of pkg/client/android.go should migrate too.
Change-Id: Id4140730bbdd218fdf7ba7cedc7870ce5164e06c
I can recreate the file, set the modification time, but can't forge creation
time, thus cannot recreate the exact same uploaded file twice.
This change targets this problem.
As this is a very rarely used flag, this is enabled only if the CAMLI_DEBUG
environment variable is true - such as the other "gross" options
(statcache, sqlitecache...).
Change-Id: I99c7805898ed97883043de4e7dcec23a472e462c
The integration tests run by devcam test require a
GPG keyblob in the default location (for camput), and
we do not track in git such a keyblob.
This change allows devcam test to create the keyblob
with devcam put init before running the tests.
Also, devcam server is now using CAMLI_KEYID instead
of hardcoding the key id in the dev server config file.
Change-Id: I030b88c9a9321016fedb62424e0e707621206922
The upload was constrained to have only 5 workers goroutines at a
time, but those goroutines could have been uploading directories and
blocked waiting for their dependent children, which never got to
upload if all 5 goroutines were uploading directories.
Now, allow an unlimited number of upload goroutines (see commit
058434449), but add a gate around things that allocate file
descriptors within the upload function.
This means that now we could have a ton of goroutines existing but
blocked on the gate, wasting ~4KB of memory each rather than the
relatively small amount of memory we used for the *node on the
container/list before.
Future CLs can reduce the memory usage and/or rewrite this.
Or maybe we forget to clean this up, but get smaller goroutine memory
usage in Go 1.3 as is being discussed. It _is_ convenient to just
launch tons of goroutines and manage dependencies with channel
operations rather than managing the dependencies state and goroutine
scheduling in your own code, just to save some memory.
Change-Id: I43d9d7795e5df4d790d6086d18e4b4b99eaacbc8
Running 'go run dev/envvardoc/envvardoc.go' now shows:
'All environment variables are documented'
I also took the liberty of cleaning-up our mishmash of logic for handling
boolean environment variables, and cleaned up a couple other spots that didn't
seem right.
This change adds docmentation for all variables starting with (CAM|DEV|AWS).
This leaves some variables still undocumented. If there are variables worth
documenting in the following list, maybe we should rename them to have a
CAM{LI} prefix for consistency's sake:
APPDATA pkg/osutil/paths.go:86
APPDATA pkg/osutil/paths.go:102
DISPLAY pkg/misc/gpgagent/gpgagent.go:126
GOPATH pkg/fileembed/genfileembed/genfileembed.go:321
GOPATH pkg/osutil/paths.go:168
GOPATH pkg/test/world.go:54
GOPATH server/appengine/build_test.go:77
GPGKEY cmd/camput/init.go:77
GPG_AGENT_INFO cmd/camput/init.go:153
GPG_AGENT_INFO pkg/misc/gpgagent/gpgagent.go:50
HOME pkg/jsonsign/keys.go:79
HOME pkg/jsonsign/signhandler/sig.go:64
HOME pkg/osutil/paths.go:36
HOMEPATH pkg/osutil/paths.go:34
PKG_CONFIG_PATH pkg/index/sqlite/dbschema.go:59
RUN_BROKEN_TESTS pkg/fs/fs_test.go:67
SKIP_DEP_TESTS pkg/test/testdep.go:29
TERM pkg/misc/gpgagent/gpgagent.go:133
TERM pkg/misc/pinentry/pinentry.go:99
TESTING_PORT_WRITE_FD pkg/webserver/webserver.go:135
TEST_GPGAGENT_LIB pkg/misc/gpgagent/gpgagent_test.go:27
USER pkg/netutil/ident.go:135
USER pkg/osutil/paths.go:45
USERNAME pkg/jsonconfig/eval.go:228
USERNAME pkg/osutil/paths.go:43
VERBOSE_FUSE pkg/fs/fs_test.go:133
VERBOSE_FUSE_STDERR pkg/fs/fs_test.go:137
XDG_CONFIG_HOME pkg/osutil/paths.go:104
Change-Id: Ief28710d3deefd1e65247cb5d3b1d8dde73e1f2d
Move up a layer to the HTTP. Also, start to remove ContextWrapper
stuff. We've done it differently for App Engine instead, and will do
it differently yet moving forward.
Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.
Bunch of other cleanup and TODO fixing too.
Much simpler and cleaner.
Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
Also made initTrustedCertsOnce a field of the Client
object, as it is a similar change.
http://camlistore.org/issue/104
Change-Id: Iabbd7f06e06d31265f390a23c4bdaac956f856f7
A new client configuration parameter, "trustedCerts" (list of strings)
is introduced. A certificate fingerprint is defined as the 10 digits
prefix of the sha1 of the whole certificate (in ASN1. DER form).
trustedCerts should contain the list of fingerprints of the
certificates we trust. If not empty, the server's certificate
is checked against that list, instead of using the full x509 validation
on it.
-added a dial function and tls configuration, which check if we're using
SSL, and if we're in "trustedCerts" mode.
pkg/client/client.go
pkg/client/config.go
-moved android specific hacks from camput to the client layer, so that
the dial and tls config could reuse/access them. Also allows future
reuse for other commands, such as camget.
pkg/client/android.go
-adapted camput to the above changes:
cmd/camput/android.go
cmd/camput/camput.go
cmd/camput/files.go
-server prints a hint when it generates the self-signed:
pkg/misc/misc.go
server/camlistored/camlistored.go
-camliactivity:
clients/android/res/xml/preferences.xml
clients/android/src/org/camlistore/Preferences.java
clients/android/src/org/camlistore/SettingsActivity.java
clients/android/src/org/camlistore/UploadService.java
clients/android/src/org/camlistore/UploadThread.java
http://camlistore.org/issue/131
Change-Id: I6be20161549a69aafc8eb7b9e96e9351dc1c5b09
Title is the attribute used in the ui, and actually recorded
by the indexer, so having "name" in camput was not consistent.
Change-Id: I6f63e54e3b19a83031f3c56c717b27336b175284
Brad Fitzpatrick