Commit Graph

226 Commits

Author SHA1 Message Date
Brad Fitzpatrick dcf5f6abe7 client: clean up. finish moving all Android stuff to pkg/client/android
Change-Id: Ia3d19a323d9c734a01ec9a12509b115191afcfdd
2013-10-26 11:08:34 -07:00
Brad Fitzpatrick 2040d2e442 Add new pkg/client/android, to clean up Android API gunk from pkg/client's godoc.
This is just a start. AndroidStatsTransport and AndroidStatusReceiver and most (all?)
of pkg/client/android.go should migrate too.

Change-Id: Id4140730bbdd218fdf7ba7cedc7870ce5164e06c
2013-10-22 07:37:15 -07:00
Brad Fitzpatrick cf15b2be90 client: use the schema.Signer API, remove Client.SignBlob method
Change-Id: I177c78f4f7f4ca54afd021d4ada866a35eea06eb
2013-10-22 07:26:44 -07:00
Salman Aljammaz a607971f5f cammount: handle SIGINT, so that we unmount too in that case.
Change-Id: I5e82c6326aab892674c775e0615a91af71532d3c
2013-10-17 15:22:18 +01:00
Tamás Gulácsi e056081be1 add capctime flag to camput
I can recreate the file, set the modification time, but can't forge creation
time, thus cannot recreate the exact same uploaded file twice.
This change targets this problem.

As this is a very rarely used flag, this is enabled only if the CAMLI_DEBUG
environment variable is true - such as the other "gross" options
(statcache, sqlitecache...).

Change-Id: I99c7805898ed97883043de4e7dcec23a472e462c
2013-10-09 21:06:51 +02:00
Steve Phillips 5dc853110f Added missing var 'which' used in a fmt.Errorf
Change-Id: I0b9a7c6a0c7e523a75f3e5beebbd860e0092a81f
2013-10-04 18:44:40 -07:00
mpl 304f5c4b38 camput attr: support -del
Change-Id: I28460fed5b572ba9a83cfcb04f38e42124fa723f
2013-10-03 15:07:43 +02:00
Brad Fitzpatrick e70f258b0b Merge two copies of StatsReceiver into common location.
Change-Id: I05bd36aef3293927fb518296b43a088f284356ea
2013-09-22 11:19:10 +01:00
mpl 0c1dc0d6e4 devcam test: generate keyblob with devcam put init
The integration tests run by devcam test require a
GPG keyblob in the default location (for camput), and
we do not track in git such a keyblob.
This change allows devcam test to create the keyblob
with devcam put init before running the tests.

Also, devcam server is now using CAMLI_KEYID instead
of hardcoding the key id in the dev server config file.

Change-Id: I030b88c9a9321016fedb62424e0e707621206922
2013-09-20 20:03:21 +02:00
Brad Fitzpatrick 79065de4b9 For SQLite compile errors, link to the wiki.
Change-Id: I34b1485daba6173d6665cfc56b6bba43164534d8
2013-09-19 10:17:57 +01:00
Brad Fitzpatrick 8319411ab4 Convert more ReceiveBlob into blobserver.Receive or blobserver.ReceiveNoHash
Change-Id: I9199555324b617167a6062a8b55ed09b449bae4f
2013-09-16 15:57:14 +01:00
Brad Fitzpatrick 59534f0ad6 Move pkg/gate to pkg/syncutil, to house more stuff.
Change-Id: I737458641a9de03da55ed627bfac10f3377a13aa
2013-09-12 16:04:10 +01:00
Brad Fitzpatrick ee4550bff4 camput: fix deadlock when uploading a large directory tree
The upload was constrained to have only 5 workers goroutines at a
time, but those goroutines could have been uploading directories and
blocked waiting for their dependent children, which never got to
upload if all 5 goroutines were uploading directories.

Now, allow an unlimited number of upload goroutines (see commit
058434449), but add a gate around things that allocate file
descriptors within the upload function.

This means that now we could have a ton of goroutines existing but
blocked on the gate, wasting ~4KB of memory each rather than the
relatively small amount of memory we used for the *node on the
container/list before.

Future CLs can reduce the memory usage and/or rewrite this.

Or maybe we forget to clean this up, but get smaller goroutine memory
usage in Go 1.3 as is being discussed.  It _is_ convenient to just
launch tons of goroutines and manage dependencies with channel
operations rather than managing the dependencies state and goroutine
scheduling in your own code, just to save some memory.

Change-Id: I43d9d7795e5df4d790d6086d18e4b4b99eaacbc8
2013-09-10 20:16:16 -07:00
Brad Fitzpatrick 058434449c camput: add an unbounded mode to chanworker.
Change-Id: I90040fc6a67c88ae54338ab9556295861170a871
2013-09-10 20:07:20 -07:00
mpl ad84a3565e camput: check stat cache value form
Addresses last comment from
http://camlistore.org/r/606

Change-Id: I5c0eb5492fdcfeb71becdeb25c6c09a1971ac09c
2013-09-09 19:08:07 +02:00
Brad Fitzpatrick 5c83627a1f camput: add debugging for camlistore.org/issue/221
Change-Id: If0ed73cb16409ac5c1f67ffc878038075cb7e432
2013-09-08 18:19:54 -07:00
Brad Fitzpatrick 4270ec4ce5 camput: remove unnecessary parens and allocation
Change-Id: I78b22458d4bd001313d6d49c6e401bc409b953fd
2013-09-08 18:17:51 -07:00
Brad Fitzpatrick 2cf4d68267 Let shares have expiration times.
Change-Id: Iecae106302ba43c354b6987163830cf78a70d12c
2013-09-08 17:46:40 -07:00
Brad Fitzpatrick d2d65912d9 Merge "Document environment variables usage." 2013-09-08 15:25:56 +00:00
Aaron Boodman 8dd7dd42cf Fix crash in `camput file` when configuration discovery has failed.
Change-Id: I8a7ac26e5d550f25eddf149dadde5013a1d50fe7
2013-09-07 19:15:44 -07:00
Bill Thiede 630f6ca61f Document environment variables usage.
Running 'go run dev/envvardoc/envvardoc.go' now shows:
'All environment variables are documented'

I also took the liberty of cleaning-up our mishmash of logic for handling
boolean environment variables, and cleaned up a couple other spots that didn't
seem right.

This change adds docmentation for all variables starting with (CAM|DEV|AWS).
This leaves some variables still undocumented.  If there are variables worth
documenting in the following list, maybe we should rename them to have a
CAM{LI} prefix for consistency's sake:

APPDATA                  pkg/osutil/paths.go:86
APPDATA                  pkg/osutil/paths.go:102
DISPLAY                  pkg/misc/gpgagent/gpgagent.go:126
GOPATH                   pkg/fileembed/genfileembed/genfileembed.go:321
GOPATH                   pkg/osutil/paths.go:168
GOPATH                   pkg/test/world.go:54
GOPATH                   server/appengine/build_test.go:77
GPGKEY                   cmd/camput/init.go:77
GPG_AGENT_INFO           cmd/camput/init.go:153
GPG_AGENT_INFO           pkg/misc/gpgagent/gpgagent.go:50
HOME                     pkg/jsonsign/keys.go:79
HOME                     pkg/jsonsign/signhandler/sig.go:64
HOME                     pkg/osutil/paths.go:36
HOMEPATH                 pkg/osutil/paths.go:34
PKG_CONFIG_PATH          pkg/index/sqlite/dbschema.go:59
RUN_BROKEN_TESTS         pkg/fs/fs_test.go:67
SKIP_DEP_TESTS           pkg/test/testdep.go:29
TERM                     pkg/misc/gpgagent/gpgagent.go:133
TERM                     pkg/misc/pinentry/pinentry.go:99
TESTING_PORT_WRITE_FD    pkg/webserver/webserver.go:135
TEST_GPGAGENT_LIB        pkg/misc/gpgagent/gpgagent_test.go:27
USER                     pkg/netutil/ident.go:135
USER                     pkg/osutil/paths.go:45
USERNAME                 pkg/jsonconfig/eval.go:228
USERNAME                 pkg/osutil/paths.go:43
VERBOSE_FUSE             pkg/fs/fs_test.go:133
VERBOSE_FUSE_STDERR      pkg/fs/fs_test.go:137
XDG_CONFIG_HOME          pkg/osutil/paths.go:104

Change-Id: Ief28710d3deefd1e65247cb5d3b1d8dde73e1f2d
2013-09-06 21:54:03 -07:00
mpl 8d71c9bb60 camput: have cache and stat cache based on github.com/cznic/kv
Added pkg/kvutil/kvutil.Open for convenience

Change-Id: Ie8db58e2c644a7358f252548bc57cccff7627497
2013-08-30 23:54:22 +02:00
Brad Fitzpatrick b24cad68dd Cleanup: remove BlobHub and time.Duration waits from storage interface
Move up a layer to the HTTP.  Also, start to remove ContextWrapper
stuff.  We've done it differently for App Engine instead, and will do
it differently yet moving forward.

Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.

Bunch of other cleanup and TODO fixing too.

Much simpler and cleaner.

Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
2013-08-21 13:57:28 -07:00
mpl 31df3635a4 camput: ignore files like .DS_Store with -filenodes
Also made initTrustedCertsOnce a field of the Client
object, as it is a similar change.

http://camlistore.org/issue/104

Change-Id: Iabbd7f06e06d31265f390a23c4bdaac956f856f7
2013-08-19 17:21:10 +02:00
mpl 9fcb4b6434 dumpconfig: typo in date
Change-Id: Iebd75442070896b588092f8bad62e0d21cab3b27
2013-08-09 18:12:47 +02:00
Brad Fitzpatrick 462eb7e4c2 Add camtool dumpconfig subcommand.
Fixes http://code.google.com/p/camlistore/issues/detail?id=192
2013-08-08 17:06:45 -07:00
Brad Fitzpatrick 0bdf20884b all: delete pkg/blobref; convert all from *blobref.BlobRef to new blob.Ref
Change-Id: Id2dfb7f19452bedf4f3c9310b36227fd8117b225
2013-08-03 19:54:30 -07:00
Bill Thiede e6b86ddb9d Fixes to appease make presubmit and make forcefull on FreeBSD.
This change adds a dependency on lsof(1) when running on FreeBSD.
lsof(1) is not part of the base system, and lsof(1) would need to be
installed (usually from ports).  If lsof(1) is not installed an error
will be returned in the form of:

ident_test.go:88: exec: "lsof": executable file not found in $PATH

Change-Id: I3cb15af369dc10a8a39aeaf82cceff3c788104b8
2013-08-03 11:48:57 -07:00
Brad Fitzpatrick 38bb307085 camtool: typo from Adrian (ajft.org) and run gofmt
Change-Id: Ibbebea7bcf9a583c0ea1e86f09eff69acd2b174e
2013-08-02 16:32:50 -07:00
Brad Fitzpatrick 98eb69b5e1 fs: bunch of read-write FUSE work, debugging, and integration tests.
Change-Id: I74807f693720effb7ae8405259797331f79f59fd
2013-07-21 12:26:05 -07:00
Brad Fitzpatrick 62b0f66f05 cammount: fix imports. lazy partial git foo before. never built what I pushed.
Change-Id: I4ebc020220f6c6c15ba77e76b09cf1f5daf2169e
2013-07-21 11:54:20 -07:00
Brad Fitzpatrick 0e28959c51 fs: move unmount from cammount to fs.Unmount
Change-Id: I2c21ccbbd81f4fa7ffc40cc769eacb8a6bf30558
2013-07-20 23:26:56 -07:00
Brad Fitzpatrick 41f3e652a9 cammount: kill xterm on exit
Change-Id: Id64901853cd2adbf9aa27c9d5cc40f93b732bbbe
2013-07-21 05:23:39 +12:00
Brad Fitzpatrick 2b879e73bc cammount, dev-cammount: add verbose/debug flag
Change-Id: If53ec139db178b44c2dd5c7f1cad33f7ab62c847
2013-07-21 05:18:18 +12:00
mpl 257fd29afa camget: allow insecure TLS with shared URLs
Change-Id: I22f610009edfecea932019c35310ae62b0613cb8
2013-07-16 15:56:32 +02:00
Brad Fitzpatrick e43e703599 dev-cammount, cammount: add xterm flag, to bring up a debug xterm in that dir.
remove child process mode. not necessary.

Change-Id: I4f6e75efe127a2b9e7867dbfac5c42fcc874e3a6
2013-07-15 21:29:00 +10:00
Brad Fitzpatrick 167138fd65 cammount: quieter. and only exit when 'q' is pressed.
Change-Id: I2c8719490f977d14e526cd34672ce784589dd460
2013-07-11 19:21:59 +10:00
Brad Fitzpatrick 02d38b07ca camput, client: move lot of camput's code into pkg/client
so cammount can use it.

Change-Id: I1bd513e6dfe6faab785c856abbd0d1337e978dd9
2013-07-11 16:46:23 +10:00
Brad Fitzpatrick ae8aec21e4 dev-cammount / cammount: sane hack/debug cycle. reliable start-up & shutdown / unmount.
especially on OS X.

Change-Id: Ibb60b82be72049dc44984e8aff7cb54560800364
2013-07-10 21:27:54 +10:00
Andrew Gerrand 1a89d81bf1 cammount: instruct user to install FUSE
Change-Id: Ia3e30bbd9713dad542e3d2f946c27892e6bb35ea
2013-07-10 19:37:31 +10:00
Michael Zuffoletti c2669a6dfc issue 23: Fix google storage blobserver as sync dest
Change-Id: I33d88dc5028ebb76d4c17d2a00d8f8346114f876
2013-07-08 11:33:10 -05:00
Brad Fitzpatrick 9468e5ba70 More docs. Every package is documented now.
misc.CountingReader moves into readerutil.

pkg/atomics is folded into pkg/types.

pkg/test/testdep is folded into pkg/test, with better name/docs.

Old cruft from pkg/webserver is deleted.

Change-Id: I3f72d8b29804254ef944995fb085837c878f79f5
2013-07-07 21:12:30 -07:00
Brad Fitzpatrick 76013089a4 Move blobref.ParseMulti to its sole caller, with error checking.
Change-Id: I6351591497879ea5a113bc8e8414b751b3d1c812
2013-07-07 11:46:21 -07:00
mpl fd8af44431 camput: stat cache: use a different key in -filenodes mode
http://camlistore.org/issue/157

Change-Id: If58116be6073e0bd160d57bb8f778e498a8763bb
2013-06-28 19:52:57 +02:00
mpl f7732db604 cammount: rm TODO
Change-Id: Id97cd0400c14ddbbfc7590055ba14d6bc1d1c554
2013-06-26 15:36:19 +02:00
mpl bb7487ea0b cammount: setup transport (trusted certs support)
Change-Id: I2970986637094d5bf581545bbedab601c62c1641
2013-06-26 15:19:14 +02:00
Brad Fitzpatrick 90ecd785ba Merge "client: factorize the transport setup, and use it in camget" 2013-06-26 01:58:07 +00:00
Josh Huckabee a5d72541c3 Enhance PostgreSQL support:
* Fix dbinit command in dev server
* Make sslmode configurable in dbinit
* Don't fail on index schema setup when plpsql language already exists

Change-Id: Ic70461963f6266172d21cda8e6f071e2442d7651
2013-06-25 16:48:38 -07:00
Josh Huckabee 65f3079498 Replace github.com/bmizerany/pq with github.com/lib/pq postgres driver.
Change-Id: If62fd5b489696171c8be6b42a988e1c7d0a634d0
2013-06-25 16:13:22 -07:00
mpl b24ac50965 client: factorize the transport setup, and use it in camget
Fixes http://camlistore.org/issue/153

Change-Id: If393b34bbfcf4a6282357c9c57ebb85bae8741a3
2013-06-25 23:37:41 +02:00