Commit Graph

147 Commits

Author SHA1 Message Date
mpl d51647ae33 camput: delete command, to send a delete claim
http://camlistore.org/issue/191

Change-Id: I053ab468420cc173e6122909bf3a0b16e4d43224
2013-11-29 00:19:42 +01:00
Brad Fitzpatrick dcf5f6abe7 client: clean up. finish moving all Android stuff to pkg/client/android
Change-Id: Ia3d19a323d9c734a01ec9a12509b115191afcfdd
2013-10-26 11:08:34 -07:00
Brad Fitzpatrick 2040d2e442 Add new pkg/client/android, to clean up Android API gunk from pkg/client's godoc.
This is just a start. AndroidStatsTransport and AndroidStatusReceiver and most (all?)
of pkg/client/android.go should migrate too.

Change-Id: Id4140730bbdd218fdf7ba7cedc7870ce5164e06c
2013-10-22 07:37:15 -07:00
Brad Fitzpatrick cf15b2be90 client: use the schema.Signer API, remove Client.SignBlob method
Change-Id: I177c78f4f7f4ca54afd021d4ada866a35eea06eb
2013-10-22 07:26:44 -07:00
Tamás Gulácsi e056081be1 add capctime flag to camput
I can recreate the file, set the modification time, but can't forge creation
time, thus cannot recreate the exact same uploaded file twice.
This change targets this problem.

As this is a very rarely used flag, this is enabled only if the CAMLI_DEBUG
environment variable is true - such as the other "gross" options
(statcache, sqlitecache...).

Change-Id: I99c7805898ed97883043de4e7dcec23a472e462c
2013-10-09 21:06:51 +02:00
mpl 304f5c4b38 camput attr: support -del
Change-Id: I28460fed5b572ba9a83cfcb04f38e42124fa723f
2013-10-03 15:07:43 +02:00
Brad Fitzpatrick e70f258b0b Merge two copies of StatsReceiver into common location.
Change-Id: I05bd36aef3293927fb518296b43a088f284356ea
2013-09-22 11:19:10 +01:00
mpl 0c1dc0d6e4 devcam test: generate keyblob with devcam put init
The integration tests run by devcam test require a
GPG keyblob in the default location (for camput), and
we do not track in git such a keyblob.
This change allows devcam test to create the keyblob
with devcam put init before running the tests.

Also, devcam server is now using CAMLI_KEYID instead
of hardcoding the key id in the dev server config file.

Change-Id: I030b88c9a9321016fedb62424e0e707621206922
2013-09-20 20:03:21 +02:00
Brad Fitzpatrick 59534f0ad6 Move pkg/gate to pkg/syncutil, to house more stuff.
Change-Id: I737458641a9de03da55ed627bfac10f3377a13aa
2013-09-12 16:04:10 +01:00
Brad Fitzpatrick ee4550bff4 camput: fix deadlock when uploading a large directory tree
The upload was constrained to have only 5 workers goroutines at a
time, but those goroutines could have been uploading directories and
blocked waiting for their dependent children, which never got to
upload if all 5 goroutines were uploading directories.

Now, allow an unlimited number of upload goroutines (see commit
058434449), but add a gate around things that allocate file
descriptors within the upload function.

This means that now we could have a ton of goroutines existing but
blocked on the gate, wasting ~4KB of memory each rather than the
relatively small amount of memory we used for the *node on the
container/list before.

Future CLs can reduce the memory usage and/or rewrite this.

Or maybe we forget to clean this up, but get smaller goroutine memory
usage in Go 1.3 as is being discussed.  It _is_ convenient to just
launch tons of goroutines and manage dependencies with channel
operations rather than managing the dependencies state and goroutine
scheduling in your own code, just to save some memory.

Change-Id: I43d9d7795e5df4d790d6086d18e4b4b99eaacbc8
2013-09-10 20:16:16 -07:00
Brad Fitzpatrick 058434449c camput: add an unbounded mode to chanworker.
Change-Id: I90040fc6a67c88ae54338ab9556295861170a871
2013-09-10 20:07:20 -07:00
mpl ad84a3565e camput: check stat cache value form
Addresses last comment from
http://camlistore.org/r/606

Change-Id: I5c0eb5492fdcfeb71becdeb25c6c09a1971ac09c
2013-09-09 19:08:07 +02:00
Brad Fitzpatrick 5c83627a1f camput: add debugging for camlistore.org/issue/221
Change-Id: If0ed73cb16409ac5c1f67ffc878038075cb7e432
2013-09-08 18:19:54 -07:00
Brad Fitzpatrick 4270ec4ce5 camput: remove unnecessary parens and allocation
Change-Id: I78b22458d4bd001313d6d49c6e401bc409b953fd
2013-09-08 18:17:51 -07:00
Brad Fitzpatrick 2cf4d68267 Let shares have expiration times.
Change-Id: Iecae106302ba43c354b6987163830cf78a70d12c
2013-09-08 17:46:40 -07:00
Brad Fitzpatrick d2d65912d9 Merge "Document environment variables usage." 2013-09-08 15:25:56 +00:00
Aaron Boodman 8dd7dd42cf Fix crash in `camput file` when configuration discovery has failed.
Change-Id: I8a7ac26e5d550f25eddf149dadde5013a1d50fe7
2013-09-07 19:15:44 -07:00
Bill Thiede 630f6ca61f Document environment variables usage.
Running 'go run dev/envvardoc/envvardoc.go' now shows:
'All environment variables are documented'

I also took the liberty of cleaning-up our mishmash of logic for handling
boolean environment variables, and cleaned up a couple other spots that didn't
seem right.

This change adds docmentation for all variables starting with (CAM|DEV|AWS).
This leaves some variables still undocumented.  If there are variables worth
documenting in the following list, maybe we should rename them to have a
CAM{LI} prefix for consistency's sake:

APPDATA                  pkg/osutil/paths.go:86
APPDATA                  pkg/osutil/paths.go:102
DISPLAY                  pkg/misc/gpgagent/gpgagent.go:126
GOPATH                   pkg/fileembed/genfileembed/genfileembed.go:321
GOPATH                   pkg/osutil/paths.go:168
GOPATH                   pkg/test/world.go:54
GOPATH                   server/appengine/build_test.go:77
GPGKEY                   cmd/camput/init.go:77
GPG_AGENT_INFO           cmd/camput/init.go:153
GPG_AGENT_INFO           pkg/misc/gpgagent/gpgagent.go:50
HOME                     pkg/jsonsign/keys.go:79
HOME                     pkg/jsonsign/signhandler/sig.go:64
HOME                     pkg/osutil/paths.go:36
HOMEPATH                 pkg/osutil/paths.go:34
PKG_CONFIG_PATH          pkg/index/sqlite/dbschema.go:59
RUN_BROKEN_TESTS         pkg/fs/fs_test.go:67
SKIP_DEP_TESTS           pkg/test/testdep.go:29
TERM                     pkg/misc/gpgagent/gpgagent.go:133
TERM                     pkg/misc/pinentry/pinentry.go:99
TESTING_PORT_WRITE_FD    pkg/webserver/webserver.go:135
TEST_GPGAGENT_LIB        pkg/misc/gpgagent/gpgagent_test.go:27
USER                     pkg/netutil/ident.go:135
USER                     pkg/osutil/paths.go:45
USERNAME                 pkg/jsonconfig/eval.go:228
USERNAME                 pkg/osutil/paths.go:43
VERBOSE_FUSE             pkg/fs/fs_test.go:133
VERBOSE_FUSE_STDERR      pkg/fs/fs_test.go:137
XDG_CONFIG_HOME          pkg/osutil/paths.go:104

Change-Id: Ief28710d3deefd1e65247cb5d3b1d8dde73e1f2d
2013-09-06 21:54:03 -07:00
mpl 8d71c9bb60 camput: have cache and stat cache based on github.com/cznic/kv
Added pkg/kvutil/kvutil.Open for convenience

Change-Id: Ie8db58e2c644a7358f252548bc57cccff7627497
2013-08-30 23:54:22 +02:00
Brad Fitzpatrick b24cad68dd Cleanup: remove BlobHub and time.Duration waits from storage interface
Move up a layer to the HTTP.  Also, start to remove ContextWrapper
stuff.  We've done it differently for App Engine instead, and will do
it differently yet moving forward.

Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.

Bunch of other cleanup and TODO fixing too.

Much simpler and cleaner.

Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
2013-08-21 13:57:28 -07:00
mpl 31df3635a4 camput: ignore files like .DS_Store with -filenodes
Also made initTrustedCertsOnce a field of the Client
object, as it is a similar change.

http://camlistore.org/issue/104

Change-Id: Iabbd7f06e06d31265f390a23c4bdaac956f856f7
2013-08-19 17:21:10 +02:00
Brad Fitzpatrick 0bdf20884b all: delete pkg/blobref; convert all from *blobref.BlobRef to new blob.Ref
Change-Id: Id2dfb7f19452bedf4f3c9310b36227fd8117b225
2013-08-03 19:54:30 -07:00
Brad Fitzpatrick 02d38b07ca camput, client: move lot of camput's code into pkg/client
so cammount can use it.

Change-Id: I1bd513e6dfe6faab785c856abbd0d1337e978dd9
2013-07-11 16:46:23 +10:00
Brad Fitzpatrick 76013089a4 Move blobref.ParseMulti to its sole caller, with error checking.
Change-Id: I6351591497879ea5a113bc8e8414b751b3d1c812
2013-07-07 11:46:21 -07:00
mpl fd8af44431 camput: stat cache: use a different key in -filenodes mode
http://camlistore.org/issue/157

Change-Id: If58116be6073e0bd160d57bb8f778e498a8763bb
2013-06-28 19:52:57 +02:00
mpl b24ac50965 client: factorize the transport setup, and use it in camget
Fixes http://camlistore.org/issue/153

Change-Id: If393b34bbfcf4a6282357c9c57ebb85bae8741a3
2013-06-25 23:37:41 +02:00
mpl 0de2881a16 client: help (wrt to the gpg key) initialize the config
Change-Id: I983e4396abacbc4d8fc354863cffeece65dd5b90
2013-06-22 01:16:11 +02:00
Brad Fitzpatrick d1fd483548 camput: remove Go 1.0 workaround.
Now that we rely on Go 1.1, we no longer need to workaround https://code.google.com/p/go/issues/detail?id=4589

Change-Id: Ib6631679e7f9bcea82ef96df0f50d9ecace31aa3
2013-06-16 20:56:58 -07:00
mpl 893b5b2f22 camput: sqlite backend for have cache and stat cache
http://camlistore.org/issue/73

Change-Id: Id6fb95f1980139f2b525321a7e1b553cb11fe727
2013-05-01 18:42:48 +02:00
Brad Fitzpatrick 2ebd1aa8e4 Merge "camput: reenable -secret-keyring, use flag.StringVar" 2013-05-01 14:34:12 +00:00
mpl db9fdb7e6e camput: reenable -secret-keyring, use flag.StringVar
http://camlistore.org/issue/76

Change-Id: I37229a1082e5d955cadcc63521ea1b13dd63a33c
2013-05-01 16:48:19 +02:00
Brad Fitzpatrick 6be77f09e4 Merge "Allow HTTPS with a self-signed certificate." 2013-05-01 02:30:23 +00:00
mpl 9b145ad1d7 camput: refactor; break up uploadNodeRegularFile
Change-Id: Ifde8f1a249eb4831757a09ea0ac881b049a90525
2013-04-24 17:51:04 +02:00
mpl ac431439f3 Allow HTTPS with a self-signed certificate.
A new client configuration parameter, "trustedCerts" (list of strings)
is introduced. A certificate fingerprint is defined as the 10 digits
prefix of the sha1 of the whole certificate (in ASN1. DER form).
trustedCerts should contain the list of fingerprints of the
certificates we trust. If not empty, the server's certificate
is checked against that list, instead of using the full x509 validation
on it.

-added a dial function and tls configuration, which check if we're using
SSL, and if we're in "trustedCerts" mode.
pkg/client/client.go
pkg/client/config.go

-moved android specific hacks from camput to the client layer, so that
the dial and tls config could reuse/access them. Also allows future
reuse for other commands, such as camget.
pkg/client/android.go

-adapted camput to the above changes:
cmd/camput/android.go
cmd/camput/camput.go
cmd/camput/files.go

-server prints a hint when it generates the self-signed:
pkg/misc/misc.go
server/camlistored/camlistored.go

-camliactivity:
clients/android/res/xml/preferences.xml
clients/android/src/org/camlistore/Preferences.java
clients/android/src/org/camlistore/SettingsActivity.java
clients/android/src/org/camlistore/UploadService.java
clients/android/src/org/camlistore/UploadThread.java

http://camlistore.org/issue/131

Change-Id: I6be20161549a69aafc8eb7b9e96e9351dc1c5b09
2013-04-23 17:10:37 +02:00
mpl 0b873649cb camput: change "name" option to "title".
Title is the attribute used in the ui, and actually recorded
by the indexer, so having "name" in camput was not consistent.

Change-Id: I6f63e54e3b19a83031f3c56c717b27336b175284
2013-04-22 17:13:50 +02:00
mpl e6a5882511 website doc: cmds: golang style + install command
Change-Id: I4f38dfb87e066ef81881bc9a2262c52109e172c9
2013-03-15 23:18:09 +01:00
mpl 698e8605e6 website: /pkg/ and /cmd/ work.
-added some doc.go for each command to make /cmd/*
work nicely.
-hacked a bit of style, mostly borrowed from golang too.

Change-Id: I011d70bca2c0151a29dfd53c78a896821730d5fb
2013-03-15 16:40:44 +01:00
mpl b5e75e9207 cmdmain: use ExitWithFailure, exit on error instead of returning it
Change-Id: I553bed110394cb8ab48b331ba74d551b840ada5c
2013-02-27 22:04:50 +01:00
mpl 81cb48980e camtool: added dbinit.
Change-Id: I26c58023a5275612dfd2bf85adfd59e8c08baab5
2013-02-27 16:07:02 +01:00
mpl 3a21c89712 camput: fix test. Add descriptions for better help.
Change-Id: I9016fca1dcf36ca5908947e84a5251bf587a83a7
2013-02-26 21:08:24 +01:00
mpl 989092a39b camput: do not parse flags before AddFlags
Change-Id: I514a89999e0e2e71b97d586bd0baa74ff0330472
2013-02-22 23:27:39 +01:00
mpl 59d0d6cb7e Create camtool cmd, with sync subcommand.
Code in common with camput was factorized in pkg/cmdmain/
Partly addresses
http://code.google.com/p/camlistore/issues/detail?id=117

Change-Id: Iabebea7ea78e56bdb8a6eccee9456c52bfe9cceb
2013-02-20 23:47:17 +01:00
mpl 981a4c9fbd camput: even with -vivify, do not reupload "big" file when server has it
This change takes advantage of Uploader.fileMapFromDuplicate to avoid
reuploading "large" files when the server already has them and the
client has lost its cache (and hence does not know about it).

Fixes: http://code.google.com/p/camlistore/issues/detail?id=88

Change-Id: I9fa819cef37e1d930a5393c9bdce5f8bb71da9c6
2013-02-12 19:02:05 +01:00
mpl f7d048d4ee Merge "rm TODO" 2013-02-03 22:50:42 +00:00
Marc-Antoine Ruel 73870dc3c2 Enable the use of localhost as a proxy for debugging purposes.
This is useful to do blackbox performance analysis at the http layer.

Change-Id: Idf26f17e1fd0cb9421b3def9c846c4bffecaa433
2013-02-03 12:26:29 -08:00
Brad Fitzpatrick a8e3cfde54 update TODO
Change-Id: I5075de51cc8fa64658c1d74038a37c3f1a92383c
2013-02-03 10:26:18 -08:00
Brad Fitzpatrick 96ea9af952 camput: add debug mechanism CAMLI_NO_FILE_DUP_SEARCH to disable fileMapFromDuplicate
Change-Id: I1772ed69cccd4245e660887e257da2ab30bd59e4
2013-02-02 21:34:05 -08:00
Brad Fitzpatrick ddcec4e92c Add --version flags to tools.
Change-Id: I1c86c3de09fb59bc20539f7d1efbd4d4710bf494
2013-02-02 10:59:39 -08:00
mpl 9ef7e5d2b9 rm TODO
Verified: if a claimDate is set, the client will create
a signRequest that has a signature time set to that claimDate.
Then when the opengpg signing is done, it will use that time
instead of time.Now().

Change-Id: I658d52dadfcea7a9b6f9291c4d8020ac0efa83b5
2013-02-01 16:09:58 +01:00
mpl 0b2532d96a camput: s/androidStatusRecevier/androidStatusReceiver/
Change-Id: I9cefc638b821948fad911c3e51e4a07c4da655b0
2013-02-01 15:52:39 +01:00