Commit Graph

54 Commits

Author SHA1 Message Date
Brad Fitzpatrick 5458321511 campit: add -blobdir flag for serverless mode.
Fixes camlistore.org/issue/362

Change-Id: If4c75b649cc4cc62f3e7a0ab07c61b088560c3e4
2014-02-07 11:02:07 -08:00
Brad Fitzpatrick 2086406552 Client fixes and cleanups:
* Fix TODO about moving camput file's havecache up a layer into
  camput itself. So now it's used for all commands (camput permanode,
  camput attr, etc). Reduces HTTP requests.

* Fix camlistore.org/issue/343 -- don't do stats before uploads
  in cases where it's useless. Adds a field to UploadHandle.

* Fix camlistore.org/issue/344: upload the public key whenever
  uploading a signed blob. Usually this is a no-op because of the
  have cache.

* Use zero (or <=0 rather), not -1, to mean unknown on UploadHandle
  Size.

* More docs on public stuff.
2014-01-15 18:32:02 -08:00
Brad Fitzpatrick 59534f0ad6 Move pkg/gate to pkg/syncutil, to house more stuff.
Change-Id: I737458641a9de03da55ed627bfac10f3377a13aa
2013-09-12 16:04:10 +01:00
Brad Fitzpatrick ee4550bff4 camput: fix deadlock when uploading a large directory tree
The upload was constrained to have only 5 workers goroutines at a
time, but those goroutines could have been uploading directories and
blocked waiting for their dependent children, which never got to
upload if all 5 goroutines were uploading directories.

Now, allow an unlimited number of upload goroutines (see commit
058434449), but add a gate around things that allocate file
descriptors within the upload function.

This means that now we could have a ton of goroutines existing but
blocked on the gate, wasting ~4KB of memory each rather than the
relatively small amount of memory we used for the *node on the
container/list before.

Future CLs can reduce the memory usage and/or rewrite this.

Or maybe we forget to clean this up, but get smaller goroutine memory
usage in Go 1.3 as is being discussed.  It _is_ convenient to just
launch tons of goroutines and manage dependencies with channel
operations rather than managing the dependencies state and goroutine
scheduling in your own code, just to save some memory.

Change-Id: I43d9d7795e5df4d790d6086d18e4b4b99eaacbc8
2013-09-10 20:16:16 -07:00
Brad Fitzpatrick 02d38b07ca camput, client: move lot of camput's code into pkg/client
so cammount can use it.

Change-Id: I1bd513e6dfe6faab785c856abbd0d1337e978dd9
2013-07-11 16:46:23 +10:00
mpl b24ac50965 client: factorize the transport setup, and use it in camget
Fixes http://camlistore.org/issue/153

Change-Id: If393b34bbfcf4a6282357c9c57ebb85bae8741a3
2013-06-25 23:37:41 +02:00
Brad Fitzpatrick 2ebd1aa8e4 Merge "camput: reenable -secret-keyring, use flag.StringVar" 2013-05-01 14:34:12 +00:00
mpl db9fdb7e6e camput: reenable -secret-keyring, use flag.StringVar
http://camlistore.org/issue/76

Change-Id: I37229a1082e5d955cadcc63521ea1b13dd63a33c
2013-05-01 16:48:19 +02:00
mpl ac431439f3 Allow HTTPS with a self-signed certificate.
A new client configuration parameter, "trustedCerts" (list of strings)
is introduced. A certificate fingerprint is defined as the 10 digits
prefix of the sha1 of the whole certificate (in ASN1. DER form).
trustedCerts should contain the list of fingerprints of the
certificates we trust. If not empty, the server's certificate
is checked against that list, instead of using the full x509 validation
on it.

-added a dial function and tls configuration, which check if we're using
SSL, and if we're in "trustedCerts" mode.
pkg/client/client.go
pkg/client/config.go

-moved android specific hacks from camput to the client layer, so that
the dial and tls config could reuse/access them. Also allows future
reuse for other commands, such as camget.
pkg/client/android.go

-adapted camput to the above changes:
cmd/camput/android.go
cmd/camput/camput.go
cmd/camput/files.go

-server prints a hint when it generates the self-signed:
pkg/misc/misc.go
server/camlistored/camlistored.go

-camliactivity:
clients/android/res/xml/preferences.xml
clients/android/src/org/camlistore/Preferences.java
clients/android/src/org/camlistore/SettingsActivity.java
clients/android/src/org/camlistore/UploadService.java
clients/android/src/org/camlistore/UploadThread.java

http://camlistore.org/issue/131

Change-Id: I6be20161549a69aafc8eb7b9e96e9351dc1c5b09
2013-04-23 17:10:37 +02:00
mpl b5e75e9207 cmdmain: use ExitWithFailure, exit on error instead of returning it
Change-Id: I553bed110394cb8ab48b331ba74d551b840ada5c
2013-02-27 22:04:50 +01:00
mpl 989092a39b camput: do not parse flags before AddFlags
Change-Id: I514a89999e0e2e71b97d586bd0baa74ff0330472
2013-02-22 23:27:39 +01:00
mpl 59d0d6cb7e Create camtool cmd, with sync subcommand.
Code in common with camput was factorized in pkg/cmdmain/
Partly addresses
http://code.google.com/p/camlistore/issues/detail?id=117

Change-Id: Iabebea7ea78e56bdb8a6eccee9456c52bfe9cceb
2013-02-20 23:47:17 +01:00
Marc-Antoine Ruel 73870dc3c2 Enable the use of localhost as a proxy for debugging purposes.
This is useful to do blackbox performance analysis at the http layer.

Change-Id: Idf26f17e1fd0cb9421b3def9c846c4bffecaa433
2013-02-03 12:26:29 -08:00
Brad Fitzpatrick ddcec4e92c Add --version flags to tools.
Change-Id: I1c86c3de09fb59bc20539f7d1efbd4d4710bf494
2013-02-02 10:59:39 -08:00
Brad Fitzpatrick e8ef81c626 android/camput: more stats
Change-Id: I668b8ee0b7c32d33075e421445af2973be1f576a
2013-01-30 17:22:53 -08:00
Brad Fitzpatrick 51e88c7434 camput: run in an Android linux non-cgo environment.
Teaches it about Android DNS resolution and Android SSL root certs.

Change-Id: I22b53d4f836f58cd3dfc8c97e993138a8a5fe318
2013-01-26 16:50:24 -08:00
Brad Fitzpatrick fe13c7c712 move camput's statsTransport to httputil.StatsTransport, use it in camget too.
Change-Id: Iab48e32ccb8a99b8ec6b058061cff9a5b99e8712
2013-01-02 20:32:13 -08:00
Brad Fitzpatrick 22a4d5e050 camput: more logging
Change-Id: I0098d42279dcf6eac07f3cedefd20213bd468b7c
2012-12-24 10:55:00 -08:00
Brad Fitzpatrick 9abbbef45e camput: gofmt 2012-11-07 22:21:52 -06:00
Brad Fitzpatrick 281a6c40f4 client: correct case of SetHTTPClient 2012-11-07 21:03:46 -06:00
Brad Fitzpatrick 9b35026497 camput: rename tinkerTransport to statsTransport 2012-11-07 21:00:22 -06:00
mpl f91f1f88c4 camput: final error printing
Change-Id: Icc51560545b20315a0383d7bda80170c5612e1f2
2012-10-24 16:02:37 +02:00
Brad Fitzpatrick 6be58c9ce0 camput: start of proper tests
Change-Id: Icc9257154fecaaf2018971ab94e0dc54086d3719
2012-10-21 14:14:51 -07:00
Brad Fitzpatrick ef8e746cb5 merge
Change-Id: I06ab89274e35bad61944e208e366cd7436381e63
2012-10-21 13:59:13 -07:00
Brad Fitzpatrick e9b5bca10e camput: start of making it more testable
Change-Id: Ie65afeaa14ea483231674433cda5a90669ca0bc7
2012-10-21 13:58:06 -07:00
mpl 159c2a5216 camput: moved/added examples, added -help as a global flag to display usage
Change-Id: I3de13057c555d68752101043f671d7694a311021
2012-10-19 17:57:17 +02:00
Brad Fitzpatrick 5315bbf232 camput: re-do the stat/have flatcaches.
still waiting on leveldb-go, but this works for now.

Change-Id: I630d7bd2757a02d3a711c48075486c5cc6339118
2012-04-22 17:18:19 -07:00
Brad Fitzpatrick da42bdf523 camput: more code movement
Change-Id: I775096c39b5c9671a3c054b7d02562cfa787f0a1
2012-04-21 07:22:32 -07:00
Brad Fitzpatrick e6e26f2017 camput: move common uploader stuff to uploader.go
Change-Id: I6137a945f89f4a0603189a5fc04b49a570a3403b
2012-04-21 07:14:57 -07:00
Brad Fitzpatrick cd79828912 camput: move nodeworker into its own file
Change-Id: Ic61d35955fe4dd59ada65a314f0c94cc3f7b0f87
2012-04-21 07:07:07 -07:00
Brad Fitzpatrick 2274bd97b1 camput: bunch of cleanups and bugs and usability stuff.
Change-Id: I029cc38d5b10a2d322e26148eb4971b3c838c889
2012-04-20 14:43:23 -07:00
Brad Fitzpatrick 0baa073b44 camput: switch to new tree upload code.
Change-Id: Iae653e4442bab63979d584caecdce58df942385d
2012-04-19 17:59:41 -07:00
Brad Fitzpatrick 0f3ce52c54 camput: memoize SumBytes
Change-Id: Ie0ea05a895f97de2b069c088d4195b533b70ea8e
2012-04-18 07:24:45 -07:00
Brad Fitzpatrick 7e94879e86 camput: in du mode, only run 1 upload worker.
Change-Id: I65b642095ca9f162f14db512f34bb7073ef7a13b
2012-04-17 17:46:05 -07:00
Brad Fitzpatrick 715c5278e0 camput: working diskusage mode.
Change-Id: I7c3df20fed7720e3535b5e618d20842d1ddaadcc
2012-04-15 17:21:19 -07:00
Brad Fitzpatrick 4f0f53a90e camput: more TreeUpload work; adds NodeWorker helper.
Change-Id: Ie5919faa34fd2f6eb1b61669598e0a06e1d8ba07
2012-04-15 17:11:25 -07:00
Brad Fitzpatrick 69ce6654e8 camput: fix bug so default switch case can be hit now.
otherwise all other files were hitting the not-directory case.

Change-Id: I03e6021e947eee301e21db3f20c2da3c2f92ab2d
2012-04-15 14:25:41 -07:00
Brad Fitzpatrick 92b10a39a4 camput: remove dead flag
Change-Id: Idf6ec9416b5878379de19b53a93b86982125f7cb
2012-04-15 14:21:38 -07:00
Brad Fitzpatrick d05a336cb0 camput: move more blob-only stuff to blobs.go
Change-Id: I967e9dc975db18088b88c00b2672353a5a72f458
2012-04-15 14:21:00 -07:00
Brad Fitzpatrick 90c254038c camput: remove dead code
Change-Id: Ifab0247b6c23d050abd822b479558a0d75930b8b
2012-04-15 14:10:08 -07:00
Brad Fitzpatrick 77fc9e8abb camput: only support '-' meaning stdin for the 'blob' subcommand, not 'file'
Change-Id: I9a8923792385e232dc42c7707c1b748f82a5ac37
2012-04-15 14:01:39 -07:00
Brad Fitzpatrick 88cff46f28 camput: more TreeUpload work
Change-Id: I8638ccdc342642447661e68918259878e04ef756
2012-04-15 13:54:21 -07:00
Brad Fitzpatrick 61a895b1f8 camput: some more TreeUpload work
Change-Id: I2b4e74f7892720bbf6b544a7abb9867d5ad3415b
2012-04-15 11:26:36 -07:00
Brad Fitzpatrick f366dfb850 some TreeUpload TODO notes
Change-Id: If26ddcb6ee8ebc1b35f569ecbf835da5f666c3a7
2012-04-15 08:55:11 -07:00
Brad Fitzpatrick 72ac3fc8b1 camput: TODOs for VFS
Change-Id: I9b403ec5e3e66f150da4277eabe99e079c61d2b8
2012-04-15 08:28:18 -07:00
Brad Fitzpatrick 683e762c79 camput: go through virtual filesystem. will be good for filtering skipDirs & testing.
Change-Id: I2fa56615b6b199ab054de388dde652749049d5e1
2012-04-15 08:11:37 -07:00
Brad Fitzpatrick 18e53cd72f rename MapToCamliJson to Go style initialism MapToCamliJSON
Change-Id: I674944c9fafa6b78c495d22c525b9ffd0b75dbf9
2012-04-15 07:45:10 -07:00
Brad Fitzpatrick a8687d112c camput: move rollSplits to Uploader field, not passed along.
Change-Id: I8e78fe88e179235a03117927bd7b36217242e055
2012-04-15 07:18:40 -07:00
Brad Fitzpatrick 91501cdbd1 remove unused code
Change-Id: I4e35ce6dbb34f3db17024f49dd98bed7d3f9b4f0
2012-04-15 07:10:10 -07:00
Brad Fitzpatrick 77f9544936 gofmt
Change-Id: If71fa78e10464bc93fc7ec748e73d935d3c555ba
2012-04-14 22:04:00 -07:00