Commit Graph

249 Commits

Author SHA1 Message Date
Brad Fitzpatrick 9fd329df80 Merge "cmd/camtool: (re)index command" 2013-12-26 17:29:03 +00:00
mpl 2d85e017ff cmd/camtool: (re)index command
http://camlistore.org/issue/193

Change-Id: I498f92bdc153f44dc84d4b47f03c47a8e7b54ad9
2013-12-26 18:23:15 +01:00
Dustin Sallings 3f967a1111 camput: set handlerConfig=true when generating a config
Change-Id: I10ca5274a90640dd72f7c678f1000cf0ddf5adf9
2013-12-25 01:53:02 -08:00
Brad Fitzpatrick 649390b9ef camtool: make diskpacked-reindex command a bit more user-friendly
Change-Id: Ic7cde2f453f5cd2716d929800b297cb766826acd
2013-12-24 20:16:36 -08:00
mpl 4a87b954fb Merge "pkg/index: move postgresql to sorted + some fixes" 2013-12-23 23:49:14 +00:00
mpl 8fec95752c pkg/index: move postgresql to sorted + some fixes
Change-Id: Ia6410256ffe8d4cbc2b9a5c056d075707105ab08
2013-12-24 00:37:14 +01:00
mpl 6e7eacea4a Merge "pkg/client: some cleanup, doc on the website" 2013-12-23 22:31:45 +00:00
mpl 1e8d46ee45 pkg/client: some cleanup, doc on the website
Changed some keys to be consistent with the server config, and removed
some others.

We're now using jsonconfig to get the keys and validate the config.

Removed some unused funcs/methods, moved some global vars as members of a client.

Also, having two possible defaults for the secret keyring
(.config/camlistore/identity-secring.gpg and .gnupg/secring.gpg)
was annoying and confusing imho, so I removed the latter.

http://camlistore.org/issue/259

Change-Id: Icf79fa8822b62b5d91843698861dd19cb553607a
2013-12-23 20:23:21 +01:00
Brad Fitzpatrick 450409d0a8 camput: rename -name flag to -title for consistency with other commands
Fixes camlistore.org/issue/285

Change-Id: Ide3fba2c40fc8e9f0ff187c183f2cc0acd97f68e
2013-12-22 09:29:32 -08:00
mpl 62f8a252ff pkg/index: move mysql to sorted
also minor cleanups in sqlite and mongo

Change-Id: I1f371358997d929c72a8f63d2630a1c3fa4a0240
2013-12-18 20:18:20 +01:00
Brad Fitzpatrick 66164b186e Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-12-17 11:27:31 -08:00
Brad Fitzpatrick 2af4040223 camtool: add env sub-command 2013-12-17 11:27:24 -08:00
Mathieu Lonjaret ac7b19de20 Merge "camput: allow ignoring files NOT only if --filenodes used" 2013-12-17 15:11:07 +00:00
Brad Fitzpatrick 8af7795f4a cammount: don't require a mount dir; add --open flag
Change-Id: I87c280b333a1ee9445b74e3ee1c87f6c08931ef3
2013-12-16 20:57:43 -08:00
mpl b62c94fdd1 pkg/index: move sqlite implementation to sorted
Change-Id: I1e300ffa14547fabeac8d255fd694054ebcb9e53
2013-12-16 17:35:27 +01:00
jakubbrzeski 33e925ffa1 camput: allow ignoring files NOT only if --filenodes used
Change-Id: I7856c5f6602e6b439ebcebbcb83eddb1d1865e88
2013-12-16 16:52:59 +01:00
mpl 8c562e9135 camtool: dbinit mongo support
Change-Id: I289759ed9bdb34a6e108af1364c4a6bcfbb5982f
2013-12-13 17:25:03 +01:00
mpl 62cbf8550c camput init: use osutil.KeyBlobsDir()
Change-Id: I1b9f29c24361ffab708af4bdaff8c870ff8730e6
2013-12-11 00:47:50 +01:00
Brad Fitzpatrick 47828b6900 camput: add --delete_after_upload flag.
But untested in tree mode. I think it's incomplete still.

Change-Id: Ia0c692e53a099b12a2b66735c228ea6f78d9c940
2013-12-10 15:05:41 +04:00
Brad Fitzpatrick b82b8efe4c Start of new context package and *context.Context type.
Will eventually be plumbed through lots of APIs, especially those requiring or benefiting from
cancelation notification and/or those needing access to the HTTP context (e.g. App Engine).

Change-Id: I591496725d620126e09d49eb07cade7707c7fc64
2013-12-02 13:20:51 -08:00
mpl d51647ae33 camput: delete command, to send a delete claim
http://camlistore.org/issue/191

Change-Id: I053ab468420cc173e6122909bf3a0b16e4d43224
2013-11-29 00:19:42 +01:00
Brad Fitzpatrick 3fd356f457 gofmt
Change-Id: I900a49b3f76aec7ef65419f2a5ecaf3eae95defb
2013-11-28 11:58:47 -08:00
Tamás Gulácsi cb6f423eeb Add diskpacked-reindex subcommand to camtool
For checking and/or rebuilding index.kv of diskpacked packs.

Change-Id: I1ad87974b2daf58b1e767bb1df6f7b64e8b6359a
2013-11-28 20:44:27 +01:00
Brad Fitzpatrick dcf5f6abe7 client: clean up. finish moving all Android stuff to pkg/client/android
Change-Id: Ia3d19a323d9c734a01ec9a12509b115191afcfdd
2013-10-26 11:08:34 -07:00
Brad Fitzpatrick 2040d2e442 Add new pkg/client/android, to clean up Android API gunk from pkg/client's godoc.
This is just a start. AndroidStatsTransport and AndroidStatusReceiver and most (all?)
of pkg/client/android.go should migrate too.

Change-Id: Id4140730bbdd218fdf7ba7cedc7870ce5164e06c
2013-10-22 07:37:15 -07:00
Brad Fitzpatrick cf15b2be90 client: use the schema.Signer API, remove Client.SignBlob method
Change-Id: I177c78f4f7f4ca54afd021d4ada866a35eea06eb
2013-10-22 07:26:44 -07:00
Salman Aljammaz a607971f5f cammount: handle SIGINT, so that we unmount too in that case.
Change-Id: I5e82c6326aab892674c775e0615a91af71532d3c
2013-10-17 15:22:18 +01:00
Tamás Gulácsi e056081be1 add capctime flag to camput
I can recreate the file, set the modification time, but can't forge creation
time, thus cannot recreate the exact same uploaded file twice.
This change targets this problem.

As this is a very rarely used flag, this is enabled only if the CAMLI_DEBUG
environment variable is true - such as the other "gross" options
(statcache, sqlitecache...).

Change-Id: I99c7805898ed97883043de4e7dcec23a472e462c
2013-10-09 21:06:51 +02:00
Steve Phillips 5dc853110f Added missing var 'which' used in a fmt.Errorf
Change-Id: I0b9a7c6a0c7e523a75f3e5beebbd860e0092a81f
2013-10-04 18:44:40 -07:00
mpl 304f5c4b38 camput attr: support -del
Change-Id: I28460fed5b572ba9a83cfcb04f38e42124fa723f
2013-10-03 15:07:43 +02:00
Brad Fitzpatrick e70f258b0b Merge two copies of StatsReceiver into common location.
Change-Id: I05bd36aef3293927fb518296b43a088f284356ea
2013-09-22 11:19:10 +01:00
mpl 0c1dc0d6e4 devcam test: generate keyblob with devcam put init
The integration tests run by devcam test require a
GPG keyblob in the default location (for camput), and
we do not track in git such a keyblob.
This change allows devcam test to create the keyblob
with devcam put init before running the tests.

Also, devcam server is now using CAMLI_KEYID instead
of hardcoding the key id in the dev server config file.

Change-Id: I030b88c9a9321016fedb62424e0e707621206922
2013-09-20 20:03:21 +02:00
Brad Fitzpatrick 79065de4b9 For SQLite compile errors, link to the wiki.
Change-Id: I34b1485daba6173d6665cfc56b6bba43164534d8
2013-09-19 10:17:57 +01:00
Brad Fitzpatrick 8319411ab4 Convert more ReceiveBlob into blobserver.Receive or blobserver.ReceiveNoHash
Change-Id: I9199555324b617167a6062a8b55ed09b449bae4f
2013-09-16 15:57:14 +01:00
Brad Fitzpatrick 59534f0ad6 Move pkg/gate to pkg/syncutil, to house more stuff.
Change-Id: I737458641a9de03da55ed627bfac10f3377a13aa
2013-09-12 16:04:10 +01:00
Brad Fitzpatrick ee4550bff4 camput: fix deadlock when uploading a large directory tree
The upload was constrained to have only 5 workers goroutines at a
time, but those goroutines could have been uploading directories and
blocked waiting for their dependent children, which never got to
upload if all 5 goroutines were uploading directories.

Now, allow an unlimited number of upload goroutines (see commit
058434449), but add a gate around things that allocate file
descriptors within the upload function.

This means that now we could have a ton of goroutines existing but
blocked on the gate, wasting ~4KB of memory each rather than the
relatively small amount of memory we used for the *node on the
container/list before.

Future CLs can reduce the memory usage and/or rewrite this.

Or maybe we forget to clean this up, but get smaller goroutine memory
usage in Go 1.3 as is being discussed.  It _is_ convenient to just
launch tons of goroutines and manage dependencies with channel
operations rather than managing the dependencies state and goroutine
scheduling in your own code, just to save some memory.

Change-Id: I43d9d7795e5df4d790d6086d18e4b4b99eaacbc8
2013-09-10 20:16:16 -07:00
Brad Fitzpatrick 058434449c camput: add an unbounded mode to chanworker.
Change-Id: I90040fc6a67c88ae54338ab9556295861170a871
2013-09-10 20:07:20 -07:00
mpl ad84a3565e camput: check stat cache value form
Addresses last comment from
http://camlistore.org/r/606

Change-Id: I5c0eb5492fdcfeb71becdeb25c6c09a1971ac09c
2013-09-09 19:08:07 +02:00
Brad Fitzpatrick 5c83627a1f camput: add debugging for camlistore.org/issue/221
Change-Id: If0ed73cb16409ac5c1f67ffc878038075cb7e432
2013-09-08 18:19:54 -07:00
Brad Fitzpatrick 4270ec4ce5 camput: remove unnecessary parens and allocation
Change-Id: I78b22458d4bd001313d6d49c6e401bc409b953fd
2013-09-08 18:17:51 -07:00
Brad Fitzpatrick 2cf4d68267 Let shares have expiration times.
Change-Id: Iecae106302ba43c354b6987163830cf78a70d12c
2013-09-08 17:46:40 -07:00
Brad Fitzpatrick d2d65912d9 Merge "Document environment variables usage." 2013-09-08 15:25:56 +00:00
Aaron Boodman 8dd7dd42cf Fix crash in `camput file` when configuration discovery has failed.
Change-Id: I8a7ac26e5d550f25eddf149dadde5013a1d50fe7
2013-09-07 19:15:44 -07:00
Bill Thiede 630f6ca61f Document environment variables usage.
Running 'go run dev/envvardoc/envvardoc.go' now shows:
'All environment variables are documented'

I also took the liberty of cleaning-up our mishmash of logic for handling
boolean environment variables, and cleaned up a couple other spots that didn't
seem right.

This change adds docmentation for all variables starting with (CAM|DEV|AWS).
This leaves some variables still undocumented.  If there are variables worth
documenting in the following list, maybe we should rename them to have a
CAM{LI} prefix for consistency's sake:

APPDATA                  pkg/osutil/paths.go:86
APPDATA                  pkg/osutil/paths.go:102
DISPLAY                  pkg/misc/gpgagent/gpgagent.go:126
GOPATH                   pkg/fileembed/genfileembed/genfileembed.go:321
GOPATH                   pkg/osutil/paths.go:168
GOPATH                   pkg/test/world.go:54
GOPATH                   server/appengine/build_test.go:77
GPGKEY                   cmd/camput/init.go:77
GPG_AGENT_INFO           cmd/camput/init.go:153
GPG_AGENT_INFO           pkg/misc/gpgagent/gpgagent.go:50
HOME                     pkg/jsonsign/keys.go:79
HOME                     pkg/jsonsign/signhandler/sig.go:64
HOME                     pkg/osutil/paths.go:36
HOMEPATH                 pkg/osutil/paths.go:34
PKG_CONFIG_PATH          pkg/index/sqlite/dbschema.go:59
RUN_BROKEN_TESTS         pkg/fs/fs_test.go:67
SKIP_DEP_TESTS           pkg/test/testdep.go:29
TERM                     pkg/misc/gpgagent/gpgagent.go:133
TERM                     pkg/misc/pinentry/pinentry.go:99
TESTING_PORT_WRITE_FD    pkg/webserver/webserver.go:135
TEST_GPGAGENT_LIB        pkg/misc/gpgagent/gpgagent_test.go:27
USER                     pkg/netutil/ident.go:135
USER                     pkg/osutil/paths.go:45
USERNAME                 pkg/jsonconfig/eval.go:228
USERNAME                 pkg/osutil/paths.go:43
VERBOSE_FUSE             pkg/fs/fs_test.go:133
VERBOSE_FUSE_STDERR      pkg/fs/fs_test.go:137
XDG_CONFIG_HOME          pkg/osutil/paths.go:104

Change-Id: Ief28710d3deefd1e65247cb5d3b1d8dde73e1f2d
2013-09-06 21:54:03 -07:00
mpl 8d71c9bb60 camput: have cache and stat cache based on github.com/cznic/kv
Added pkg/kvutil/kvutil.Open for convenience

Change-Id: Ie8db58e2c644a7358f252548bc57cccff7627497
2013-08-30 23:54:22 +02:00
Brad Fitzpatrick b24cad68dd Cleanup: remove BlobHub and time.Duration waits from storage interface
Move up a layer to the HTTP.  Also, start to remove ContextWrapper
stuff.  We've done it differently for App Engine instead, and will do
it differently yet moving forward.

Also add blobserver.Receive and use it in most places, moving checksum
verification up a layer.

Bunch of other cleanup and TODO fixing too.

Much simpler and cleaner.

Change-Id: I12e56c5d4e53bfcf82bdd8fb0b6d57c248ff605c
2013-08-21 13:57:28 -07:00
mpl 31df3635a4 camput: ignore files like .DS_Store with -filenodes
Also made initTrustedCertsOnce a field of the Client
object, as it is a similar change.

http://camlistore.org/issue/104

Change-Id: Iabbd7f06e06d31265f390a23c4bdaac956f856f7
2013-08-19 17:21:10 +02:00
mpl 9fcb4b6434 dumpconfig: typo in date
Change-Id: Iebd75442070896b588092f8bad62e0d21cab3b27
2013-08-09 18:12:47 +02:00
Brad Fitzpatrick 462eb7e4c2 Add camtool dumpconfig subcommand.
Fixes http://code.google.com/p/camlistore/issues/detail?id=192
2013-08-08 17:06:45 -07:00
Brad Fitzpatrick 0bdf20884b all: delete pkg/blobref; convert all from *blobref.BlobRef to new blob.Ref
Change-Id: Id2dfb7f19452bedf4f3c9310b36227fd8117b225
2013-08-03 19:54:30 -07:00