Commit Graph

3095 Commits

Author SHA1 Message Date
Brad Fitzpatrick b34c6198e4 sqlite: add start of an fd leak test
Change-Id: I84857e7a57bc4d95fa1a0ddc1c7538d1924cb93a
2013-09-13 11:21:48 +01:00
Bill Thiede 12427b1fb1 server/image: properly set image format.
The value of format was empty and causing cr2 and jpeg images to fall through
to the the default case, encoding thumbnails as PNG.

This change sets the default case as jpeg, and ensures that cr2 is encoded as
jpeg.  Png files now have their own explicit branch in the switch.

Tested by uploading a cr2, jpeg and png file to camlistore, then viewed the
mime type in the thumbnail response for each.

Change-Id: Ibe4400f220014b3fd72d9ae420c7d60c2f8cb33a
2013-09-12 19:53:38 -07:00
Brad Fitzpatrick afb81d0f5d Add checker for iterator Close calls
Change-Id: I90ba84635dd55ee056acedb8d0c0f5fa5d2a0ef5
2013-09-12 18:13:29 +01:00
Brad Fitzpatrick 45034814d8 Add pkg/leak, to help find leaks.
Change-Id: I52b15f168f63c5ac40c40f258d1f3ff2ffd4965e
2013-09-12 17:51:09 +01:00
Brad Fitzpatrick 0375ee2d47 Do directory describe queries in parallel.
Change-Id: I9e3a4d97a97b177356d11d4ff0a735d80bfc63db
TODO: use syncutil.Group in many more places.
2013-09-12 16:22:00 +01:00
Brad Fitzpatrick c8044d02ea Add syncutil.Group: an easy way to do multiple things at once.
Change-Id: Ie28885a5f42019dd1c3fb14f92c7cc5d40e08742
2013-09-12 16:18:39 +01:00
Brad Fitzpatrick 1a99796288 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-09-12 16:10:51 +01:00
Brad Fitzpatrick 59534f0ad6 Move pkg/gate to pkg/syncutil, to house more stuff.
Change-Id: I737458641a9de03da55ed627bfac10f3377a13aa
2013-09-12 16:04:10 +01:00
mpl 6cf55d78b0 index: GetDirMembers, forgot to close iterator
Could fix too many open files problem with sqlite reported by ajft

Change-Id: Icde85b11dcd96a336d1bba1e15a7665a1b4a874a
2013-09-12 16:35:12 +02:00
Bill Thiede 53397ce117 Add ./dev/... to multiple build targets.
Adds ./dev/... to forcefull, full, & presubmit.

Change-Id: I64d947b92dcda09ceb6bd2c536f1b698e2367cc0
2013-09-11 19:36:49 -07:00
Brad Fitzpatrick 1decd09846 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-09-11 07:46:58 +01:00
Brad Fitzpatrick 13691c699f make.go: build camput, etc faster, ignoring embedded UI resources
Change-Id: I1f7f60f4d5de97f69ddaf8c370889320ca8d5efe
2013-09-10 22:42:24 -07:00
Brad Fitzpatrick ee4550bff4 camput: fix deadlock when uploading a large directory tree
The upload was constrained to have only 5 workers goroutines at a
time, but those goroutines could have been uploading directories and
blocked waiting for their dependent children, which never got to
upload if all 5 goroutines were uploading directories.

Now, allow an unlimited number of upload goroutines (see commit
058434449), but add a gate around things that allocate file
descriptors within the upload function.

This means that now we could have a ton of goroutines existing but
blocked on the gate, wasting ~4KB of memory each rather than the
relatively small amount of memory we used for the *node on the
container/list before.

Future CLs can reduce the memory usage and/or rewrite this.

Or maybe we forget to clean this up, but get smaller goroutine memory
usage in Go 1.3 as is being discussed.  It _is_ convenient to just
launch tons of goroutines and manage dependencies with channel
operations rather than managing the dependencies state and goroutine
scheduling in your own code, just to save some memory.

Change-Id: I43d9d7795e5df4d790d6086d18e4b4b99eaacbc8
2013-09-10 20:16:16 -07:00
Brad Fitzpatrick 058434449c camput: add an unbounded mode to chanworker.
Change-Id: I90040fc6a67c88ae54338ab9556295861170a871
2013-09-10 20:07:20 -07:00
Brad Fitzpatrick e731931edd Some TODO items.
Change-Id: I4236764cb1dc4a1aea003d86b43687576c1fe3c5
2013-09-10 20:02:26 -07:00
Brad Fitzpatrick 6fca50a9c7 devcam: add -nobuild flag to put.
But really this should be a global option, and/or environment controlled,
like dev/demo.sh set up previously.

Change-Id: Iabf0c02e4a44ed574088e2938f3b4f53cabf3a43
2013-09-10 20:02:15 -07:00
mpl d488c576fc search: support for static directory children
This change introduces a new index entry
to help with finding the children of a static directory.
It also fixes ResolvePrefixHop so that it takes
into account static directories, and not only collections.

This is the first step to support publishing static directories.

http://camlistore.org/issue/179

Change-Id: I5666e5caa6c782004054ae4c19a6b6119d4fda8b
2013-09-10 23:06:48 +02:00
Brad Fitzpatrick 6f75f451c8 localdisk: another attempt at fixing Mkdir/Rmdir locking for Issue 177
Change-Id: I0569727b3cc1c2cf4c1b48f9c4b70eb645434f5a
2013-09-10 13:57:36 -07:00
mpl 69367a7e5a Merge "devcam: do not depend on GOPATH, rebuild camget and camput" 2013-09-10 19:25:58 +00:00
mpl 50c07cf7be devcam: do not depend on GOPATH, rebuild camget and camput
http://camlistore.org/issue/218

Change-Id: I4f48fac38cad095cd6d4c4d5aea4efeb35fff306
2013-09-10 21:15:12 +02:00
Brad Fitzpatrick cdaaa2f746 Merge "serverconfig: add test for mongo indexer, fix dev mongo config" 2013-09-10 18:25:32 +00:00
Brad Fitzpatrick 00115d2bcc Merge "support thumbnailing or CR2 files" 2013-09-10 18:24:16 +00:00
Brad Fitzpatrick 08fea5ab9b Update cznic/exp/lldb to 9825efece0 (Fixes Camlistore Issue 216) 2013-09-09 15:42:12 -07:00
mpl edd7f68799 serverconfig: add test for mongo indexer, fix dev mongo config
Change-Id: Ifb7f8cef208dc93a4cf350ff197bb2af2c0e0e81
2013-09-09 22:54:32 +02:00
mpl ad84a3565e camput: check stat cache value form
Addresses last comment from
http://camlistore.org/r/606

Change-Id: I5c0eb5492fdcfeb71becdeb25c6c09a1971ac09c
2013-09-09 19:08:07 +02:00
Andrew Gerrand 7b60af2ac4 support thumbnailing or CR2 files
Change-Id: I76f102a8cd4283f6fcb54985c52a16ddb8f6a44f
2013-09-09 23:49:07 +10:00
Brad Fitzpatrick 5c83627a1f camput: add debugging for camlistore.org/issue/221
Change-Id: If0ed73cb16409ac5c1f67ffc878038075cb7e432
2013-09-08 18:19:54 -07:00
Brad Fitzpatrick 4270ec4ce5 camput: remove unnecessary parens and allocation
Change-Id: I78b22458d4bd001313d6d49c6e401bc409b953fd
2013-09-08 18:17:51 -07:00
Brad Fitzpatrick c1edf39880 Add devcam to gitignore
Change-Id: If55797b29fc4f192501c335e3a663d939c23e4b0
2013-09-08 18:08:16 -07:00
Brad Fitzpatrick abd2393c06 share: fix shares with no expiration time.
with tests.

Change-Id: I5861104c223de8b4381a69e0ab16998979ca9106
2013-09-08 18:04:54 -07:00
Brad Fitzpatrick 2cf4d68267 Let shares have expiration times.
Change-Id: Iecae106302ba43c354b6987163830cf78a70d12c
2013-09-08 17:46:40 -07:00
Brad Fitzpatrick 13a21a61df replica: allow optional readBackends config option. Default is same as write backends.
Change-Id: I8aae98f76449c41d937469cf91d6788c348807ed
2013-09-08 16:35:56 -07:00
Aaron Boodman 97fab00c71 Fix bug parsing baseURL config
Change-Id: Ia8a5698ae5e0421672bd91f9a1c1497bcd36eb7c
2013-09-08 16:31:43 -07:00
Brad Fitzpatrick acc90f184c replica: use blob.Ref as map key. add TODO comments.
Change-Id: Ie697c32b296df02134ab3aeda6defa96e3ff1aaa
2013-09-08 16:06:59 -07:00
Brad Fitzpatrick a51edb5e77 localdisk: simplify directory locking.
Tries to fix https://code.google.com/p/camlistore/issues/detail?id=177

Change-Id: I2ea089437528d39ecdb8a3a3129e03ba11a76e67
2013-09-08 14:55:41 -07:00
Hunter Freyer 019cffac5c Reorganizes the "schema" page on the website, and promotes much of the
permanode documentation to the website in its own page.

One of the goals here is to create an exhaustive list of camliTypes
and permanode attributes, and make it clear what they're used for.

Change-Id: I68a95fe5f870a9b25551b1e9c2eec805920d8931
2013-09-08 17:01:09 -04:00
Brad Fitzpatrick 00d8ff5275 index: remove now-longer-necessary blob hash check
Change-Id: Ia2a79655832a840d37666b94a1f101042861c8ff
2013-09-08 12:38:20 -07:00
Brad Fitzpatrick b80c7ca907 replica: close uploader goroutines on receive read error
Change-Id: I0f6a3013192b4d883ce6810718bb604a9a5fa46b
2013-09-08 12:37:07 -07:00
Brad Fitzpatrick ff4e491b3b devcam: fix breakage caused from envvar strconv changes.
Change-Id: Id05d00d9b82924242905ccbc7a7e8002dc4859a7
2013-09-08 12:32:13 -07:00
Brad Fitzpatrick d2d65912d9 Merge "Document environment variables usage." 2013-09-08 15:25:56 +00:00
Aaron Boodman 8dd7dd42cf Fix crash in `camput file` when configuration discovery has failed.
Change-Id: I8a7ac26e5d550f25eddf149dadde5013a1d50fe7
2013-09-07 19:15:44 -07:00
Bill Thiede 630f6ca61f Document environment variables usage.
Running 'go run dev/envvardoc/envvardoc.go' now shows:
'All environment variables are documented'

I also took the liberty of cleaning-up our mishmash of logic for handling
boolean environment variables, and cleaned up a couple other spots that didn't
seem right.

This change adds docmentation for all variables starting with (CAM|DEV|AWS).
This leaves some variables still undocumented.  If there are variables worth
documenting in the following list, maybe we should rename them to have a
CAM{LI} prefix for consistency's sake:

APPDATA                  pkg/osutil/paths.go:86
APPDATA                  pkg/osutil/paths.go:102
DISPLAY                  pkg/misc/gpgagent/gpgagent.go:126
GOPATH                   pkg/fileembed/genfileembed/genfileembed.go:321
GOPATH                   pkg/osutil/paths.go:168
GOPATH                   pkg/test/world.go:54
GOPATH                   server/appengine/build_test.go:77
GPGKEY                   cmd/camput/init.go:77
GPG_AGENT_INFO           cmd/camput/init.go:153
GPG_AGENT_INFO           pkg/misc/gpgagent/gpgagent.go:50
HOME                     pkg/jsonsign/keys.go:79
HOME                     pkg/jsonsign/signhandler/sig.go:64
HOME                     pkg/osutil/paths.go:36
HOMEPATH                 pkg/osutil/paths.go:34
PKG_CONFIG_PATH          pkg/index/sqlite/dbschema.go:59
RUN_BROKEN_TESTS         pkg/fs/fs_test.go:67
SKIP_DEP_TESTS           pkg/test/testdep.go:29
TERM                     pkg/misc/gpgagent/gpgagent.go:133
TERM                     pkg/misc/pinentry/pinentry.go:99
TESTING_PORT_WRITE_FD    pkg/webserver/webserver.go:135
TEST_GPGAGENT_LIB        pkg/misc/gpgagent/gpgagent_test.go:27
USER                     pkg/netutil/ident.go:135
USER                     pkg/osutil/paths.go:45
USERNAME                 pkg/jsonconfig/eval.go:228
USERNAME                 pkg/osutil/paths.go:43
VERBOSE_FUSE             pkg/fs/fs_test.go:133
VERBOSE_FUSE_STDERR      pkg/fs/fs_test.go:137
XDG_CONFIG_HOME          pkg/osutil/paths.go:104

Change-Id: Ief28710d3deefd1e65247cb5d3b1d8dde73e1f2d
2013-09-06 21:54:03 -07:00
Brad Fitzpatrick 85473f6849 Merge "More work on OS X launcher." 2013-09-07 00:58:43 +00:00
Nick O'Neill cb40c185f6 More work on OS X launcher.
Change-Id: Id88e5f1c26b536b4a296f25b08bb4375ea014dc5
2013-09-06 17:55:51 -07:00
Brad Fitzpatrick 7a34e5a52a Upstream bug is fixed.
http://golang.org/issue/6319

Change-Id: I9467a7727fdb4b27cb1c4b5ceb093c2b1635427f
2013-09-04 13:44:43 -07:00
Brad Fitzpatrick 501c6dcd45 Change make.go to build devcam. Update HACKING file.
Change-Id: I6938b519cc721ca23ab9c6de8e86a6a1b944010f
2013-09-03 19:56:42 -07:00
Brad Fitzpatrick dabe4355af Change how the Android camput.arm asset is built to use make.go
Change-Id: I00a6e22ef088a173f78ffca83c984d915670180c
2013-09-03 19:48:32 -07:00
Brad Fitzpatrick 83ab8a62e8 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-09-03 18:22:53 -07:00
Brad Fitzpatrick 3a5793b743 search: actually use result of SearchPermanodesWithAttr and don't leak a goroutine
Change-Id: Ia1ff8acc1d396901c6b589dbe9a972e3dd4df05f
2013-09-03 18:22:05 -07:00
Andrew Gerrand bcfeeb47f6 diskpacked: cleanups
Change-Id: I91566b120698b618ad4f83468a0cd4ea2c0ae765
2013-09-03 10:27:33 +10:00