Commit Graph

65 Commits

Author SHA1 Message Date
mpl 748fd21e96 importer: wire run (context) HTTP client with host HTTP client
Killing two birds with one stone: we don't need to set it in devcam
anymore now.

Also switched to using the short form (run's one) everywhere, for
consistency.

Change-Id: I1e72ab99a07392c4970b1ed19b6e460ff422a24a
2014-07-22 15:40:06 +02:00
mpl b82459424d devcam server --makethings
Only creates twitter things for now.

http://camlistore.org/issue/417

Change-Id: I05d548e012586279e165c01468524ceb055ff17b
2014-07-21 23:24:33 +02:00
mpl 6f5133a728 server/app: first look for app executable in CAMLI_APP_BINDIR
Change-Id: I2d8e525c33da5813d67fbd74ca9af8059864a205
2014-06-16 00:02:09 +02:00
mpl a34f9e2669 publisher app
http://camlistore.org/issue/365

Change-Id: I281fdcbbe6a2bdf15607e75a21bc93b453f82c85
2014-06-15 00:03:03 +02:00
mpl 9723b5c814 apps: more generic app handler work, to prepare for publisher
http://camlistore.org/issue/365

Change-Id: I3c34ba1f09debc3aaaf68ad9fb11b595470b6b5d
2014-06-14 18:36:10 +02:00
mpl 4569a7f475 devcam review: submit changes for review
Context: http://camlistore.org/issue/408

Change-Id: I587de65c9dc8d87c02541096e1f4f7765e644962
2014-05-13 23:06:30 +02:00
Nick O'Neill 2d70c95aef Tweet importing with some basic info
Change-Id: I73455eef50dcdc495ddc042d266b3a4a8020c877
2014-03-20 18:57:43 -07:00
mpl 05058091e9 kvutil: add VerifyDb*, use kvutil in sorted
http://camlistore.org/issue/393

Change-Id: If9482aa05f2346644196e3218dcde571e4d1a1a1
2014-03-20 01:21:27 +01:00
mpl 4ac7601315 sorted/kv: enable VerifyDb* options with env var
http://camlistore.org/issue/393

Change-Id: I97bbe3d4ab6eb186c3960db0995cedeec3a34ddf
2014-03-20 00:04:18 +01:00
Tamás Gulácsi 65fa5d6d3e Picasa: add importer.
Imports all images from all albums, and puts each image in its album.
Each album is linked under "Picasa (<username>)".
Tries to skip already imported files (check based on album/filename).
See Issue #391.

Change-Id: If00751671a429891c05bbe41f59c1421b2ff2de8
2014-03-17 10:52:51 +01:00
Andrew Gerrand d136d91000 add devcam server -fullindex flag
Change-Id: I5ef7968d122c9a64665ab70a5f9571f6e3147836
2014-02-11 10:37:41 +11:00
Brad Fitzpatrick cc4843ed22 devcam: mount: ditch xterm flag
Change-Id: Id0b003698aa28c3c506df8031597c4bd19c034f8
2014-01-31 12:00:42 +01:00
Brad Fitzpatrick 77dd30fbd1 devcam: mount: extra args go before mountpoint
Change-Id: Ibe0686c6e2c847c16b916133edbbb56915ff8ae6
2014-01-31 11:58:42 +01:00
Brad Fitzpatrick 99bc9f7664 devcam: let test command take arguments, to run subset of tests 2014-01-23 13:57:13 -08:00
Brad Fitzpatrick a384ff188d Get rid of CAMLI_DEV_KEYBLOBS / keyblobs / etc.
Fixes camlistore.org/issue/277

Change-Id: I7f380f9d18785f600fa0c442d5a19cd118782788
2014-01-20 13:47:08 -08:00
Brad Fitzpatrick d1f2ae227f Start of Foursquare importer.
Change-Id: I56b0e71a3aad697b82710ebea6fc941436a3c119
2014-01-14 20:45:23 -08:00
Brad Fitzpatrick 48e67cab16 devcam: add --publish=false and --mini option to start a lite server with no blobs
Change-Id: Idcde77f72126800d610c632940552f04c396634d
2014-01-13 20:11:43 -08:00
Aaron Boodman 63ec18c3f3 Make the devcam server search handler honor the secring and identity flags.
Change-Id: I7bfbbc82f9ed429b2fa61c84e8c0d514641444fe
2013-12-28 16:24:28 -08:00
Aaron Boodman dc562622a7 Enable -secretring and -keyid for devcam server. Also fix bug in devcam server -wipe.
Change-Id: Ic615935c6ea9aa150a23aee0a18361b498574d76
2013-12-28 14:35:16 -08:00
Aaron Boodman 246b2b9ca8 Allow devcam to use a persistent directory rather than a temp dir.
Change-Id: I234ba75e36dd55f339f679e45ba4c1ac3159f1f0
2013-12-27 15:25:08 -08:00
Bill Thiede bab46f1db5 devcam: option to add args to camlistored exec.
Handy catch-all for camlistored args we haven't plummed through to
devcam server in the proper manner.  Example use:

	$ devcam server -extraargs=-reindex

Change-Id: I4af922e4c95728dc7858fa2db9e60c7efd37a3cb
2013-12-24 21:35:48 -08:00
Brad Fitzpatrick bda8ad3c4b devcam: fix slash-vs-backslash bug on Windows
Change-Id: Id9c952d406db36319919ddbfd1695b5405277bde
2013-12-24 18:44:51 -08:00
mpl 6d19d64b88 devcam server: properly wipe mongo
Change-Id: Ib75a460a4be84e21affc662d45b3cf1e321b15bd
2013-12-13 17:37:19 +01:00
Brad Fitzpatrick 9f8e9e57fd Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-12-13 11:15:29 +04:00
mpl fcbbf2a4df index: move mongo implementation to sorted
Also devcam server -wipe wasn't wiping for mongo anymore, now fixed.

Change-Id: Iecc9d8025ddfba8d8ae9417ee170baf02be1d52f
2013-12-13 00:25:40 +01:00
Brad Fitzpatrick 5eeef19472 Make CAMLI_FAST_DEV work again.
Change-Id: I294de23a50d48c7c2325780b50141a7f02d55aab
2013-12-12 16:18:24 +04:00
Brad Fitzpatrick 2d2a0c1479 devcam: add -race flag
Change-Id: I1d477431b917faba3d2ee99ef5241dda14e748f3
2013-12-11 22:13:06 +04:00
Brad Fitzpatrick bf54591e8d Remove the memIndex option from genconfig and devcam; default memoryIndex to on.
We had the legacy "memIndex" option to put the Camlistore index in
memory (the leveldb memdb) as an option in genconfig called
"memIndex", and we also had an option called "memoryIndex" for whether
to slurp the on-disk index to memory on start-up.  Too confusing!

Instead, delete "memIndex" (since it's not the default anyway, now
that we have kv).

Then, also (the original point of this change): default the
memoryIndex option to true, so search works for people by default.
This option might go away in the future if it becomes the only required
way.

Also, document this.

Change-Id: Iddffa6e19adbf09c5aacd063aa44de362d90633b
2013-12-11 12:20:22 +04:00
Brad Fitzpatrick efd51a08ed devcam: print out temp dir on start-up
Change-Id: I7f84ffdecc5cb63baa931d5b5de8aca564970a67
2013-11-24 17:11:06 -08:00
Aaron Boodman 93b0bbbee5 Flickr importer: store credentials in Camlistore, not weird JSON file.
Change-Id: I07ed4e9043c9fa448425144886dc3b7047c4e85b
2013-11-22 09:50:03 -08:00
Aaron Boodman 2a747ccd56 Add ability to specify flickr config for devcam.
Change-Id: Ic1139730d969558e2d6375dd86c6741e3dfd2b9b
2013-11-18 21:16:09 -08:00
mpl fdaf030bc0 index: add deletedAt method, and cache for deletion status
This method will be used to find out when a permanode or claim
was (un)deleted for the last time.

This change also introduces the keyDeletes index entry which
allows to find the permanodes/claims that a delete claim deletes.

Finally, since some index keys have changed, this CL introduces
a check for an index schema version, to force users to reindex
when needed.

http://camlistore.org/issue/191

Change-Id: I6f8e5d3030a6ddc8b21ff5dc5db311f2ea16f613
2013-11-06 23:34:13 +01:00
mpl 8ca1378869 devcam: reintroduce --sqlite when building server
Change-Id: I2f0fadc380803237057a0dd774070ff79a980a98
2013-10-25 01:00:43 +02:00
Salman Aljammaz a3686d6ec0 devcam: Add mount command, also factor out build() and vebose flags
Change-Id: Ia501cad493d1707c63ebea9640a4d8fa6355da4c
2013-10-23 17:49:09 +01:00
Brad Fitzpatrick 18de812d8b Merge "devcam: fix up package doc strings" 2013-10-22 15:38:46 +00:00
Salman Aljammaz bfd1a65724 devcam: fix up package doc strings
Change-Id: I5f1b85e872a2d66d3c68babd2a2b53b76fb23dc3
2013-10-22 16:36:57 +01:00
Salman Aljammaz b0a8dee500 Add camtool to devcam.
Change-Id: Ife10222ca8f1ba08be923f9433781910376ddf05
2013-10-22 16:24:28 +01:00
mpl b4a5a36457 devcam put/get: read CAMLI_QUIET from current env
reverts 0015237713

Change-Id: Ie8ec46c1971a014cafe05d160d2f65a84d8cd466
2013-09-23 23:21:48 +02:00
mpl 0015237713 devcam put/get: do not use os.Getenv
Change-Id: I2e5b247835ec3febe7f312cabcfbf73f58bf43dd
2013-09-23 12:21:39 +02:00
mpl 70a167e922 devcam server: do not use os.Getenv
Since 0d854e106c, we do not
use the env directly, but a copy of it, so os.Getenv does
not work to retrieve vars that were set in that copy.

Change-Id: Ibc6ab40ea2ee51e1dea6a5e9ceac782304e0039a
2013-09-23 12:07:04 +02:00
Brad Fitzpatrick 0d854e106c devcam: don't use current environment as scratchpad for future process's environment
Remove calls to os.Setenv.

Change-Id: I9a0c611eb534b278459ddf724b8e7d181f2395ef
2013-09-22 20:13:28 +01:00
mpl 0c1dc0d6e4 devcam test: generate keyblob with devcam put init
The integration tests run by devcam test require a
GPG keyblob in the default location (for camput), and
we do not track in git such a keyblob.
This change allows devcam test to create the keyblob
with devcam put init before running the tests.

Also, devcam server is now using CAMLI_KEYID instead
of hardcoding the key id in the dev server config file.

Change-Id: I030b88c9a9321016fedb62424e0e707621206922
2013-09-20 20:03:21 +02:00
mpl 6210cc8d56 devcam server: sqlite support.
Also makes kv use CAMLI_DBNAME for the index file instead of
CAMLI_KVINDEX_FILE; one less env var to care about.

http://camlistore.org/issue/225

Change-Id: I954cc2e5ff8ed1bb213435af18f1ecc2948c4e59
2013-09-20 14:04:03 +02:00
mpl ba8f88c21a devcam test, replaces make presubmit
http://camlistore.org/issue/223

Change-Id: Iaaea9926a98d7d3335fd6ee52daa225e51ec4703
2013-09-19 17:00:59 +02:00
Brad Fitzpatrick 1decd09846 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-09-11 07:46:58 +01:00
Brad Fitzpatrick 6fca50a9c7 devcam: add -nobuild flag to put.
But really this should be a global option, and/or environment controlled,
like dev/demo.sh set up previously.

Change-Id: Iabf0c02e4a44ed574088e2938f3b4f53cabf3a43
2013-09-10 20:02:15 -07:00
mpl 50c07cf7be devcam: do not depend on GOPATH, rebuild camget and camput
http://camlistore.org/issue/218

Change-Id: I4f48fac38cad095cd6d4c4d5aea4efeb35fff306
2013-09-10 21:15:12 +02:00
Brad Fitzpatrick ff4e491b3b devcam: fix breakage caused from envvar strconv changes.
Change-Id: Id05d00d9b82924242905ccbc7a7e8002dc4859a7
2013-09-08 12:32:13 -07:00
Bill Thiede 630f6ca61f Document environment variables usage.
Running 'go run dev/envvardoc/envvardoc.go' now shows:
'All environment variables are documented'

I also took the liberty of cleaning-up our mishmash of logic for handling
boolean environment variables, and cleaned up a couple other spots that didn't
seem right.

This change adds docmentation for all variables starting with (CAM|DEV|AWS).
This leaves some variables still undocumented.  If there are variables worth
documenting in the following list, maybe we should rename them to have a
CAM{LI} prefix for consistency's sake:

APPDATA                  pkg/osutil/paths.go:86
APPDATA                  pkg/osutil/paths.go:102
DISPLAY                  pkg/misc/gpgagent/gpgagent.go:126
GOPATH                   pkg/fileembed/genfileembed/genfileembed.go:321
GOPATH                   pkg/osutil/paths.go:168
GOPATH                   pkg/test/world.go:54
GOPATH                   server/appengine/build_test.go:77
GPGKEY                   cmd/camput/init.go:77
GPG_AGENT_INFO           cmd/camput/init.go:153
GPG_AGENT_INFO           pkg/misc/gpgagent/gpgagent.go:50
HOME                     pkg/jsonsign/keys.go:79
HOME                     pkg/jsonsign/signhandler/sig.go:64
HOME                     pkg/osutil/paths.go:36
HOMEPATH                 pkg/osutil/paths.go:34
PKG_CONFIG_PATH          pkg/index/sqlite/dbschema.go:59
RUN_BROKEN_TESTS         pkg/fs/fs_test.go:67
SKIP_DEP_TESTS           pkg/test/testdep.go:29
TERM                     pkg/misc/gpgagent/gpgagent.go:133
TERM                     pkg/misc/pinentry/pinentry.go:99
TESTING_PORT_WRITE_FD    pkg/webserver/webserver.go:135
TEST_GPGAGENT_LIB        pkg/misc/gpgagent/gpgagent_test.go:27
USER                     pkg/netutil/ident.go:135
USER                     pkg/osutil/paths.go:45
USERNAME                 pkg/jsonconfig/eval.go:228
USERNAME                 pkg/osutil/paths.go:43
VERBOSE_FUSE             pkg/fs/fs_test.go:133
VERBOSE_FUSE_STDERR      pkg/fs/fs_test.go:137
XDG_CONFIG_HOME          pkg/osutil/paths.go:104

Change-Id: Ief28710d3deefd1e65247cb5d3b1d8dde73e1f2d
2013-09-06 21:54:03 -07:00
Brad Fitzpatrick 4aeb310a4c devcam: fix build
Change-Id: If5f8d6e0e4e315cd8ca01aa7bf9b0f59407c52fe
2013-09-02 08:56:08 -07:00