Commit Graph

87 Commits

Author SHA1 Message Date
Brad Fitzpatrick a384ff188d Get rid of CAMLI_DEV_KEYBLOBS / keyblobs / etc.
Fixes camlistore.org/issue/277

Change-Id: I7f380f9d18785f600fa0c442d5a19cd118782788
2014-01-20 13:47:08 -08:00
Brad Fitzpatrick d1f2ae227f Start of Foursquare importer.
Change-Id: I56b0e71a3aad697b82710ebea6fc941436a3c119
2014-01-14 20:45:23 -08:00
Brad Fitzpatrick 48e67cab16 devcam: add --publish=false and --mini option to start a lite server with no blobs
Change-Id: Idcde77f72126800d610c632940552f04c396634d
2014-01-13 20:11:43 -08:00
mpl bf8c463d0a pkg/client: multi servers config
http://camlistore.org/issue/309

Change-Id: I22bbbf6a808a772272f153b2535d693bd986d13a
2014-01-05 23:19:43 -08:00
Aaron Boodman 63ec18c3f3 Make the devcam server search handler honor the secring and identity flags.
Change-Id: I7bfbbc82f9ed429b2fa61c84e8c0d514641444fe
2013-12-28 16:24:28 -08:00
mpl 6e7eacea4a Merge "pkg/client: some cleanup, doc on the website" 2013-12-23 22:31:45 +00:00
mpl 1e8d46ee45 pkg/client: some cleanup, doc on the website
Changed some keys to be consistent with the server config, and removed
some others.

We're now using jsonconfig to get the keys and validate the config.

Removed some unused funcs/methods, moved some global vars as members of a client.

Also, having two possible defaults for the secret keyring
(.config/camlistore/identity-secring.gpg and .gnupg/secring.gpg)
was annoying and confusing imho, so I removed the latter.

http://camlistore.org/issue/259

Change-Id: Icf79fa8822b62b5d91843698861dd19cb553607a
2013-12-23 20:23:21 +01:00
Brad Fitzpatrick 5c5666d037 Thumbnail cleanups and let the meta map persist on disk with a sorted.KeyValue
Previously, every time you restarted the server, it forgot about all
thumbnails previously generated.  (in practice it didn't/doesn't
matter with a single user, though, since they're still cached in the
browser and we always reply to If-Modified-Since immediately without
checking the cache)  But it'll matter more with the Publish handler.

Also, rename some stuff, clean up some stuff, drop an unused interface.

And then necessarily change the serverconfig low-level generator to use
a kvfile for the thumbmeta map when using local disk for blobs.

--

Change-Id: I4dcfcb21429a440aa118794c03f7abf7bd69c33b
2013-12-14 09:37:56 -08:00
Brad Fitzpatrick bf54591e8d Remove the memIndex option from genconfig and devcam; default memoryIndex to on.
We had the legacy "memIndex" option to put the Camlistore index in
memory (the leveldb memdb) as an option in genconfig called
"memIndex", and we also had an option called "memoryIndex" for whether
to slurp the on-disk index to memory on start-up.  Too confusing!

Instead, delete "memIndex" (since it's not the default anyway, now
that we have kv).

Then, also (the original point of this change): default the
memoryIndex option to true, so search works for people by default.
This option might go away in the future if it becomes the only required
way.

Also, document this.

Change-Id: Iddffa6e19adbf09c5aacd063aa44de362d90633b
2013-12-11 12:20:22 +04:00
Brad Fitzpatrick f544114844 Fix diskpacked regression, add tests, start of storagetest.
Regressed from rev cb6f423e. Eventually pkg storagetest should test all methods of blobserver.Storage
for all storage target types.

Change-Id: I2c1c93b76fd9280a3eb429b1d71c64a693ed1ace
2013-11-30 13:06:04 -08:00
Brad Fitzpatrick 9261c9b303 dev-server-config: no longer need to replicate to a sync. the sync subscribes to its source.
Change-Id: I7734f47f039cbdd40fcb672ba60d6a2488b42bdc
2013-11-24 17:11:40 -08:00
Brad Fitzpatrick 62fbbb61a0 encrypt: use key metaIndex for index, not meta; that was used. fixes devcam server.
Change-Id: I483c696dcc3402537bec136f8847c17767760b3b
2013-11-23 11:11:05 -08:00
Brad Fitzpatrick cfd02e6c89 Fix dev server config's root storage's fetch/stat.
Don't want to read from the indexer or a async target.

Change-Id: I72831a6ce3fba6e005c71c3e8ac73557cd268c34
2013-11-22 22:50:40 -08:00
Brad Fitzpatrick 70475701d1 Get rid of QueueCreator and all its associated complexity.
Previous TODO entry was:

-- Get rid of QueueCreator entirely. Plan:
     -- sync handler still has a source and dest (one pair) but
        instead of calling CreateQueue on the source, it instead
        has an index.Storage (configured via a RequiredObject
        so it can be a kvfile, leveldb, mysql, postgres etc)
     -- make all the index.Storage types be instantiable
        from a jsonconfig Object, perhaps with constructors keyed
        on a "type" field.
     -- make sync handler support blobserver.Receiver (or StatReceiver)
        like indexes, so it can receive blobs.  but all it needs to
        do to acknowledge the ReceiveBlob is write and flush to its
        index.Storage. the syncing is async by default. (otherwise callers
        could just use "replica" if they wanted sync replication).
        But maybe for ease of configuration switching, we could also
        support a sync mode.  when it needs to replicate a blob,
        it uses the source.
     -- future option: sync mirror to an alternate path on ReceiveBlob
        that can delete. e.g. you're uploading to s3 and google,
        but don't want to upload to both at once, so you use the localdisk
        as a buffer to spread out your upstream bandwidth.
     -- end result: no more hardlinks or queue creator.

Change-Id: I6244fc4f3a655f08470ae3160502659399f468ed
2013-11-22 14:33:31 -08:00
Aaron Boodman 93b0bbbee5 Flickr importer: store credentials in Camlistore, not weird JSON file.
Change-Id: I07ed4e9043c9fa448425144886dc3b7047c4e85b
2013-11-22 09:50:03 -08:00
mpl f8b41e236e Merge "server/publish.go: use go html template for publishing" 2013-11-19 22:22:17 +00:00
mpl a6aeff034c server/publish.go: use go html template for publishing
Change-Id: Ic1c55d35e52e407864d339927b453de58a64e03c
2013-11-19 17:50:26 +01:00
Aaron Boodman 2a747ccd56 Add ability to specify flickr config for devcam.
Change-Id: Ic1139730d969558e2d6375dd86c6741e3dfd2b9b
2013-11-18 21:16:09 -08:00
Aaron Boodman 9c671f76eb Revert "Revert "Add genconfig support for flickr importer""
This reverts commit 8367de6e95.

Change-Id: I1fdfa4070572b1badc0f0980348404f905fa13a3
2013-11-17 14:15:12 -08:00
Brad Fitzpatrick b33a785e62 Merge "Rearrange the Flickr importer to use OAuth instead of Flickr's custom importer." 2013-11-17 06:39:24 +00:00
Aaron Boodman 9ae23e52b0 Rearrange the Flickr importer to use OAuth instead of Flickr's custom importer.
Change-Id: I8284431fa6572a08d3c7ff0d0e655297627a4eda
2013-11-16 22:30:21 -08:00
Brad Fitzpatrick c573feb226 Make the search handler conditionally request the index be slurped into memory.
Change-Id: I7fadeba41c3b1bbcfbb0fd3470ff30cdf04b1b90
2013-11-16 15:18:16 -08:00
Aaron Boodman e4f25a2444 First little bit of Flickr importer.
Change-Id: Ia42b07a81b01904f5722d06653c4bf9de2e81c64
2013-10-20 11:18:53 -07:00
Brad Fitzpatrick cf116c6da9 importer: work on infrastructure for third-party import from other sites
Change-Id: Ib4e5e4d543a1b2a913b99fa0a4d92f1a5b94c5dc
2013-10-19 15:48:05 -07:00
mpl 0c1dc0d6e4 devcam test: generate keyblob with devcam put init
The integration tests run by devcam test require a
GPG keyblob in the default location (for camput), and
we do not track in git such a keyblob.
This change allows devcam test to create the keyblob
with devcam put init before running the tests.

Also, devcam server is now using CAMLI_KEYID instead
of hardcoding the key id in the dev server config file.

Change-Id: I030b88c9a9321016fedb62424e0e707621206922
2013-09-20 20:03:21 +02:00
mpl 6210cc8d56 devcam server: sqlite support.
Also makes kv use CAMLI_DBNAME for the index file instead of
CAMLI_KVINDEX_FILE; one less env var to care about.

http://camlistore.org/issue/225

Change-Id: I954cc2e5ff8ed1bb213435af18f1ecc2948c4e59
2013-09-20 14:04:03 +02:00
mpl edd7f68799 serverconfig: add test for mongo indexer, fix dev mongo config
Change-Id: Ifb7f8cef208dc93a4cf350ff197bb2af2c0e0e81
2013-09-09 22:54:32 +02:00
mpl 3a26c433db devcam: 'put' and 'get' commands, replace dev-camput and dev-camget
Also added -port flag, and untied the port to the
password in devcam server, always 'pass3179' now.

http://camlistore.org/issue/212

Change-Id: Iadf693951763d47907be17c53d4807a4706eb150
2013-09-01 10:25:09 -07:00
Brad Fitzpatrick 500ba50b97 devcam: make kvfile the default dev-server indexer.
Memory index requires -memindex now.

Change-Id: I6df424e88c3783e397c528cd8633402a9edf3d95
2013-08-23 17:19:21 -05:00
mpl 31df3635a4 camput: ignore files like .DS_Store with -filenodes
Also made initTrustedCertsOnce a field of the Client
object, as it is a similar change.

http://camlistore.org/issue/104

Change-Id: Iabbd7f06e06d31265f390a23c4bdaac956f856f7
2013-08-19 17:21:10 +02:00
Brad Fitzpatrick 13fe0608fb Start of integration tests and a library to make them easy.
Change-Id: I24c55252d81d2170205f090a11a5c45473707e5d
2013-07-20 22:36:53 -07:00
Josh Huckabee bdac95b218 Fix dev server TLS config.
Change-Id: I630b7381d72ce4538de489db69e6d8300afb65ce
2013-06-25 07:44:49 -07:00
Brad Fitzpatrick 0cc0cadee9 dev-server: add a share handler
Change-Id: I211fe078d1bc32b5abb4d3a2c48b0a8b566236de
2013-06-23 13:50:02 -07:00
mpl da5056c0d1 dev-clients: rename config file to match xdg compliant changes
Change-Id: I51d6dffb38fa178db151da8c7d2bcc041f996413
2013-06-22 16:40:19 +02:00
Brad Fitzpatrick 10d000d900 ui: serve Closure from embedded zip data when available
Change-Id: I9bb6bb4f13f69b293fd98441d82068c0677ffbd5
2013-06-18 23:14:36 -07:00
Brad Fitzpatrick 3176c5be43 encrypt: require big ugly I_AGREE configuration in order to use it, so people don't get misled
Change-Id: I4259e423d6a4b733f353a1e8792c6b8cb3c95c43
2013-06-15 20:08:37 -07:00
Brad Fitzpatrick d12007af52 encrypt: enough of a shell to start up in dev-server, even if nothing works yet
Change-Id: I7e3c17d952a943afd4cfb57939f25bf181aaa6fd
2013-06-15 15:05:27 -07:00
Brad Fitzpatrick 5761b57d69 Work on implementing new static resource plan, as documented in the big TODO item.
Change-Id: I79d7665a24633630c39cceb5e1ab49cbb0300150
2013-06-12 10:10:24 -07:00
mpl 7059a616c0 show server status on index page
We're only showing build version for now.
Next step would be status handler.

See http://camlistore.org/issue/111

Change-Id: I9e0792abe9ffed8139d38478d186b6e75f99de7e
2013-06-09 18:46:24 +02:00
mpl 11a3f78691 WIP: switch from ui to newui
This CL proposes 4 changes:

1) When making the embeds, any .html files in newui is rewwritten
on the fly so that the resulting embed uses all.js instead of
individual .js files. all.js is the only javascript file embedded
for newui.

2) Various bugs and paths in the newui/*.js files were fixed so that
everything works when using all.js

3) pkg/server/ui.go was modified so that all features from ui should
now have an equivalent in newui, that is served through ui/new/.
In a next step, we can move back everything from newui/ into ui/ and
completely get rid of the old ui.

4) The published gallery now uses closure too.
In another next step, we should probably work on serving its static
part directly from an .html file or (include it in the zembed somehow?)
instead of generating it at every request. Or at least cache it?

This commit includes the zembeds in newui, except for zembed_all.js.go
since it is large and it would take a lot of space in the git history
everytime we change it.
The plan is to roll-sum it into chunks to address that problem.

Change-Id: I381d8ddf3a2bbe37e3c4b1c4bf423f16beffaaf4
2013-06-04 16:34:51 +02:00
Brad Fitzpatrick c7bbccf435 config: Rename blobServer config key and flag to just server. use auth from environment when an explicit server is given
Change-Id: I07cfe3df3b644e143fcb92456e56d238bbbdd52e
2013-01-25 16:23:31 -08:00
Brad Fitzpatrick abb0e307d6 Show owner name on index page
Change-Id: Ic845283da88eea57a43a13ea13b2993070856102
2012-12-11 18:41:23 -08:00
Brad Fitzpatrick db55ee77d9 camput: automatic blobroot path discovery 2012-11-07 22:23:45 -06:00
Brad Fitzpatrick ff60ec2785 Get rid of some old camput config cruft 2012-11-07 21:19:07 -06:00
Brad Fitzpatrick a41269e78e Reindex all dev-server blobs into memindex on restart.
Required some sync work (full syncs on start, blocking full syncs on
start, and also adding a dev-only hack to force a depedency from
search -> sync, to control the handler initialization order, otherwise
publish handlers would race with the sync handler and they'd create
new "blog" and "pics" permanodes and we'd end up with duplicates).
2012-11-07 22:40:17 +01:00
Brad Fitzpatrick bccf8ed575 Move discovery up from the UI handler to the Root handler. 2012-11-07 18:57:43 +01:00
mpl 295cbade1b Add postgres support
Also added the -short option for make presubmit
because postgres test is too slow.

Change-Id: I6be21c4d4cd67671eb44ab65333fc32c1e261bbd
2012-11-03 19:59:22 +01:00
Brad Fitzpatrick 73cd34a6a0 serverconfig & camlistored: simplify low/high-level config bootstrapping.
Change-Id: I22e003ff2eaa29b09a5c4cbb59afa1ca9bd624df
2012-03-19 13:09:00 -07:00
Brad Fitzpatrick f745487a49 In dev server config, use /bs-and-maybe-also-index/ to speed public permanode bootstrap blob uploads.
Change-Id: Ib32164ca9e3e958d34906077a1841800026d42ee
2012-03-03 12:54:20 -08:00
mpl a7b2879e43 mongodb indexer + dependencies (mgo, bson, gocheck).
Change-Id: I6a54ff99777b6283829f3dfcafb0295b6022d5a7
2011-12-31 23:36:16 +01:00