Commit Graph

99 Commits

Author SHA1 Message Date
Salmān Aljammāz 4e39af1086 Generate self-signed certs with the correct hostname.
This change makes gce/create.go generate a self-signed certificate
with the hostname from the -hostname argument and upload it to GCS
before creating a new camlistore instance.

It also makes camlistored use baseURL to figure out the hostname
when generating its self-signed certificate.

Change-Id: I64f85853dab34a7ce95e5d5997e58f2e5da43496
2014-11-13 15:53:40 +00:00
mpl bb49185463 serverinit: add support for memory-only indexer. Enable in devcam.
To avoid confusion, the previously named "memoryIndex" configuration
parameter, which actually meant to slurp the index into the memory
corpus, has been renamed to "copyIndexToMemory". "memoryIndex" is now
the parameter to enable a memory-only indexer.

http://camlistore.org/issue/416

Change-Id: I7026e5463fda3a1887a88e0b07d4e3431c60c8b7
2014-08-26 22:46:13 +02:00
mpl a34f9e2669 publisher app
http://camlistore.org/issue/365

Change-Id: I281fdcbbe6a2bdf15607e75a21bc93b453f82c85
2014-06-15 00:03:03 +02:00
mpl 173adebc19 Merge "apps: more generic app handler work, to prepare for publisher" 2014-06-10 21:29:01 +00:00
mpl 9723b5c814 apps: more generic app handler work, to prepare for publisher
http://camlistore.org/issue/365

Change-Id: I3c34ba1f09debc3aaaf68ad9fb11b595470b6b5d
2014-06-14 18:36:10 +02:00
Brad Fitzpatrick d5fb600ac2 Better demo setup
Change-Id: I61e55e3773e30332e6a7d3b426090fbe946a2462
2014-06-02 15:55:10 +09:00
mpl 21dda2b4ef app/hello: dummy server application (hello world)
Change-Id: I6690b9459325af5a76d1de679d56701eefdd195e
2014-05-08 22:15:01 +02:00
mpl 291320d451 pkg/index: cleanup, refactor sub pkgs
http://camlistore.org/issue/263

Change-Id: I319bb097f0ce30b2bd5271b5c3bbff92b8dcc318
2014-04-05 01:22:05 +02:00
Brad Fitzpatrick bf2a7b60a3 Importer overhaul. Only Foursquare is currently working.
The rest are currently stubbed out and will need updating
to the new APIs.

Change-Id: I9d70302b3ac1026192413bf9dcd3c8f1eb420349
2014-03-31 10:56:30 -07:00
Nick O'Neill 2d70c95aef Tweet importing with some basic info
Change-Id: I73455eef50dcdc495ddc042d266b3a4a8020c877
2014-03-20 18:57:43 -07:00
Tamás Gulácsi 65fa5d6d3e Picasa: add importer.
Imports all images from all albums, and puts each image in its album.
Each album is linked under "Picasa (<username>)".
Tries to skip already imported files (check based on album/filename).
See Issue #391.

Change-Id: If00751671a429891c05bbe41f59c1421b2ff2de8
2014-03-17 10:52:51 +01:00
mpl 6707837806 pkg/types/serverconfig: json-tagged struct
Also changed the TLS/HTTPS config keys for consistency.

http://camlistore.org/issue/339

Change-Id: I704ec129f91b93ebb20bc1191816166a2f10692d
2014-02-10 21:49:11 +01:00
Brad Fitzpatrick a384ff188d Get rid of CAMLI_DEV_KEYBLOBS / keyblobs / etc.
Fixes camlistore.org/issue/277

Change-Id: I7f380f9d18785f600fa0c442d5a19cd118782788
2014-01-20 13:47:08 -08:00
Brad Fitzpatrick d1f2ae227f Start of Foursquare importer.
Change-Id: I56b0e71a3aad697b82710ebea6fc941436a3c119
2014-01-14 20:45:23 -08:00
Brad Fitzpatrick 48e67cab16 devcam: add --publish=false and --mini option to start a lite server with no blobs
Change-Id: Idcde77f72126800d610c632940552f04c396634d
2014-01-13 20:11:43 -08:00
mpl bf8c463d0a pkg/client: multi servers config
http://camlistore.org/issue/309

Change-Id: I22bbbf6a808a772272f153b2535d693bd986d13a
2014-01-05 23:19:43 -08:00
Aaron Boodman 63ec18c3f3 Make the devcam server search handler honor the secring and identity flags.
Change-Id: I7bfbbc82f9ed429b2fa61c84e8c0d514641444fe
2013-12-28 16:24:28 -08:00
mpl 6e7eacea4a Merge "pkg/client: some cleanup, doc on the website" 2013-12-23 22:31:45 +00:00
mpl 1e8d46ee45 pkg/client: some cleanup, doc on the website
Changed some keys to be consistent with the server config, and removed
some others.

We're now using jsonconfig to get the keys and validate the config.

Removed some unused funcs/methods, moved some global vars as members of a client.

Also, having two possible defaults for the secret keyring
(.config/camlistore/identity-secring.gpg and .gnupg/secring.gpg)
was annoying and confusing imho, so I removed the latter.

http://camlistore.org/issue/259

Change-Id: Icf79fa8822b62b5d91843698861dd19cb553607a
2013-12-23 20:23:21 +01:00
Brad Fitzpatrick 5c5666d037 Thumbnail cleanups and let the meta map persist on disk with a sorted.KeyValue
Previously, every time you restarted the server, it forgot about all
thumbnails previously generated.  (in practice it didn't/doesn't
matter with a single user, though, since they're still cached in the
browser and we always reply to If-Modified-Since immediately without
checking the cache)  But it'll matter more with the Publish handler.

Also, rename some stuff, clean up some stuff, drop an unused interface.

And then necessarily change the serverconfig low-level generator to use
a kvfile for the thumbmeta map when using local disk for blobs.

--

Change-Id: I4dcfcb21429a440aa118794c03f7abf7bd69c33b
2013-12-14 09:37:56 -08:00
Brad Fitzpatrick bf54591e8d Remove the memIndex option from genconfig and devcam; default memoryIndex to on.
We had the legacy "memIndex" option to put the Camlistore index in
memory (the leveldb memdb) as an option in genconfig called
"memIndex", and we also had an option called "memoryIndex" for whether
to slurp the on-disk index to memory on start-up.  Too confusing!

Instead, delete "memIndex" (since it's not the default anyway, now
that we have kv).

Then, also (the original point of this change): default the
memoryIndex option to true, so search works for people by default.
This option might go away in the future if it becomes the only required
way.

Also, document this.

Change-Id: Iddffa6e19adbf09c5aacd063aa44de362d90633b
2013-12-11 12:20:22 +04:00
Brad Fitzpatrick f544114844 Fix diskpacked regression, add tests, start of storagetest.
Regressed from rev cb6f423e. Eventually pkg storagetest should test all methods of blobserver.Storage
for all storage target types.

Change-Id: I2c1c93b76fd9280a3eb429b1d71c64a693ed1ace
2013-11-30 13:06:04 -08:00
Brad Fitzpatrick 9261c9b303 dev-server-config: no longer need to replicate to a sync. the sync subscribes to its source.
Change-Id: I7734f47f039cbdd40fcb672ba60d6a2488b42bdc
2013-11-24 17:11:40 -08:00
Brad Fitzpatrick 62fbbb61a0 encrypt: use key metaIndex for index, not meta; that was used. fixes devcam server.
Change-Id: I483c696dcc3402537bec136f8847c17767760b3b
2013-11-23 11:11:05 -08:00
Brad Fitzpatrick cfd02e6c89 Fix dev server config's root storage's fetch/stat.
Don't want to read from the indexer or a async target.

Change-Id: I72831a6ce3fba6e005c71c3e8ac73557cd268c34
2013-11-22 22:50:40 -08:00
Brad Fitzpatrick 70475701d1 Get rid of QueueCreator and all its associated complexity.
Previous TODO entry was:

-- Get rid of QueueCreator entirely. Plan:
     -- sync handler still has a source and dest (one pair) but
        instead of calling CreateQueue on the source, it instead
        has an index.Storage (configured via a RequiredObject
        so it can be a kvfile, leveldb, mysql, postgres etc)
     -- make all the index.Storage types be instantiable
        from a jsonconfig Object, perhaps with constructors keyed
        on a "type" field.
     -- make sync handler support blobserver.Receiver (or StatReceiver)
        like indexes, so it can receive blobs.  but all it needs to
        do to acknowledge the ReceiveBlob is write and flush to its
        index.Storage. the syncing is async by default. (otherwise callers
        could just use "replica" if they wanted sync replication).
        But maybe for ease of configuration switching, we could also
        support a sync mode.  when it needs to replicate a blob,
        it uses the source.
     -- future option: sync mirror to an alternate path on ReceiveBlob
        that can delete. e.g. you're uploading to s3 and google,
        but don't want to upload to both at once, so you use the localdisk
        as a buffer to spread out your upstream bandwidth.
     -- end result: no more hardlinks or queue creator.

Change-Id: I6244fc4f3a655f08470ae3160502659399f468ed
2013-11-22 14:33:31 -08:00
Aaron Boodman 93b0bbbee5 Flickr importer: store credentials in Camlistore, not weird JSON file.
Change-Id: I07ed4e9043c9fa448425144886dc3b7047c4e85b
2013-11-22 09:50:03 -08:00
mpl f8b41e236e Merge "server/publish.go: use go html template for publishing" 2013-11-19 22:22:17 +00:00
mpl a6aeff034c server/publish.go: use go html template for publishing
Change-Id: Ic1c55d35e52e407864d339927b453de58a64e03c
2013-11-19 17:50:26 +01:00
Aaron Boodman 2a747ccd56 Add ability to specify flickr config for devcam.
Change-Id: Ic1139730d969558e2d6375dd86c6741e3dfd2b9b
2013-11-18 21:16:09 -08:00
Aaron Boodman 9c671f76eb Revert "Revert "Add genconfig support for flickr importer""
This reverts commit 8367de6e95.

Change-Id: I1fdfa4070572b1badc0f0980348404f905fa13a3
2013-11-17 14:15:12 -08:00
Brad Fitzpatrick b33a785e62 Merge "Rearrange the Flickr importer to use OAuth instead of Flickr's custom importer." 2013-11-17 06:39:24 +00:00
Aaron Boodman 9ae23e52b0 Rearrange the Flickr importer to use OAuth instead of Flickr's custom importer.
Change-Id: I8284431fa6572a08d3c7ff0d0e655297627a4eda
2013-11-16 22:30:21 -08:00
Brad Fitzpatrick c573feb226 Make the search handler conditionally request the index be slurped into memory.
Change-Id: I7fadeba41c3b1bbcfbb0fd3470ff30cdf04b1b90
2013-11-16 15:18:16 -08:00
Aaron Boodman e4f25a2444 First little bit of Flickr importer.
Change-Id: Ia42b07a81b01904f5722d06653c4bf9de2e81c64
2013-10-20 11:18:53 -07:00
Brad Fitzpatrick cf116c6da9 importer: work on infrastructure for third-party import from other sites
Change-Id: Ib4e5e4d543a1b2a913b99fa0a4d92f1a5b94c5dc
2013-10-19 15:48:05 -07:00
mpl 0c1dc0d6e4 devcam test: generate keyblob with devcam put init
The integration tests run by devcam test require a
GPG keyblob in the default location (for camput), and
we do not track in git such a keyblob.
This change allows devcam test to create the keyblob
with devcam put init before running the tests.

Also, devcam server is now using CAMLI_KEYID instead
of hardcoding the key id in the dev server config file.

Change-Id: I030b88c9a9321016fedb62424e0e707621206922
2013-09-20 20:03:21 +02:00
mpl 6210cc8d56 devcam server: sqlite support.
Also makes kv use CAMLI_DBNAME for the index file instead of
CAMLI_KVINDEX_FILE; one less env var to care about.

http://camlistore.org/issue/225

Change-Id: I954cc2e5ff8ed1bb213435af18f1ecc2948c4e59
2013-09-20 14:04:03 +02:00
mpl edd7f68799 serverconfig: add test for mongo indexer, fix dev mongo config
Change-Id: Ifb7f8cef208dc93a4cf350ff197bb2af2c0e0e81
2013-09-09 22:54:32 +02:00
mpl 3a26c433db devcam: 'put' and 'get' commands, replace dev-camput and dev-camget
Also added -port flag, and untied the port to the
password in devcam server, always 'pass3179' now.

http://camlistore.org/issue/212

Change-Id: Iadf693951763d47907be17c53d4807a4706eb150
2013-09-01 10:25:09 -07:00
Brad Fitzpatrick 500ba50b97 devcam: make kvfile the default dev-server indexer.
Memory index requires -memindex now.

Change-Id: I6df424e88c3783e397c528cd8633402a9edf3d95
2013-08-23 17:19:21 -05:00
mpl 31df3635a4 camput: ignore files like .DS_Store with -filenodes
Also made initTrustedCertsOnce a field of the Client
object, as it is a similar change.

http://camlistore.org/issue/104

Change-Id: Iabbd7f06e06d31265f390a23c4bdaac956f856f7
2013-08-19 17:21:10 +02:00
Brad Fitzpatrick 13fe0608fb Start of integration tests and a library to make them easy.
Change-Id: I24c55252d81d2170205f090a11a5c45473707e5d
2013-07-20 22:36:53 -07:00
Josh Huckabee bdac95b218 Fix dev server TLS config.
Change-Id: I630b7381d72ce4538de489db69e6d8300afb65ce
2013-06-25 07:44:49 -07:00
Brad Fitzpatrick 0cc0cadee9 dev-server: add a share handler
Change-Id: I211fe078d1bc32b5abb4d3a2c48b0a8b566236de
2013-06-23 13:50:02 -07:00
mpl da5056c0d1 dev-clients: rename config file to match xdg compliant changes
Change-Id: I51d6dffb38fa178db151da8c7d2bcc041f996413
2013-06-22 16:40:19 +02:00
Brad Fitzpatrick 10d000d900 ui: serve Closure from embedded zip data when available
Change-Id: I9bb6bb4f13f69b293fd98441d82068c0677ffbd5
2013-06-18 23:14:36 -07:00
Brad Fitzpatrick 3176c5be43 encrypt: require big ugly I_AGREE configuration in order to use it, so people don't get misled
Change-Id: I4259e423d6a4b733f353a1e8792c6b8cb3c95c43
2013-06-15 20:08:37 -07:00
Brad Fitzpatrick d12007af52 encrypt: enough of a shell to start up in dev-server, even if nothing works yet
Change-Id: I7e3c17d952a943afd4cfb57939f25bf181aaa6fd
2013-06-15 15:05:27 -07:00
Brad Fitzpatrick 5761b57d69 Work on implementing new static resource plan, as documented in the big TODO item.
Change-Id: I79d7665a24633630c39cceb5e1ab49cbb0300150
2013-06-12 10:10:24 -07:00