Commit Graph

3400 Commits

Author SHA1 Message Date
Aaron Boodman 376fb91747 Merge "Flickr importer: store credentials in Camlistore, not weird JSON file." 2013-11-22 17:50:49 +00:00
Aaron Boodman 93b0bbbee5 Flickr importer: store credentials in Camlistore, not weird JSON file.
Change-Id: I07ed4e9043c9fa448425144886dc3b7047c4e85b
2013-11-22 09:50:03 -08:00
Brad Fitzpatrick 7bd24db271 Merge "website: more redirects to the issues tracker" 2013-11-22 17:22:50 +00:00
mpl 8f5d54361f serverconfig: fix tests because of previous publish config changes
Change-Id: I643b31be80dcc86b4cdb76bc8a9012dcb57e65ee
2013-11-22 18:14:01 +01:00
Mathieu Lonjaret f8348c5875 Merge "buildbot/master: add Basic Auth support." 2013-11-22 15:29:06 +00:00
mpl 14fdd3f066 website: more redirects to the issues tracker
http://camlistore.org/issue/243

Change-Id: I25424ae5e69f602e8a8bb8d69954f37f28b8f687
2013-11-22 15:58:57 +01:00
Bill Thiede 738783ebf4 buildbot/master: add Basic Auth support.
Moved BasicAuth parsing and localhost detection code from pkg/auth ->
pkg/httputil for use by buildbot master.

Added user config file for remote access.  The file's name is
"masterbot-config.json" and is located in osutil.CamliConfigDir(),
which on Unix will resolve to $XDG_CONFIG_HOME/camlistore/, if
XDG_CONFIG_HOME set, or ~/.config/camlistore/.  On Windows it will be
under %APPDATA%\Camlistore\.  The expected format is a json object
with usernames as the keys and sha1 sums of the password as the
values, i.e.:

    {
        "user1": "1234567890abcdef12341234567890abcdef1234",
        "user2": "1234abcdef12345678901234abcdef1234567890"
    }

This file is polled at a 1 minute interval and reparsed if the file's
modification time is more recent then the previous parse attempt.  It
is ok for the file to go missing, it will zero out the remote user
list. A malformed file will result in the master exiting.

New commandline flags, -tlsCertFile & -tlsKeyFile, added.  Specifying
both will enable TLS on the listener specified by -host.  The go
source contains generate_cert.go in crypto/tls that can be used to
generate self-signed cert.pem and key.pem for testing.

Added -skiptlscheck commandline option to builder.  This allows the
builder to report to https:// addresses with self-signed certs as we
don't currently have a way to specify the cert chains to be used
for TLS verification.  This is a stop-gap solution.

When launching a master that listens for secure connections, we
currently need tell the builders to skip certificate validation. Add
'-builderopts="-skiptlscheck"' to the master's commandline to skip
cerfication verification.

Change-Id: I0750b5c9fa8f4def67fc05a841087b50abded2f7
2013-11-21 21:22:12 -08:00
Brad Fitzpatrick 02eb1a6e97 fs: put all importer roots in the roots folder too 2013-11-21 17:51:34 -08:00
Brad Fitzpatrick ed889d38be dummy importer: upload a file and make a node. 2013-11-21 17:51:13 -08:00
Aaron Boodman 7c3a7bd30a Ugh. Make importer.Object.SetAttr() actually ignore duplicate values.
I'm going to spend some time writing unit tests for this next I think.

Change-Id: Id19f24882f33adea4060ff852a986626ea8de171
2013-11-20 23:57:11 -08:00
Brad Fitzpatrick c5037b76e6 Fix thumbnails, broken in cb31dd58d2
Change-Id: I82defd54f53d4f6c1a594185d5e9b7d3a4a92c07
2013-11-20 17:44:10 -08:00
Brad Fitzpatrick 838a59c56f Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-19 15:28:20 -08:00
Brad Fitzpatrick bbabdfc8a6 index: move Storage interface to its own file, start of constructor registration system
Change-Id: I49547ec8cd12e2f559876fa17063d7f16e73195d
2013-11-19 15:28:13 -08:00
mpl f8b41e236e Merge "server/publish.go: use go html template for publishing" 2013-11-19 22:22:17 +00:00
mpl 8350f15547 Merge "pkg/index: write relevant keys when receiving a delete claim" 2013-11-19 17:03:04 +00:00
mpl c81f3147f6 pkg/index: write relevant keys when receiving a delete claim
This change:

1) Checks if the incoming claim is a delete claim with the use
of GetBlobMeta.

2) write the keyDeleted and keyDeletes keys when it's a delete
claim, plus the usual keys when the target is a permanode.

Yet to be done in the next CLs:
1) update the index deletes cache upon reception of a delete claim
2) update most of the search functions so they use deletedAt properly
3) add new keys necessary for GetRecentPermanodes to give a fully
correct result.

I also made indextest.DumpIndex public because it turned to be useful
to debug within pkg/search/ as well.

http://camlistore.org/issue/191

Change-Id: I8d8b9d12a535b8b1de0018b4a0e359241f14d52a
2013-11-19 18:02:12 +01:00
mpl a6aeff034c server/publish.go: use go html template for publishing
Change-Id: Ic1c55d35e52e407864d339927b453de58a64e03c
2013-11-19 17:50:26 +01:00
Aaron Boodman 5e54d24464 Merge "Add ability to specify flickr config for devcam." 2013-11-19 05:18:26 +00:00
Aaron Boodman 2a747ccd56 Add ability to specify flickr config for devcam.
Change-Id: Ic1139730d969558e2d6375dd86c6741e3dfd2b9b
2013-11-18 21:16:09 -08:00
Brad Fitzpatrick 27b19787e0 TODO: new plan to kill QueueCreator
Change-Id: I098307063f7645889a39dedafc4d3f862a3f798c
2013-11-18 18:33:59 -08:00
Brad Fitzpatrick 531b4dc32b search: remove MIME from DescribedBlob; fix Handler tests.
Change-Id: I2d92155e3446325520d91379ac24becef49ca56f
2013-11-18 16:36:08 -08:00
Brad Fitzpatrick d0916a6802 Merge "Actually import the first 100 photos from Flickr." 2013-11-18 06:29:33 +00:00
Aaron Boodman e3d71c34d1 Actually import the first 100 photos from Flickr.
Currently this just imports the title, description, and image data.

Change-Id: I95de17cddb0ac18eec05f8ee288764b04cd7f406
2013-11-17 21:58:06 -08:00
Brad Fitzpatrick 48dba353e9 query: make all tests test in all three modes.
Change-Id: I945830ddb3bfe68a7223994b94d780b46503c95e
2013-11-17 20:20:27 -08:00
Brad Fitzpatrick cb31dd58d2 In-memory search: include FileInfo. More tests pass.
Change-Id: Ic33357b63d1d57dc174fe18360e920730016c903
2013-11-17 19:49:19 -08:00
Brad Fitzpatrick 84077b6d34 New package: strutil
Change-Id: I74ea770ed59ce9a5aa2fdea10b37c418fab50598
2013-11-17 19:31:12 -08:00
Brad Fitzpatrick e8603b1293 Put claims in memory too for in-memory search. Required index schema version bump.
Change-Id: I194d65476bddea111277cd0b1472c56b5527226b
2013-11-17 16:52:51 -08:00
Brad Fitzpatrick bd11fd5cb9 importer: some TODO notes
Change-Id: Ia92be507092722d5751f42ec3c21e18cfc4cb7f0
2013-11-17 15:00:46 -08:00
Brad Fitzpatrick 27fd9e71a7 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-17 15:00:21 -08:00
Brad Fitzpatrick 0b07171c0c index: change GetOwnerClaims interface for efficiency, rename to AppendClaims
Change-Id: Icec235961ed8a09256df87c5fe1455cbad81ae3f
2013-11-17 14:54:30 -08:00
Aaron Boodman de9e5103e7 Grab the userID from the flickr oauth handshake.
Remove the mutex from auth.go because I believe the
mutex in importer.go should guaranteeing only one
flickr importer is running at a time.

Change-Id: Ice971c6c041cf21fdb2bc93cb15f861dd1c8c1d7
2013-11-17 14:26:42 -08:00
Aaron Boodman 9c671f76eb Revert "Revert "Add genconfig support for flickr importer""
This reverts commit 8367de6e95.

Change-Id: I1fdfa4070572b1badc0f0980348404f905fa13a3
2013-11-17 14:15:12 -08:00
Brad Fitzpatrick 8abaddfa80 index: rearrange interface definition.
more stable stuff at the top. stuff that needs a re-think below.

Change-Id: Id1682b393718a2e9928088d38a4ddea477698e98
2013-11-17 13:44:49 -08:00
Brad Fitzpatrick 8367de6e95 Revert "Add genconfig support for flickr importer"
This reverts commit b53e79614d.

It broke the tests.
2013-11-17 13:43:06 -08:00
Brad Fitzpatrick 1d91f3a627 Merge "Add genconfig support for flickr importer" 2013-11-17 21:31:05 +00:00
Aaron Boodman b53e79614d Add genconfig support for flickr importer
Change-Id: I8be13f81fae15acfe87cce9d65318524efc3035d
2013-11-17 13:29:49 -08:00
Brad Fitzpatrick 04a116fed0 TODOs
Change-Id: I6a7866d889f7446bc89c25a19c57d0c8220bf99f
2013-11-17 11:01:09 -08:00
Brad Fitzpatrick d82f02db50 Search optimization: keep blobs arranged by camliType in memory; use when possible.
Change-Id: I8b3ebcf7dfece21d17074da32dc20e4d600e4c50
2013-11-17 10:52:37 -08:00
Brad Fitzpatrick 8e8ac9092a corpus: add a generation number for future query caching
Change-Id: I07dd61756537688f4f5e68bbcaf93bff75aaddc9
2013-11-17 10:05:40 -08:00
Brad Fitzpatrick 494c9f37e5 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2013-11-17 09:49:51 -08:00
Brad Fitzpatrick 3eb493599e in-memory search: better structure for keeping memory corpus and kv
index in sync, both at start-up and while running and receiving blobs.
They both use the same mechanism now.

Also adds KeyId to the index and Corpus, as the next step. Plenty more
row types remain...

Change-Id: Id79955ba25dc79d5fbd94b0e5248d33dcf71d97e
2013-11-17 09:41:45 -08:00
Brad Fitzpatrick 6fbe62f081 Merge "Bugfix: incorrect baseURL passed to InstallHandlers" 2013-11-17 16:28:07 +00:00
Aaron Boodman 3f7543853a Bugfix: incorrect baseURL passed to InstallHandlers
Change-Id: Iaf26a806358cd82ca09c48d83ca86155e27c289d
2013-11-17 00:41:01 -08:00
Aaron Boodman fde2936507 Remove some unnecessary code from dummy importer.
Change-Id: Ifd1cb70f6f5e11261353e8fe16d17c348ea95f12
2013-11-16 22:43:17 -08:00
Brad Fitzpatrick b33a785e62 Merge "Rearrange the Flickr importer to use OAuth instead of Flickr's custom importer." 2013-11-17 06:39:24 +00:00
Aaron Boodman 9ae23e52b0 Rearrange the Flickr importer to use OAuth instead of Flickr's custom importer.
Change-Id: I8284431fa6572a08d3c7ff0d0e655297627a4eda
2013-11-16 22:30:21 -08:00
Brad Fitzpatrick 5ce4fab0f9 Work around lack of cgo when cross-compiling to arm.
Change-Id: Ic230362b9f19af0c99f296f14330c970bf5d010b
2013-11-16 22:22:48 -08:00
Brad Fitzpatrick 272db6545a index: change GetBlobMIMEType to GetBlobMeta. mime types on a blob was wrong anyway.
And some more of in-memory search, which this makes easier / less garbage-y.

Change-Id: If1fa7dd44842244ae5e5440f84e92d66dc3321fb
2013-11-16 19:40:14 -08:00
Brad Fitzpatrick bbabc52723 test: add Diff function.
forgot to commit earlier.

Change-Id: I9986b6bd2266d455837bc20e47b43e4c971654d7
2013-11-16 19:06:30 -08:00
Brad Fitzpatrick f3cc3c7ed9 search: more in-memory search work. make tests verify Scan doesn't hit Storage.
also some string interning work.

Change-Id: I7864b56eb97318bce943afdca3b1212f4729a9a8
2013-11-16 18:50:01 -08:00