Commit Graph

50 Commits

Author SHA1 Message Date
Brad Fitzpatrick 36c1ecebd9 Let camtool sync's --dest and --thirdleg flags take local disk paths, instead of servers.
Change-Id: I8daddf06a7201384edff2cb22ecfd5872fc84928
2013-06-09 11:36:26 +02:00
mpl f21cea5131 share: refactored
-pkg/server/share.go handles share(d) blobs.
-pkg/blobserver/gethandler no longer deals with share blobs
and has been rewritten like the other handlers (no more auth
in there).
-client and signhandler changed accordingly.
-serverconfig: auth moved up in makeCamliHandler
-share parameter added to the user level and low level config.
-share URL is now http(s)://host[:port]/[prefix/]share/sha1-xxxx.

Addresses http://camlistore.org/issue/107

Change-Id: I0a2c07ac1a1e435b141702e0ff06dc8182721d65
2013-03-06 19:11:00 +01:00
Brad Fitzpatrick 83002f748c localdisk: work on Linux w/ non-Unixy filesystem
Change-Id: I97f062a44d536866c6c5cd690f2c2409a08cbdf1
2013-02-13 16:48:28 -08:00
Brad Fitzpatrick 4c7e985c14 uploadhelper: use a type for JSON response, not maps.
Change-Id: I642af4daf97829a813bf7dad61cc7683ec780e69
2013-02-11 20:33:53 -08:00
Brad Fitzpatrick 6d562c65d5 localdisk: minor cleanups, copy one paranoia check earlier
Change-Id: I81cc2bf054dae6fdc5428d8e44cb710b2aae6d88
2013-02-07 10:04:03 -08:00
Brad Fitzpatrick f1f670079b server.vivify: use the right claim date on all 3 relevant dates.
Fixes https://code.google.com/p/camlistore/issues/detail?id=93

Change-Id: I9d9ab02d47f2ef78f6f690414a04fc5c24f95fac
2013-01-31 17:52:11 -08:00
Brad Fitzpatrick bac9664784 cond: don't use Superset
Change-Id: I6cf975c68b6f8f39fd7432e3716e4e308660c54c
2013-01-22 09:42:13 -08:00
Brad Fitzpatrick 1bf01d7315 schema: lot of Blob/Builder work, and kill schema.Map.
Next up: unexport schema.Superset.

Change-Id: Ia4bb34790abba75bdb4d4101df7a4f15875f4237
2013-01-21 20:56:12 -08:00
Brad Fitzpatrick 5f6130b13b gethandler: use http.ServeContent; delete pre-Go 1 HTTP Range stuff
Change-Id: Icef01ea04aa7d38fc39c829618ff2856449eb326
2013-01-20 14:06:55 -08:00
Brad Fitzpatrick 7ceaaa0012 blobref: simplify the FromHash func. Make type implicit.
Change-Id: I2e01c3663bdb1151c11dfc9a1d59c7081940ffac
2013-01-20 13:36:27 -08:00
Brad Fitzpatrick 37bf028633 s3: verify configured bucket exists at start-up.
Change-Id: I43d0795641fe9dbc345a1f0916b828ede76cf4a6
2013-01-20 11:33:41 -08:00
mpl 7e448fe05e serverconfig: remove TODOs about FindHandlerByType since its behaviour now complies with the docs
Change-Id: I7ca754c0a27aa0ffc1bc1b4fe034a1c17f1876ef
2013-01-17 16:54:07 +01:00
mpl 306c6f9c19 blobserver: drop support for PUT as it was replaced with a POST upload action for uploading
Change-Id: Iffaea22064f057b93c33268546470d71efb02274
2013-01-17 16:05:11 +01:00
Pawel Szczur f1798f2c75 Removed redundant code.
Change-Id: I172091bd84fcc88561969882ff19396c963a582f
2013-01-12 01:48:54 +01:00
Brad Fitzpatrick 88e92a6cc5 Document blobserver.Loader methods, fix GetHandlerType to not force-load, diagnose loops better.
Change-Id: I03e989267e80ad610d3877c97fbf8adc8a88bb60
2013-01-11 12:42:30 -08:00
Brad Fitzpatrick d80cad8e2d auth stuff: clean up, allow scheme registration, integrate App Engine auth.
Change-Id: Idc7ac3782d3669a8f68255bd54d03f583814d5e2
2013-01-10 23:04:24 -08:00
mpl 77d044f4b7 schema: add FileTime to guess the file
creation time from potential file metadata.
camput: add the -exiftime option that allows
the use of the above for -filenodes and -vivify.

Fixes: http://code.google.com/p/camlistore/issues/detail?id=77

Change-Id: I05d1af3d875bb984e47e06775fd149e5ed0ed7b1
2013-01-10 17:43:05 +01:00
Brad Fitzpatrick c5df8cf7b6 get handler: still set content-type text/plain when it's a small all-utf-8 blob (for JSON metadata)
Change-Id: I8a55bb44199c015515ea3aa46f912d8d5a472a5c
2013-01-06 15:40:25 -08:00
Brad Fitzpatrick 906388d91a get handler: remove old sniffing for demos which no longer works and makes things much slower
Change-Id: I1d9e7368881894cba9a1febab6c5efcf765b0d59
2013-01-05 17:17:38 -08:00
mpl 12213c058e Changed auth to take into account not only the credentials,
but the requested operation/action too.

This allows to restrict vivify credentials to only upload
(as well as get and stat, because they're needed) to the
blobserver.

Change-Id: Idaed60d1f0d679cb9795ba9a11f094f964774335
2013-01-04 22:17:12 +01:00
Brad Fitzpatrick 1a8eee298a Misc cleanups and TODOs
Change-Id: I9903366427e41ebadba289bf66b294690675f78f
2012-12-31 17:36:28 -08:00
Brad Fitzpatrick e01b19fb32 upload handler: remove r60->go1 notes
Change-Id: I92ab4406890d2aebe957d168aa1c1032ab99f191
2012-12-31 17:21:00 -08:00
mpl ed20da7592 vivify: support in camput and upload handler.
no special credentials/auth yet.
pkg/blobserver/handlers/get.go moved to
pkg/blobserver/gethandler/get.go to avoid
a dependency loop with the json sign helper.
pkg/server/sig.go was moved to pkg/jsonsign/signhandler
because it seemed inapproriate to import in
pkg/blobserver/handlers/upload.go something from
pkg/server

Change-Id: Ifeb14512e182e8a101d4fced6d6d4184e2b9cb99
2012-12-30 23:39:22 +01:00
Brad Fitzpatrick 31094d825d blobref: make blobref.SeekerFromStreamingFetcher always succeed, falling back to buffering impl
Change-Id: I370782b5f9d122b5609a98887a5780b29a40c213
2012-12-25 10:19:33 -08:00
mpl a536e2a6d3 Merge "camput: Preparation for vivify." 2012-12-23 00:11:08 +00:00
mpl 6cb7ecd644 camput: Preparation for vivify.
Refactor writeFileMapRolling so we can write the chunks,
then the top level contents, separately.

Change-Id: Ib56cd4b2aa295516fafbe7072ad5b352d1aaaa89
2012-12-23 01:15:29 +01:00
Brad Fitzpatrick d0af61a04a Fix bugs in enumerate handler and s3's MaxEnumerateConfig.
Change-Id: Ib9cd52fe7d015a8c70416ec67ff6fd99a7a03d24
2012-12-22 13:50:55 -08:00
Brad Fitzpatrick d2b20395e9 localdisk: Ignore 'cache' directory when enumeration.
Change-Id: Iebc3d71b13f21e6f9a80c8bbfccced5fe6bcfb1d
2012-12-21 17:48:07 -08:00
Brad Fitzpatrick a41269e78e Reindex all dev-server blobs into memindex on restart.
Required some sync work (full syncs on start, blocking full syncs on
start, and also adding a dev-only hack to force a depedency from
search -> sync, to control the handler initialization order, otherwise
publish handlers would race with the sync handler and they'd create
new "blog" and "pics" permanodes and we'd end up with duplicates).
2012-11-07 22:40:17 +01:00
Brad Fitzpatrick 71d7cf9e88 Implement storage generations, init and discovery 2012-11-07 21:14:56 +01:00
Brad Fitzpatrick a6d3151645 gofmt handlers 2012-11-07 20:54:55 +01:00
Brad Fitzpatrick b887087606 Do a TODO: s/errorIsNoEnt/os.IsNotExist/g 2012-11-07 20:02:34 +01:00
Maxime Lavigne 5a7510c384 Modification to allow windows build and deployement
Webserver were initialized with "tcp" and ":3179" by default and
listenURL assumed that it would be treated as IPv6 and replaced [::]
by localhost. Host that were listening on IPv4 0.0.0.0 didn't get
the modification.

Receive in localdisk were using link that failed on windows plateforms.

Camlistored didn't use Json Marshaling which caused problem with the
way Windows stores its paths.

Change-Id: I9f62f7d46399c3514707383efcb2752dbaf1f420
2012-10-30 23:17:58 -04:00
mpl 844551d0b2 If baseURL was not user defined, this change makes sure it is
dynamically defined from incoming requests.
The listen flag has moved from pkg/webserver to server/camlistored
and is not using runsit anymore.
httputil.ServerError now only outputs the full error if we're
in internal/debug mode.
http://code.google.com/p/camlistore/issues/detail?id=53

Change-Id: I55a1c6d43a549d9ee2022742a83aa142dc19e02a
2012-10-15 16:26:50 +02:00
Brad Fitzpatrick 8c293e34b6 Per Go initialism style, rename Json to JSON 2012-07-28 15:42:56 -07:00
mpl 326d62aea4 camput: actually catch errors when trying to upload,
also added startupcheck for remote blobserver

Change-Id: Id9f5383c6fd3fdf697f2b75052f3e63b0972205d
2012-07-23 11:05:57 +02:00
Brad Fitzpatrick 50fe7a2fda Make UI discovery have absolute paths, so root discovery using UI's discovery handler works.
Change-Id: Ieb1b5d143c528b4b15875aa810038c3ebdf30251
2012-05-13 13:02:20 -07:00
Brad Fitzpatrick d29cbb5e35 rename FindHandlerByTypeIfLoaded to be shorter. document better.
Change-Id: I20b1a191d2d67972ba1f7a7fd3678d4ab4adae55
2012-05-14 03:48:51 +10:00
Brad Fitzpatrick 610e708924 fix more references to lib/go/camli
Change-Id: Ic35c67b0255d713185e8c77fdeb91c17c79a11c7
2012-05-14 01:07:13 +10:00
Brad Fitzpatrick 7b6a26ffeb localdisk: fix TODO which resulted in a race/crash.
Change-Id: I7ff0be046c07e5fd33c861afd10b5e725066a033
2012-04-22 18:51:41 -07:00
Brad Fitzpatrick 1d3703f7ef Clean up some logging.
Change-Id: I92ff6e68e9866784e643682c5e6db5d03f877c5b
2012-04-22 17:56:52 -07:00
Brad Fitzpatrick ebc77dabd2 cond: implement more efficient JSON sniffing.
Change-Id: I2c1e3f70832dc38a1103fbbfb3a4d8efff8f51bf
2012-04-22 17:50:35 -07:00
Brad Fitzpatrick 9d57c52c34 localdisk: attempt #3 at fixing queue directory delete/create race.
this version seems to work for quite awhile now, at least.

Change-Id: I1484838a9b1b2b12a8af72e57e6c7ba6bd06e07d
2012-04-22 17:17:51 -07:00
Brad Fitzpatrick b5bccf2261 localdisk: fix double slash in full path, preventing race lock from working
Change-Id: I171099214b06ec99d0ad766d1604442879fb3183
2012-04-21 08:29:32 -07:00
Brad Fitzpatrick 0e5d27a05c localdisk: fix race deleting empty directories w/ uploading new blobs
Change-Id: Icae0b7dd3de051bba23d1a583b47f5421adedb47
2012-04-20 13:04:42 -07:00
Brad Fitzpatrick 67e32df2bd Fix a bunch of broken tests, lost in the Go 1 conversion.
Change-Id: I7c16563bc25f52868984c63e42125cd9bd9d5812
2012-03-19 18:44:50 -07:00
Brad Fitzpatrick 35e0f4805f cond: use new bytes.NewReader instead of bytes.NewBuffer
Change-Id: If55c959983b5a894fc838d3d5aa2dd45e1e86892
2012-03-03 12:43:37 -08:00
ranveer d1e1f2bcb1 Add path/filepath
Fixes broken build.

Change-Id: Ia691b76b9a4d3c92f310223166f53d65f1286ec1
2012-02-22 10:52:21 -02:00
mpl 28caffab1f remove empty blob dirs in queue. fixes issue 47.
Change-Id: If07576825123be910f040f0c5a0dc6988d55de08
2012-02-20 15:07:14 +01:00
Brad Fitzpatrick 0714a463c9 Update from r60 to [almost] Go 1.
A lot is still broken, but most stuff at least compiles now.

The directory tree has been rearranged now too.  Go libraries are now
under "pkg".  Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".

The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.

For now, put the camlistore root under $GOPATH/src.  Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.

Then you can:

$ go build ./server/camlistored

... etc

The build.pl script is currently disabled.  It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).

Many things are certainly broken.

Many things are disabled.  (MySQL, all indexing, etc).

Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.

The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.

Help wanted.

Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
2012-02-18 21:53:06 -08:00