Commit Graph

37 Commits

Author SHA1 Message Date
Brad Fitzpatrick bf2f09cab3 index: reschedule indexing a claim blob if public key blob isn't yet available
Change-Id: Ie0174bf830eb4790080b2b5e7cdc4ea0af25406f
2014-04-02 13:39:36 -07:00
mpl cdc4ed5ae2 camput init: ring/key related fixes
1) Removed exec call to gpg, because it automatically looks in .gnupg/,
which we don't use anymore as a default.
2) Now taking into account global --secret-keyring flag.
This flag is now in osutil.
3) New or modified funcs in osutil
4) Made sure --gpgkey works too.
5) Cleaned up error messages and hints.

Context: http://camlistore.org/issue/364
         http://camlistore.org/issue/368

Change-Id: I2e51032ed0597da656db100d72f5588b37308e1a
2014-03-28 19:55:18 +01:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
Brad Fitzpatrick abcea5421c Merge "third_party,jsonsign: bump openpgp to 188:61c59dda830a" 2014-02-09 23:49:46 +00:00
mpl c4f3a5b215 third_party,jsonsign: bump openpgp to 188:61c59dda830a
http://camlistore.org/issue/370

Change-Id: I0066b1ad1455ff7960bda1f64ee953daa1195e1c
2014-02-09 01:04:36 +01:00
mpl 5a29733d3c jsonsign: clean up remnants of old secret ring default
http://camlistore.org/issue/368

Change-Id: I2708a97fdffd218f15ed9a1d482712ed7ed07197
2014-02-09 00:15:37 +01:00
Brad Fitzpatrick 2a3d84824a jsonsign: update old code to use fmt.Errorf
Change-Id: Iedd969e336dd02af6b3bc60590bfc7aa9ea9c89e
2014-01-28 21:52:36 -08:00
Brad Fitzpatrick 0f7b7951db jsonsign: don't upload public key until the first thing is signed
Change-Id: Ibe1f69bc82caae7abf98ff9fac0def944686378d
2014-01-13 20:11:14 -08:00
Brad Fitzpatrick 3f68f21f56 camerrors: rename MissingKeyBlob to ErrMissingKeyBlob
Change-Id: I15b7e9c1d82b8933f7969d50030ae02fa27deae7
2014-01-13 20:09:54 -08:00
mpl 1e8d46ee45 pkg/client: some cleanup, doc on the website
Changed some keys to be consistent with the server config, and removed
some others.

We're now using jsonconfig to get the keys and validate the config.

Removed some unused funcs/methods, moved some global vars as members of a client.

Also, having two possible defaults for the secret keyring
(.config/camlistore/identity-secring.gpg and .gnupg/secring.gpg)
was annoying and confusing imho, so I removed the latter.

http://camlistore.org/issue/259

Change-Id: Icf79fa8822b62b5d91843698861dd19cb553607a
2013-12-23 20:23:21 +01:00
Brad Fitzpatrick 705107ad80 search/index: invert depedency. search now depends on index.
creates new package types/camtypes for misc types needed by both. might eventually go away as
search matures.

Change-Id: Ib771ead7bea39936ba478b7e5d58de997060861b
2013-11-16 15:00:30 -08:00
Brad Fitzpatrick 0324143892 importer: make the schema.Signer available to the importer
Change-Id: Iad1a2f3bfe381b31a381957edb85d6d4aa21b38c
2013-10-24 22:45:28 -07:00
Brad Fitzpatrick 5c3b57682e schema: integrate with the jsonsign package.
Signing schema blobs (permanodes and claims) was too hard before, and
the code was scattered in weird places.

The pkg/jsonsign code tries pretty hard to be Camlistore-neutral, so
it remains so.

The pkg/client code (which deals mostly with HTTP access to a remote
camlistore server) had the signing the logic in it before, mostly
because that's where we originally needed it (for camput), but that
should be fixed in subsequent changes, to remove most of its signing
logic and use pkg/schema's instead.

The schema package now depends on jsonsign, which I think is an
acceptable dependency edge, since the Camlistore schema does logically
depend no signed claims (and that was a gaping hole in its public
API), and it seems safe cyclic-wise, especially if jsonsign continues
to be a leaf package.

Change-Id: I10229ec5d880ddd801cb1ba77ac26b654693036d
2013-10-21 22:08:45 -07:00
Brad Fitzpatrick d84df65cad jsonsign: add missing Close on public key blob reader
Change-Id: I1837fe65433e3a713939ae178e278f52884ab114
2013-10-21 20:39:08 -07:00
Brad Fitzpatrick 8319411ab4 Convert more ReceiveBlob into blobserver.Receive or blobserver.ReceiveNoHash
Change-Id: I9199555324b617167a6062a8b55ed09b449bae4f
2013-09-16 15:57:14 +01:00
Brad Fitzpatrick 128f56887b Handle GET and HEAD the same in more (all?) places.
Change-Id: I486a95f0a6fc127d17dcf0abeb94eab292980156
2013-09-15 20:12:26 +01:00
mpl e036f96488 sync: delay copy retry on specific errors
http://camlistore.org/issue/206

Change-Id: I1dd07149352e3af6b39bcb86ed2312f19c3bae30
2013-08-30 19:51:04 +02:00
mpl 99aaf3b55e client: take into account "selfPubKeyDir" from config when signing
http://camlistore.org/issue/210

Change-Id: Id6d75393a90bdf3141acc10fddc2b3cedee64422
2013-08-23 21:31:03 +02:00
Brad Fitzpatrick ae20a0f7de Ditch App Engine ContextWrapper noise; add test that App Engine always compiles.
I had broken App Engine in my previous commit, since it wasn't part of 'make presubmit'

Change-Id: I32c25efbc7c348ff516128c87b6e817ae0685386
2013-08-21 18:51:10 -05:00
mpl e682cda008 appengine: fix build issues
split pkg/jsonconfig/sign.go and pkg/server/root.go
because of unsupported stuff on appengine

Change-Id: I6f97ec2e043dabbe54971a88dc401a45ba36a60a
2013-08-11 19:42:57 +02:00
Brad Fitzpatrick 0bdf20884b all: delete pkg/blobref; convert all from *blobref.BlobRef to new blob.Ref
Change-Id: Id2dfb7f19452bedf4f3c9310b36227fd8117b225
2013-08-03 19:54:30 -07:00
Brad Fitzpatrick 9468e5ba70 More docs. Every package is documented now.
misc.CountingReader moves into readerutil.

pkg/atomics is folded into pkg/types.

pkg/test/testdep is folded into pkg/test, with better name/docs.

Old cruft from pkg/webserver is deleted.

Change-Id: I3f72d8b29804254ef944995fb085837c878f79f5
2013-07-07 21:12:30 -07:00
mpl 0de2881a16 client: help (wrt to the gpg key) initialize the config
Change-Id: I983e4396abacbc4d8fc354863cffeece65dd5b90
2013-06-22 01:16:11 +02:00
Brad Fitzpatrick ca58d8e2e0 Remove noisy var _ = log.Printf lines.
Change-Id: Ia58b8ef5f271f542ae4fe61c7fb1497322770322
2013-06-14 12:55:55 -07:00
Brad Fitzpatrick d832b6a06f Document PrefixHandler and clean up users with accessor functions.
Change-Id: Ib0b23ff3cc548a35c35eae1cdc59a8673dc19638
2013-06-12 11:17:30 +02:00
mpl db9fdb7e6e camput: reenable -secret-keyring, use flag.StringVar
http://camlistore.org/issue/76

Change-Id: I37229a1082e5d955cadcc63521ea1b13dd63a33c
2013-05-01 16:48:19 +02:00
mpl f21cea5131 share: refactored
-pkg/server/share.go handles share(d) blobs.
-pkg/blobserver/gethandler no longer deals with share blobs
and has been rewritten like the other handlers (no more auth
in there).
-client and signhandler changed accordingly.
-serverconfig: auth moved up in makeCamliHandler
-share parameter added to the user level and low level config.
-share URL is now http(s)://host[:port]/[prefix/]share/sha1-xxxx.

Addresses http://camlistore.org/issue/107

Change-Id: I0a2c07ac1a1e435b141702e0ff06dc8182721d65
2013-03-06 19:11:00 +01:00
mpl 83fa932f3c signhandler: inject the claimdate (if present) in the signrequest
This allows to set a signature time (instead of using time.Now()),
so that identical files will result in identical permanodes.

Change-Id: I315c06bd30e8ac5e4d36b0e3f758483c22a31063
2013-02-02 02:48:00 +01:00
Brad Fitzpatrick 1bf01d7315 schema: lot of Blob/Builder work, and kill schema.Map.
Next up: unexport schema.Superset.

Change-Id: Ia4bb34790abba75bdb4d4101df7a4f15875f4237
2013-01-21 20:56:12 -08:00
mpl ed20da7592 vivify: support in camput and upload handler.
no special credentials/auth yet.
pkg/blobserver/handlers/get.go moved to
pkg/blobserver/gethandler/get.go to avoid
a dependency loop with the json sign helper.
pkg/server/sig.go was moved to pkg/jsonsign/signhandler
because it seemed inapproriate to import in
pkg/blobserver/handlers/upload.go something from
pkg/server

Change-Id: Ifeb14512e182e8a101d4fced6d6d4184e2b9cb99
2012-12-30 23:39:22 +01:00
Brad Fitzpatrick 51e88cac65 Support for "planned permanodes" in pkg/jsonsign, pkg/schema and cmd/camput.
A planned permanode involves setting the contents of the permanode
(instead of a random string) as well as the OpenPGP signing time so
the resultant bytes of the blob (and thus its blobref) is deterministic.

This allows multiple independent devices (my laptops) to create the
same permanodes for the same files (photos backed up from my phone)
when offline (airplane) and then when they sync later, still only have
one permanode per unique file.  This means that tagging and other
metadata applied to permanodes on one laptop merge cleanly with
metadata from the other.
2012-07-28 16:32:31 -07:00
Brad Fitzpatrick 8c293e34b6 Per Go initialism style, rename Json to JSON 2012-07-28 15:42:56 -07:00
Brad Fitzpatrick 10cf23c700 Get the MySQL indexer working again.
It now uses the generic indexing layer, rather than its own schema.
Also, move it under pkg/index/, like mongo.
Also, link in mongo to the main binary.
Not using the tests yet (leaving for mpl :))

Change-Id: I47a2203ca6c0b5244cd458aedec46227c4363c62
2012-03-26 13:57:53 -07:00
Brad Fitzpatrick ea2342bf35 jsonsign: add WriteKeyRing function & tests.
Change-Id: Ic90751fd7834bf0e3bb0e1073fca647370561165
2012-03-19 21:00:51 -07:00
Brad Fitzpatrick a7200af73e jsonsign: start of WriteKeyRing
Change-Id: Id312578956d36b234d67d0c3f2ea3a261d122d48
2012-03-19 19:02:18 -07:00
Brad Fitzpatrick 67e32df2bd Fix a bunch of broken tests, lost in the Go 1 conversion.
Change-Id: I7c16563bc25f52868984c63e42125cd9bd9d5812
2012-03-19 18:44:50 -07:00
Brad Fitzpatrick 0714a463c9 Update from r60 to [almost] Go 1.
A lot is still broken, but most stuff at least compiles now.

The directory tree has been rearranged now too.  Go libraries are now
under "pkg".  Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".

The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.

For now, put the camlistore root under $GOPATH/src.  Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.

Then you can:

$ go build ./server/camlistored

... etc

The build.pl script is currently disabled.  It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).

Many things are certainly broken.

Many things are disabled.  (MySQL, all indexing, etc).

Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.

The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.

Help wanted.

Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
2012-02-18 21:53:06 -08:00