Commit Graph

28 Commits

Author SHA1 Message Date
Brian Gitonga Marete e9dd1af195 cmd/camget, cmd/camput, pkg/schema: Support FIFOs.
This patch:

1. Introduces a JSON schema for FIFOs (named pipes).
2. Adds support for uploading and restoring FIFOs to camput and camget
respectively.

It includes tests for the basic schema support in pkg/schema and
integration tests for camput and camget.

This CL does not add FIFO support to the HTTP server and fs (cammount
support) code. Those will be addressed in a subsequent patch.

Change-Id: Iab926073c36e653d25e693a559178bea9bf19b9a
2014-05-21 14:52:23 +03:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
Brian Gitonga Marete 0c4b67d3bf cmd/camget: Support restoring symbolic links when writing to filesystem.
This patch adds support for restoring symbolic links when camget is
called with the -o option.

This fixes an issue which caused `camget -o' to write an incomplete
snapshot to the target directory when it encountered a symlink.

Note that no attempt is made to restore the symlink metadata since: a)
A symlink's permissions do not matter and are fixed to 1777 on some
systems b) Go's os.Chtimes() always acts on the symlink target c)
os.Chown() is not currently called for any file in the current code.

Tested on Linux.

Change-Id: I7099592dcd5cc513c2685671be6f6b165f8f8053
2014-03-08 05:14:59 +03:00
mpl b5d29a181f client,camget: add -cert to specify trusted cert with share URLs
http://camlistore.org/issue/330

Change-Id: I6d060fe4804d6cf9baa3686243e3ab93839ec9ac
2014-01-09 12:05:38 -08:00
Brad Fitzpatrick 0bdf20884b all: delete pkg/blobref; convert all from *blobref.BlobRef to new blob.Ref
Change-Id: Id2dfb7f19452bedf4f3c9310b36227fd8117b225
2013-08-03 19:54:30 -07:00
mpl 257fd29afa camget: allow insecure TLS with shared URLs
Change-Id: I22f610009edfecea932019c35310ae62b0613cb8
2013-07-16 15:56:32 +02:00
mpl b24ac50965 client: factorize the transport setup, and use it in camget
Fixes http://camlistore.org/issue/153

Change-Id: If393b34bbfcf4a6282357c9c57ebb85bae8741a3
2013-06-25 23:37:41 +02:00
mpl e6a5882511 website doc: cmds: golang style + install command
Change-Id: I4f38dfb87e066ef81881bc9a2262c52109e172c9
2013-03-15 23:18:09 +01:00
mpl 698e8605e6 website: /pkg/ and /cmd/ work.
-added some doc.go for each command to make /cmd/*
work nicely.
-hacked a bit of style, mostly borrowed from golang too.

Change-Id: I011d70bca2c0151a29dfd53c78a896821730d5fb
2013-03-15 16:40:44 +01:00
Brad Fitzpatrick ddcec4e92c Add --version flags to tools.
Change-Id: I1c86c3de09fb59bc20539f7d1efbd4d4710bf494
2013-02-02 10:59:39 -08:00
Brad Fitzpatrick cfc32e4a05 schema, camget: more work on deleting the Superset type.
not much more remains.

Change-Id: I6cfe4145f67b100a0e2509f88ce6e1c580b7f9fe
2013-01-22 09:32:40 -08:00
Brad Fitzpatrick 7c319ca284 pkg/cacher: unify the cammount and camget disk caching code
Change-Id: I8e53d00f1f5459856a4e2b863d74c2b7c3f515bc
2013-01-20 09:52:03 -08:00
Brad Fitzpatrick dd72cd4434 schema: verify no slashes in fileName in FileNameString accessor.
Change-Id: I1b02e67144fed1f80123530786af55c74bd64f36
2013-01-13 20:43:55 -08:00
Brad Fitzpatrick d689b8df88 camget: say the cache dir if verbose
Change-Id: Ic9e8fa9379e354d920e8c896fbcd4f3a861dfe20
2013-01-06 10:44:50 -08:00
Brad Fitzpatrick 3e5940a7ff camget, schema: read chunks more aggressively
Change-Id: I31bc3a145fc2f3a51c3cecbaa33872da077e7bd7
2013-01-05 23:12:42 -08:00
Brad Fitzpatrick ff9492c6cd camget: buffer the workc channel
Change-Id: Ic6bf443c4d91f240b0b86c34a33bf74a1f2b627f
2013-01-05 16:44:43 -08:00
Brad Fitzpatrick 9f6776dddb camget, httputil: more verbose HTTP debugging
Change-Id: I2fc1a1fe0fff99202245d391a8a76e9fab07adc2
2013-01-05 16:44:34 -08:00
Brad Fitzpatrick c89dbc7130 camget: put a local disk cache in front of the client.
a simple localhost test with only 4.4 MB and 4 files drops from 350 HTTP requests to 76.

more work to be done, but this is better.

Change-Id: I66ed67fe95ddd30aeefe742395874b2eabc237c7
2013-01-03 17:55:30 -08:00
Brad Fitzpatrick fe13c7c712 move camput's statsTransport to httputil.StatsTransport, use it in camget too.
Change-Id: Iab48e32ccb8a99b8ec6b058061cff9a5b99e8712
2013-01-02 20:32:13 -08:00
Brad Fitzpatrick 8866e88db0 camget: do 10 at once. better, much pref work remains.
Change-Id: Ib2e159a8a290aff249fbec109d4e76099fa0491a
2013-01-02 15:13:22 -08:00
Brad Fitzpatrick 827feaa3ac camget, client, schema: start of camget --shared support. see flag docs.
Change-Id: I5dd43129cb0032821a5913a8f20da0ddb38c63da
2013-01-02 12:55:12 -08:00
Brad Fitzpatrick edd5cff32a camget: fix directory restoring
Change-Id: I16c433fdda8a3220befd33309b1a31bbb69c0e58
2013-01-02 11:23:00 -08:00
Brad Fitzpatrick 378b854479 camget: use bytes.NewReader instead of bytes.NewBuffer
Change-Id: If51c47f356460b32355836e67a1ce443c071b65c
2013-01-02 09:20:35 -08:00
Brad Fitzpatrick e3247edafb Change blobref.SeekerFromStreamingFetcher signature to not return an error.
Change-Id: I77f693e3b3d0d116e08bca3d3f4cb45ef2a00b27
2012-12-25 10:27:35 -08:00
Brad Fitzpatrick 826bb388b4 camget: clean up, fix file dumping (to use FileReader), add --contents flag, update docs
Change-Id: I1fb2ed8691e24f7cd89e92b9a1ae3afbeb7fb21a
2012-12-25 10:20:14 -08:00
Brad Fitzpatrick 0ddc9dcadb camget: add a graphviz output mode with -graph
Change-Id: I472a1e7fda0f1a15bdcba1a7be8e0d586741bf08
2012-10-28 19:32:25 +01:00
Brad Fitzpatrick a229abeebd camget: fix build. update dev-camget.
Change-Id: I5289801de7b50caedeabd9cc2513d768aafd424b
2012-03-05 08:04:42 -08:00
Brad Fitzpatrick 0714a463c9 Update from r60 to [almost] Go 1.
A lot is still broken, but most stuff at least compiles now.

The directory tree has been rearranged now too.  Go libraries are now
under "pkg".  Fully qualified, they are e.g. "camlistore.org/pkg/jsonsign".

The go tool cannot yet fetch from arbitrary domains, but discussion is
happening now on which mechanism to use to allow that.

For now, put the camlistore root under $GOPATH/src.  Typically $GOPATH
is $HOME, so Camlistore should be at $HOME/src/camlistore.org.

Then you can:

$ go build ./server/camlistored

... etc

The build.pl script is currently disabled.  It'll be resurrected at
some point, but with a very different role (helping create a fake
GOPATH and running the go build command, if things are installed at
the wrong place, and/or running fileembed generators).

Many things are certainly broken.

Many things are disabled.  (MySQL, all indexing, etc).

Many things need to be moved into
camlistore.org/third_party/{code.google.com,github.com} and updated
from their r60 to Go 1 versions, where applicable.

The GoMySQL stuff should be updated to use database/sql and the ziutek
library implementing database/sql/driver.

Help wanted.

Change-Id: If71217dc5c8f0e70dbe46e9504ca5131c6eeacde
2012-02-18 21:53:06 -08:00