Commit Graph

348 Commits

Author SHA1 Message Date
mpl f0c2f2db43 camtool makestatic: create dir on top of static-set
Change-Id: I6d71211450a195174ef8ac0ac5a66d44d6955927
2014-10-20 23:07:39 +02:00
Brad Fitzpatrick 3a43fd6430 camtool: add 'makestatic' subcommand
It converts a permanode set (with a bunch of camliMember) into a static-set.
Mostly for sharing a snapshot. Will likely evolve over time.

Change-Id: I9906750223a27e9a491c91f1ce2490c15d1d3986
2014-10-19 18:25:45 +02:00
Tamás Gulácsi bea59b29fa Remove kvfile-specific code from diskpacked.
In hope to use leveldb.

Change-Id: Ie9c166d48d6a2c1f52f9cfa29e703d8b8ad7f162
2014-09-23 22:40:07 +02:00
Brad Fitzpatrick 8f2bdd3a82 Few little fixes to 51ef6152f
Change-Id: Id319d1fa4ad725f1ce9178fd0fbd7f787880a6fa
2014-09-06 19:19:36 -07:00
Aaron Boodman 51ef6152fd Add -search argument to camput share.
Bug: https://code.google.com/p/camlistore/issues/detail?id=512

Change-Id: Ic0dd559c98589107fcdeade8784699cf6b2f1761
2014-09-06 18:50:21 -07:00
Fabian Wickborn 2aed1b8241 Renamed goexif folder to match upstream URL
In the advent of github.com/camlistore/goexif to be closed, this
commit renames the goexif folder in third_party to match the
upstream on GitHub.

The affected import paths have been rewritten accordingly.

Change-Id: I5a8871efd01987944b7f5e93979307857ae16fe7
2014-09-05 17:27:59 +02:00
mpl 3fbc92a4ae Merge "client&camput: use cmdmain.Stderr for logging" 2014-08-12 20:33:50 +00:00
mpl 0fa50c4355 client&camput: use cmdmain.Stderr for logging
And use -verbose in tests, for easier debugging.

Change-Id: Iac5e1a8e9c987f82fc221ce73c7e3ca10a86a924
2014-08-12 22:41:55 +02:00
mpl df7952b9ba mysql: create database now distinct from create tables
Because we do not want the database name to be optional with a
placeholder (/*DB*/) in the statement that creates the database, since
it is not actually optional there - as opposed to when creating tables.

Change-Id: I05351d76d95071492d763758a11454f219524510
2014-08-08 20:06:02 +02:00
mpl 44ac2e4a3e camtool/dbinit: replace dbname placeholder before creating table
Change-Id: I1a6c9f10c0731f5422a1d9780815e6e5235bb090
2014-08-08 17:44:17 +02:00
mpl 98d967ce39 camtool/dbinit: fix error printing
Change-Id: I749eae0207f5aa96bb342cf961d8511344acc26c
2014-08-08 17:24:48 +02:00
Brad Fitzpatrick c0a149ac3e Move GCE stuff out of camlistored into new package. Use from camtool.
Now camtool dumpconfig works on GCE too using the same mechanisms as camlistored.

Change-Id: I11c5311a6b7ace35fb0c6dd661ba9bdcad7820ed
2014-08-07 12:33:52 -07:00
Brad Fitzpatrick 1bfd1b7d98 camlistored, serverinit: allow server config from URLs.
paves the way for server config from GCE metadata, too.

Change-Id: I9e54e3ad79eb53e7ae96159f2aa02f4a017e8e11
2014-08-02 16:53:58 -07:00
mpl 26cb64e825 Introduce search.QueryDescriber interface. *client.Client implements it.
There are several places where we're using a *search.Handler, when we
only want to be able to do query and describe requests. When these
places need to be moved also out of the context of camlistored, (like
when making a server app such as the publisher), the QueryDescriber can
be used, and the adequate implementation (*search.Handler,
*client.Client, or any other) is used depending on the context.

Change-Id: Ie1c5bb6f1bbba32c24a8eb2d1175af95ffcc8335
2014-07-21 22:20:24 +02:00
Brad Fitzpatrick cc418dac8f Merge "legal: package for compiled-in licenses" 2014-07-11 17:08:59 +00:00
Piotr S. Staszewski 5e61e2d84e legal: package for compiled-in licenses
Initial implementation, printing helper and 'wiring' for cam* tools.

Change-Id: Icf8baa77ee1f12495be8f1a57b7d575d6918cb5c
2014-07-10 23:01:54 +02:00
Brad Fitzpatrick 4ac89ec630 Merge "Clarified use of camtool googinit" 2014-07-05 03:02:58 +00:00
Brad Fitzpatrick cc956829f3 Rename android.go files to androidx.go
Go 1.4 (Go tip currently) defines a new GOOS=android, and a GOOS.go file is ignored on
other operating systems. But these android.go aren't actually meant to be only run on
GOOS=android, since we just build Linux binaries on Android for now.
2014-07-08 11:18:40 -07:00
Robert Kroeger f40e2e6e61 Clarified use of camtool googinit
To avoid user confusion, force camtool googinit use to actually
specify if Google Cloud Storage or Google Drive auth setup is
desired.

Change-Id: I6541b4a589de2dda73b725fcaf7bcdadd3f60b65
2014-07-07 08:33:53 -06:00
tnt 5d8eeb534b Fixed wrong calls to formatting functions.
E.g: fmt.Println("%s", "hello world") -> fmt.Printf("%s", "hello world")

Change-Id: I51c0bd54c14173e074fafd86c83a5287fcdb1520
2014-06-30 22:05:22 +02:00
Brad Fitzpatrick 8a9ff86371 camget: add -skip_irregular flag for Windows users mostly
Change-Id: I8037d2f6a54e375eeeb7c79ecfedbbeaf547efed
2014-06-27 13:48:05 -07:00
Brad Fitzpatrick 20e952ce83 types: add NewOnceCloser, use it in camget
Change-Id: I465207031468a57f2840d5071fc27713d6184325
2014-05-31 16:08:02 +09:00
Brad Fitzpatrick 6914346584 Merge "Fix running out of FDs in camget" 2014-05-31 06:46:28 +00:00
Brad Fitzpatrick 26bdefc4b0 osutil: remove err error stutter, add osutil.ErrNotSupported
Change-Id: I483f30d3d4db39a9ffa0f83c1e055d7a59edba4b
2014-05-31 10:42:44 +09:00
Brian Gitonga Marete 53ab352fa7 pkg/schema, cmd/camget, cmd/camput: Support sockets.
This patch:

1. Introduces a JSON schema for unix domain sockets (socket files).
2. Adds support for the new schema to pkg/schema.
3. Adds support for uploading and restoring sockets to camput and
camget respectively.

Depends on CL 2706 (https://camlistore-review.googlesource.com/2706/)

Change-Id: Ib0d6e9f9ebd348f55efee5a08cd774f3969d0760
2014-05-21 18:51:14 +03:00
Brian Gitonga Marete e9dd1af195 cmd/camget, cmd/camput, pkg/schema: Support FIFOs.
This patch:

1. Introduces a JSON schema for FIFOs (named pipes).
2. Adds support for uploading and restoring FIFOs to camput and camget
respectively.

It includes tests for the basic schema support in pkg/schema and
integration tests for camput and camget.

This CL does not add FIFO support to the HTTP server and fs (cammount
support) code. Those will be addressed in a subsequent patch.

Change-Id: Iab926073c36e653d25e693a559178bea9bf19b9a
2014-05-21 14:52:23 +03:00
Robert Kroeger c6259885bf Fix running out of FDs in camget
camget runs out of FDs when fetching the contents of a directory with
many entries. Close the connections to the server more aggressively
to fix this.

Change-Id: Iabef20481d3ec12cc5f15e75edc01d0e1e883486
2014-05-18 11:16:48 -06:00
Steven L. Speek db1a4cf6aa search: Self-documenting atoms
Parse errors in the atoms have a position too
Easier to add new search predicates

Change-Id: I9d2e890b9afcfb524aafa8f1e4e1debc683555ac
2014-05-15 00:42:32 +02:00
mpl a001c61339 camtool sync, localdisk: cancel enumerate to avoid channel lock
http://camlistore.org/issue/433

Change-Id: Id20d759dc5192dd34e69a866d8dcd44ca106d63b
2014-05-09 01:36:38 +02:00
mpl 45ce6cfdd3 mysql: replace mymysql with go-sql-driver
from github.com/go-sql-driver/mysql at 9a7aa3606b82e2081a13a008ada88dfdb96c20fd

Context: http://camlistore.org/issue/428

Change-Id: Ia93ac35da02933bd03d2c6b7b8e5e549ba0472cb
2014-05-01 17:42:51 +02:00
Mathieu Lonjaret bd52105da6 Merge "camtool: added googinit command" 2014-04-24 13:57:25 +00:00
jakubbrzeski e38dfda67a camtool: added googinit command
It initializes drive or googlestorage.
Deleted gsinit command.

Change-Id: I4d3b82018df9eec4d726cf91fd11be527c9a778a
2014-04-24 13:12:47 +02:00
mpl 2b091db606 camput init: do not call DefaultSecretRingFile in init()
Because we fail in tests when calling methods that rely on default
paths. (See buildinfo.TestingLinked)

Change-Id: I4d3a99dd2ea83a8351a993353e5e7c29d3cb17a7
2014-03-31 18:01:04 +02:00
mpl cdc4ed5ae2 camput init: ring/key related fixes
1) Removed exec call to gpg, because it automatically looks in .gnupg/,
which we don't use anymore as a default.
2) Now taking into account global --secret-keyring flag.
This flag is now in osutil.
3) New or modified funcs in osutil
4) Made sure --gpgkey works too.
5) Cleaned up error messages and hints.

Context: http://camlistore.org/issue/364
         http://camlistore.org/issue/368

Change-Id: I2e51032ed0597da656db100d72f5588b37308e1a
2014-03-28 19:55:18 +01:00
Hernan Grecco a6e4f87bc7 Implemented an optional parameter (attr) in 'camtool claims' to filter by claim type
For example, to obtain all the versions for a given file use:

   $ camtool claims <permanode> --attr="camliContent"

Change-Id: Ibb267a0f95702faa0e7e763c99f99ee446559aac
2014-03-19 00:45:51 -03:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
mpl dab702ea49 camtool sync: ignore diff identity with localdisk
When either the source or the destination is a localdisk, there is no
need to warn about differing identities.

Change-Id: Ia61779dd7907099c5ab58d9ca876cf4690f6fe05
2014-03-13 16:10:34 +01:00
Brad Fitzpatrick d10c9f070b Merge "cmd/camget: Support restoring symbolic links when writing to filesystem." 2014-03-11 03:51:01 +00:00
Brad Fitzpatrick bf892c7637 camtool: add search subcommand
Change-Id: I8738c2eb7eae4417778be39a4237e02d42965615
2014-03-10 19:47:30 -07:00
Brad Fitzpatrick 34d4513eb4 Merge "camtool list/sync: fix 3 bugs" 2014-03-09 05:22:31 +00:00
mpl 73526917a1 camtool list/sync: fix 3 bugs
1) revert to printing to stdout (broke in
08923e1c00) for actual output

2) do not warn about different identities when "syncing" to stdout

3) check blob was actually described before using its fields

Change-Id: Ibb3aa4c560e1b0036aba080807400e8820163cc9
2014-03-08 19:15:46 +01:00
Brian Gitonga Marete 0c4b67d3bf cmd/camget: Support restoring symbolic links when writing to filesystem.
This patch adds support for restoring symbolic links when camget is
called with the -o option.

This fixes an issue which caused `camget -o' to write an incomplete
snapshot to the target directory when it encountered a symlink.

Note that no attempt is made to restore the symlink metadata since: a)
A symlink's permissions do not matter and are fixed to 1777 on some
systems b) Go's os.Chtimes() always acts on the symlink target c)
os.Chown() is not currently called for any file in the current code.

Tested on Linux.

Change-Id: I7099592dcd5cc513c2685671be6f6b165f8f8053
2014-03-08 05:14:59 +03:00
Brad Fitzpatrick 76ba33b27a Move ListMissingDestinationBlobs from pkg/client to blobserver
Change-Id: I7489531fe2a1a5127b2212d5a5f80d4c4c614353
2014-03-06 15:04:15 -08:00
Brad Fitzpatrick f1a4aae0da camput: add --blobdir=discard debug option
Does a dry-run and just shows the final blobref, without writing data anywhere.

Change-Id: Ia234b642b5b9029ee4fd9c61bc7edcdcfa5ca858
2014-03-04 22:47:58 -08:00
Brad Fitzpatrick e3491e5515 camput: fix tests 2014-03-03 12:53:40 -08:00
mpl 08923e1c00 camtool sync: warn when src and dest have different identities
http://camlistore.org/issue/281

Change-Id: I2e2d595534b14476293a031b88e312de3e0b3f4a
2014-03-01 23:47:49 +01:00
Brad Fitzpatrick bbfcc27d85 sync: use blob.Ref.Less instead of stringify to compare, fix API wart, more tests
Change-Id: Iffd98463becbcf6d62dd666b38fb00572b34725b
2014-03-01 11:34:57 -08:00
Brad Fitzpatrick 26cbd36e0b camput, camtool: update docs
Change-Id: If54df42be3b95ec9f10782454bc247449e1a74e6
2014-02-27 21:23:01 -08:00
Brad Fitzpatrick 593e0f0f81 cmdmain, camput: PreExit should always run, not just in verbose mode
Change-Id: I6001a3474071a00f0ab56b031ac8c5504e5e9da5
2014-02-27 20:47:15 -08:00
mpl a73715c92a Merge "client: default to None auth if no auth in config" 2014-02-27 15:23:02 +00:00