Commit Graph

342 Commits

Author SHA1 Message Date
mpl 3fbc92a4ae Merge "client&camput: use cmdmain.Stderr for logging" 2014-08-12 20:33:50 +00:00
mpl 0fa50c4355 client&camput: use cmdmain.Stderr for logging
And use -verbose in tests, for easier debugging.

Change-Id: Iac5e1a8e9c987f82fc221ce73c7e3ca10a86a924
2014-08-12 22:41:55 +02:00
mpl df7952b9ba mysql: create database now distinct from create tables
Because we do not want the database name to be optional with a
placeholder (/*DB*/) in the statement that creates the database, since
it is not actually optional there - as opposed to when creating tables.

Change-Id: I05351d76d95071492d763758a11454f219524510
2014-08-08 20:06:02 +02:00
mpl 44ac2e4a3e camtool/dbinit: replace dbname placeholder before creating table
Change-Id: I1a6c9f10c0731f5422a1d9780815e6e5235bb090
2014-08-08 17:44:17 +02:00
mpl 98d967ce39 camtool/dbinit: fix error printing
Change-Id: I749eae0207f5aa96bb342cf961d8511344acc26c
2014-08-08 17:24:48 +02:00
Brad Fitzpatrick c0a149ac3e Move GCE stuff out of camlistored into new package. Use from camtool.
Now camtool dumpconfig works on GCE too using the same mechanisms as camlistored.

Change-Id: I11c5311a6b7ace35fb0c6dd661ba9bdcad7820ed
2014-08-07 12:33:52 -07:00
Brad Fitzpatrick 1bfd1b7d98 camlistored, serverinit: allow server config from URLs.
paves the way for server config from GCE metadata, too.

Change-Id: I9e54e3ad79eb53e7ae96159f2aa02f4a017e8e11
2014-08-02 16:53:58 -07:00
mpl 26cb64e825 Introduce search.QueryDescriber interface. *client.Client implements it.
There are several places where we're using a *search.Handler, when we
only want to be able to do query and describe requests. When these
places need to be moved also out of the context of camlistored, (like
when making a server app such as the publisher), the QueryDescriber can
be used, and the adequate implementation (*search.Handler,
*client.Client, or any other) is used depending on the context.

Change-Id: Ie1c5bb6f1bbba32c24a8eb2d1175af95ffcc8335
2014-07-21 22:20:24 +02:00
Brad Fitzpatrick cc418dac8f Merge "legal: package for compiled-in licenses" 2014-07-11 17:08:59 +00:00
Piotr S. Staszewski 5e61e2d84e legal: package for compiled-in licenses
Initial implementation, printing helper and 'wiring' for cam* tools.

Change-Id: Icf8baa77ee1f12495be8f1a57b7d575d6918cb5c
2014-07-10 23:01:54 +02:00
Brad Fitzpatrick 4ac89ec630 Merge "Clarified use of camtool googinit" 2014-07-05 03:02:58 +00:00
Brad Fitzpatrick cc956829f3 Rename android.go files to androidx.go
Go 1.4 (Go tip currently) defines a new GOOS=android, and a GOOS.go file is ignored on
other operating systems. But these android.go aren't actually meant to be only run on
GOOS=android, since we just build Linux binaries on Android for now.
2014-07-08 11:18:40 -07:00
Robert Kroeger f40e2e6e61 Clarified use of camtool googinit
To avoid user confusion, force camtool googinit use to actually
specify if Google Cloud Storage or Google Drive auth setup is
desired.

Change-Id: I6541b4a589de2dda73b725fcaf7bcdadd3f60b65
2014-07-07 08:33:53 -06:00
tnt 5d8eeb534b Fixed wrong calls to formatting functions.
E.g: fmt.Println("%s", "hello world") -> fmt.Printf("%s", "hello world")

Change-Id: I51c0bd54c14173e074fafd86c83a5287fcdb1520
2014-06-30 22:05:22 +02:00
Brad Fitzpatrick 8a9ff86371 camget: add -skip_irregular flag for Windows users mostly
Change-Id: I8037d2f6a54e375eeeb7c79ecfedbbeaf547efed
2014-06-27 13:48:05 -07:00
Brad Fitzpatrick 20e952ce83 types: add NewOnceCloser, use it in camget
Change-Id: I465207031468a57f2840d5071fc27713d6184325
2014-05-31 16:08:02 +09:00
Brad Fitzpatrick 6914346584 Merge "Fix running out of FDs in camget" 2014-05-31 06:46:28 +00:00
Brad Fitzpatrick 26bdefc4b0 osutil: remove err error stutter, add osutil.ErrNotSupported
Change-Id: I483f30d3d4db39a9ffa0f83c1e055d7a59edba4b
2014-05-31 10:42:44 +09:00
Brian Gitonga Marete 53ab352fa7 pkg/schema, cmd/camget, cmd/camput: Support sockets.
This patch:

1. Introduces a JSON schema for unix domain sockets (socket files).
2. Adds support for the new schema to pkg/schema.
3. Adds support for uploading and restoring sockets to camput and
camget respectively.

Depends on CL 2706 (https://camlistore-review.googlesource.com/2706/)

Change-Id: Ib0d6e9f9ebd348f55efee5a08cd774f3969d0760
2014-05-21 18:51:14 +03:00
Brian Gitonga Marete e9dd1af195 cmd/camget, cmd/camput, pkg/schema: Support FIFOs.
This patch:

1. Introduces a JSON schema for FIFOs (named pipes).
2. Adds support for uploading and restoring FIFOs to camput and camget
respectively.

It includes tests for the basic schema support in pkg/schema and
integration tests for camput and camget.

This CL does not add FIFO support to the HTTP server and fs (cammount
support) code. Those will be addressed in a subsequent patch.

Change-Id: Iab926073c36e653d25e693a559178bea9bf19b9a
2014-05-21 14:52:23 +03:00
Robert Kroeger c6259885bf Fix running out of FDs in camget
camget runs out of FDs when fetching the contents of a directory with
many entries. Close the connections to the server more aggressively
to fix this.

Change-Id: Iabef20481d3ec12cc5f15e75edc01d0e1e883486
2014-05-18 11:16:48 -06:00
Steven L. Speek db1a4cf6aa search: Self-documenting atoms
Parse errors in the atoms have a position too
Easier to add new search predicates

Change-Id: I9d2e890b9afcfb524aafa8f1e4e1debc683555ac
2014-05-15 00:42:32 +02:00
mpl a001c61339 camtool sync, localdisk: cancel enumerate to avoid channel lock
http://camlistore.org/issue/433

Change-Id: Id20d759dc5192dd34e69a866d8dcd44ca106d63b
2014-05-09 01:36:38 +02:00
mpl 45ce6cfdd3 mysql: replace mymysql with go-sql-driver
from github.com/go-sql-driver/mysql at 9a7aa3606b82e2081a13a008ada88dfdb96c20fd

Context: http://camlistore.org/issue/428

Change-Id: Ia93ac35da02933bd03d2c6b7b8e5e549ba0472cb
2014-05-01 17:42:51 +02:00
Mathieu Lonjaret bd52105da6 Merge "camtool: added googinit command" 2014-04-24 13:57:25 +00:00
jakubbrzeski e38dfda67a camtool: added googinit command
It initializes drive or googlestorage.
Deleted gsinit command.

Change-Id: I4d3b82018df9eec4d726cf91fd11be527c9a778a
2014-04-24 13:12:47 +02:00
mpl 2b091db606 camput init: do not call DefaultSecretRingFile in init()
Because we fail in tests when calling methods that rely on default
paths. (See buildinfo.TestingLinked)

Change-Id: I4d3a99dd2ea83a8351a993353e5e7c29d3cb17a7
2014-03-31 18:01:04 +02:00
mpl cdc4ed5ae2 camput init: ring/key related fixes
1) Removed exec call to gpg, because it automatically looks in .gnupg/,
which we don't use anymore as a default.
2) Now taking into account global --secret-keyring flag.
This flag is now in osutil.
3) New or modified funcs in osutil
4) Made sure --gpgkey works too.
5) Cleaned up error messages and hints.

Context: http://camlistore.org/issue/364
         http://camlistore.org/issue/368

Change-Id: I2e51032ed0597da656db100d72f5588b37308e1a
2014-03-28 19:55:18 +01:00
Hernan Grecco a6e4f87bc7 Implemented an optional parameter (attr) in 'camtool claims' to filter by claim type
For example, to obtain all the versions for a given file use:

   $ camtool claims <permanode> --attr="camliContent"

Change-Id: Ibb267a0f95702faa0e7e763c99f99ee446559aac
2014-03-19 00:45:51 -03:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
mpl dab702ea49 camtool sync: ignore diff identity with localdisk
When either the source or the destination is a localdisk, there is no
need to warn about differing identities.

Change-Id: Ia61779dd7907099c5ab58d9ca876cf4690f6fe05
2014-03-13 16:10:34 +01:00
Brad Fitzpatrick d10c9f070b Merge "cmd/camget: Support restoring symbolic links when writing to filesystem." 2014-03-11 03:51:01 +00:00
Brad Fitzpatrick bf892c7637 camtool: add search subcommand
Change-Id: I8738c2eb7eae4417778be39a4237e02d42965615
2014-03-10 19:47:30 -07:00
Brad Fitzpatrick 34d4513eb4 Merge "camtool list/sync: fix 3 bugs" 2014-03-09 05:22:31 +00:00
mpl 73526917a1 camtool list/sync: fix 3 bugs
1) revert to printing to stdout (broke in
08923e1c00) for actual output

2) do not warn about different identities when "syncing" to stdout

3) check blob was actually described before using its fields

Change-Id: Ibb3aa4c560e1b0036aba080807400e8820163cc9
2014-03-08 19:15:46 +01:00
Brian Gitonga Marete 0c4b67d3bf cmd/camget: Support restoring symbolic links when writing to filesystem.
This patch adds support for restoring symbolic links when camget is
called with the -o option.

This fixes an issue which caused `camget -o' to write an incomplete
snapshot to the target directory when it encountered a symlink.

Note that no attempt is made to restore the symlink metadata since: a)
A symlink's permissions do not matter and are fixed to 1777 on some
systems b) Go's os.Chtimes() always acts on the symlink target c)
os.Chown() is not currently called for any file in the current code.

Tested on Linux.

Change-Id: I7099592dcd5cc513c2685671be6f6b165f8f8053
2014-03-08 05:14:59 +03:00
Brad Fitzpatrick 76ba33b27a Move ListMissingDestinationBlobs from pkg/client to blobserver
Change-Id: I7489531fe2a1a5127b2212d5a5f80d4c4c614353
2014-03-06 15:04:15 -08:00
Brad Fitzpatrick f1a4aae0da camput: add --blobdir=discard debug option
Does a dry-run and just shows the final blobref, without writing data anywhere.

Change-Id: Ia234b642b5b9029ee4fd9c61bc7edcdcfa5ca858
2014-03-04 22:47:58 -08:00
Brad Fitzpatrick e3491e5515 camput: fix tests 2014-03-03 12:53:40 -08:00
mpl 08923e1c00 camtool sync: warn when src and dest have different identities
http://camlistore.org/issue/281

Change-Id: I2e2d595534b14476293a031b88e312de3e0b3f4a
2014-03-01 23:47:49 +01:00
Brad Fitzpatrick bbfcc27d85 sync: use blob.Ref.Less instead of stringify to compare, fix API wart, more tests
Change-Id: Iffd98463becbcf6d62dd666b38fb00572b34725b
2014-03-01 11:34:57 -08:00
Brad Fitzpatrick 26cbd36e0b camput, camtool: update docs
Change-Id: If54df42be3b95ec9f10782454bc247449e1a74e6
2014-02-27 21:23:01 -08:00
Brad Fitzpatrick 593e0f0f81 cmdmain, camput: PreExit should always run, not just in verbose mode
Change-Id: I6001a3474071a00f0ab56b031ac8c5504e5e9da5
2014-02-27 20:47:15 -08:00
mpl a73715c92a Merge "client: default to None auth if no auth in config" 2014-02-27 15:23:02 +00:00
mpl c4c4d6407a client: default to None auth if no auth in config
Defer warning until discovery fails.
Also fix SetupAuth error checks in camtool.

Change-Id: Ib554595603994e929d465e4e0de6acaf319d5b42
2014-02-27 01:37:59 +01:00
Brad Fitzpatrick bf35e5f011 camput: fix camput blob from stdin
Change-Id: Ic9997c408e1241e16dd4470b3f733ef3fb4f79c9
2014-02-25 14:34:26 -08:00
Brad Fitzpatrick 897591a444 camput: hermetic client config for test
Change-Id: I3c77b500da2d445851b0d8383a11e7f0942b2ff2
2014-02-23 10:59:41 -08:00
Brad Fitzpatrick ccacf76451 camput: re-enable deadlock test, now that it's fixed
And tweak limits a bit lower.

Change-Id: I7a38ffce37a997aa6ce9943e13605e76476f5646
2014-02-21 15:38:21 -08:00
Tamás Gulácsi 89fc2bb83e camput: limit upload concurrency (number of goroutines)
Use separate workers for directory uploads.
Limit upload concurrency to 3+2+3 (normal+directories+stat)
goroutines on Android, and 10+5+10 otherwise.

Thanks to the predictably failing test (1333551)
which we can test against, we can eliminate the need for
unbounded number of upload goroutines (see ee4550b).

Change-Id: Idac4c94b7d7ec4fc6c6a77fed111e0ece0843545
2014-02-21 21:31:13 +01:00
Brad Fitzpatrick 85e2e5e301 Fix crash with "go test -v -cpu=1,1 ./cmd/camput/"
Don't double-register the help flag. Only affects tests run twice, like above.

Change-Id: Ia235f2e2aa4a70e3b5a5c6fe21d8add90069664f
2014-02-21 10:59:34 -08:00