Commit Graph

321 Commits

Author SHA1 Message Date
Steven L. Speek db1a4cf6aa search: Self-documenting atoms
Parse errors in the atoms have a position too
Easier to add new search predicates

Change-Id: I9d2e890b9afcfb524aafa8f1e4e1debc683555ac
2014-05-15 00:42:32 +02:00
mpl a001c61339 camtool sync, localdisk: cancel enumerate to avoid channel lock
http://camlistore.org/issue/433

Change-Id: Id20d759dc5192dd34e69a866d8dcd44ca106d63b
2014-05-09 01:36:38 +02:00
mpl 45ce6cfdd3 mysql: replace mymysql with go-sql-driver
from github.com/go-sql-driver/mysql at 9a7aa3606b82e2081a13a008ada88dfdb96c20fd

Context: http://camlistore.org/issue/428

Change-Id: Ia93ac35da02933bd03d2c6b7b8e5e549ba0472cb
2014-05-01 17:42:51 +02:00
Mathieu Lonjaret bd52105da6 Merge "camtool: added googinit command" 2014-04-24 13:57:25 +00:00
jakubbrzeski e38dfda67a camtool: added googinit command
It initializes drive or googlestorage.
Deleted gsinit command.

Change-Id: I4d3b82018df9eec4d726cf91fd11be527c9a778a
2014-04-24 13:12:47 +02:00
mpl 2b091db606 camput init: do not call DefaultSecretRingFile in init()
Because we fail in tests when calling methods that rely on default
paths. (See buildinfo.TestingLinked)

Change-Id: I4d3a99dd2ea83a8351a993353e5e7c29d3cb17a7
2014-03-31 18:01:04 +02:00
mpl cdc4ed5ae2 camput init: ring/key related fixes
1) Removed exec call to gpg, because it automatically looks in .gnupg/,
which we don't use anymore as a default.
2) Now taking into account global --secret-keyring flag.
This flag is now in osutil.
3) New or modified funcs in osutil
4) Made sure --gpgkey works too.
5) Cleaned up error messages and hints.

Context: http://camlistore.org/issue/364
         http://camlistore.org/issue/368

Change-Id: I2e51032ed0597da656db100d72f5588b37308e1a
2014-03-28 19:55:18 +01:00
Hernan Grecco a6e4f87bc7 Implemented an optional parameter (attr) in 'camtool claims' to filter by claim type
For example, to obtain all the versions for a given file use:

   $ camtool claims <permanode> --attr="camliContent"

Change-Id: Ibb267a0f95702faa0e7e763c99f99ee446559aac
2014-03-19 00:45:51 -03:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
mpl dab702ea49 camtool sync: ignore diff identity with localdisk
When either the source or the destination is a localdisk, there is no
need to warn about differing identities.

Change-Id: Ia61779dd7907099c5ab58d9ca876cf4690f6fe05
2014-03-13 16:10:34 +01:00
Brad Fitzpatrick d10c9f070b Merge "cmd/camget: Support restoring symbolic links when writing to filesystem." 2014-03-11 03:51:01 +00:00
Brad Fitzpatrick bf892c7637 camtool: add search subcommand
Change-Id: I8738c2eb7eae4417778be39a4237e02d42965615
2014-03-10 19:47:30 -07:00
Brad Fitzpatrick 34d4513eb4 Merge "camtool list/sync: fix 3 bugs" 2014-03-09 05:22:31 +00:00
mpl 73526917a1 camtool list/sync: fix 3 bugs
1) revert to printing to stdout (broke in
08923e1c00) for actual output

2) do not warn about different identities when "syncing" to stdout

3) check blob was actually described before using its fields

Change-Id: Ibb3aa4c560e1b0036aba080807400e8820163cc9
2014-03-08 19:15:46 +01:00
Brian Gitonga Marete 0c4b67d3bf cmd/camget: Support restoring symbolic links when writing to filesystem.
This patch adds support for restoring symbolic links when camget is
called with the -o option.

This fixes an issue which caused `camget -o' to write an incomplete
snapshot to the target directory when it encountered a symlink.

Note that no attempt is made to restore the symlink metadata since: a)
A symlink's permissions do not matter and are fixed to 1777 on some
systems b) Go's os.Chtimes() always acts on the symlink target c)
os.Chown() is not currently called for any file in the current code.

Tested on Linux.

Change-Id: I7099592dcd5cc513c2685671be6f6b165f8f8053
2014-03-08 05:14:59 +03:00
Brad Fitzpatrick 76ba33b27a Move ListMissingDestinationBlobs from pkg/client to blobserver
Change-Id: I7489531fe2a1a5127b2212d5a5f80d4c4c614353
2014-03-06 15:04:15 -08:00
Brad Fitzpatrick f1a4aae0da camput: add --blobdir=discard debug option
Does a dry-run and just shows the final blobref, without writing data anywhere.

Change-Id: Ia234b642b5b9029ee4fd9c61bc7edcdcfa5ca858
2014-03-04 22:47:58 -08:00
Brad Fitzpatrick e3491e5515 camput: fix tests 2014-03-03 12:53:40 -08:00
mpl 08923e1c00 camtool sync: warn when src and dest have different identities
http://camlistore.org/issue/281

Change-Id: I2e2d595534b14476293a031b88e312de3e0b3f4a
2014-03-01 23:47:49 +01:00
Brad Fitzpatrick bbfcc27d85 sync: use blob.Ref.Less instead of stringify to compare, fix API wart, more tests
Change-Id: Iffd98463becbcf6d62dd666b38fb00572b34725b
2014-03-01 11:34:57 -08:00
Brad Fitzpatrick 26cbd36e0b camput, camtool: update docs
Change-Id: If54df42be3b95ec9f10782454bc247449e1a74e6
2014-02-27 21:23:01 -08:00
Brad Fitzpatrick 593e0f0f81 cmdmain, camput: PreExit should always run, not just in verbose mode
Change-Id: I6001a3474071a00f0ab56b031ac8c5504e5e9da5
2014-02-27 20:47:15 -08:00
mpl a73715c92a Merge "client: default to None auth if no auth in config" 2014-02-27 15:23:02 +00:00
mpl c4c4d6407a client: default to None auth if no auth in config
Defer warning until discovery fails.
Also fix SetupAuth error checks in camtool.

Change-Id: Ib554595603994e929d465e4e0de6acaf319d5b42
2014-02-27 01:37:59 +01:00
Brad Fitzpatrick bf35e5f011 camput: fix camput blob from stdin
Change-Id: Ic9997c408e1241e16dd4470b3f733ef3fb4f79c9
2014-02-25 14:34:26 -08:00
Brad Fitzpatrick 897591a444 camput: hermetic client config for test
Change-Id: I3c77b500da2d445851b0d8383a11e7f0942b2ff2
2014-02-23 10:59:41 -08:00
Brad Fitzpatrick ccacf76451 camput: re-enable deadlock test, now that it's fixed
And tweak limits a bit lower.

Change-Id: I7a38ffce37a997aa6ce9943e13605e76476f5646
2014-02-21 15:38:21 -08:00
Tamás Gulácsi 89fc2bb83e camput: limit upload concurrency (number of goroutines)
Use separate workers for directory uploads.
Limit upload concurrency to 3+2+3 (normal+directories+stat)
goroutines on Android, and 10+5+10 otherwise.

Thanks to the predictably failing test (1333551)
which we can test against, we can eliminate the need for
unbounded number of upload goroutines (see ee4550b).

Change-Id: Idac4c94b7d7ec4fc6c6a77fed111e0ece0843545
2014-02-21 21:31:13 +01:00
Brad Fitzpatrick 85e2e5e301 Fix crash with "go test -v -cpu=1,1 ./cmd/camput/"
Don't double-register the help flag. Only affects tests run twice, like above.

Change-Id: Ia235f2e2aa4a70e3b5a5c6fe21d8add90069664f
2014-02-21 10:59:34 -08:00
Brad Fitzpatrick 1333551de1 camput: predictably-failing disabled deadlock test
Change-Id: Ibf0877ac1b4e32ad32fe9592c487915130f9f488
2014-02-20 12:08:03 -08:00
Brad Fitzpatrick dfd94e505d camput: start of a deadlock test
Currently disabled until it's working reliably. See comments in code.

Much of this is from Tamás Gulácsi in https://camlistore-review.googlesource.com/#/c/2128/

Change-Id: I2fe2abfdee98584c698a2ac52202f13dda9beef2
2014-02-19 00:07:28 -08:00
Brad Fitzpatrick 77a34e060c camput: add a var for uploadWorkers
Change-Id: I529336ece16a35e25c33960589f706ebf8b2c5ec
2014-02-18 23:59:44 -08:00
Brad Fitzpatrick b023265043 camput: configurable timeout on env
Change-Id: Iffd0b404e48e4fd5235c22bdd00ad22c564fb0a0
2014-02-18 23:58:09 -08:00
Brad Fitzpatrick 685b44cd43 camput: don't show HTTP req stats when not using HTTP
Change-Id: I4d0c0dc44de84fe62f3ee2385bcd3875eb27be97
2014-02-18 23:56:22 -08:00
Brad Fitzpatrick 53ec1b496c camput: move debug flag registration to its own func, for use by tests later.
Also, don't call TransportForConfig on client when we're in blobdir mode.
Avoids a warning.

Change-Id: I5d618b7ab7d701b4c9d9086c8adedef7b447f62b
2014-02-18 21:24:52 -08:00
Brad Fitzpatrick 1509f266be camput: set os.Args from env.Run.args before test's cmdmain.Main runs
Not required now, but will be for future tests.

Change-Id: I50c469d4f4fa589494036f306c25fe60914fb902
2014-02-18 18:47:11 -08:00
Brad Fitzpatrick a199802898 camtool: document reindex-diskpacked better
Change-Id: Ib2f669ddf4fa493572a811b0564f104fbaeea05a
2014-02-11 19:24:08 -08:00
Brad Fitzpatrick 9a3a3795bb camtool: remove TODO
Change-Id: I3d9ac173335bd68b66bd8dfb5e0e40894aef3f79
2014-02-09 15:49:24 -08:00
Brad Fitzpatrick ca58231336 camtool: remove duplicated code
Change-Id: I2009b132a1c10e60132333b9942aa4b5943b3932
2014-02-09 15:48:07 -08:00
Brad Fitzpatrick 16b4d5c7e3 camtool: add discovery subcommand
Change-Id: Ic4cbc2777719301d3e4f3cdb0b14c10be6f64b3e
2014-02-09 15:16:47 -08:00
Tamás Gulácsi 97520583b8 Use 'uint32' instead of 'int64' for blob sizes everywhere.
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.

Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
2014-02-08 17:58:12 +01:00
Brad Fitzpatrick 5458321511 campit: add -blobdir flag for serverless mode.
Fixes camlistore.org/issue/362

Change-Id: If4c75b649cc4cc62f3e7a0ab07c61b088560c3e4
2014-02-07 11:02:07 -08:00
mpl 9add1d2c87 Merge "camtool list: introduce -describe to print camliType for each blob" 2014-01-31 23:10:54 +00:00
mpl 8084d6a4c7 camtool list: introduce -describe to print camliType for each blob
Change-Id: I73630dc135a169f5b24a403b5662e7ad617cb40d
2014-01-31 19:12:29 +01:00
Brad Fitzpatrick 48373fc8a7 camput: clean stale stuf in cachedir on start
Change-Id: Ie7b8d7da49703e5c430b8195202791445223cf25
2014-01-31 12:42:59 +01:00
Brad Fitzpatrick bda047eef7 camtool: add env camsrcroot
Change-Id: I701e81ab732d7574a227e8f01ca09ac91c608481
2014-01-31 12:21:04 +01:00
Brad Fitzpatrick 63d4bafd0c cammount: add -term flag, like -xterm but different
Change-Id: Idd81e345c4995a0ee939a2b4fe01d339f44ba526
2014-01-31 12:20:50 +01:00
Brad Fitzpatrick e2dc7a9256 camput: make uploading a directory tree less verbose. maybe too much.
Change-Id: Ia33588f4adc5af76bdca9ea8d26cec2340a94b30
2014-01-31 11:50:08 +01:00
Brad Fitzpatrick d77426ed6e camput: remove some android TODOs, now done
Change-Id: I206808f59bbf5a8f8dae35586b61180e0d404374
2014-01-30 17:29:41 +01:00
Brad Fitzpatrick 3f27e4f708 Add android camput pre-exit hook.
Change-Id: I30f9771e5562ece6a35d44cc8ab57c305a2899cb
2014-01-30 17:29:28 +01:00