Commit Graph

101 Commits

Author SHA1 Message Date
mpl e0d719ba21 pkg/types: remove
Most of it replaced with vendor/go4.org/types and
vendor/go4.org/readerutil

u32 went where needed in pkg/blobserver/*
invertedBool went in pkg/types/serverconfig
atomics64 went in pkg/fs

Change-Id: I230426cda35be4b45ed67e869f14e6fdae89be22
2016-02-05 18:28:47 +01:00
Attila Tajti f0343c0fae camtool sync: local path support on Windows
Change-Id: I2d3e0c0bbd3f3ae50165d366e6a1093fceb36e5c
2016-01-11 12:00:37 +01:00
Brad Fitzpatrick 66f0ddac0e pkg/client: fix directory upload deadlock, clean up HTTP handling
Unexports pkg/client.Client.TransportForConfig in the process.

Fixes camlistore/camlistore#662

Change-Id: Ifc2d6b0fee70d5db86654897d2d377b343b6d38b
2015-12-29 23:39:29 -08:00
Brad Fitzpatrick fba8b457de pkg/client: wire up http2 support, unexported Client.InsecureTLS
Also add a flag to camtool discovery:

$ camtool discovery -httpversion
HTTP/2.0

Fixes camlistore/camlistore#661

Change-Id: I6db392e01bd176a2757ebdb4a0e881eebaec9b03
2015-12-29 11:33:38 -08:00
mpl 6af01f6c71 vendor: move pkg/images dependencies from third_party
This change is in anticipation of moving pkg/images to go4.org, where it
should not depend on packages in third_party.

So:
third_party/github.com/nf/cr2 -> vendor/github.com/nf/cr2
third_party/github.com/rwcarlsen/goexif -> vendor/github.com/rwcarlsen/goexif
third_party/golang.org/x/image/tiff -> vendor/golang.org/x/image/tiff

Note that third_party/go/pkg/image/jpeg was also a dependency of
pkg/images. We had vendored image/jpeg from tip at the time because it
offered advantages over the version from Go1.3
(https://github.com/camlistore/camlistore/issues/463).
Since we now depend on Go1.5, we can go back to depend on the stdlib
version, so we simply remove third_party/go/pkg/image/jpeg and adjust
the imports accordingly.

Change-Id: Ifc8ffae0551102e644a0a0c67f3ff89e04df15c7
2015-12-18 22:15:33 +01:00
Tamás Gulácsi 9f7e1df32b Remove pkg/context
Use golang.org/x/net/context instead.

Fixes #648.

Change-Id: I676a2f25458be97610a49d6f954f2102cbd373fa
2015-12-12 23:09:02 +01:00
mpl 1ffd5895e0 camtool/googinit: use cloud/storage Client instead of googlestorage
Issue #652

Change-Id: I54b2041acfa8f998e1fb082c54f6c7069615d8a7
2015-12-08 00:11:42 +01:00
mpl 40be4d8da2 vendor: go4.org/jsonconfig (and errorutil)
Previously pkg/jsonconfig and pkg/errorutil

Copied from go4.org at rev d1b8a2fb2de6160036e4801aa5e4d855571078b8

Change-Id: I673ed55b0825baa2607289b6082f205100261d7a
2015-12-01 17:21:49 +01:00
mpl 3dbaa49395 vendor: go4.org/oauthutil
Previously pkg/oauthutil

Imported from go4.org at rev
3970dd22c0e7e2397304ad626a20b72f8db41b58

Change-Id: Iffb751d03af9bc0c3008e5cd3833ed42fba0bdb1
2015-11-27 00:31:40 +01:00
mpl 12eddf9c19 vendor: go4.org/strutil go4.org/syncutil
They were internal packages (under pkg), which we are now moving to
go4.org, so we in turn need to vendor them in now.

Change-Id: I92224f731404d0bd4ca1c57492bed37cb3367ed4
2015-11-21 00:21:20 +01:00
Brad Fitzpatrick 6f58fc2426 Fix build, move more stuff from third_party to vendor. 2015-08-18 10:19:49 +02:00
Tamás Gulácsi 533433f163 cmd/camtool reindex-diskpacked: don't panic on high-level conf
If no metaIndex is specified, just go on, but don't panic.
Also try to clear the logic a little bit (don't overwrite indexConf).

Change-Id: I43e61ac3d5cd239cc113cb034ff3dd61eaaa14fd
2015-07-23 07:16:14 +02:00
mpl 5147aff319 google blobservers: migrate to golang.org/x/oauth2
Previous oauth2 (code.google.com/p/goauth2/oauth) still used in:
third_party/github.com/tgulacsi/picago
pkg/importer/picasa
pkg/importer/foursquare

Final cleanup in a subsequent CL.

Change-Id: I805d81fcaa651e03a17823c78abe5040d51346c3
2015-04-20 23:17:28 +02:00
Brad Fitzpatrick e255841ca8 camtool: fix packblobs usage
I removed the --start option.

Change-Id: I66a2bfaf380131dc589fd67e721091d0ba59e8b6
2015-01-19 15:09:07 -08:00
Brad Fitzpatrick 7f954485be camtool: add 'packblobs' command
Update #532

Change-Id: I2012b750b482da878f5b9ced9e7cea5a2124bec8
2015-01-19 15:06:04 -08:00
Brad Fitzpatrick a99dc14885 cmd/camtool: add flag to send a raw SearchQuery; permits sending a sort field
Part of upcoming blobpacked migration tool.

Change-Id: If0e91aa90aab78936976e8b0d6b3bfe2e5a4c27a
2015-01-17 20:29:05 -08:00
mpl 21a61dad40 Merge "index,camtool: try and cope better with broken exif" 2014-11-18 15:24:20 +00:00
mpl fda1399e9c index,camtool: try and cope better with broken exif
http://camlistore.org/issue/493

Change-Id: I40aebd67252cf82a3a5a143af6c258d7ed2aecda
2014-11-10 19:05:35 +01:00
Brad Fitzpatrick dedfeaff29 strutil: add IsPlausibleJSON
Use it in camtool search, and for https://camlistore-review.googlesource.com/#/c/2888/

Change-Id: Ibae0b4aa346cc8c18ff6a4b4eef994cfddd42dda
2014-11-08 18:16:47 -03:00
mpl f0c2f2db43 camtool makestatic: create dir on top of static-set
Change-Id: I6d71211450a195174ef8ac0ac5a66d44d6955927
2014-10-20 23:07:39 +02:00
Brad Fitzpatrick 3a43fd6430 camtool: add 'makestatic' subcommand
It converts a permanode set (with a bunch of camliMember) into a static-set.
Mostly for sharing a snapshot. Will likely evolve over time.

Change-Id: I9906750223a27e9a491c91f1ce2490c15d1d3986
2014-10-19 18:25:45 +02:00
Tamás Gulácsi bea59b29fa Remove kvfile-specific code from diskpacked.
In hope to use leveldb.

Change-Id: Ie9c166d48d6a2c1f52f9cfa29e703d8b8ad7f162
2014-09-23 22:40:07 +02:00
Fabian Wickborn 2aed1b8241 Renamed goexif folder to match upstream URL
In the advent of github.com/camlistore/goexif to be closed, this
commit renames the goexif folder in third_party to match the
upstream on GitHub.

The affected import paths have been rewritten accordingly.

Change-Id: I5a8871efd01987944b7f5e93979307857ae16fe7
2014-09-05 17:27:59 +02:00
mpl df7952b9ba mysql: create database now distinct from create tables
Because we do not want the database name to be optional with a
placeholder (/*DB*/) in the statement that creates the database, since
it is not actually optional there - as opposed to when creating tables.

Change-Id: I05351d76d95071492d763758a11454f219524510
2014-08-08 20:06:02 +02:00
mpl 44ac2e4a3e camtool/dbinit: replace dbname placeholder before creating table
Change-Id: I1a6c9f10c0731f5422a1d9780815e6e5235bb090
2014-08-08 17:44:17 +02:00
mpl 98d967ce39 camtool/dbinit: fix error printing
Change-Id: I749eae0207f5aa96bb342cf961d8511344acc26c
2014-08-08 17:24:48 +02:00
Brad Fitzpatrick c0a149ac3e Move GCE stuff out of camlistored into new package. Use from camtool.
Now camtool dumpconfig works on GCE too using the same mechanisms as camlistored.

Change-Id: I11c5311a6b7ace35fb0c6dd661ba9bdcad7820ed
2014-08-07 12:33:52 -07:00
Brad Fitzpatrick 1bfd1b7d98 camlistored, serverinit: allow server config from URLs.
paves the way for server config from GCE metadata, too.

Change-Id: I9e54e3ad79eb53e7ae96159f2aa02f4a017e8e11
2014-08-02 16:53:58 -07:00
mpl 26cb64e825 Introduce search.QueryDescriber interface. *client.Client implements it.
There are several places where we're using a *search.Handler, when we
only want to be able to do query and describe requests. When these
places need to be moved also out of the context of camlistored, (like
when making a server app such as the publisher), the QueryDescriber can
be used, and the adequate implementation (*search.Handler,
*client.Client, or any other) is used depending on the context.

Change-Id: Ie1c5bb6f1bbba32c24a8eb2d1175af95ffcc8335
2014-07-21 22:20:24 +02:00
Robert Kroeger f40e2e6e61 Clarified use of camtool googinit
To avoid user confusion, force camtool googinit use to actually
specify if Google Cloud Storage or Google Drive auth setup is
desired.

Change-Id: I6541b4a589de2dda73b725fcaf7bcdadd3f60b65
2014-07-07 08:33:53 -06:00
tnt 5d8eeb534b Fixed wrong calls to formatting functions.
E.g: fmt.Println("%s", "hello world") -> fmt.Printf("%s", "hello world")

Change-Id: I51c0bd54c14173e074fafd86c83a5287fcdb1520
2014-06-30 22:05:22 +02:00
Steven L. Speek db1a4cf6aa search: Self-documenting atoms
Parse errors in the atoms have a position too
Easier to add new search predicates

Change-Id: I9d2e890b9afcfb524aafa8f1e4e1debc683555ac
2014-05-15 00:42:32 +02:00
mpl a001c61339 camtool sync, localdisk: cancel enumerate to avoid channel lock
http://camlistore.org/issue/433

Change-Id: Id20d759dc5192dd34e69a866d8dcd44ca106d63b
2014-05-09 01:36:38 +02:00
mpl 45ce6cfdd3 mysql: replace mymysql with go-sql-driver
from github.com/go-sql-driver/mysql at 9a7aa3606b82e2081a13a008ada88dfdb96c20fd

Context: http://camlistore.org/issue/428

Change-Id: Ia93ac35da02933bd03d2c6b7b8e5e549ba0472cb
2014-05-01 17:42:51 +02:00
jakubbrzeski e38dfda67a camtool: added googinit command
It initializes drive or googlestorage.
Deleted gsinit command.

Change-Id: I4d3b82018df9eec4d726cf91fd11be527c9a778a
2014-04-24 13:12:47 +02:00
Hernan Grecco a6e4f87bc7 Implemented an optional parameter (attr) in 'camtool claims' to filter by claim type
For example, to obtain all the versions for a given file use:

   $ camtool claims <permanode> --attr="camliContent"

Change-Id: Ibb267a0f95702faa0e7e763c99f99ee446559aac
2014-03-19 00:45:51 -03:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
mpl dab702ea49 camtool sync: ignore diff identity with localdisk
When either the source or the destination is a localdisk, there is no
need to warn about differing identities.

Change-Id: Ia61779dd7907099c5ab58d9ca876cf4690f6fe05
2014-03-13 16:10:34 +01:00
Brad Fitzpatrick bf892c7637 camtool: add search subcommand
Change-Id: I8738c2eb7eae4417778be39a4237e02d42965615
2014-03-10 19:47:30 -07:00
Brad Fitzpatrick 34d4513eb4 Merge "camtool list/sync: fix 3 bugs" 2014-03-09 05:22:31 +00:00
mpl 73526917a1 camtool list/sync: fix 3 bugs
1) revert to printing to stdout (broke in
08923e1c00) for actual output

2) do not warn about different identities when "syncing" to stdout

3) check blob was actually described before using its fields

Change-Id: Ibb3aa4c560e1b0036aba080807400e8820163cc9
2014-03-08 19:15:46 +01:00
Brad Fitzpatrick 76ba33b27a Move ListMissingDestinationBlobs from pkg/client to blobserver
Change-Id: I7489531fe2a1a5127b2212d5a5f80d4c4c614353
2014-03-06 15:04:15 -08:00
mpl 08923e1c00 camtool sync: warn when src and dest have different identities
http://camlistore.org/issue/281

Change-Id: I2e2d595534b14476293a031b88e312de3e0b3f4a
2014-03-01 23:47:49 +01:00
Brad Fitzpatrick bbfcc27d85 sync: use blob.Ref.Less instead of stringify to compare, fix API wart, more tests
Change-Id: Iffd98463becbcf6d62dd666b38fb00572b34725b
2014-03-01 11:34:57 -08:00
Brad Fitzpatrick 26cbd36e0b camput, camtool: update docs
Change-Id: If54df42be3b95ec9f10782454bc247449e1a74e6
2014-02-27 21:23:01 -08:00
mpl c4c4d6407a client: default to None auth if no auth in config
Defer warning until discovery fails.
Also fix SetupAuth error checks in camtool.

Change-Id: Ib554595603994e929d465e4e0de6acaf319d5b42
2014-02-27 01:37:59 +01:00
Brad Fitzpatrick a199802898 camtool: document reindex-diskpacked better
Change-Id: Ib2f669ddf4fa493572a811b0564f104fbaeea05a
2014-02-11 19:24:08 -08:00
Brad Fitzpatrick 9a3a3795bb camtool: remove TODO
Change-Id: I3d9ac173335bd68b66bd8dfb5e0e40894aef3f79
2014-02-09 15:49:24 -08:00
Brad Fitzpatrick ca58231336 camtool: remove duplicated code
Change-Id: I2009b132a1c10e60132333b9942aa4b5943b3932
2014-02-09 15:48:07 -08:00
Brad Fitzpatrick 16b4d5c7e3 camtool: add discovery subcommand
Change-Id: Ic4cbc2777719301d3e4f3cdb0b14c10be6f64b3e
2014-02-09 15:16:47 -08:00