Commit Graph

42 Commits

Author SHA1 Message Date
Brad Fitzpatrick 75d60962f6 Move remaining stuff in third_party/* to vendor/*
Change-Id: Ifbcc02817083cba68d8c1acec3e6ec50e8f61149
2016-04-20 16:49:15 -07:00
Tamás Gulácsi 7402cc0efd Delete misc unused objects
Using honnef.co/go/unused/cmd/unused

Change-Id: I672b3cb77f09e9bd80dcdc149cde4f7f2939e451
2016-04-06 17:59:51 +02:00
Will Norris 77ed42edf8 add canonical import paths
The import path was added to the go file that included the package
documentation if one existed.  Otherwise, I used what seemed to be the
primary file for the package.

Fixes #689

Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
2016-03-13 19:57:14 -07:00
mpl e0d719ba21 pkg/types: remove
Most of it replaced with vendor/go4.org/types and
vendor/go4.org/readerutil

u32 went where needed in pkg/blobserver/*
invertedBool went in pkg/types/serverconfig
atomics64 went in pkg/fs

Change-Id: I230426cda35be4b45ed67e869f14e6fdae89be22
2016-02-05 18:28:47 +01:00
mpl 8a93e80a8b Merge "blob, server: show hash functions in discovery" 2015-03-02 18:30:31 +00:00
mpl dfe854e168 blob, server: show hash functions in discovery
Issue #122

Change-Id: Iddc3edd4d10065b4cf94f4d554cc6c24ca78a26c
2015-03-02 18:45:58 +01:00
mpl b6eb85631c blob SubFetcher: explicitely states with errors the testSubFetcher constraints
testSubFetcher in blobserver/storagetest was already checking that we'd
get specific error messages in the case of negative input parameters or
an out of range offset.

This change rationalizes these constraints with named errors
(ErrNegativeSubFetch and ErrOutOfRangeOffsetSubFetch) specified
in the SubFetcher interface.

It also fixes the googlestorage and s3 implementations so that they pass
the aforementioned test.

Change-Id: I25b72b842855b90ee3cab44c90654581dccf4b8e
2015-02-26 15:18:27 +01:00
Brad Fitzpatrick e9dade20fc blob: add weird StringMinusOne method
Change-Id: If1cb450e2131002eaa80e7f5ee2a449c745b99a6
2014-12-23 19:10:13 -08:00
Emil Hessman e8ffb9a66c pkg: fix various vet reports
Change-Id: I0b7fa09849650fe696e59ad5c5c73a034ab80faa
2014-11-12 23:25:53 +01:00
Brad Fitzpatrick 5ac4e18a62 blob: add func ReaderAt
Change-Id: Ic8afd01951d3fdb792a5cbe64ab60c8be0dec859
2014-10-28 19:29:33 -03:00
Brad Fitzpatrick 55f0c91c25 blobpacked: start of untested blob streaming code
Will be used for recovery.

Change-Id: I8a6d219086a306e53b1b621b7bc7cf59b984c765
2014-10-22 15:12:41 -07:00
Brad Fitzpatrick cd5fa05a18 blob: document, clean up ChanPeeker
Change-Id: I7c721f1e3fb81011cf2911be8889dcb9abc66dd5
2014-09-22 22:05:48 -04:00
mpl 5036d6b456 blobserver: add memory storage, mainly for tests.
Mostly code moved from pkg/test/fetcher.go

Also got rid of blob.MemoryStorage

http://camlistore.org/issue/416

Change-Id: I8d17b80942e2fde1c869f71738bfb8a67ed8ab31
2014-08-31 01:17:52 +02:00
Brad Fitzpatrick 77a5f0f375 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-08-25 16:50:42 -07:00
Brad Fitzpatrick 4609f65918 blob: add SubFetcher interface
Change-Id: If33430e78fb51c2b5750d2d52cdbda350e0f8a24
2014-08-25 16:43:50 -07:00
Brad Fitzpatrick 12e595d567 Move pkg/depcheck up a level so it's not caught by our fmt checks.
gofmt barfs on it.

Change-Id: I9522d5036f75737d32b46684c894648a41e1cd82
2014-08-23 20:26:51 -07:00
Brad Fitzpatrick b38077ff0f New pkg/depcheck package to check Go version.
Based on kr's example (syntantically valid Go code, to make cmd/go happy, but
will fail to compile and include the failure text)

Change-Id: I11f9eeabf6bf2c77055752014c71183b7eb7b23a
2014-08-19 20:03:36 -07:00
mpl d36b42862c blobserver/protocol: use blob.SizedRef to upload
Change-Id: I4b38b07869e08c2ccdcb76898dc908e6358501b0
2014-08-18 19:36:34 +02:00
Brad Fitzpatrick 26a7db6c80 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-04-11 15:51:05 -07:00
Brad Fitzpatrick bf83a52bef blob: add ParseKnown
Change-Id: I96e031951d1f490a700faabf5aeb864f8c0b9bc3
2014-04-11 15:50:31 -07:00
Brad Fitzpatrick bfc201f7a6 Add blobserver.RefTypes and blob.TypeAlphabet
To be implemented later, but adding now so googledrive's blob storage implementation
can depend on them.

Change-Id: Ief374e8592bd696c79aa2b80ded11e301063750b
2014-04-06 23:07:55 -07:00
Govert Versluis 96637ffcd3 pkg/blobserver/mongo: Add support for blob storage in MongoDB
Related changes:
Split docker-related test-helper functions from pkg/sorted/mongo.
These helper functions are now also used in the pkg/blobserver/mongo
tests too.
Also fixed a typo in a comment in pkg/blob/fetcher.go and a missing variable in
debug output in pkg/blobserver/storagetest/storagetest.go

Addresses http://camlistore.org/issue/127

Change-Id: I8b6f57f9ced066d6f83788fdcc87be6619c65c3c

Conflicts:
	pkg/blob/fetcher.go
2014-03-22 12:36:59 -07:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
Brad Fitzpatrick bfcd26a2ff blob: move Blob type to its own file, add FromFetcher, FromReader
Prep for killing SeekFetcher.

Change-Id: Iade678ccf74d5ebc9613dbb9dbf44b9ed9ff4dea
2014-03-14 11:47:37 -07:00
Tamás Gulácsi 97520583b8 Use 'uint32' instead of 'int64' for blob sizes everywhere.
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.

Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
2014-02-08 17:58:12 +01:00
Brad Fitzpatrick a384ff188d Get rid of CAMLI_DEV_KEYBLOBS / keyblobs / etc.
Fixes camlistore.org/issue/277

Change-Id: I7f380f9d18785f600fa0c442d5a19cd118782788
2014-01-20 13:47:08 -08:00
Tamás Gulácsi 36643ff986 Fix Fetch -> FetchStreaming conversion
As blob.fetcherToSeekerWrapper.Fetch erroneously asserts that FetchStreaming
returns a ReadSeekCloser everytime, it had to be changed.

Move MaxBlobSize from blobserver to constants (new package).

Change-Id: I4b4f22c302cbec84d77d21454e0c9e8aebdf73e5
2014-01-06 18:56:39 +01:00
Bill Thiede 9aeb0aa2c7 pkg/client: Search fix zero-value Describe queries.
Adds support for zero value blob.Ref {Unm,M}arshalJSON.

Fix for:
https://camlistore.org/issue/308

Change-Id: I910fc4f05015b0c3ddc57eba9d1d8fe1bfe992bf
2014-01-04 08:11:23 -08:00
Brian Gitonga Marete 97c52b7233 pkg/blob: Introuduce a type that wraps a blob ref with the blob data.
This new type (blob.Blob) will be used to implement a blob streaming
interface for the storage engines.

Change-Id: I05f3a68022368cf9c7b4a98cced3098ba3965799
2013-12-23 21:50:50 +03:00
Brad Fitzpatrick cde3176c0d blob: speed up parsing slightly.
BenchmarkParseBlob  2000000   727 ns/op   64 B/op   2 allocs/op
BenchmarkParseBlob  5000000   531 ns/op   64 B/op   2 allocs/op

Kinda pointless and mostly the result of a failed refactor anyway,
but checking it in.

Change-Id: I7cdd0a8c5d03d7e223626ef6ddb77d8d10dd06a7
2013-12-06 11:15:51 +04:00
Brad Fitzpatrick f1a13214af blob: implement the parser's buffer pool TODO
Change-Id: I24366c5f42fd18103df68676047c7e4cdf6b7af6
2013-12-06 10:19:20 +04:00
Brad Fitzpatrick 6c1df9d633 blob: add ParseBytes
Will be used by index->corpus slurping of blob meta rows on
start-up, to reduce garbage.

Change-Id: Ied6605bd44ae38537d3661579c7a8b6dea2b2d9e
2013-12-04 14:38:13 +01:00
Brad Fitzpatrick d324eecaba blob: add slice sorter types for []blob.Ref and []blob.SizedRef
Change-Id: I23224a9408a9d6fd2bbd7dd0fa94dba64f4586c1
2013-12-02 11:55:39 -08:00
Brad Fitzpatrick b0b054ca27 Doc tweaks
Change-Id: I7c4cdd9b0cc37b2a40084f1cf11b81d8ce3a39fa
2013-11-23 09:00:18 -08:00
Brad Fitzpatrick 0dfaad41a7 blob: add RefFromString
Change-Id: I1e98874fd43c1570c2e79907dbdbc7e44ca64bf7
2013-10-25 00:22:10 -07:00
mpl 99aaf3b55e client: take into account "selfPubKeyDir" from config when signing
http://camlistore.org/issue/210

Change-Id: Id6d75393a90bdf3141acc10fddc2b3cedee64422
2013-08-23 21:31:03 +02:00
Brad Fitzpatrick 9bfb89967c pkg/blob: optimize MarshalJSON.
No real reason. Just finishing a TODO.

Change-Id: I9266cee9d3872f9b27baa8287b967e9446e1314d
2013-08-23 13:41:20 -05:00
Brad Fitzpatrick ec4af83d19 pkg/blob: implement Go 1.2's encoding.BinaryMarshaler
Change-Id: Ic9fbbd4cdd99fbf19ed2467f9fc151c688ec048f
2013-08-23 13:30:18 -05:00
Brad Fitzpatrick 41fe3660c2 blob: add == test
Change-Id: If3a8096e885d51316e3841d9a6bde625ee76f51e
2013-08-03 22:25:39 -07:00
Brad Fitzpatrick 0bdf20884b all: delete pkg/blobref; convert all from *blobref.BlobRef to new blob.Ref
Change-Id: Id2dfb7f19452bedf4f3c9310b36227fd8117b225
2013-08-03 19:54:30 -07:00
Brad Fitzpatrick c44d89e22f blob: more porting blobref to new pkg blob
Change-Id: Id7f0dd80dd6ce5fa55a2c04e2e2a882a5ef8add6
2013-08-03 19:08:17 -07:00
Brad Fitzpatrick e56a944843 blob: replacement for *blobref.BlobRef type.
Shorter name (blob.Ref instead of *blobref.BlobRef) and value
semantics instead of pointer.

Also, equality (==) works on it, so it can be used as a map key.

It's also more memory efficient, for the upcoming search server that
needs to store tons (or all) of your blobrefs in memory.

Change-Id: I494a2d72d31865211fa388fedf938ed25f99ca13
2013-08-03 13:48:17 -07:00