Commit Graph

15 Commits

Author SHA1 Message Date
mpl db2604f981 pkg/schema: break static-sets in subsets for large directories
The current maximum size for a schema blob is 1MB. For a large enough
directory (~20000 children), the resulting static-set JSON schema is
over that maximum size.

We could increase that maximum, but we would eventually hit the maximum
blob size (16MB), which would only allow for ~300000 children. Even if
that is an uncommon size, it is technically possible to have such large
directories, so I don't think it would be reasonable to restrict users
to such a limit. So it does not seems like enough of a solution.

The solution proposed in this CL is to spread the children of a
directory (when they are more numerous than a given maximum, here set to
10000) onto several static-sets, recursively if needed. These
static-sets (subsets of the whole lot of children) are stored in the new
"mergeSets" field of their parent static-set schema. The actual fileRefs
or dirRefs, are still stored in the "members" field of the subset they were
spread in. The "mergeSets" and "members" field of a static-set are therefore
mutually exclusive.

Fixes 

Change-Id: Ibe47b50795d5288fe904d3cce0cc7f780d313408
2018-02-09 01:36:38 +01:00
Brad Fitzpatrick 194d4f9443 blobserver, all: add contexts to ReceiveBlob, Fetch & million resulting deps
I had intended for this to be a small change.

I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.

And then it got a bit infectious and ended up touching everything.

I ended up doing SubFetch in the process by necessity.

At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.

Updates 

Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
2018-01-18 16:22:16 -08:00
Brad Fitzpatrick 57648c6b83 all: update copyright holder from Google Inc to The Perkeep Authors
The AUTHORS file is the list of copyright holders.
2018-01-03 16:52:49 -08:00
Brad Fitzpatrick d6a0b05df0 Rename import paths from camlistore.org to perkeep.org.
Part of the project renaming, issue .

After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.

This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.

Also, this only moves the lru package to internal. More will move to
internal later.

Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.

This updates some docs, but not all.

devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).

Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
2018-01-01 16:03:34 -08:00
Filippo Valsorda 3d4b74d86e schema/dirreader: limit I/O concurrency in Readdir
The comment was right: a directory with too many entries depleted all my
file descriptors.  I want my file descriptors back.

Change-Id: Iacdd95f06a7f5b2a2e7d4676b17eb33f86060ad2
2016-05-22 00:27:49 +01:00
aarzilli b92df8065d DirReader.Readdir incorrectly logs the contents of err where it should really log the contents of res.err
Change-Id: Ic607d823af4b6147a54f896ffabaabf67b7b1c02
2014-11-24 18:41:50 +01:00
Jingguo Yao 9b6a9c587a schema: fix typos in error messages
1. Error messages in dirreader should mention dirreader instead of
   filereader.
2. Remove \n at the ends of error messages.
3. Use errors.New for plain text error messages.
4. Give precise static-set information in one error message.

Change-Id: Iae0f46bca492e9dbbe41b4c1bebf5f5950d722b5
2014-11-03 20:44:57 +08:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
Brad Fitzpatrick 740b8eec48 Close static-set blob after done parsing it.
Fixes http://camlistore.org/issue/230 I think.

No test, though.

Change-Id: I3b23d7e4c39bd19ba6ca9e33ec382fe3e0fd33f2
2013-09-20 12:23:28 +01:00
Brad Fitzpatrick 0bdf20884b all: delete pkg/blobref; convert all from *blobref.BlobRef to new blob.Ref
Change-Id: Id2dfb7f19452bedf4f3c9310b36227fd8117b225
2013-08-03 19:54:30 -07:00
Brad Fitzpatrick d2bc6f6cd3 fs: use schema.DirReader, rather than doing it by hand
Change-Id: I7e79621d621314ed1f50219f0cafc44b1b82ab1f
2013-02-18 10:17:38 -08:00
Brad Fitzpatrick 9efdf83d3e schema: use better types in superset: blobref and Time3339
Change-Id: I7004220789e1048c1ebcc8703bf2948a733bae10
2013-02-10 12:58:51 -08:00
Brad Fitzpatrick 070592444a schema: unexport Superset
Change-Id: If45569b1b6d72b1cb6450b9d634f59e88fae4851
2013-01-22 10:32:15 -08:00
Brad Fitzpatrick cfc32e4a05 schema, camget: more work on deleting the Superset type.
not much more remains.

Change-Id: I6cfe4145f67b100a0e2509f88ce6e1c580b7f9fe
2013-01-22 09:32:40 -08:00
Brad Fitzpatrick 997e52d6b1 schema: split DirReader off into its own file
Change-Id: I69fcbe43912f37cc32567b3d46cabe8579ae00de
2013-01-03 17:16:26 -08:00