The current maximum size for a schema blob is 1MB. For a large enough
directory (~20000 children), the resulting static-set JSON schema is
over that maximum size.
We could increase that maximum, but we would eventually hit the maximum
blob size (16MB), which would only allow for ~300000 children. Even if
that is an uncommon size, it is technically possible to have such large
directories, so I don't think it would be reasonable to restrict users
to such a limit. So it does not seems like enough of a solution.
The solution proposed in this CL is to spread the children of a
directory (when they are more numerous than a given maximum, here set to
10000) onto several static-sets, recursively if needed. These
static-sets (subsets of the whole lot of children) are stored in the new
"mergeSets" field of their parent static-set schema. The actual fileRefs
or dirRefs, are still stored in the "members" field of the subset they were
spread in. The "mergeSets" and "members" field of a static-set are therefore
mutually exclusive.
Fixes#924
Change-Id: Ibe47b50795d5288fe904d3cce0cc7f780d313408
when its wholeref index lines contain any "sha1-" wholeref lines.
Then, have the client use that info when uploading files to also
compute the sha-1 (in addition to the sha-224) and ask the server for
both.
By client, we mean changes to cmd/camput, to pkg/client functions (for
camput and other Go clients, such as the apps), and to the web UI.
Also, add a -sha1 mode to devcam server, as well as the
CAMLI_SHA1_ENABLED env var, to facilitate testing/debugging of the
legacy sha1 mode. Fix/document unrelated options in devcam server
while we're at it.
Change-Id: I369679d2a4719f6f8b409f78dd4d065e13a3f62f
DBNames is supposed to provide configuration for the various databases
names. However,
1) I contend that nobody needs or wants to configure them as long as we
provide sane defaults.
2) it seems the only obvious user we have for this is to set up some of
the names on GCE.
3) having another external source for names complicates the code
further, especially when we already have the distinction between
database names for DBMS and file names for file-based databases.
4) writing a correct documentation for it is awkward.
Therefore, in this CL, I propose that we remove DBNames. Instead,
genconfig.go now sets some consistent default names for the various
queues and indexes set up on a DBMS (MySQL, PostGres, Mongo). To that
end, we introduce the new, but optional, DBUnique configuration
parameter, that is used as a part of all the database names, in order to
be able to run several Perkeep instances on the same DBMS, without name
conflicts.
In addition, the queue for the bs->index synchandler is now set up on
the same DBMS that is already in use for the index itself, instead of
using a file-base database.
And i think we could proceed likewise for the other queues.
Fixes#951
Change-Id: Ib6a638f088a563d881e3957e4042e932382b44f4
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
Remove gerrit redirect for /doc/* links, and instead link directly to
the on-site doc page. This allows us to remove special handling of any
/gw/doc/* URLs. However, we also need to add special handling for
/gw/server/* URLs to fix a couple of broken links on /doc/status.md.
Change-Id: Ica73c3d78b47dd363c5c3e736123af2977dedb0d
Vendoring:
- update gopherjs to rev: b40cd48c38f9a18eb3db20d163bad78de12cf0b7
- as usual, replace the fsnotify import statements with our dummy
pkg/misc/fakefsnotify, as described in
https://camlistore-review.googlesource.com/c/9409
- add vendor/golang.org/x/tools/go/gcimporter15 dependency from
rev: e531a2a1c15f94033f6fa87666caeb19a688175f
- make.go + .travis.yml: make go1.9 the required version
- Update comments on doc/environments-vars.md for CAMLI_GOPHERJS_GOROOT
Fixes#948Fixes#949
Change-Id: I9285aebadad1502708d164a9dc69e044a325cb3a
-keep the browser URL bar in sync with the current search/zoom-level
-introduce the "map:" predicate, to be used as the current viewport in
the map aspect. This was previously achieved with the locrect predicate. We
try to keep this new predicate unknown to the server, and replaced on
the fly by an equivalent locrect predicate.
-stay on the map aspect when the search expression has a map predicate.
Same thing when loading a URL with a map predicate parameter: try to
load directly on map aspect, instead of going to search aspect first.
-make sure there's only at most one map query in flight at all times
Change-Id: Ibf928ee5c926670c221f212f14ca9799e1e7bb79
This change adds the "locrect" search predicate, which works like the
"loc" predicate, except it allows to specify the coordinates of a
rectangular area as a location, instead of a named location.
The coordinates are the latitude and longitude of the North-East corner
of the rectangle, followed by the latitude and longitude of the
South-West corner. For example: locrect:48.63,-123.37,46.59,-121.28
Related: issue #934
Change-Id: I0cf39c1d0b49d557b2081f07b2c8b4508ccfc052
rev b9bcb1da229a59cc1e1d168401662cb6450aae08
Also added pkg/misc/fakefsnotify just so we only have to do:
- "github.com/fsnotify/fsnotify"
+ fsnotify "camlistore.org/pkg/misc/fakefsnotify"
in vendor/github.com/gopherjs/gopherjs/build/build.go
in order to avoid vendoring fsnotify, without having to grossly
patch the gopherjs code as we previously did.
Change-Id: Ic106a6afa1117d2061415355c236433f835895f1
mpl