* Update goleveldb to v1.0.0
* Update goleveldb to 64b5b1c739545ed311fb9d9924d19d188fabdc83
- fix data race (758128399b1df3a87e92df6c26c1d2063da8fabe)
- optimizations from bytes.Buffer (42cf80bcefdf184caead3785a11b06fe6dfe9649)
- fsync after creating new manifest (eb135432c5aa4c841c91f3fdc871f07a94aa5a44)
* Update all dependencies.
Keep google.golang.org/api on v0.42.0 as that's the latest not borking
servicemanagement/v1 APIService.Enable.
* Pin bazil.org to latest version that supports macOS
Support has been removed in mid-2018 - see
https://github.com/bazil/fuse/issues/224
for details.
* pkg/blobserver/sftp: Make it pass on Windows
Translate paths with filepath.ToSlash
* Remove macos test cache
* go mod tidy
Remove zembed*.go as pkg/fileembed is history
* Remove caching in tests-linux
Co-authored-by: Tamás Gulácsi <tamas@gulacsi.eu>
As demonstrated by the test case added in this commit, a
search panics if the DirConstraint contains a LogicalConstraint
with Op set to "not". This is because the isFileOrDirConstraint
function doesn't account for unset "B".
Change-Id: I111ace62486160254ba22ab1498de78cde2a5e9b
The old directory aspect (directory_detail.js), was done before the
current (React-based) look and layout of the web UI. It is still
functional, but does not look very good, and does not match the look of
the rest of the web UI.
Therefore, this change adds a new directory layout, with a
blob container approach and layout similar to the one for permanode
sets, and replaces the old one as the default Directory aspect.
It seems to have performance issues (slow loads), but I think it's
better to enable it now, rather than wait again for many months to
land it. Especially if it forces us to address said performance issues.
If it's too bad and users want the old one in the meantime, it should be
easy to re-enable the old as the default.
Change-Id: Ieb0cde8290f0a058008cdc73b596ad1a330ded1a
When looking for the children of a directory, we used
to return an os.ErrNotExist if no children were found.
We now only return os.ErrNotExist if the directory
itself does not exist.
Fixes#1152
Change-Id: I8dd827f5597535d98c898302d0a1ca5399b03476
The DirConstraint type already existed but wasn't functional as none of
the matching for any of its fields was implemented.
That functionality is required by two other features:
1) Browsing directories in the publisher requires getting information
about the children of the directory. In practice that can be achieved
with a search on the directory, accompanied with some describe rules.
But that comes with limitations, such as:
-no control over the sorting of the described children in the response
-max number of children in the response (and no way to overcome it since
you can't "continue" on a describe)
hence the need for a direct search of a directory's children.
This is implemented as DirConstraint/FileConstraint.ParentDir and will
be used in https://camlistore-review.googlesource.com/8286
2) Looking for files/directories shared by transitivity.
Knowing if an item is the target of a share claim is easy enough once
enough of ClaimConstraint is implemented. But in order to find out if an
item is effectively shared because one of its ancestors (directory) is
the target of a share claim, with transitivity set, we need some sort of
search that can do directories traversal. This is implemented as
DirConstraint.RecursiveContains and it will be used as a subquery in
https://camlistore-review.googlesource.com/9866.
Now implemented:
DirConstraint.FileName
DirConstraint.BlobRefPrefix
DirConstraint.ParentDir
DirConstraint.TopFileCount
DirConstraint.Contains
DirConstraint.RecursiveContains
ParentDir will also allow us to quit relying on the treeHandler in the
web UI,
and to use the more generic search queries mechanism instead.
Change-Id: I022cb51732ee4271906271fd75c6f737856b6165
Even when AltLookupFun was defined, Lookup used to fallthrough after the
AltLookupFn, depending on what AltLookupFn returned.
This means in some cases, our tests in pkg/search did end up calling the
Google API (which was not intended). We found out because with the new
dependency on an API key for the Google service, we now rely on reading
the key from the config dir, which is wired to panic if it is accessed
during tests.
This change therefore simplifies Lookup, by making it return the results
of the AltLookupFn call, if AltLookupFn exists.
Change-Id: Ie196f892f36d309c725a6e869094364d49cffdc8
We had checks in place to return an error if a search query had both
no constraint and no expression. However:
1) we accepted " " as a valid search expression (which is the
work-around that was used by the web UI for the index page)
2) the rest of the code seems to be coping well if we remove these
checks, which this CL does.
This allows us to remove the q=" " trick in the web UI, which does not
seem to be breaking anything, and has the side-effect of improving the
map aspect story (issue 1081). Because with this change, the index page
is not considered anymore as the result of a search query, which means
if one then clicks on an image, one then gets to the image permanode as
a "standalone" item, instead of as one item part of set.
As a consequence:
1) If one then clicks on the map aspect, one gets only this item on the
map (with the proper zoom level), instead of all the items from the
index page.
2) The other side of the coin is left/right navigation does not "work"
anymore in this context, since the item is not part of a set anymore.
In other words, this change makes the index page (i.e. the empty search)
a somewhat particular case, and makes it behave a little differently
from the result of any other (non-empty) search.
Updates issue #1081
Change-Id: Ibbb1711c67c7cf8adaf94ab5d2c68c3755e4983b
corpus.AppendPermanodeAttrValues and corpus.PermanodeAttrValue now take
a GPG key ID as their signer filter attribute, instead of a blob Ref.
Fix search package accordingly.
Fixes#1069
Change-Id: I0ef519fc16cf357adb84c32fa6e5bf92d8a0de26
Introduction of an Owner type in pkg/index.
Related changes in pkg/index and pkg/serverinit.
Fixes#1047
Change-Id: Ifc316865833349d5202b5085e8f2c1235f3f2220
- correct logging that logged functions instead of their value
- use ID vs Id naming
- use correct function names in comments
Change-Id: I61562cef7ebac7337ec6c85312cdf7915cb1a84b
when its wholeref index lines contain any "sha1-" wholeref lines.
Then, have the client use that info when uploading files to also
compute the sha-1 (in addition to the sha-224) and ask the server for
both.
By client, we mean changes to cmd/camput, to pkg/client functions (for
camput and other Go clients, such as the apps), and to the web UI.
Also, add a -sha1 mode to devcam server, as well as the
CAMLI_SHA1_ENABLED env var, to facilitate testing/debugging of the
legacy sha1 mode. Fix/document unrelated options in devcam server
while we're at it.
Change-Id: I369679d2a4719f6f8b409f78dd4d065e13a3f62f
I had intended for this to be a small change.
I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.
And then it got a bit infectious and ended up touching everything.
I ended up doing SubFetch in the process by necessity.
At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.
Updates #733
Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
Remove the blob.SHA{1,224}From{Bytes,String} constructors too. No
longer used. This adds blob.RefFromBytes which was missing. We had
blob.RefFromString. Now everything uses blob.RefFrom* instead of
specifying a hash function.
Some tests set a flag to force use of SHA-1 because there was too much
golden data to update. We can remove those one-by-one over time as we
fix up tests.
Updates #537
Change-Id: Ibe6428089a6221594c2b751f53f98b03b5a28dc2
Notably: pkg/misc all moves.
And pkg/googlestorage is deleted, since it's not used. Only the
x/net/http2/h2demo code used to use it, but that ended in
https://go-review.googlesource.com/33230 (our vendored code is old).
So just nuke that dir for now. When it's refreshed, it'll either be
gone (dep prune) or new enough to not need googlestorage.
Also move pkg/pools, pkg/leak, and pkg/geocode to internal.
More remains.
Change-Id: I2640c4d18424062fdb8461ba451f1ce26719ae9d
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
The search handler can store and retrieve search aliases.
Keyword namedSearch handles these new atoms of the form
named:foo.
Creating an alias has been implemented using a client in
the camtool subcommands named-search-get and named-search-set.
Change-Id: I7960f83bad464eb1a971c07f33631744a5eea814
This switches most usages of the pre-1.7 context library to use the
standard library. Remaining usages are in:
app/publisher/main.go
pkg/fs/...
Change-Id: Ia74acc39499dcb39892342a2c9a2776537cf49f1
-keep the browser URL bar in sync with the current search/zoom-level
-introduce the "map:" predicate, to be used as the current viewport in
the map aspect. This was previously achieved with the locrect predicate. We
try to keep this new predicate unknown to the server, and replaced on
the fly by an equivalent locrect predicate.
-stay on the map aspect when the search expression has a map predicate.
Same thing when loading a URL with a map predicate parameter: try to
load directly on map aspect, instead of going to search aspect first.
-make sure there's only at most one map query in flight at all times
Change-Id: Ibf928ee5c926670c221f212f14ca9799e1e7bb79
Avoid select overhead in hot paths. Just use funcs.
Also, for sort-by-map searches, don't do a describe and pass over all
the results doing location lookups a second time. Remember the
location from the initial matching. Cache it on the search value.
Reduces some sort-by-map searches from 10 seconds to 3 seconds for
me. (still too slow, but good start)
Change-Id: I632954738df9accd802f28364ed11e48ddba0d14
The "ref:prefix" search predicate is simply the equivalent of the
Constraint {
BlobRefPrefix: prefix,
}
search constraint.
The "ref:prefix" search expression was already supported by the search
box of the web UI, but as opposed to (all, I think) other search
expressions, it was not supported server-side. Which means, it had to be
converted to a search Constraint as the above, before being sent in the
query.
This change therefore fixes this inconsistency. In addition, but
relatedly, since the map aspect relies on expressing the zoom-level as a
locrect expression, it is much simpler if the search query it uses only
has to be constructed from search expressions, and without search
constraints. So if we want to be able to support marking a single node
search with the map aspect, while dealing only with search expressions,
this change is necessary.
Fixes#939
Change-Id: Ia58f410198ecd1f7e0981321da370d687df3a120
This change makes zooming and panning on the map aspect send a new
"MapSort" search query, so that the (1000, by default) most relevant
results for the currently displayed area always appear as markers
after a zoom/pan.
This required completely "lifting up" (in React lingo) the currentSearch
state out of the Header class, and into the Index class, which should
have already been done in a56372830a. This
is necessary because we need the expression in the search box to be mutable
both by the user('s input) and by the map aspect, which represents the
zoom level as a trailing "locrect" predicate in the search expression.
Fixes#938
Change-Id: I0004c9ff09f03b4f1d95a35e54605689eebf0c1a
The search for the Around blobRef was using the equality operator,
instead of an ordering operator, which was incorrect.
Also, if the Around blob.Ref had not been found during the search, we
were only returning early for special cases, which is incorrect. As soon
as we know that an Around blob is wanted, and we know it has not been
found, the search should abort.
Relatedly, a formal check to enforce that Around is incompatible with
MapSorted was added.
Finally, some tests to check the Around algorithm, in the case where we
don't use a sorted source, were added.
Change-Id: I94fc984cf4130badc879cdadaba718ef6361c9b7
luz paz