The directory created by `T.TempDir()` and is automatically removed when
the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Use modernc.org/sqlite
This way it can be enabled by default, as it is a cgo-free, Go-only package.
No need for build tags, conditional compilation (whether libsqlite3-dev is installed).
* make.go: Remove unused -sqlite flag
* Remove use of -sqlite flag
- Document that we use an '|indexed' suffix in the 'have:' key to mark
indexed blobs.
- Don't refer to GetOwnerClaims. This method no longer exists and was
replaced by AppendClaims. Don't bother naming the exact Index
interface method this key serves but instead focus on explaining what
information we store in that key.
Some code was merged without being fmt-compliant. This
PR runs `go fmt ./...` and adds a test to github workflows
to verify that pull requests are fmt-compliant.
Calling 'GetPermanodesWithAttr', as currently written, will return an
empty list for any unindexed parameters.
Rather than always returning an empty list if the user asks for a
unindexed attribute, error out.
Also add documentation of this potential foot-gun for users of the
client.
Change-Id: Iadc1b5a00aa709584affb89a1397e82aaffb692f
In order to facilitate manual recovery in severe scenarios, the
--keep-going flag will make the server come back online even when there
are index or recovery errors. This will allow further querying on the
unhealthy server to identify missing data or to extract data for
migration to a new/healthy instance.
Fixes#1166
Change-Id: I4153905757d9e7d8014780dd5660a862cb8ba1ab
When looking for the children of a directory, we used
to return an os.ErrNotExist if no children were found.
We now only return os.ErrNotExist if the directory
itself does not exist.
Fixes#1152
Change-Id: I8dd827f5597535d98c898302d0a1ca5399b03476
The DirConstraint type already existed but wasn't functional as none of
the matching for any of its fields was implemented.
That functionality is required by two other features:
1) Browsing directories in the publisher requires getting information
about the children of the directory. In practice that can be achieved
with a search on the directory, accompanied with some describe rules.
But that comes with limitations, such as:
-no control over the sorting of the described children in the response
-max number of children in the response (and no way to overcome it since
you can't "continue" on a describe)
hence the need for a direct search of a directory's children.
This is implemented as DirConstraint/FileConstraint.ParentDir and will
be used in https://camlistore-review.googlesource.com/8286
2) Looking for files/directories shared by transitivity.
Knowing if an item is the target of a share claim is easy enough once
enough of ClaimConstraint is implemented. But in order to find out if an
item is effectively shared because one of its ancestors (directory) is
the target of a share claim, with transitivity set, we need some sort of
search that can do directories traversal. This is implemented as
DirConstraint.RecursiveContains and it will be used as a subquery in
https://camlistore-review.googlesource.com/9866.
Now implemented:
DirConstraint.FileName
DirConstraint.BlobRefPrefix
DirConstraint.ParentDir
DirConstraint.TopFileCount
DirConstraint.Contains
DirConstraint.RecursiveContains
ParentDir will also allow us to quit relying on the treeHandler in the
web UI,
and to use the more generic search queries mechanism instead.
Change-Id: I022cb51732ee4271906271fd75c6f737856b6165
A bunch of mine had a larger initial ftyp box, which broke the second
part of the pattern. But the second part of the pattern doesn't matter
anyway. This only needs to casually recognize them. A later full
parse will determine what they really are.
This also adds some new debugging when CAMLI_DEBUG is true.
Change-Id: Ib4adc9b5447a64ba4682624e42b55f1d65779ef7
If a GPS coordinate is bogus, we should have caught it when receiving
the blob, and not write it out as an index row.
If we did not, we end up catching it when trying to slurp it to the
corpus. At this point there's no reason to penalize the user by
panicking, we can instead skip it and log about the error.
Change-Id: Ie40479befca78d8b5da343ef1a8420f703f07cb7
the changes to DecodeConfig in ce955adf48
moved the call to image.DecodeConfig to before the call to exif.Decode,
because we need to do that for the HEIF case.
After the call to exif.Decode, if err is nil and is not a short read
error, we ignore the error and the code goes on. To mark that intent,
we should have explicitly set the err to nil at this point. However,
that omission did not matter before
ce955adf48 because image.DecodeConfig
was the last setter of the error.
Now that it has moved up, we do have to explicitly reset the err to nil.
While we were at it, we rewrote the whole logic to be easier to read.
This seems to fix all the tests in pkg/index and pkg/search.
Change-Id: Id0b711179ccb1365b851d3e86ecb70d47ac358b4
corpus.AppendPermanodeAttrValues and corpus.PermanodeAttrValue now take
a GPG key ID as their signer filter attribute, instead of a blob Ref.
Fix search package accordingly.
Fixes#1069
Change-Id: I0ef519fc16cf357adb84c32fa6e5bf92d8a0de26
The current maximum size for a schema blob is 1MB. For a large enough
directory (~20000 children), the resulting static-set JSON schema is
over that maximum size.
We could increase that maximum, but we would eventually hit the maximum
blob size (16MB), which would only allow for ~300000 children. Even if
that is an uncommon size, it is technically possible to have such large
directories, so I don't think it would be reasonable to restrict users
to such a limit. So it does not seems like enough of a solution.
The solution proposed in this CL is to spread the children of a
directory (when they are more numerous than a given maximum, here set to
10000) onto several static-sets, recursively if needed. These
static-sets (subsets of the whole lot of children) are stored in the new
"mergeSets" field of their parent static-set schema. The actual fileRefs
or dirRefs, are still stored in the "members" field of the subset they were
spread in. The "mergeSets" and "members" field of a static-set are therefore
mutually exclusive.
Fixes#924
Change-Id: Ibe47b50795d5288fe904d3cce0cc7f780d313408
Introduction of an Owner type in pkg/index.
Related changes in pkg/index and pkg/serverinit.
Fixes#1047
Change-Id: Ifc316865833349d5202b5085e8f2c1235f3f2220
- correct logging that logged functions instead of their value
- use ID vs Id naming
- use correct function names in comments
Change-Id: I61562cef7ebac7337ec6c85312cdf7915cb1a84b
when its wholeref index lines contain any "sha1-" wholeref lines.
Then, have the client use that info when uploading files to also
compute the sha-1 (in addition to the sha-224) and ask the server for
both.
By client, we mean changes to cmd/camput, to pkg/client functions (for
camput and other Go clients, such as the apps), and to the web UI.
Also, add a -sha1 mode to devcam server, as well as the
CAMLI_SHA1_ENABLED env var, to facilitate testing/debugging of the
legacy sha1 mode. Fix/document unrelated options in devcam server
while we're at it.
Change-Id: I369679d2a4719f6f8b409f78dd4d065e13a3f62f
Follow-up of ec66bcc871
This change should be the last of its kind, in the corpus at least, and
should make most searches and the web UI usable again, with both kinds
of hashes (sha1 and sha224).
Updates #537
Change-Id: Icfe9e8aaab031313612c555b7601895aeba16a7c
Follow-up of ec66bcc871
I had run the pkg/search tests to see if no high-level search was
broken, for forgot about pkg/index itself, whoops.
Change-Id: Iec3aa28aca82ac6773983ea9df9ade26a48fc4a7
Otherwise claims that are actually from the same signer, end up being
treated as from different signers, because some of the claims were
signed by the sha1 version and others by the sha224 one.
TODO in follow-up CLs: similar fixes in rest of the corpus, such as with
claimPtrsAttrValue. See if non-corpus index functions/methods suffer
from the same problem.
Change-Id: Icbc70e97edc569f46e575d79aaf4359b33996053
I had intended for this to be a small change.
I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.
And then it got a bit infectious and ended up touching everything.
I ended up doing SubFetch in the process by necessity.
At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.
Updates #733
Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
Remove the blob.SHA{1,224}From{Bytes,String} constructors too. No
longer used. This adds blob.RefFromBytes which was missing. We had
blob.RefFromString. Now everything uses blob.RefFrom* instead of
specifying a hash function.
Some tests set a flag to force use of SHA-1 because there was too much
golden data to update. We can remove those one-by-one over time as we
fix up tests.
Updates #537
Change-Id: Ibe6428089a6221594c2b751f53f98b03b5a28dc2
This addresses a long-standing TODO in the BlobStatter interface to
clean it up. Just like all new Go programmers, I misused channels in
APIs. I should've cleaned this up years ago.
While here, I also added a context.
The rest should get contexts later.
This also cleans up a few things here & there.
The pkg/client statting no longer does batching, which added a lot of
complexity. There was a comment saying something like "once we have
SPDY, we can delete this". Well, we have HTTP/2 now, so seems
deletable.
All tests pass.
Change-Id: I034ce07d9b70e5cc9e5482213368993e638d4bc8
luz paz