We've had issue #756 making reindexing tests fail for so long now, that
having Travis CI monitor builds has become useless, since we don't
expect any other result than a fail.
This CL skips the Reindexing tests when on Travis, so that it becomes
useful again in helping us catching regressions.
Change-Id: I7b837f62a4e2d7471a08155346150130af74f48f
As Let's Encrypt DNS server (Unbound) is pretty strict, it
wouldn't resolve names camlistore.net until we implemented more of the
DNS protocol and fix various things.
Since I had no way at first to know what exactly it didn't like, I
started by fixing all errors and warnings reported at:
http://dnsviz.net/d/camlistore.net/dnssec/
Therefore, this CL adds:
-TCP support
-NS response
-SOA response
-MX (empty) response
-DNSKEY (empty) response
-TXT (empty) response
-explicit non-support of EDNS
Then I found out we also needed this:
-https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00
-CAA response. A proper response is optional here, as Let's Encrypt only
enforces the response if it finds one. But we do have to reply.
Fixes#867
Change-Id: Ib45f8a642cd83cf19c8ab36435644a2c645a70e7
On some devices, updating notifications and status text at a high rate
causes whole-system UI lag and even complete system freezes. On a
Nexus 5, the uploader is completely unusable and requires a hard phone
restart almost every time "upload all" is triggered from the UI. The
device freezes happen across multiple stock Android releases.
This commit reduces the update rate of the UI significantly by updating
the notification only when the integer percent value of the upload
progress changes. It also reduces the update rate of the status text to
around 30 fps, which should still be plenty. With both of the changes,
the uploader behaves as expected. Either change alone is not enough to
get rid of the device freezes.
Change-Id: I2c170772d29c4670c8c10d09cdaea96207fc9e61
vendor: add golang.org/x/crypto/acme/{autocert}
at 8e06e8ddd9629eb88639aba897641bff8031f1d3
remove rsc.io/letsencrypt and its deps.
Issue #859
Change-Id: I292543e88476bd4658846496f96a2a07bb25a186
Before that change, one would only get a response to a DNS query, if
directly sending the query to the dns server.
I think the main problem was that one has to copy the question section
of the query in the response message, which is part of what SetReply
does.
With this change, I seem to get responses (for e.g. www.camlistore.net
and 6401800c.camlistore.net), without having to specify a server to
query to dig (or nslookup), so by letting the resolver and the DNS
hierarchy follow its normal course.
Issue #722
Change-Id: I530529920fdaa3e5fb025c7684b563e3f05772ac
server/camnetdns: update to cloud.google.com/go imports.
vendor: add github.com/miekg/dns at rev
3f1f7c8ec9ead89493df11f2c3d8bec353a2c2c0 as a dep of server/camnetdns
Change-Id: Ief8e0ad3e09a6c3511c72a7991309808f855d3df
This change follows the changes at
ab06dbd80d
google.golang.org/genproto/googleapis/datastore updated to
44808ed2d86e258615bb701d395cbbfe6686a3e6
As a consequence:
google.golang.org/genproto/protobuf added at
44808ed2d86e258615bb701d395cbbfe6686a3e6
google.golang.org/genproto/googleapis/api/{label, metric, monitoredres,
serviceconfig} added/updated to 44808ed2d86e258615bb701d395cbbfe6686a3e6
github.com/golang/protobuf/ptypes{any, struct, timestamp, wrappers}
added/updated to 1f49d83d9aa00e6ce4fc8258c71cc7786aec968a
Test locally, and in "pseudo-production", i.e. with -cloudlaunch but with
a different projectID (hence with some of the hard-coded stuff changed).
Change-Id: Iad0caa78b19619bf153bf05870e6cb3a2ab669dd
When fetching shared blobs, we rely on the share chain to verify if a
blob can be reached. This chain is updated whenever we fetch an
additional link of the chain, by updating the Client.via map. However,
when some blobs of the chain are already cached in camget's DiskCache,
because we get them from the cache, we don't fetch them with
Client.FetchVia, which means the Client.via map isn't updated. And thus
the chain is broken.
This change adds Client.UpdateShareChain, and sets it as a hook to be
called by the CachingFetcher in the event of a cache hit. That way, we
ensure that the share chain is updated even when we get blobs from the
cache (instead of from the Client).
We also add a mutex to guard Client.via, because it is accessed by
concurrent smartFetch calls in case of a static-set.
As FetchVia was undocumented and not used by anyone, I made it
unexported. We can always export it again later when needed.
Fixes#856
Change-Id: I767cbec4b6f382cbccc25c0b97782b2a7472deb8
We would have needed a sharing test with a file large enough to have
several parts to detect the bugs introduced in rev
383f118ccf and fixed in rev
ac4f053414
The sharing test introduced in this change is such a test.
Fixes#527
Change-Id: I7cd8a0b9b8e2acf794723f30ae38bd97d9eced80
at rev 63cb68f1e3834e44683ca062ddf06cb9a889380a
Forgot to add it to the commit at
ab06dbd80dFixes#854
Change-Id: Ic8a9a3d0fd279b2bcf20c1dd77bee56512ae3f39
rm google.golang.org/cloud
add cloud.google.com/go at a47b182e769f5e75f5fc927ff6ee2678f7f552cf
update google.golang.org/api to 63cb68f1e3834e44683ca062ddf06cb9a889380a
update google.golang.org/grpc to
0e6ec3a4501ee9ee2d023abe92e436fd04ed4081
update go4.org to f5283521d7365fb2875408726e9cbf349f173767
fix in cmd/ pkg/ server/
TODO(mpl): fix misc/docker tools as well. next CL.
Fixes#832
Change-Id: I842b968a0afea8a5822913bd614d67cdbe50ee63
When receiving a file, we were only trying to guess its MIME type
through its contents (pkg/magic). We're now making a better effort at it
by guessing from the filename extension if needed.
Also:
pkg/magic: get rid of all the extra video extensions that are already
covered by mime.TypeByExtension. Because it's redundant and
confusing.
app/publisher, pkg/types/camtypes: also use mime.TypeByExtension as an
extra effort. Especially since a reindex would be necessary to benefit
from the pkg/index change.
There are other places in Camlistore that could use such an effort.
Maybe we should have a camtypes.*FileInfo.MIME() method that tries all
the ways to guess the MIME type of the file?
Change-Id: Ib9a2bc42af77c5394dac578ae415524b5111ad4e
All the camweb features depending on docker had been stuck for a couple
of weeks because one of the docker containers was wedged somehow. I
don't know exactly what was wrong, but I figure it can't hurt to add the
same kind of cleanup that we have for the demo blobserver.
So in this CL, we name each container running a git command, so we can
stop and remove said container the next time we run the same command.
Change-Id: I66592fbbde73ea30e4cee7477ada450e0c6a645e
Because our go generate line is:
//go:generate go run gensearchtypes.go -out zsearch.go
which will run a binary of gensearchtypes.go built for whatever $GOOS is
set to. Which will fail if $GOOS is different from runtime.GOOS (the
cross-compiling case).
I suppose it means that the day pkg/search becomes GOOS differentiated,
we may have to introduce an -os flag to gensearchtypes.go, since it
calls go doc on camlistore.org/pkg/search, whose output might depend on
GOOS?
Change-Id: I1ea32bb9190300120887ee8614dcdd2d1391a954
To decide whether a search submitted to the app search proxy is allowed,
we compare its results to the domain blobs, result of the master query,
that we cache when the master query is set.
However, since the results of the master query are liable to change when
new blobs arrive (e.g. a new camliMember is added to the set that is
published), that cache may need to be invalidated. Otherwise, we might
reply with a 403 to search query that is actually allowed.
Therefore, this CL adds a refresh of the cache on two instances:
-When the app handler gets a search query that seems to be forbidden.
Before replying with a 403, we refresh the cache with the master query,
and recheck whether the search query is allowed.
-When the publisher gets a request for a "members" page, or the "file"
page, it preemptively asks the app handler to refresh. Now that a lot of
the client workflow has been moved to javascript/the browser, these
kinds of requests should not happen too often, so it seems a reasonable
place to ask for a refresh. But this might change, so we should of
course be careful not to flood the app handler with refresh requests in
the future.
In any case, the app handler is suppressing the refresh requests, so
that it does not perform refreshes at more that one per minute.
As a smarter approach, we could later imagine a way for the app handler
to be aware of when new blobs get to the blobserver (akin to the blob
hub that the sync handler uses?), so that it only ever refreshes when
needed.
Fixes#851
Change-Id: Idc14cce5018053deac01ec454e5c936ed93e5a05
Add Handler.GetPermanodeLocation, based on the existing logic of loc:* and has:loc
search predicates, and that of index/Corpus.PermanodeLatLong:
1. Permanode attributes "latitude" and "longitude"
2. Referenced permanode attributes (eg. for "foursquare.com:checkin"
its "foursquareVenuePermanode")
3. Location in permanode camliContent file metadata
The sources are checked in this order, the location from
the first source yielding a valid result is returned.
camtypes: add new Location type
index: Add GetFileLocation to Index/Interface, to make
indexed location info accessible without a corpus.
This was unlike other file metadata like image info or media tags
which had accessors in both Index and Corpus.
Fixes#777
Change-Id: I63cf143d67a12732ca2c941de64b63736be5de6e
So far only images were served with their MIME types set properly, so
they would display directly in the browser, instead of being served as a
file download.
Now the same is done for a subset of text types: i.e. text/plain,
text/html, text/xml, and text/json. Aside from the browsing convenience,
the obvious advantage is being able to serve HTML directly, which should
allow us to build other things on top of the publisher.
Also a bit of related refactoring: moving the extension matching to
pkg/magic
Change-Id: Id98065c7c685036a272d1d2e293bfcbca33015ee
mpl