As the priority is to fix GCE instances, the port for the http-01
challenge is not configurable for now (80) even not on GCE, but it will
be in a follow-up change.
update golang.org/x/crypto/* (for acme) to rev
13931e22f9e72ea58bb73048bc752b48c6d4d4ac
update golang.org/x/sys/* to rev
fff93fa7cd278d84afc205751523809c464168ab (because unix is a dep of
crypto)
remove warning about Let's Encrypt security issue from pkg/deploy/gce
I had to manually exclude vendor/golang.org/x/crypto/acme/jws_test.go
for now because it contains a private key, and git whines about it, and
i could not override it.
Fixes#1033
Change-Id: Ie4f2049e97892dee9ab513300a5f12e64976aec8
This CL is about levelDB as the HaveCache for camput, and there are
several aspects to it. To describe it, I'll take the particular example
where you want to add many permanodes (~33k) to a given set, with
camput. Something like:
for _, blob := range blobs {
do("camput attr -add sha1-foobar camliMember " + blob)
}
In a "normal" levelDB use case, everytime the number of level-0 .ldb
files goes over 4 (by default), a background compaction task is
started to transform these SST into level-1 ones, and remove the level-0
ones.
However, since our particular camput call is very short lived
(especially on a local Perkeep), not only might there be not enough time
for the compaction to be triggered, but even if it is, when the DB is
flushed (on a Close call), any ongoing compactions are cancelled. This
makes level-0 compactions very unlikely to happen on short-lived camput
calls. As a result, the number of level-0 files keeps growing until
levelDB fails while trying to open them all, because it hits the current
process ulimit.
Now, in this CL, what we propose is to systematically force a compaction
as soon as the HaveCache is opened. It is not scheduled concurrently, so
we are sure that the compaction happens before the DB actually gets used
by camput. This seems to make sure that the number of level-0 tables
never grows too much. With this change, I was able to run the above
example on 33K blobs without hitting the ulimit error.
However, it should be noted that potential problems might remain. The
compaction for levels above 0 is triggered based only on the total size
of the level (e.g. at 100MB by default for level-1), and not on the
number of files. Since we're creating many tiny tables (basically 1
entry per table), the number of files grows very fast while the total
size does not, and the compaction does not get triggered, even if forced
with CompactRange. This does not seem to be a problem for our use case,
as levelDB does not seem to need to open many of the level-1 files at
the same time, so we're not hitting the ulimit problem because of that.
If needed, there's at least one way this problem (if it is one?) could
be fixed: make the compaction trigger on other conditions, such as
number of files per level. I've experimented with it (forcing the
level-1 compaction to trigger at the 100 files limit), and it seems to
be working. But I had to do change the goleveldb code itself, and I
don't think levelDB implementations are supposed to do that.
For information, at the end of the run on the 33K blobs:
$ du -sch *.ldb
...
83M total
$ ll | wc -l
20988
And indeed, when asking for leveldb.stats on the table:
Level | Tables | Size(MB) |
-------+------------+---------------+
0 | 1 | 0.00015 |
1 | 20981 | 3.47307 |
Also, update github.com/syndtr/goleveldb to
34011bf325bce385408353a30b101fe5e923eb6e
And remove github.com/syndtr/gosnappy as goleveldb does not use it
anymore.
Also apply this change to StatCache.
Fixes#1008
Change-Id: If9f790a003e67f3c075881470e52e5f2174afa73
This upgrades the markdown conversion library to v2 and
switches camweb to use the new, simplified API.
Change-Id: I710110431fdfa32104c591abdf9b01c74a214f48
move calculation of domainName into main rather than init. This is
necessary because the root var, which domainName is based on, is not
passed as a command line flag when run in production.
Also clean up a few minor things in godoc.go from previous CLs, such as
removing the docRx var which is no longer used.
Fixes#1036
Change-Id: I7145b87d5f1ed13e143bd3750ce22f2d41d6169d
Upgrades mysql driver to a versioned release, fixes a minor issue
in dbinit that caused SHOW DATABASES queries to fail and improves
error messages to help diagnose such problems.
Changelog: vendor/github.com/go-sql-driver/mysql/CHANGELOG.md
Revision: go-sql-driver/mysql@a0583e0143
Change-Id: I4f5fdee60b30eca85564c452f72774e9c15828f3
In rev: 837fe8ac46 a new
version of websocket was imported and changes were made to use
origin checking. This integration test was missed during those
changes.
This change fixes this test and adds the origin data to the
request, uses the correct 'ws' scheme and uses the new Dialer
API to make requests.
Change-Id: I93e8228794665012f15370532cdeda3cb702ea00
This blobserver is just "cat"ing the given "read" storages.
This is read-only, so you should use some other storage to augment this for
writing and removing - for example the "cond" storage is perfect for this.
My use-case is to use blobpacked with large=diskpacked, small=filesystem,
but consolidate the small blob storage into a diskpacked + filesystem
after the filesystem becomes huge.
Another use-case is joining separately built camlistore servers into one.
(For me, they have to be separated later, so I've built them separately,
but I've to use it joined for a month).
Change-Id: I4e7e42cd59286f0f34da2f6ff01e44439771d53c
Remove the blob.SHA{1,224}From{Bytes,String} constructors too. No
longer used. This adds blob.RefFromBytes which was missing. We had
blob.RefFromString. Now everything uses blob.RefFrom* instead of
specifying a hash function.
Some tests set a flag to force use of SHA-1 because there was too much
golden data to update. We can remove those one-by-one over time as we
fix up tests.
Updates #537
Change-Id: Ibe6428089a6221594c2b751f53f98b03b5a28dc2
Gopkg.toml / Gopkg.lock contain the full set of revisions used by
Perkeep now. You can remove your entire vendor tree and
dep ensure && dep prune
will restore you to a known good state.
This final change contains various files that were missing (like
LICENSE/README.md/tests) and also prunes a number of packages that
are transitively unused.
Fixes#889
Change-Id: I6ee6c4a4e14e2de6b6f33620eb7e2c44e97ce91b
This addresses a long-standing TODO in the BlobStatter interface to
clean it up. Just like all new Go programmers, I misused channels in
APIs. I should've cleaned this up years ago.
While here, I also added a context.
The rest should get contexts later.
This also cleans up a few things here & there.
The pkg/client statting no longer does batching, which added a lot of
complexity. There was a comment saying something like "once we have
SPDY, we can delete this". Well, we have HTTP/2 now, so seems
deletable.
All tests pass.
Change-Id: I034ce07d9b70e5cc9e5482213368993e638d4bc8
Uses revisions as close to current usage as possible.
Addresses issues with repos that have different per-directory
revisions.
- cloud.google.com/go
rev: b70ccc799b9d019708c3eb9395acef6e3f6b7bc8
- github.com/golang/protobuf
rev: 1e59b77b52bf8e4b449a57e6f79f21226d571845
- google.golang.org/api
rev: 48e49d1645e228d1c50c3d54fb476b2224477303
- google.golang.org/appengine
rev: 150dc57a1b433e64154302bdc40b6bb8aefa313a
version: v1.0.0
- google.golang.org/genproto
rev: 08f135d1a31b6ba454287638a3ce23a55adace6f
Addresses #889
Change-Id: I41610d2f409d7a558a59e3018094a124efc100c5
The test added for proper markdown title generation
do not work when run via devcam. This is caused by
depending on doc/uses.md as the input testdata.
devcam does not copy markdown files into the target tmp
directory so this test fails when run that way.
Solution is create a markdown file in a testdata directory.
These are copied to the target tmp directory and the test
succeeds.
Change-Id: Iae654dca2374b925e6c895a71d4d93b12ccc188b
It appears that markdown generation added class ids at some point.
This means that all pages on perkeep.org have the title "Perkeep"
Also added a test for this case.
This issue was flagged by the Google Webmaster Tools.
Change-Id: I7ddab56e03a506b84aab415d94856b8f1c126250
this is a straight revert of 02938281 plus the additional logic to
derive the domain name from the serving directory. This allows package
docs to also be served from localhost during development.
Updates #1010
Change-Id: I07f817edb0b83b03e363373945d143ed3cb82fbf
mpl