It is being automatically removed by dep ensure since
3dfd895def but its removal breaks the web
UI.
This change alone is therefore probably not enough, we need to teach dep not
to remove it.
Change-Id: I61d50e703e786fdebc10446784dc84d43a0c1620
Use the now renamed libsqlite3 build pragma. This requires
having sqlite3 installed on the system instead of using the
c code bundled in vendor (which is removed by the dep tool
by default)
1.6.0 corresponds with rev
github.com/mattn/go-sqlite3@6c771bb9887719704b210e87e934f08be014bdb1
Changelog: See https://github.com/mattn/go-sqlite3/releases
Change-Id: I48fc0bc005715b83805837f544e7fa7875e6d56a
As the priority is to fix GCE instances, the port for the http-01
challenge is not configurable for now (80) even not on GCE, but it will
be in a follow-up change.
update golang.org/x/crypto/* (for acme) to rev
13931e22f9e72ea58bb73048bc752b48c6d4d4ac
update golang.org/x/sys/* to rev
fff93fa7cd278d84afc205751523809c464168ab (because unix is a dep of
crypto)
remove warning about Let's Encrypt security issue from pkg/deploy/gce
I had to manually exclude vendor/golang.org/x/crypto/acme/jws_test.go
for now because it contains a private key, and git whines about it, and
i could not override it.
Fixes#1033
Change-Id: Ie4f2049e97892dee9ab513300a5f12e64976aec8
This CL is about levelDB as the HaveCache for camput, and there are
several aspects to it. To describe it, I'll take the particular example
where you want to add many permanodes (~33k) to a given set, with
camput. Something like:
for _, blob := range blobs {
do("camput attr -add sha1-foobar camliMember " + blob)
}
In a "normal" levelDB use case, everytime the number of level-0 .ldb
files goes over 4 (by default), a background compaction task is
started to transform these SST into level-1 ones, and remove the level-0
ones.
However, since our particular camput call is very short lived
(especially on a local Perkeep), not only might there be not enough time
for the compaction to be triggered, but even if it is, when the DB is
flushed (on a Close call), any ongoing compactions are cancelled. This
makes level-0 compactions very unlikely to happen on short-lived camput
calls. As a result, the number of level-0 files keeps growing until
levelDB fails while trying to open them all, because it hits the current
process ulimit.
Now, in this CL, what we propose is to systematically force a compaction
as soon as the HaveCache is opened. It is not scheduled concurrently, so
we are sure that the compaction happens before the DB actually gets used
by camput. This seems to make sure that the number of level-0 tables
never grows too much. With this change, I was able to run the above
example on 33K blobs without hitting the ulimit error.
However, it should be noted that potential problems might remain. The
compaction for levels above 0 is triggered based only on the total size
of the level (e.g. at 100MB by default for level-1), and not on the
number of files. Since we're creating many tiny tables (basically 1
entry per table), the number of files grows very fast while the total
size does not, and the compaction does not get triggered, even if forced
with CompactRange. This does not seem to be a problem for our use case,
as levelDB does not seem to need to open many of the level-1 files at
the same time, so we're not hitting the ulimit problem because of that.
If needed, there's at least one way this problem (if it is one?) could
be fixed: make the compaction trigger on other conditions, such as
number of files per level. I've experimented with it (forcing the
level-1 compaction to trigger at the 100 files limit), and it seems to
be working. But I had to do change the goleveldb code itself, and I
don't think levelDB implementations are supposed to do that.
For information, at the end of the run on the 33K blobs:
$ du -sch *.ldb
...
83M total
$ ll | wc -l
20988
And indeed, when asking for leveldb.stats on the table:
Level | Tables | Size(MB) |
-------+------------+---------------+
0 | 1 | 0.00015 |
1 | 20981 | 3.47307 |
Also, update github.com/syndtr/goleveldb to
34011bf325bce385408353a30b101fe5e923eb6e
And remove github.com/syndtr/gosnappy as goleveldb does not use it
anymore.
Also apply this change to StatCache.
Fixes#1008
Change-Id: If9f790a003e67f3c075881470e52e5f2174afa73
This upgrades the markdown conversion library to v2 and
switches camweb to use the new, simplified API.
Change-Id: I710110431fdfa32104c591abdf9b01c74a214f48
Upgrades mysql driver to a versioned release, fixes a minor issue
in dbinit that caused SHOW DATABASES queries to fail and improves
error messages to help diagnose such problems.
Changelog: vendor/github.com/go-sql-driver/mysql/CHANGELOG.md
Revision: go-sql-driver/mysql@a0583e0143
Change-Id: I4f5fdee60b30eca85564c452f72774e9c15828f3
Gopkg.toml / Gopkg.lock contain the full set of revisions used by
Perkeep now. You can remove your entire vendor tree and
dep ensure && dep prune
will restore you to a known good state.
This final change contains various files that were missing (like
LICENSE/README.md/tests) and also prunes a number of packages that
are transitively unused.
Fixes#889
Change-Id: I6ee6c4a4e14e2de6b6f33620eb7e2c44e97ce91b
Uses revisions as close to current usage as possible.
Addresses issues with repos that have different per-directory
revisions.
- cloud.google.com/go
rev: b70ccc799b9d019708c3eb9395acef6e3f6b7bc8
- github.com/golang/protobuf
rev: 1e59b77b52bf8e4b449a57e6f79f21226d571845
- google.golang.org/api
rev: 48e49d1645e228d1c50c3d54fb476b2224477303
- google.golang.org/appengine
rev: 150dc57a1b433e64154302bdc40b6bb8aefa313a
version: v1.0.0
- google.golang.org/genproto
rev: 08f135d1a31b6ba454287638a3ce23a55adace6f
Addresses #889
Change-Id: I41610d2f409d7a558a59e3018094a124efc100c5
Remove goexif local changes (a TODO, README and gofmt)
Update to rev: 709fab3d192d7c62f86043caff1e7e3fb0f42bd8
Use dep to prune unused files
Change-Id: Icce8885917b798b25eaeed4c98ff393ce0db66b8
This pulls in a clean upstream version of go-sqlite3 at revision
919cf4144a09abfd877330bbe735f3a243a84537 using the dep tool.
The current method of using the -sqlite flag in make.go appears to
enable conditional builds with/without sqlite3, obsoleting the
need for local modifications.
We also manually remove sqlite
Addresses: #889
Change-Id: I4d2c107cd12608ac0bcb7c0607629ecb2d7f880f
The mongo integration was using a very old package. It's using
a new namespace now. Upgrade and adjust all call points
Removes labix.org/v2/mgo
Introduces gopkg.in/mgo.v2 from branch v2 with revision
3f83fa5005286a7fe593b055f0d7771a7dce4655
Change-Id: I2784fca941998460f58e0ac8d3d51286401590b5
Various text packages were previously imported from
rev 88f656faf3f37f690df1a32515b479415e1a6769
This includes the dep included/pruned version of x/text so that
we can update golang.org/x/net to the latest revision.
Addresses: #889
Change-Id: I54e8b02aef70d16519801b8fafd6bfd5548af4cd
This syncs golang.org/x/image using the dep tool to the most recent set of
changes from upstream.
Addresses: #889
Change-Id: I2bfde5a1c146656365062d73773ac2754df591b9
Notably: pkg/misc all moves.
And pkg/googlestorage is deleted, since it's not used. Only the
x/net/http2/h2demo code used to use it, but that ended in
https://go-review.googlesource.com/33230 (our vendored code is old).
So just nuke that dir for now. When it's refreshed, it'll either be
gone (dep prune) or new enough to not need googlestorage.
Also move pkg/pools, pkg/leak, and pkg/geocode to internal.
More remains.
Change-Id: I2640c4d18424062fdb8461ba451f1ce26719ae9d
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
Picago was very old and had local changes, update to latest release.
Also update to latest oauth2 as a transitive dependency.
Actual versions:
github.com/tgulacsi/picago: 9e1ac2306c701ca7477a169b2b49902b7b4c58bf
golang.org/x/oauth2: 197281d4e0ecd78c33865daf9c6e51626feefcb2
manually omit oauth2 testdata and tests that contain private keys to
make "devcam review" happy
This change was prepared using the forthcoming migration to the dep
tool.
Manually tested by setting up a new Picasa oauth config and importing
many photos.
Addresses: #889
Change-Id: I4d21941ccb4ade84ec658b5cfc8624e29f52be60
This is the most recent committed vendoring of this directory introduced in
commit bace8b0d on Dec 9, 2016
f9cfd754 introduced the Apr 16, 2016 versions of a number of crypto packages
and this brings them up to date.
This change was prepared using the forthcoming migration to the dep tool.
Change-Id: Id29faa7fe57fe3fbb15208f615f951025ae0fbd9
NaCl offers authenticated encryption, which means that the blobstore
can't tamper with the data. Since SHA-1 were checked one could not
change a blob outright, but could add new blobs by tampering with the
meta blobs, too. It's true that only signed blobs should cause actions
just by being present, but we are already far too deep in the chain of
assumptions, just not to spend a bit of CPU adding a MAC. The new
scheme is much easier to prove secure.
Also simplified the meta by removing the IV (which is in the encrypted
blob anyway) and the encrypted size (which is plaintext size + overhead).
Finally, added tests (including a storagetest) and tried to make this
sort of production-ready.
Still to do are meta compaction and a way to regenerate the meta from
the blobs, in case of meta corruption (which now we can do securely
thanks to NaCl authentication).
golang.org/x/crypto/nacl/secretbox:
golang.org/x/crypto/poly1305:
golang.org/x/crypto/salsa20/salsa:
golang.org/x/crypto/scrypt:
golang.org/x/crypto/pbkdf2:
1e61df8d9ea476e2e1504cd9a32b40280c7c6c7e
Change-Id: I095c6204ac093f6292c7943dbb77655d2c51aba6
Previous vendoring removed a number of files manually. This enumerates the
existing removals in .gitignore and makes it easier to transition to another
dependency management system. We also take this opportunity to add travis and
gitignore files to the list and remove those from the vendor tree.
Part of solving #889
Change-Id: Ia1b1b11ea5f2065de3a069022c2ee346e2befe78
The runsit package is obsolete. Pull the listen code directly into webserver and
remove support for the runsit specific named ports. Update TODO.
Change-Id: I0d8ea798375d0eb4abea86ed9e6454376233e992
github.com/mailgun/mailgun-go vendored in at rev
17e8bd11e87cb660ba5da8d635bbeae44b9443ac
github.com/pkg/errors vendored in at rev
f15c970de5b76fac0b59abb32d62c17cc7bed265
Fixes#980
Change-Id: I824ee3aa7493e062e6d916030dd62e0bd51ce477
Update myitcv.io/react to revision:
bca7c66b77ed8a5b86fb77cff70914c4a7cc3ce5
Update react, react-dom to v15.6.2 based on instructions in
https://reactjs.org/blog/2017/09/25/react-v15.6.2.html
Add vendor/embed/react/update.sh script to assist with future updates
Update mobile UI to fix problems documented here:
https://reactjs.org/warnings/legacy-factories.html
Update mobile.html and debug_console.html to use minified js
Fixes#977
Change-Id: I15ea81822e4f85669b321d5b3eb8b169f534def7
Paul Lindner