This change has two major parts, which were interwoven enough to do
them in one change:
1) make serverinit.Config fully opaque, in prep for TOML configs #1134
2) shrink the massive server/perkeepd/camlistored.go file. It was out
of control and had a bunch of code that better belonged
elsewhere. This change moves a few hundred lines of code from
camlistored.go into more logical places: internal/osutil/gce for
GCE stuff, serverinit for config stuff (KeyRingAndId), etc.
I also added a TODO to make it possible to compile perkeepd without
any GCE stuff, which I saw as a possible and worthy goal only after
moving everything away.
Updates #1134
Change-Id: Iea6f84c5aca9c70b97806f4a201ec35e0f630e3b
This also changed a few of the magic strings in the SetAttrs call to use
the nodeattr constants.
One thing I'm unsure about is if "link" should be changed to
nodeattr.URL, but that's left out of this initial revision.
This also sneaks in a change to devimport that was blocking
the testing of the feed service, as it doesn't use authentication.
Fixes#1148
Change-Id: Ic3cf85dc30c446954f3780683cba99f118b46fb6
Remove explicit version checking using pkg-config and just check if
setting journal_mode=WAL succeeds. Remove -wal command line
flag from pk tool and also remove duplicated logic for db
handling.
Fixes#840
Change-Id: I72874a5bb9c5bb3a7be552a65e5ddb0c12fd726e
Also removed misc/buildbot while at it (which contained camput
references) since we don't use it anymore at all.
TODO: the OSX app seems to be relying on finding a binary in ../bin,
which we do not use anymore. This will probably need fixing.
Updates #981
Change-Id: I14220fbad2e81181330fca4bb2d2e5fe170e1bd6
A new "put" mode is added to the pk command, so that the "pk put"
command can be used to create and upload blobs.
What this command does is actually just call the previously named
"camput" executable, which is renamed to "pk-put" in this change.
This involves adding a new way to register a mode in cmdmain, when such
a mode is just meant to call an external binary. To emphasize the
distinction, the existing func (to register a sub-command, or a mode) is
renamed from RegisterCommand to RegisterMode, and RegisterCommand is now
the name of the new func/way.
Updates #981
Updates #1056
Change-Id: Ief954c17aa88a376f551df7de4b4e9fe41ad96d1
search -1 will output one blob ref per line instead of JSON, suitable
for piping into xargs.
Fixes#1034
Change-Id: I7bf812767fa9859ff5f5d98858abb688906f20f6
The current maximum size for a schema blob is 1MB. For a large enough
directory (~20000 children), the resulting static-set JSON schema is
over that maximum size.
We could increase that maximum, but we would eventually hit the maximum
blob size (16MB), which would only allow for ~300000 children. Even if
that is an uncommon size, it is technically possible to have such large
directories, so I don't think it would be reasonable to restrict users
to such a limit. So it does not seems like enough of a solution.
The solution proposed in this CL is to spread the children of a
directory (when they are more numerous than a given maximum, here set to
10000) onto several static-sets, recursively if needed. These
static-sets (subsets of the whole lot of children) are stored in the new
"mergeSets" field of their parent static-set schema. The actual fileRefs
or dirRefs, are still stored in the "members" field of the subset they were
spread in. The "mergeSets" and "members" field of a static-set are therefore
mutually exclusive.
Fixes#924
Change-Id: Ibe47b50795d5288fe904d3cce0cc7f780d313408
when its wholeref index lines contain any "sha1-" wholeref lines.
Then, have the client use that info when uploading files to also
compute the sha-1 (in addition to the sha-224) and ask the server for
both.
By client, we mean changes to cmd/camput, to pkg/client functions (for
camput and other Go clients, such as the apps), and to the web UI.
Also, add a -sha1 mode to devcam server, as well as the
CAMLI_SHA1_ENABLED env var, to facilitate testing/debugging of the
legacy sha1 mode. Fix/document unrelated options in devcam server
while we're at it.
Change-Id: I369679d2a4719f6f8b409f78dd4d065e13a3f62f
This removes NewDefault and NewFromParams.
Now the default way to create a client is:
client.New() -> (*Client, error)
Specifying a server is optional and now requires
client.OptionServer(server).
If the caller really wants to log.Fatal on error, they can use
client.NewOrFail.
Also, some of the boilerplate from GopherJS callers is now promoted to
be the default behavior in the client package.
Change-Id: Icb106cf3e13cc492fe5b2f7f240e1ad4227eaf33
Actually add the file, missing from previous commit
38c2a87ad2.
That's what I get for skipping code review. Sigh.
Change-Id: Icfaa7f6922c4ceaae69bd228b8d6498d0a85a7d8
I had intended for this to be a small change.
I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.
And then it got a bit infectious and ended up touching everything.
I ended up doing SubFetch in the process by necessity.
At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.
Updates #733
Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
This CL is about levelDB as the HaveCache for camput, and there are
several aspects to it. To describe it, I'll take the particular example
where you want to add many permanodes (~33k) to a given set, with
camput. Something like:
for _, blob := range blobs {
do("camput attr -add sha1-foobar camliMember " + blob)
}
In a "normal" levelDB use case, everytime the number of level-0 .ldb
files goes over 4 (by default), a background compaction task is
started to transform these SST into level-1 ones, and remove the level-0
ones.
However, since our particular camput call is very short lived
(especially on a local Perkeep), not only might there be not enough time
for the compaction to be triggered, but even if it is, when the DB is
flushed (on a Close call), any ongoing compactions are cancelled. This
makes level-0 compactions very unlikely to happen on short-lived camput
calls. As a result, the number of level-0 files keeps growing until
levelDB fails while trying to open them all, because it hits the current
process ulimit.
Now, in this CL, what we propose is to systematically force a compaction
as soon as the HaveCache is opened. It is not scheduled concurrently, so
we are sure that the compaction happens before the DB actually gets used
by camput. This seems to make sure that the number of level-0 tables
never grows too much. With this change, I was able to run the above
example on 33K blobs without hitting the ulimit error.
However, it should be noted that potential problems might remain. The
compaction for levels above 0 is triggered based only on the total size
of the level (e.g. at 100MB by default for level-1), and not on the
number of files. Since we're creating many tiny tables (basically 1
entry per table), the number of files grows very fast while the total
size does not, and the compaction does not get triggered, even if forced
with CompactRange. This does not seem to be a problem for our use case,
as levelDB does not seem to need to open many of the level-1 files at
the same time, so we're not hitting the ulimit problem because of that.
If needed, there's at least one way this problem (if it is one?) could
be fixed: make the compaction trigger on other conditions, such as
number of files per level. I've experimented with it (forcing the
level-1 compaction to trigger at the 100 files limit), and it seems to
be working. But I had to do change the goleveldb code itself, and I
don't think levelDB implementations are supposed to do that.
For information, at the end of the run on the 33K blobs:
$ du -sch *.ldb
...
83M total
$ ll | wc -l
20988
And indeed, when asking for leveldb.stats on the table:
Level | Tables | Size(MB) |
-------+------------+---------------+
0 | 1 | 0.00015 |
1 | 20981 | 3.47307 |
Also, update github.com/syndtr/goleveldb to
34011bf325bce385408353a30b101fe5e923eb6e
And remove github.com/syndtr/gosnappy as goleveldb does not use it
anymore.
Also apply this change to StatCache.
Fixes#1008
Change-Id: If9f790a003e67f3c075881470e52e5f2174afa73
Upgrades mysql driver to a versioned release, fixes a minor issue
in dbinit that caused SHOW DATABASES queries to fail and improves
error messages to help diagnose such problems.
Changelog: vendor/github.com/go-sql-driver/mysql/CHANGELOG.md
Revision: go-sql-driver/mysql@a0583e0143
Change-Id: I4f5fdee60b30eca85564c452f72774e9c15828f3
Remove the blob.SHA{1,224}From{Bytes,String} constructors too. No
longer used. This adds blob.RefFromBytes which was missing. We had
blob.RefFromString. Now everything uses blob.RefFrom* instead of
specifying a hash function.
Some tests set a flag to force use of SHA-1 because there was too much
golden data to update. We can remove those one-by-one over time as we
fix up tests.
Updates #537
Change-Id: Ibe6428089a6221594c2b751f53f98b03b5a28dc2
This addresses a long-standing TODO in the BlobStatter interface to
clean it up. Just like all new Go programmers, I misused channels in
APIs. I should've cleaned this up years ago.
While here, I also added a context.
The rest should get contexts later.
This also cleans up a few things here & there.
The pkg/client statting no longer does batching, which added a lot of
complexity. There was a comment saying something like "once we have
SPDY, we can delete this". Well, we have HTTP/2 now, so seems
deletable.
All tests pass.
Change-Id: I034ce07d9b70e5cc9e5482213368993e638d4bc8
The mongo integration was using a very old package. It's using
a new namespace now. Upgrade and adjust all call points
Removes labix.org/v2/mgo
Introduces gopkg.in/mgo.v2 from branch v2 with revision
3f83fa5005286a7fe593b055f0d7771a7dce4655
Change-Id: I2784fca941998460f58e0ac8d3d51286401590b5
mpl