Commit Graph

100 Commits

Author SHA1 Message Date
Brad Fitzpatrick 54578ea062 pkg/blobserver/blobpacked: log before starting integrity check
Fixes 

Change-Id: I3abea84a9cee090309098634e655721272386092
2018-04-21 12:05:41 -07:00
mpl 4b47f45535 pkg/blobserver/blobpacked: reindex duplicates as well
For reasons yet to be explained, it can happen that zip blobs with the
exact same overall data contents get written to the large blobpacked.
Some of their packed schema blobs might differ though, e.g. if the
file names are different.

This is a problem, at least because of the added storage space used. And
also because it is not supposed to happen as there is a wholeRef check
on reception of a file.

However, it seems harmless to let these duplicates get indexed as z:
rows.

Therefore, this change improves a bit the inspection of duplicates and
the logging around it, to help users delete them by hand if they decide
to do so, but also adds detected duplicates as found zip blobs to the
index. This means finding duplicates is not longer considered a failure
of the recovery process.

Updates issue 

Change-Id: I5fec2f2226818eefd03ffce82c768de867fad6b0
2018-04-07 20:14:02 +02:00
mpl d917c62d76 blobserver/blobpacked: add new z: meta rows
As discussed in issue , part of the problem with reindexing and
then checking the integrity of the index, is that neither b: nor w:
meta rows are keyed by the blobRef of the packed blob (the "zipRef").

This can lead, among other things, to reindexing actually erasing all
trace of a zipRef as it may have been in the value of a row, that gets
subsequently overwritten by a row that has a different value, but an
identical key (since e.g. w: rows are keyed by wholeRef).

To address that issue, this change introduces a new kind of meta row,
prefixed by z: , which indexes the following information:
key: blobref of the zip, prefixed by "z:"
value: size of the zip, blobref of the contents of the whole file, size
of the whole file, position in the whole file of the data in the zip,
size of the data in the zip.

Fixes 

Change-Id: Iae61fe823cda0accb22e55ea075407a2b8fd11f8
2018-03-30 20:47:18 +02:00
Brad Fitzpatrick 77ef2484c2 blobserver/blobpacked: re-validate large integrity after repair
Also:
* lowercase the SHOUTYCASE log prefix
* close the meta index on Close (from mpl's CL)

For 

Change-Id: I26d9d77338ac850a20d9d631c0424129f6f98fe2
2018-03-23 11:34:57 -07:00
Brad Fitzpatrick 93422ae168 blobserver/blobpacked: add more debug
For 

Change-Id: I4c1210480d869f8d4fb1b53d28d69cc02ad51d72
2018-03-22 22:22:11 -07:00
mpl 4d9b0b1306 blobserver/blobpacked: fix incomplete Errorf call
Caught by Go1.10 yay

Change-Id: Ibb64ca8830b61d69350f3eddd50574d7d85b7c94
2018-03-02 19:48:37 +01:00
mpl 0d96057201 pk/blobserver/blobpacked: check large storage integrity
Check that all blobpacked zips are in the blobpacked meta, and
vice-versa (that all entries in meta do exist as a zip in large
storage).

As the current recovery code would not fix the case of stale entries
(large blobRefs in the blobpacked index of large blobs that don't exist
anymore), this change also adds a new recovery mode, which wipes the
existing blobpacked index, before rebuilding it.

In doing so, the recovery var in blobpacked pkg, as well as the
flagRecovery in camlistored.go have been changed to ints instead of bools,
to take into account that we now have several modes of operation for
recovery.

Fixes 

Change-Id: I1fe76b805af34933e362d70c9f27bfd5403e3f3a
2018-02-17 02:53:24 +01:00
Paul Lindner 84b2c6b3e4 all: various lint fixes
- correct logging that logged functions instead of their value
- use ID vs Id naming
- use correct function names in comments

Change-Id: I61562cef7ebac7337ec6c85312cdf7915cb1a84b
2018-02-05 11:59:00 -08:00
Paul Lindner 459c75410e all: more renaming of Camlistore to Perkeep
Change-Id: I118e3cbcf20d80afeffc84f001388c4556f21628
2018-01-30 03:02:56 -08:00
Brad Fitzpatrick 8eec428c0a Merge "blobserver: add context to BlobRemover" 2018-01-23 18:33:20 +00:00
Paul Lindner 6d2d9714de all: simpify constructs by running gofmt -s on all code
Change-Id: Idc12ddcfe8f735d77c6baa942f5bb7a2c7d9b40b
2018-01-21 10:27:12 -08:00
Brad Fitzpatrick 66db09453f blobserver: add context to BlobRemover
Updates 

Change-Id: I2fffb5cad59aa994441ee82ac5d940270113ee5a
2018-01-19 09:54:46 -08:00
Brad Fitzpatrick 194d4f9443 blobserver, all: add contexts to ReceiveBlob, Fetch & million resulting deps
I had intended for this to be a small change.

I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.

And then it got a bit infectious and ended up touching everything.

I ended up doing SubFetch in the process by necessity.

At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.

Updates 

Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
2018-01-18 16:22:16 -08:00
Brad Fitzpatrick 38f10a7bd0 all, testhooks: use sha224 by default, add hook for some tests to use sha-1
Remove the blob.SHA{1,224}From{Bytes,String} constructors too. No
longer used. This adds blob.RefFromBytes which was missing. We had
blob.RefFromString. Now everything uses blob.RefFrom* instead of
specifying a hash function.

Some tests set a flag to force use of SHA-1 because there was too much
golden data to update. We can remove those one-by-one over time as we
fix up tests.

Updates 

Change-Id: Ibe6428089a6221594c2b751f53f98b03b5a28dc2
2018-01-09 20:03:38 -08:00
Brad Fitzpatrick 0e8980b54b blobserver: change BlobStatter interface, simplify proxycache
This addresses a long-standing TODO in the BlobStatter interface to
clean it up. Just like all new Go programmers, I misused channels in
APIs. I should've cleaned this up years ago.

While here, I also added a context.

The rest should get contexts later.

This also cleans up a few things here & there.

The pkg/client statting no longer does batching, which added a lot of
complexity. There was a comment saying something like "once we have
SPDY, we can delete this". Well, we have HTTP/2 now, so seems
deletable.

All tests pass.

Change-Id: I034ce07d9b70e5cc9e5482213368993e638d4bc8
2018-01-08 16:54:52 -08:00
Brad Fitzpatrick 11e9c5567c Move some packages from perkeep.org/pkg to perkeep.org/internal
Notably: pkg/misc all moves.

And pkg/googlestorage is deleted, since it's not used. Only the
x/net/http2/h2demo code used to use it, but that ended in
https://go-review.googlesource.com/33230 (our vendored code is old).
So just nuke that dir for now. When it's refreshed, it'll either be
gone (dep prune) or new enough to not need googlestorage.

Also move pkg/pools, pkg/leak, and pkg/geocode to internal.

More remains.

Change-Id: I2640c4d18424062fdb8461ba451f1ce26719ae9d
2018-01-01 20:54:48 -08:00
Brad Fitzpatrick d6a0b05df0 Rename import paths from camlistore.org to perkeep.org.
Part of the project renaming, issue .

After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.

This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.

Also, this only moves the lru package to internal. More will move to
internal later.

Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.

This updates some docs, but not all.

devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).

Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
2018-01-01 16:03:34 -08:00
mpl c366c68a84 pkg/blobpacked: adjust tests for rollsum change
After the rollsum fix in 4723d0f452
landed, the way files were truncated in blobs changed, and hence some
expected hashsums as well.

This CL adjusts such expectations.

Change-Id: I44fc1f5ce1922d7bc99f9a8096ef4b8d212571dc
2018-01-02 00:33:47 +01:00
Brad Fitzpatrick 26e0ff0c96 Merge "blobserver/blobpacked: log reindexing progress" 2017-12-31 02:05:45 +00:00
Paul Lindner 15feaeb24c all: lint fixes for 'error strings should not be capitalized or end with punctuation or a newline'
Change-Id: I9c3766a51ac8be694ae76befff4b6fa9a85e34eb
2017-12-11 06:13:25 -08:00
Paul Lindner 1afc3450c2 Remove blank line in blobpacked to apply comments to the package
Change-Id: I040cd9e36584c7b17b809691495f81295733c6a1
2017-12-03 17:01:29 -08:00
Paul Lindner b09cd377d7 Switch to stdlib context from golang.org/x/net/context
This switches most usages of the pre-1.7 context library to use the
standard library.  Remaining usages are in:

  app/publisher/main.go
  pkg/fs/...

Change-Id: Ia74acc39499dcb39892342a2c9a2776537cf49f1
2017-11-26 01:12:26 -08:00
Paul Lindner fa46c3935d Correct various misspelled words
Change-Id: I236e880526e4c2b0bd318da041983d557e0aa885
2017-09-11 08:33:31 -07:00
mpl 04afc9ae8a pkg/server: add recovery option on web UI and instance page
When the blobpacked index gets corrupted/destroyed, it needs to be
rebuilt and hence camlistored refuses to start without the -recovery
flag being set if it detects it is needed.

This is a problem on GCE, because camlistored is handled by systemd, and
so the only way to restart camlistored with -recovery is to edit the
systemd service, then issue 'systemctl daemon-reload', and 'systemctl
restart camlistored'. Then revert the change for the next time
camlistored restarts. This is not user-friendly.

This change adds a check on camlistored startup based on the presence
and value (as a boolean) of the "camlistore-recovery" key as an instance
attribute. Therefore, the user only has to add the
("camlistore-recovery", "true") attribute in the Custom metadata of
their Google Cloud instance page to switch to recovery mode. (And
restart the instance).

As an additional feature (for non-GCE users), this change also adds the
option to restart the server in recovery mode from the status page of
the web UI.

Change-Id: I44f5ca293ddd0a0033fc5d9c2edca1bac0ee9c8f
2017-06-07 23:00:50 +02:00
Mathieu Lonjaret 8a17e7252b Merge "pkg/sorted/mysql: drop tables on reindex" 2017-01-18 18:14:06 +00:00
mpl af77128123 pkg/sorted/mysql: drop tables on reindex
When reindexing on a (My)SQL based sorted.KeyValue, we should recreate
the database schema from scratch, which means dropping the tables.

However, index.Reindex just calls Wipe on the newly created
sorted.KeyValue, which only deletes the rows, and does not drop the
tables.

Therefore, this CL changes the implementation of Wipe in the MySQL case,
so that it takes care of dropping the tables, and doing everything that
needs to be done afterwards to set up the sorted.KeyValue.

In addition, with the introduction of the sorted.NeedWipeError, we detect
upon initialization of a sorted.KeyValue if it failed because it needed
a schema update. If that is the case, and we're in reindex mode, we can
fix the sorted.KeyValue with a Wipe and carry on.

Finally, we introduce the new sorted.NewKeyValueMaybeWipe function that
automatically wipes a KeyValue when a NeedWipeError was returned upon
its creation.

Next, do the same with other sorted SQLs.

Fixes 

Change-Id: I2032781cbf453a364880bd3e2e8b3c09aac7aed9
2017-01-16 19:10:05 +01:00
mpl 326a7125d2 blobserver/blobpacked: log reindexing progress
Fixes 

Change-Id: If05e30f34520f76c4342e3be5415e5cbd195ee7c
2016-11-08 00:23:29 +01:00
mpl 14322f8afe pkg/blobserver/blobpacked: change the meaning of -recovery
The -recovery flag from camlistored, now forces the blobpacked index to
be rebuilt, regardless of its state.

Fixes 

Change-Id: I4e6bd5374ec68d7bb32de9fc119abbc881707625
2016-11-07 22:44:59 +01:00
Brad Fitzpatrick 177ad3807a blobserver/blobpacked: fix missing error check
Noticed while debugging mysql kv problems.

Change-Id: Id65f7ce98964afad1b7f88187f95bd31c174428a
2016-05-23 10:26:37 -07:00
Tilman Dilo d208b531c9 Fix permissions for files
Use mode 0644 instead of 0755 for non-vendored files.

Change-Id: Ifbe1d5e21f24814531248a8a6c84d6f0a72c071c
2016-04-28 21:15:09 +02:00
Brad Fitzpatrick c2e4fc7de6 blobserver/blobpacked: fix TestReindex at Go master
Fixes 

Change-Id: I0c58900fc566812e1a48c04d3bfba5060c0c4160
2016-04-20 18:51:07 -07:00
Brad Fitzpatrick 290569f9ae blobserver/blobpacked: make TestPackLarge work with Go tip
compress/flate changed at tip. compressed zips can have different
digests now.

Updates 

Change-Id: Ida572ca88fe4729c57772db85f69876d28c3742f
2016-04-20 17:48:01 -07:00
Brad Fitzpatrick 75d60962f6 Move remaining stuff in third_party/* to vendor/*
Change-Id: Ifbcc02817083cba68d8c1acec3e6ec50e8f61149
2016-04-20 16:49:15 -07:00
Tamás Gulácsi 7402cc0efd Delete misc unused objects
Using honnef.co/go/unused/cmd/unused

Change-Id: I672b3cb77f09e9bd80dcdc149cde4f7f2939e451
2016-04-06 17:59:51 +02:00
Will Norris 77ed42edf8 add canonical import paths
The import path was added to the go file that included the package
documentation if one existed.  Otherwise, I used what seemed to be the
primary file for the package.

Fixes 

Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
2016-03-13 19:57:14 -07:00
Mathieu Lonjaret c86007e232 Merge "blobserver/blobpacked: reindex meta from packed blobs" 2016-03-11 20:51:46 +00:00
mpl 4dd31658a5 blobserver/blobpacked: reindex meta from packed blobs
Added recovery mode to trigger reindex.

Fixes 

Change-Id: Ifb9c26e1fe8e562c1dab5f1ce793f1c9165be988
2016-03-11 17:02:38 +01:00
Mathieu Lonjaret 1383f340db Merge "pkg/blobserver: fix deadlock in MergedEnumerate" 2016-02-10 22:18:51 +00:00
Attila Tajti b306c1fccd pkg/blobserver: fix deadlock in MergedEnumerate
Change-Id: I45a13d5dbc1659640c44a5898ee1068a4a335a66
2016-02-10 23:14:17 +01:00
mpl e0d719ba21 pkg/types: remove
Most of it replaced with vendor/go4.org/types and
vendor/go4.org/readerutil

u32 went where needed in pkg/blobserver/*
invertedBool went in pkg/types/serverconfig
atomics64 went in pkg/fs

Change-Id: I230426cda35be4b45ed67e869f14e6fdae89be22
2016-02-05 18:28:47 +01:00
mpl ba0126e899 blobserver/blobpacked: fix manifest doc example
Change-Id: Ibfd212d714b568565d09470ee1d6edce58731088
2016-01-26 15:25:50 +01:00
Stephen Searles 23457fb56a adding keys to fields to make go vet happy
Change-Id: I28e38da6f5499c3284e647b1c123bcfc882120f7
2016-01-09 00:34:55 -08:00
Mathieu Lonjaret 54c5c41d07 Merge "serverinit,blobpacked: wipe blobpacked meta before reindexing" 2015-12-24 23:28:47 +00:00
Tamás Gulácsi 9f7e1df32b Remove pkg/context
Use golang.org/x/net/context instead.

Fixes .

Change-Id: I676a2f25458be97610a49d6f954f2102cbd373fa
2015-12-12 23:09:02 +01:00
mpl 40be4d8da2 vendor: go4.org/jsonconfig (and errorutil)
Previously pkg/jsonconfig and pkg/errorutil

Copied from go4.org at rev d1b8a2fb2de6160036e4801aa5e4d855571078b8

Change-Id: I673ed55b0825baa2607289b6082f205100261d7a
2015-12-01 17:21:49 +01:00
mpl 12eddf9c19 vendor: go4.org/strutil go4.org/syncutil
They were internal packages (under pkg), which we are now moving to
go4.org, so we in turn need to vendor them in now.

Change-Id: I92224f731404d0bd4ca1c57492bed37cb3367ed4
2015-11-21 00:21:20 +01:00
mpl 59d7e89d87 serverinit,blobpacked: wipe blobpacked meta before reindexing
Fixes 

Change-Id: Ic202e4478e4979a29b71404c08c3188cbb3c292b
2015-09-08 17:02:11 +02:00
Brad Fitzpatrick dda34f8fb7 blobpacked: implement blobserver.WholeRefFetcher
Fixes 

Change-Id: I4fc052b99b2e7f05c12a53adf8eb16e333efe8c4
2015-04-13 20:38:42 +02:00
mpl b6eb85631c blob SubFetcher: explicitely states with errors the testSubFetcher constraints
testSubFetcher in blobserver/storagetest was already checking that we'd
get specific error messages in the case of negative input parameters or
an out of range offset.

This change rationalizes these constraints with named errors
(ErrNegativeSubFetch and ErrOutOfRangeOffsetSubFetch) specified
in the SubFetcher interface.

It also fixes the googlestorage and s3 implementations so that they pass
the aforementioned test.

Change-Id: I25b72b842855b90ee3cab44c90654581dccf4b8e
2015-02-26 15:18:27 +01:00
Brad Fitzpatrick b4255c3d72 blobpacked: stub TODO implementation of blobserver.WholeRefFetcher
Change-Id: I96d986fe1cec8ceb093d59f0a725e4f542824f69
2015-02-04 21:07:05 -08:00