This is a redo of the previously-reverted b663eacb06 but with
tests fixed this time. (I accidentally ran them earlier with
--exec=true from an earlier refactor.)
Signed-off-by: Brad Fitzpatrick <brad@danga.com>
These were all part of an earlier effort to let it be easy for people
to easily deploy their own Perkeep servers and get them on the
internet. The idea was that we'd run a small DNS server which would
map from GPG public keys to their rented cloud IPs which users would
prove to us with the gpgchallenge stuff.
The recently added Tailscale support (see 43f34e5cc5, #1668) makes
most of this redundant.
Also I'd stopped running this infrastructure ages ago and removed the
launcher code recently in b5823a65b9 (and disabled it in
c9f78d02ad). So this was all basically dead code.
Signed-off-by: Brad Fitzpatrick <brad@danga.com>
We used to have a web-based Perkeep launcher at perkeep.org/launch
that created a GCE-based Perkeep instance for users, where they pay
Google for compute time. (One of those "one click deploy" template things)
Unfortunately, Google broke their APIs for doing the third party VM
creations and we disabled it some years ago. But the code remains. And
now, updating it again, we find that they've broken it again:
Error: pkg/deploy/gce/deploy.go:358:4: servicemanagement.NewServicesService(s).Enable undefined (type *servicemanagement.ServicesService has no field or method Enable)
It's not worth fighting Google's API breakages. Just remove the GCE
launcher support as it's been unused for years.
We can always resurrect this code from git if really needed. But a
Digital Ocean or Fly launcher would probably be much easier.
Signed-off-by: Brad Fitzpatrick <brad@danga.com>
* Use modernc.org/sqlite
This way it can be enabled by default, as it is a cgo-free, Go-only package.
No need for build tags, conditional compilation (whether libsqlite3-dev is installed).
* make.go: Remove unused -sqlite flag
* Remove use of -sqlite flag
- Key ids in the `0xXXXXXXXXXXXXXXXX` format are now accepted.
- Key ids in the `02A6 ACA0 5F16 9329 D94A FFB8 C804 D0BA 9F70 883D`
format are also accepted. With or without spaces.
- Help texts now use long keyids so that its clear that we support
them.
Cloud-backed storage currently does not use blobpacked
if they are a replica. This causes issues if they become
the main storage.
Example use case where this would cause issues:
- Create a configuration with s3, blobPath, and packRelated.
- Add objects
- Remove blobPath
- `pk list`
- All blobs are gone because the s3-only config uses blobpacked
but the s3 replica was not.
A working importer for the Instapaper service. The importer imports
bookmarks, full text articles, and highlights. A blob item handler for
Instapaper highlight permanodes is included to show summarized highlight
content from the server UI.
Also, this CL updates github.com/garyburd.com/go-auth to rev
bca2e7f09a178fd36b034107a00e2323bca6a82e in order to get support for
XAuth requests.
Fixes#1208
Change-Id: I72e0c40b245c7eec4a44bb475fcaa96a0ee9a1c5
In order to facilitate manual recovery in severe scenarios, the
--keep-going flag will make the server come back online even when there
are index or recovery errors. This will allow further querying on the
unhealthy server to identify missing data or to extract data for
migration to a new/healthy instance.
Fixes#1166
Change-Id: I4153905757d9e7d8014780dd5660a862cb8ba1ab
My previous commit (f3f38f0c76) made a change that https became true
if httpsCert or httpsKey was enabled. This broke devcam server, which
always sets the cert+key fields in config/dev-server-config.json but
always set "https" to either true or false.
Instead, make it an error to omit "https" if the cert or key is
specified. (It's still currently legal to specify https==false but
specify a cert or key files, so devcam server now works, but it's not
clear that that's a good idea. I might revisit this later when I
update dev-server-config.json to the upcoming TOML format.)
Fixes#1168
Change-Id: I29d000c5eb9472cc901fbd4cb91159a6c293570e
They were named conn because nearly 8 years ago the pre-ResponseWriter
type was named *http.Conn:
https://golang.org/doc/devel/weekly.html#2010-09-29
Time to modernize.
Change-Id: I68d1a5f16dfa6fc2e7a1f863e5d9edb7308527dc
This change has two major parts, which were interwoven enough to do
them in one change:
1) make serverinit.Config fully opaque, in prep for TOML configs #1134
2) shrink the massive server/perkeepd/camlistored.go file. It was out
of control and had a bunch of code that better belonged
elsewhere. This change moves a few hundred lines of code from
camlistored.go into more logical places: internal/osutil/gce for
GCE stuff, serverinit for config stuff (KeyRingAndId), etc.
I also added a TODO to make it possible to compile perkeepd without
any GCE stuff, which I saw as a possible and worthy goal only after
moving everything away.
Updates #1134
Change-Id: Iea6f84c5aca9c70b97806f4a201ec35e0f630e3b
The index and search pkg now rely on the GPG identity to be in the long
form (16 chars). But the server configuration historically was using the
short form (8 chars).
This change therefore forces the GPG identity ending up in the generated
low-level config to be in the long form.
Also, fix the functions in pkg/jsonsign to generate the long form as well,
to ease the consistent use of it everywhere else.
Fixes#1047
Change-Id: I1333150316caa2c5e179dde0c92b850a362ad17e
Introduction of an Owner type in pkg/index.
Related changes in pkg/index and pkg/serverinit.
Fixes#1047
Change-Id: Ifc316865833349d5202b5085e8f2c1235f3f2220
- correct logging that logged functions instead of their value
- use ID vs Id naming
- use correct function names in comments
Change-Id: I61562cef7ebac7337ec6c85312cdf7915cb1a84b
I had intended for this to be a small change.
I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.
And then it got a bit infectious and ended up touching everything.
I ended up doing SubFetch in the process by necessity.
At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.
Updates #733
Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
Remove the blob.SHA{1,224}From{Bytes,String} constructors too. No
longer used. This adds blob.RefFromBytes which was missing. We had
blob.RefFromString. Now everything uses blob.RefFrom* instead of
specifying a hash function.
Some tests set a flag to force use of SHA-1 because there was too much
golden data to update. We can remove those one-by-one over time as we
fix up tests.
Updates #537
Change-Id: Ibe6428089a6221594c2b751f53f98b03b5a28dc2
followup of 7eda9fd502
We want existing Perkeep instances on GCE to be able to keep on running
with their DBNames-style existing databases.
To that end, we introduce the "perkeep-config-version" metadata key,
which will be set by the launcher from now on.
When perkeep configuration starts, it can lookup that key. If it is set,
it means we're in a newly created instance, and we don't need to care
about DBNames compatibility. If not, we modify the low level
configuration on the fly, so that it keeps on using the old DBNames
values that were set for a GCE instance.
Change-Id: I611811fdb9c68777c2ba799e9047d00ec0bae040
DBNames is supposed to provide configuration for the various databases
names. However,
1) I contend that nobody needs or wants to configure them as long as we
provide sane defaults.
2) it seems the only obvious user we have for this is to set up some of
the names on GCE.
3) having another external source for names complicates the code
further, especially when we already have the distinction between
database names for DBMS and file names for file-based databases.
4) writing a correct documentation for it is awkward.
Therefore, in this CL, I propose that we remove DBNames. Instead,
genconfig.go now sets some consistent default names for the various
queues and indexes set up on a DBMS (MySQL, PostGres, Mongo). To that
end, we introduce the new, but optional, DBUnique configuration
parameter, that is used as a part of all the database names, in order to
be able to run several Perkeep instances on the same DBMS, without name
conflicts.
In addition, the queue for the bs->index synchandler is now set up on
the same DBMS that is already in use for the index itself, instead of
using a file-base database.
And i think we could proceed likewise for the other queues.
Fixes#951
Change-Id: Ib6a638f088a563d881e3957e4042e932382b44f4
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
Brad Fitzpatrick