Commit Graph

35 Commits

Author SHA1 Message Date
Tamás Gulácsi c986ee3c62
Use modernc.org/sqlite (#1581)
* Use modernc.org/sqlite

This way it can be enabled by default, as it is a cgo-free, Go-only package.
No need for build tags, conditional compilation (whether libsqlite3-dev is installed).

* make.go: Remove unused -sqlite flag

* Remove use of -sqlite flag
2021-12-27 12:18:08 -08:00
mpl c243487bdb pkg/index/sqlite: fix tests
Long broken since the sha224 switch.

Updates #1085

Change-Id: I2dda3dac6bd3c8006be0974bdc9be402e25b7429
2018-08-29 03:39:05 +02:00
Brad Fitzpatrick d6a0b05df0 Rename import paths from camlistore.org to perkeep.org.
Part of the project renaming, issue #981.

After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.

This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.

Also, this only moves the lru package to internal. More will move to
internal later.

Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.

This updates some docs, but not all.

devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).

Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
2018-01-01 16:03:34 -08:00
Paul Lindner fa46c3935d Correct various misspelled words
Change-Id: I236e880526e4c2b0bd318da041983d557e0aa885
2017-09-11 08:33:31 -07:00
Brad Fitzpatrick 75d60962f6 Move remaining stuff in third_party/* to vendor/*
Change-Id: Ifbcc02817083cba68d8c1acec3e6ec50e8f61149
2016-04-20 16:49:15 -07:00
Will Norris 77ed42edf8 add canonical import paths
The import path was added to the go file that included the package
documentation if one existed.  Otherwise, I used what seemed to be the
primary file for the package.

Fixes #689

Change-Id: If51be0e86529fd6f179e80af6781e639f8550fd2
2016-03-13 19:57:14 -07:00
mpl e0d719ba21 pkg/types: remove
Most of it replaced with vendor/go4.org/types and
vendor/go4.org/readerutil

u32 went where needed in pkg/blobserver/*
invertedBool went in pkg/types/serverconfig
atomics64 went in pkg/fs

Change-Id: I230426cda35be4b45ed67e869f14e6fdae89be22
2016-02-05 18:28:47 +01:00
mpl 40be4d8da2 vendor: go4.org/jsonconfig (and errorutil)
Previously pkg/jsonconfig and pkg/errorutil

Copied from go4.org at rev d1b8a2fb2de6160036e4801aa5e4d855571078b8

Change-Id: I673ed55b0825baa2607289b6082f205100261d7a
2015-12-01 17:21:49 +01:00
mpl 59e30b9433 pkg/index/sqlite: add build tag for test
so we don't get a failed test when we're not running with
--tags=with_sqlite

Change-Id: I2de4c21b4f3ee673132f56b12c2dfff8272d5d79
2015-11-18 23:17:20 +01:00
mpl ac6acf5d70 index/sqlite tests: let sorted/sqlite pkg do initDB and enable WAL
The problem was newSorted was taking care of doing the DB
initialization. It was not only unnecessary, since newKeyValueFromConfig
from sorted/sqlite will do it by calling initDB, but also incorrect
because WAL was not getting enabled. Which is what was causing the
"database is locked" error "fixed" in commit
f6bf779cd6

Change-Id: Ib4badeda4ed4c6c4be43d5f7fc4c543a467b0cd1
2015-03-13 19:11:33 +01:00
Brad Fitzpatrick 4188fefaa0 sqlite: Empty file to make the go tool happy with a test-only directory.
Not sure if this is still needed but had it sitting on my filesystem, never
checked in.

Change-Id: I06a5efd265032b6a4f867a8d0504cfb4f061304b
2014-08-02 16:51:47 -07:00
mpl 899acb8e72 pkg/sorted: remove API for each distinct implementation
Change-Id: I42446683189cdb00dcfbec1f221885092fdd83d6
2014-04-08 01:37:08 +02:00
mpl 291320d451 pkg/index: cleanup, refactor sub pkgs
http://camlistore.org/issue/263

Change-Id: I319bb097f0ce30b2bd5271b5c3bbff92b8dcc318
2014-04-05 01:22:05 +02:00
Brad Fitzpatrick bfc607fee7 index: reindex blobs when dependent blobs arrive out-of-order
Keep track of missing dependencies both in memory and in the index's
underlying sorted.KeyValue. When we see a dependent blob arrive, see
if we can reindex things.

Fixes camlistore.org/issue/102

Change-Id: I3d8cfc463e4b8c9d158be8f9656e772839b093b9
2014-03-15 08:44:09 -07:00
mpl 62f8a252ff pkg/index: move mysql to sorted
also minor cleanups in sqlite and mongo

Change-Id: I1f371358997d929c72a8f63d2630a1c3fa4a0240
2013-12-18 20:18:20 +01:00
mpl b62c94fdd1 pkg/index: move sqlite implementation to sorted
Change-Id: I1e300ffa14547fabeac8d255fd694054ebcb9e53
2013-12-16 17:35:27 +01:00
Brad Fitzpatrick 76171ddb3d Change sorted.KeyValue.Find to take an optional end bound; add tests.
The new package sorted/kvtest provides a generic KeyValue test for all
implementations. Memory, SQLite, and kvfile now use it.

This speeds up the index slurping start-up of my personal Camlistore
server from 30 seconds (when it was doing 17,000+ queries in small
windows) to now just 5 seconds. That 5 seconds can be improved yet
further.

Change-Id: Idd55ba9ccd3ed12a26868a41db1af676aff7b67b
2013-12-07 08:43:18 -08:00
mpl 15bb5f142e pkg/index: use IsDeleted in some of the search methods
1) Done in AppendClaims, PermanodeOfSignerAttrValue,
SearchPermanodesWithAttr, PathsOfSignerTarget,
and PathsLookup.

2) camtypes.Path.DateClaim is now a time.Time, and used
as such in Path search methods.

3) Added kvSignerAttrValue, kvPathBackward, and kvPathForward

4) More Delete tests. We probably want more cases, but that's
a start.

http://camlistore.org/issue/191

Change-Id: I1d42485f2c6d1eea024f70378daacaaad9197768
2013-11-28 16:08:23 +01:00
Brad Fitzpatrick 90c1e48afe Rename index.Storage to sorted.KeyValue and move it into a new package.
Having index.Index and index.Storage both in the same package led to
confusing discussions about "an index". Better names now, and smaller
packages.
2013-11-22 23:24:54 -08:00
mpl fdaf030bc0 index: add deletedAt method, and cache for deletion status
This method will be used to find out when a permanode or claim
was (un)deleted for the last time.

This change also introduces the keyDeletes index entry which
allows to find the permanodes/claims that a delete claim deletes.

Finally, since some index keys have changed, this CL introduces
a check for an index schema version, to force users to reindex
when needed.

http://camlistore.org/issue/191

Change-Id: I6f8e5d3030a6ddc8b21ff5dc5db311f2ea16f613
2013-11-06 23:34:13 +01:00
Brad Fitzpatrick b34c6198e4 sqlite: add start of an fd leak test
Change-Id: I84857e7a57bc4d95fa1a0ddc1c7538d1924cb93a
2013-09-13 11:21:48 +01:00
mpl d488c576fc search: support for static directory children
This change introduces a new index entry
to help with finding the children of a static directory.
It also fixes ResolvePrefixHop so that it takes
into account static directories, and not only collections.

This is the first step to support publishing static directories.

http://camlistore.org/issue/179

Change-Id: I5666e5caa6c782004054ae4c19a6b6119d4fda8b
2013-09-10 23:06:48 +02:00
Brad Fitzpatrick e20c7f44a8 auth: fix security problem where 'localhost' allowed all access.
Also, add tests. And remove CAMLI_ADVERTISED_PASSWORD, replacing it with explicit devauth mode.

Change-Id: I9cdfe2d537ecefcf75df0b9fb8538258703b1eb7
2013-08-31 13:44:44 -07:00
mpl f4ef8c0375 dev-server: remove. replaced by devcam server.
Fix related docs, config files, buildbot...

Change-Id: I638d3856f42953603b13204fb56782974d0db9ee
2013-08-03 01:12:38 +02:00
Brad Fitzpatrick eb1208a12e sqlite: updated ErrNotCompiled installation hint
Change-Id: I96a5fde5bf3d34cc29f38180272a624c353a6c4f
2013-07-28 18:23:04 -07:00
Brad Fitzpatrick 9468e5ba70 More docs. Every package is documented now.
misc.CountingReader moves into readerutil.

pkg/atomics is folded into pkg/types.

pkg/test/testdep is folded into pkg/test, with better name/docs.

Old cruft from pkg/webserver is deleted.

Change-Id: I3f72d8b29804254ef944995fb085837c878f79f5
2013-07-07 21:12:30 -07:00
Brad Fitzpatrick ded8f8d0d7 Be helpful about how to setup SQLite, especially on a Mac.
Change-Id: I87dbcd077383f0148a8c50f4120e6606139fcacd
2013-06-22 17:51:09 -07:00
mpl a4aba6671a sqlite: Write-Ahead Logging, for improved concurrency
http://camlistore.org/issue/114

Change-Id: I93c3d48f06a6cb0703bd9ce3f89b03fc59826a73
2013-05-01 18:04:03 +02:00
Brad Fitzpatrick 2fd4a22694 Make sqlite (more) reliable. camlistore.org/issue/114
Change-Id: Ia759de784bf4363bd9f5141ddddfa2a441a53f1d
2013-02-17 22:10:28 -08:00
Brad Fitzpatrick 9720652971 Add disabled failing test for http://camlistore.org/issue/114
Change-Id: Iae5dbf1f28ca89793f43bc5e2df2493cf2747a77
2013-02-17 21:42:50 -08:00
Brad Fitzpatrick 75360ee9b3 Rename index.IndexStorage to index.Storage, to reduce stutter.
Change-Id: I52b2a8460c3957a1c8211a470e1df9a8dc5b7067
2013-01-13 20:19:36 -08:00
Brad Fitzpatrick 6ca0efac8c camlistored: much better sqlite support, by default, and help/docs/hints 2013-01-10 16:16:10 -08:00
Brad Fitzpatrick 02b8b35b51 Make sqlite support optional and detectable. 2013-01-10 14:29:20 -08:00
Brad Fitzpatrick f696007a14 index: implement EdgesTo
For finding back references in the graph.
2012-11-05 10:29:42 +01:00
Brad Fitzpatrick a519105585 index/sqlite: new indexer, using sqlite3
Change-Id: If1405fe3ee7c632c05431f4b0be1077d6d42c4a3
2012-04-13 16:11:16 -07:00