perkeep/pkg/lru
Stephen Searles 6b426cb10d improve proxycache and stats blobservers
improving proxycache
- added fuller sample config to the package documentation
- switched the stats caching from sorted.kv to the stats blobserver
- added a cleaning mechanism to evict the least recently used blobs
- implemented StatBlobs to actually inspect the local cache. It still
  always consults the origin, but only for the blobs necessary after
  giving the cache a 50ms headstart.
- logging a few errors that were previously ignored
- added tests modeled after the tests for the localdisk blobstore
- added a method to verify the cache, and call it on initialization
- added a strictStats option to always get stats from the origin
- filling in cacheBytes on initialization

improving stats blobserver
- implemented a few more of the blobserver interfaces, Enumerator and
  Remover
- Fixed a bug(?) in ReceiveBlob that seemed to prevent it from actually
  storing stats
- added a test

minor improvements include:
- blobserver/memory: allowing the memory blobserver to hold actually
  infinite items, if desired
- blobserver: closing dest in the NoImpl blobserver, as required by the
  BlobEnumerator interface
- storagetest: not closing dest leads to deadlock
- lru: max entries of 0 now means infinite (maybe do anything <0?)
- test: a helper function to create a random blob using a global random
  source that is, by default, deterministic, to make test results more
  consistent.

In the future, an improved BlobHub or similar interface could allow a
tighter feedback loop in providing cache consistency. i.e. the cache
could register with backend stores to be notified of content updates,
minimizing the time between backend changes and cache correction.

The proxycache will verify itself at startup, reporting an error if
any of its blobs do not exist in the backend storage or if the backend
storage has a different size for the content than the cache.

Fixes #443

Change-Id: I9ee1efd8c1d0eed49bb82930c2489a64122d3e00
2017-12-29 10:09:15 -08:00
..
cache.go improve proxycache and stats blobservers 2017-12-29 10:09:15 -08:00
cache_test.go