From 27b19787e08692dfbafd18b0a8503826a419e1a4 Mon Sep 17 00:00:00 2001 From: Brad Fitzpatrick Date: Mon, 18 Nov 2013 18:33:59 -0800 Subject: [PATCH] TODO: new plan to kill QueueCreator Change-Id: I098307063f7645889a39dedafc4d3f862a3f798c --- TODO | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/TODO b/TODO index ba08ec678..fb7627793 100644 --- a/TODO +++ b/TODO @@ -35,12 +35,27 @@ Offline list: where the lifetime of the background daemon will be the same as the user's login session. --- support syncing from a storage type that doesn't support - QueueCreator by using local disk instead. writes go to both 1) that - storage target not supporting QueueCreator and 2) local disk before - being acknowledged. Then sync goes from local disk to wherever, - deleting from local disk when done as normal (except in that case, - the hardlink refcount would drop from 1 to 0, not 2 to 1) +-- Get rid of QueueCreator entirely. Plan: + -- sync handler still has a source and dest (one pair) but + instead of calling CreateQueue on the source, it instead + has an index.Storage (configured via a RequiredObject + so it can be a kvfile, leveldb, mysql, postgres etc) + -- make all the index.Storage types be instantiable + from a jsonconfig Object, perhaps with constructors keyed + on a "type" field. + -- make sync handler support blobserver.Receiver (or StatReceiver) + like indexes, so it can receive blobs. but all it needs to + do to acknowledge the ReceiveBlob is write and flush to its + index.Storage. the syncing is async by default. (otherwise callers + could just use "replica" if they wanted sync replication). + But maybe for ease of configuration switching, we could also + support a sync mode. when it needs to replicate a blob, + it uses the source. + -- future option: sync mirror to an alternate path on ReceiveBlob + that can delete. e.g. you're uploading to s3 and google, + but don't want to upload to both at once, so you use the localdisk + as a buffer to spread out your upstream bandwidth. + -- end result: no more hardlinks or queue creator. -- website: remove the "Installation" heading for /cmd/*, since they're misleading and people should use "go run make.go" in the