A working importer for the Instapaper service. The importer imports
bookmarks, full text articles, and highlights. A blob item handler for
Instapaper highlight permanodes is included to show summarized highlight
content from the server UI.
Also, this CL updates github.com/garyburd.com/go-auth to rev
bca2e7f09a178fd36b034107a00e2323bca6a82e in order to get support for
XAuth requests.
Fixes#1208
Change-Id: I72e0c40b245c7eec4a44bb475fcaa96a0ee9a1c5
Foursquare was rebranded Swarm (for check-ins) some time ago, so
rename our implementation to match the upstream branding. But still
store "foursquare" as the permanode importer type, to not break old
users. That override is now added to the new Properties struct, and
all importers now return their properties.
More stuff will be added to Properties later: WIP flag, bug link,
proper title, icon?
Change-Id: I31cbe2feec3dbf9c6bdb0c866056d9c6966313e3
Introduction of an Owner type in pkg/index.
Related changes in pkg/index and pkg/serverinit.
Fixes#1047
Change-Id: Ifc316865833349d5202b5085e8f2c1235f3f2220
when its wholeref index lines contain any "sha1-" wholeref lines.
Then, have the client use that info when uploading files to also
compute the sha-1 (in addition to the sha-224) and ask the server for
both.
By client, we mean changes to cmd/camput, to pkg/client functions (for
camput and other Go clients, such as the apps), and to the web UI.
Also, add a -sha1 mode to devcam server, as well as the
CAMLI_SHA1_ENABLED env var, to facilitate testing/debugging of the
legacy sha1 mode. Fix/document unrelated options in devcam server
while we're at it.
Change-Id: I369679d2a4719f6f8b409f78dd4d065e13a3f62f
NaCl offers authenticated encryption, which means that the blobstore
can't tamper with the data. Since SHA-1 were checked one could not
change a blob outright, but could add new blobs by tampering with the
meta blobs, too. It's true that only signed blobs should cause actions
just by being present, but we are already far too deep in the chain of
assumptions, just not to spend a bit of CPU adding a MAC. The new
scheme is much easier to prove secure.
Also simplified the meta by removing the IV (which is in the encrypted
blob anyway) and the encrypted size (which is plaintext size + overhead).
Finally, added tests (including a storagetest) and tried to make this
sort of production-ready.
Still to do are meta compaction and a way to regenerate the meta from
the blobs, in case of meta corruption (which now we can do securely
thanks to NaCl authentication).
golang.org/x/crypto/nacl/secretbox:
golang.org/x/crypto/poly1305:
golang.org/x/crypto/salsa20/salsa:
golang.org/x/crypto/scrypt:
golang.org/x/crypto/pbkdf2:
1e61df8d9ea476e2e1504cd9a32b40280c7c6c7e
Change-Id: I095c6204ac093f6292c7943dbb77655d2c51aba6
Uses the Google Drive API to import photos from the Google Photos folder
in Google Drive.
Still a work in progress, so not yet added to allimporters.
Issues #874#896
Change-Id: I9b9bffe071434b2a55be667579c3699a3addb8ed
Since we don't actually ever remove blobs (until we add garbage
collection, and even then), if a share claim gets deleted (with a delete
claim), the only knowledge of the deletion resides in the Index.
So when the share handler verifies the sharing chain, there is nothing
preventing it from reading a supposedly deleted share claim (from the
blobserver), and concluding that the share chain is valid.
This change adds the index handler to the share handler, so it can check
the deletion status of a share claim, and hence support "share
cancellation".
Fixes#914
Change-Id: I572fdddee30e745aa2d2a6720c83c8e8c916515d
This update the default configuration such that `devcam server` and
`devcam server -hello` work again. Changes in whether prefix and listen
were required on the application left both of those targets broken.
The default server target was only broken briefly, when the scanning
cabinet was introduced.
The `server -hello` command has been broken for months.
This change is a temporary one. A more proper fix is present in
https://camlistore-review.googlesource.com/#/c/9206/.
Once that change is merged, this commit can be reverted.
Change-Id: If692760da265502fece3b981bdd34d4d5bea5690
WARNING: this app is still experimental, and even its data schema might
change. Do not use in production.
This change adds a Camlistore-based port of the scanning cabinet app
originally created by Brad Fitzpatrick:
https://github.com/bradfitz/scanningcabinet
Some of it is inspired from the App Engine Go port of Patrick Borgeest:
https://bitbucket.org/pborgeest/nometicland
The data schema is roughly as follows:
-a scan is a permanode, with the node type: "scanningcabinet:scan".
-a scan's camliContent attribute is set to the actual image file.
-a scan also holds the "dateCreated" attribute, as well as the
"document" attribute, which references the document this scan is a part
of (if any).
-a document is a permanode, with the node type: "scanningcabinet:doc".
-a document page, is modeled by the "camliPath:sha1-xxx" = "pageNumber"
relation, where sha1-xxx is the blobRef of a scan.
-a document can also hold the following attributes: "dateCreated",
"tag", "locationText", "title", "startDate", and "paymentDueDate".
Known caveats, in decreasing order of concern:
-the data schema might still change.
-the scancab tool, to actually create and upload the files from physical
documents, is practically untested (since I do not own a scanner).
-some parts, in particular related to searches, are probably
sub-optimized.
-the usual unavoidable bugs.
Change-Id: If6afc509e13f7c21164a3abd276fec075a3813bb
This change should have been with
65343d114d , to keep up with the app
handler changes.
It is ok to hardcode 3179 as the server's listen port (even if that is
not the actually chosen port), because the app handler will only use the
host part of the listen address.
Change-Id: Ib0a40a0376680d67a0617f184d5986117c704246
- Available at '/help/'
- Loads and parses server config upon init and generates the necessary client config for access
- Links to 'Downloadable Tools' are not functional (task is not yet completed)
- A link was added to the default server page as well as a 'Help' option to the Piggy menu
Issue: #536
Change-Id: I10629de034c908ebc76f6feacbbfca1df7bf1c2b
Signhandler was moved from the ui to the root handler and genconfig
adjusted accordingly.
New discovery struct types to replace the JSON maps are defined in
pkg/types/camtypes. The changes are related to Issue #539.
Change-Id: Ib3f12feeaf6fa81107de2d468cab491723e590ae
This change makes gce/create.go generate a self-signed certificate
with the hostname from the -hostname argument and upload it to GCS
before creating a new camlistore instance.
It also makes camlistored use baseURL to figure out the hostname
when generating its self-signed certificate.
Change-Id: I64f85853dab34a7ce95e5d5997e58f2e5da43496
To avoid confusion, the previously named "memoryIndex" configuration
parameter, which actually meant to slurp the index into the memory
corpus, has been renamed to "copyIndexToMemory". "memoryIndex" is now
the parameter to enable a memory-only indexer.
http://camlistore.org/issue/416
Change-Id: I7026e5463fda3a1887a88e0b07d4e3431c60c8b7
Imports all images from all albums, and puts each image in its album.
Each album is linked under "Picasa (<username>)".
Tries to skip already imported files (check based on album/filename).
See Issue #391.
Change-Id: If00751671a429891c05bbe41f59c1421b2ff2de8
Previously, every time you restarted the server, it forgot about all
thumbnails previously generated. (in practice it didn't/doesn't
matter with a single user, though, since they're still cached in the
browser and we always reply to If-Modified-Since immediately without
checking the cache) But it'll matter more with the Publish handler.
Also, rename some stuff, clean up some stuff, drop an unused interface.
And then necessarily change the serverconfig low-level generator to use
a kvfile for the thumbmeta map when using local disk for blobs.
--
Change-Id: I4dcfcb21429a440aa118794c03f7abf7bd69c33b
We had the legacy "memIndex" option to put the Camlistore index in
memory (the leveldb memdb) as an option in genconfig called
"memIndex", and we also had an option called "memoryIndex" for whether
to slurp the on-disk index to memory on start-up. Too confusing!
Instead, delete "memIndex" (since it's not the default anyway, now
that we have kv).
Then, also (the original point of this change): default the
memoryIndex option to true, so search works for people by default.
This option might go away in the future if it becomes the only required
way.
Also, document this.
Change-Id: Iddffa6e19adbf09c5aacd063aa44de362d90633b
Regressed from rev cb6f423e. Eventually pkg storagetest should test all methods of blobserver.Storage
for all storage target types.
Change-Id: I2c1c93b76fd9280a3eb429b1d71c64a693ed1ace
Previous TODO entry was:
-- Get rid of QueueCreator entirely. Plan:
-- sync handler still has a source and dest (one pair) but
instead of calling CreateQueue on the source, it instead
has an index.Storage (configured via a RequiredObject
so it can be a kvfile, leveldb, mysql, postgres etc)
-- make all the index.Storage types be instantiable
from a jsonconfig Object, perhaps with constructors keyed
on a "type" field.
-- make sync handler support blobserver.Receiver (or StatReceiver)
like indexes, so it can receive blobs. but all it needs to
do to acknowledge the ReceiveBlob is write and flush to its
index.Storage. the syncing is async by default. (otherwise callers
could just use "replica" if they wanted sync replication).
But maybe for ease of configuration switching, we could also
support a sync mode. when it needs to replicate a blob,
it uses the source.
-- future option: sync mirror to an alternate path on ReceiveBlob
that can delete. e.g. you're uploading to s3 and google,
but don't want to upload to both at once, so you use the localdisk
as a buffer to spread out your upstream bandwidth.
-- end result: no more hardlinks or queue creator.
Change-Id: I6244fc4f3a655f08470ae3160502659399f468ed
Michael Hoffmann