And also speed up /importer/TYPE page's query by saying it's unsorted, which
gets us a better index. Hacky workaround. Should fix it in the query planner
later instead.
Fixes#1105
Change-Id: I140ffb40456ec802844584568d2f1a7a38b26e41
Foursquare was rebranded Swarm (for check-ins) some time ago, so
rename our implementation to match the upstream branding. But still
store "foursquare" as the permanode importer type, to not break old
users. That override is now added to the new Properties struct, and
all importers now return their properties.
More stuff will be added to Properties later: WIP flag, bug link,
proper title, icon?
Change-Id: I31cbe2feec3dbf9c6bdb0c866056d9c6966313e3
A new "put" mode is added to the pk command, so that the "pk put"
command can be used to create and upload blobs.
What this command does is actually just call the previously named
"camput" executable, which is renamed to "pk-put" in this change.
This involves adding a new way to register a mode in cmdmain, when such
a mode is just meant to call an external binary. To emphasize the
distinction, the existing func (to register a sub-command, or a mode) is
renamed from RegisterCommand to RegisterMode, and RegisterCommand is now
the name of the new func/way.
Updates #981
Updates #1056
Change-Id: Ief954c17aa88a376f551df7de4b4e9fe41ad96d1
I have not found any doc stating that the photos Space is not supposed
to work anymore, but Brad said so in the bug, and some testing seemed to
confirm it. So this change switches to using the "drive" Space which
contains all files in Google Drive, including the photos from Google
Photos, as long as the option to have the Google Photos folder in Drive
is enabled.
Fortunately, we can still request the Spaces property of the file, and
use that to filter out all of the non-photos files. For now...
Fixes#987
Change-Id: I7f257b70804fd72c730ded158173ec6ae64fde5b
We still need to figure out the root cause of the problem, i.e.
why a sha224 importer node was created in the first place, when a
valid sha1 one already existed. Next CL.
Updates #1069
Change-Id: I00b4aab9a4f67b9c4ee198774378630615f6d11d
- correct logging that logged functions instead of their value
- use ID vs Id naming
- use correct function names in comments
Change-Id: I61562cef7ebac7337ec6c85312cdf7915cb1a84b
This removes NewDefault and NewFromParams.
Now the default way to create a client is:
client.New() -> (*Client, error)
Specifying a server is optional and now requires
client.OptionServer(server).
If the caller really wants to log.Fatal on error, they can use
client.NewOrFail.
Also, some of the boilerplate from GopherJS callers is now promoted to
be the default behavior in the client package.
Change-Id: Icb106cf3e13cc492fe5b2f7f240e1ad4227eaf33
I had intended for this to be a small change.
I was going to just add context.Context to the BlobReceiver interface,
but then I saw blob.Fetcher could also use one, so I decided to do two
in one CL.
And then it got a bit infectious and ended up touching everything.
I ended up doing SubFetch in the process by necessity.
At a certain point I finally started using context.TODO() in a few
spots, but not too many. But removing context.TODO() will come in the
future. There are more blob storage interfaces lacking context, too,
like RemoveBlobs.
Updates #733
Change-Id: Idf273180b3f8e397ac5929c6d7f520ccc5cdce08
Remove the blob.SHA{1,224}From{Bytes,String} constructors too. No
longer used. This adds blob.RefFromBytes which was missing. We had
blob.RefFromString. Now everything uses blob.RefFrom* instead of
specifying a hash function.
Some tests set a flag to force use of SHA-1 because there was too much
golden data to update. We can remove those one-by-one over time as we
fix up tests.
Updates #537
Change-Id: Ibe6428089a6221594c2b751f53f98b03b5a28dc2
Part of the project renaming, issue #981.
After this, users will need to mv their $GOPATH/src/camlistore.org to
$GOPATH/src/perkeep.org. Sorry.
This doesn't yet rename the tools like camlistored, camput, camget,
camtool, etc.
Also, this only moves the lru package to internal. More will move to
internal later.
Also, this doesn't yet remove the "/pkg/" directory. That'll likely
happen later.
This updates some docs, but not all.
devcam test now passes again, even with Go 1.10 (which requires vet
checks are clean too). So a bunch of vet tests are fixed in this CL
too, and a bunch of other broken tests are now fixed (introduced from
the past week of merging the CL backlog).
Change-Id: If580db1691b5b99f8ed6195070789b1f44877dd4
This code previously had methods returning channels. Such APIs are
always error-prone and difficult to use. Switch to a synchronous func
callback pattern instead, with contexts propagated.
Change-Id: Iaa1b91227c0daf4c8562fcba8d27dbcd7ab755c5
Remove dynamic rate limit adjustment for now. It was racy.
No need to be super fast, anyway, as long as it catches up eventually.
But we can make it smarter later. I wanted to get it correct first.
Change-Id: Id5b5fc946546d8d9c0720f1c0ec2f341a17cdd01
This switches most usages of the pre-1.7 context library to use the
standard library. Remaining usages are in:
app/publisher/main.go
pkg/fs/...
Change-Id: Ia74acc39499dcb39892342a2c9a2776537cf49f1
to Rev 9dfe39835686865bff950a07b394c12a98ddc811 for golang.org/x/html
to Rev 88f656faf3f37f690df1a32515b479415e1a6769 for golang.org/x/text
These packages moved from code.google.com to their new home
in golang.org/x/html and golang.org/x/text
Change-Id: I4ee45ae1e18eb05ef7b0a8ec69e2f1f11d140340
The picasa importer and the gphotos importer fetch items in different
ways (one using the Picasa API, and the other the Google Drive API). But
most of the time a photo object downloaded from Picasa should result in
the same file as if downloaded from the Google Photos folder in Google
Drive (which is as it should be, really). Therefore the corresponding
file schemas in Camlistore should be identical as well. When that's the
case, there is no reason for the two importers to each create a
permanode if they're going to have the same camliContent. Especially
since it would look like duplicates (in the web UI in particular).
This change fixes the gphotos importer so that when it imports a photo,
and creates a file schema for it, it looks for an existing picasa
permanode having the same file schema as a camliContent. If one is found
it is reused, and the permanode ends up with an attributes set that is a
merge of the picasa-based attributes and the drive-based ones.
One caveat: the files in the Google Photos folder in Google Drive are
never updated with whatever modifications are done "at the source" (i.e.
on the items in Google Photos), they always stay as they were originally
uploaded.
(https://productforums.google.com/forum/#!msg/drive/HbNOd1o40CQ/VfIJCncyAAAJ).
As a result, such photos are different when imported from Picasa and
when imported from Drive, so they result in different files in
Camlistore.
In a subsequent CL, we'll modify the picasa importer in a similar manner
as what's done in this here CL.
Issues #874#896
Change-Id: I6a3c89de1af404556f01ca61d92861933fd35158
Uses the Google Drive API to import photos from the Google Photos folder
in Google Drive.
Still a work in progress, so not yet added to allimporters.
Issues #874#896
Change-Id: I9b9bffe071434b2a55be667579c3699a3addb8ed
Bumps twitter.runCompleteVersion to 5 to fix previously
imported text containing & or <.
Adds an integration test for importing tweets from the API
and from a zip archive. Test data was exported from account
camlistore_test, which we can update later if needed. The zip
archive was skinnyfied to include only the data/ directory.
Factored out importer integration test setup code from pinboard
importer tests.
Fixes#476
Change-Id: Idc7355caefaa2173823dd1f86fe02ae81f974a0e
Brad Fitzpatrick