Commit Graph

4237 Commits

Author SHA1 Message Date
Brad Fitzpatrick bfafb8321f TODO: clean up sync+index incoming blob handling
Change-Id: I64cd4818dead81f5a0dac0e275994df17b7740ef
2014-03-18 17:35:14 -07:00
Steven L. Speek 146a42cc51 search: accept 'and', 'or', and parentheses in expressions
'and' has precendence over 'or'.
both operators are left associative
parenthesized expressions are evaluated first
Parser refactored, parseAtom split up.

Change-Id: I1f194cc75df49bad9d30d041d689d8ba833076f1
2014-03-18 16:40:44 -07:00
mpl b43373c45c Merge "publish: use generic queries" 2014-03-18 22:35:40 +00:00
Brad Fitzpatrick 7c2852263d Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-03-18 13:47:10 -07:00
Brad Fitzpatrick bfe61b1297 storagetest: add a test of EnumerateBlobs w/ prefix of a blobref
Change-Id: Id865da5ba6c65e0110b0eb6c6d0bd45980d33105
2014-03-18 13:47:02 -07:00
Brad Fitzpatrick bfa8e0c6fe HACKING: update github part
Change-Id: I1e8bc0381a084357dd0a008a1a1e2563f85a9ea6
2014-03-18 13:21:42 -07:00
Brad Fitzpatrick bfd3b3fdc8 TODO: ditch the importer.Interrupt type
Change-Id: I3a7eac29713a5079220b2e56508bd8ca105b14cf
2014-03-18 10:38:49 -07:00
Brad Fitzpatrick bf4116eb93 genconfig: picasaa importer doesn't have a path
Was removed during codereview, but not removed from here.

Change-Id: I51309f46fe9747bcc0b2476813acac9120c32b28
2014-03-18 10:10:10 -07:00
Brad Fitzpatrick bfbdd57ad6 more TODO
Change-Id: Ibb03612c1a06287bf781b96f478d4d627b159c84
2014-03-18 10:08:53 -07:00
Brad Fitzpatrick bf428453c7 website: add Picasaweb to 0.8 notes
Change-Id: I0de6dbfa506a9153a73c61e638ce4ddbfbfde702
2014-03-18 10:08:42 -07:00
mpl 9cadbcd5bc publish: use generic queries
Use generic queries instead of specialized index queries. This is a step
towards the publisher app, because its client will have to use generic
queries.

Context: http://camlistore.org/issue/365

Change-Id: I2781a345e024174e3bea8511b6cdc6f342d5a7c1
2014-03-18 17:26:29 +01:00
Brad Fitzpatrick 90404a4129 Merge "Picasa: add importer." 2014-03-18 16:05:42 +00:00
Brad Fitzpatrick bf2e1fa585 sync: don't replicate a shard's missing blobs until enumeration is complete
Prevents spurious replication of blobs on enumeration error.

Change-Id: I38db7406f6ea52137cb757b32599b18eb7fcf3da
2014-03-17 23:21:53 -07:00
Brad Fitzpatrick bf2d86738e blobserver: give ListMissingDestinationBlobs the ability to exit earlier
If there's an enumeration error for src or dst, sending a zero value
will shut down ListMissingDestinationBlobs.

Change-Id: I95598e9dc2607610436faa06f40485d1abd2342f
2014-03-17 23:20:37 -07:00
Brad Fitzpatrick bf1ec32e39 sync: add paranoia around checking storage's Enumerate implementation
Didn't find anything, but is useful to keep in, to maybe find bugs in the future
for other storage types.

Change-Id: If0fd37e03578de233be8da95ca45623c5f12156b
2014-03-17 23:05:27 -07:00
Brad Fitzpatrick bf8f4b2423 sync: fix bug in prefix enumeration. could send one extra item.
Depending on timing, could lead to ListMissingDestinationBlobs getting out
of sync and causing a lot of blobs to be replicated that were fine and already
on the server.

Change-Id: I3710e59088f1fe4e526f8f11bc9d1837a727e512
2014-03-17 23:02:01 -07:00
Brad Fitzpatrick bf88f5f06c sync: missing handler return + wording change
Change-Id: Iebd5344a0a0e1418cb48c92b91858ebf2f9486c8
2014-03-17 23:00:18 -07:00
Brad Fitzpatrick bfde349851 s3: do some retries in ListBucket, to work around Amazon-vs-Go's HTTP/TLS bug
Change-Id: Idfc8187d07f558f2e7c590dc6c14c66791f46e57
2014-03-17 22:59:02 -07:00
Brad Fitzpatrick bfbe29fc4a Linkify status HTML, and link to sync handler status pages too.
Change-Id: I18467511fd34f3bc232f38c2b91075c616c220a1
2014-03-17 21:52:28 -07:00
Brad Fitzpatrick bf3d052314 C+A: add Nolan Darilek (Individual CLA)
Change-Id: Ic4f94e5fc01ae8637f9f9441f961dca903c310ae
2014-03-17 11:44:18 -07:00
Brad Fitzpatrick bfc76ff0fe serverinit: allow S3-only configurations to also have a working index
... with kv or sqlite indexes, at least. Other types will need more
genconfig work.

Change-Id: I5fb9fa0a122693e08780c3fa20eae67fa850d98f
2014-03-17 08:45:01 -07:00
Tamás Gulácsi 65fa5d6d3e Picasa: add importer.
Imports all images from all albums, and puts each image in its album.
Each album is linked under "Picasa (<username>)".
Tries to skip already imported files (check based on album/filename).
See Issue #391.

Change-Id: If00751671a429891c05bbe41f59c1421b2ff2de8
2014-03-17 10:52:51 +01:00
Brad Fitzpatrick e44d49a557 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2014-03-16 22:23:13 -07:00
Aaron Boodman f44173135f Merge "Fix bug where "Select as current set" item never shows up." 2014-03-17 04:39:40 +00:00
Aaron Boodman 4677e5ec97 Fix bug where "Select as current set" item never shows up.
Change-Id: Ibc6b98c3feb65f28c3768cfaf6169044e41ad160
2014-03-16 21:38:43 -07:00
Brad Fitzpatrick bf28dd4488 More status handler HTML+JSON, more sync status.
Change-Id: I0381853191d5b871af649d102b976e592def791f
2014-03-16 20:14:57 -07:00
Brad Fitzpatrick bfb0d1e8de s3: more fault injection, and simplify receive, removing spilling to disk
Change-Id: Id7fb01c40dc863bee483887bbf14934161eb03a0
2014-03-16 19:44:52 -07:00
Brad Fitzpatrick bfa4aea031 fault: new package for fault injection
Doesn't do much.

Change-Id: Iab038564a753611a8bc8f44f33a57dbfe96d76e0
2014-03-16 19:27:40 -07:00
Brad Fitzpatrick cb656260c5 Merge "pkg/schema: Move two non-utf8-handling utility funcs to schema.go." 2014-03-15 22:34:04 +00:00
Brad Fitzpatrick bfc607fee7 index: reindex blobs when dependent blobs arrive out-of-order
Keep track of missing dependencies both in memory and in the index's
underlying sorted.KeyValue. When we see a dependent blob arrive, see
if we can reindex things.

Fixes camlistore.org/issue/102

Change-Id: I3d8cfc463e4b8c9d158be8f9656e772839b093b9
2014-03-15 08:44:09 -07:00
Brian Gitonga Marete 88b94b686d pkg/schema: Move two non-utf8-handling utility funcs to schema.go.
As per review comments.

Change-Id: I0240f697d0cb180a9fb2e9b1ddda7ac18869b4ad
2014-03-15 07:39:08 +03:00
Brad Fitzpatrick bf94a73859 Get rid of SeekFetcher vs StreamingFetcher distinction and complexity.
StreamingFetcher is now just Fetcher, and its FetchStreaming is now
just Fetch.

SeekFetcher is gone. Blobs are max 16 MB anyway, so we can slurp to
memory when needed. The main thing that cared about SeekFetcher
was the GET handler, ServeBlobref, because http.ServeContent needed
one for range requests. That's rewritten in an earlier commit, using
the FakeSeeker from another earlier commit.

Lot of code got simpler as a result.

Change-Id: Ib819413e48a8f9b8d97f596d0fbf771dab211f11
2014-03-14 12:29:13 -07:00
Brad Fitzpatrick bfbf660cd1 serverinit: fix flaky test. sort publishRoot prefixes, for predictable output
Change-Id: I230a7d344f9c982539b3ed6d3a88eb1066322ea5
2014-03-14 12:09:17 -07:00
Brad Fitzpatrick bf4e8f7238 gethandler: rewrite to not use SeekerFromStreamingFetcher; add tests
Change-Id: I785dd56f7bc7c1f113575700c74f910b4f4ea4a0
2014-03-14 11:52:11 -07:00
Brad Fitzpatrick bfcd26a2ff blob: move Blob type to its own file, add FromFetcher, FromReader
Prep for killing SeekFetcher.

Change-Id: Iade678ccf74d5ebc9613dbb9dbf44b9ed9ff4dea
2014-03-14 11:47:37 -07:00
Brad Fitzpatrick bf96b2af24 types: add NewFakeSeeker
NewFakeSeeker returns a ReadSeeker that can pretend to Seek (based
on the provided total size of the reader's content), but any reads
will fail if the fake seek position doesn't match reality.

Change-Id: I4e63aa78c80fdbcd87b2133018009ed72bedb9f3
2014-03-14 11:09:00 -07:00
Brad Fitzpatrick bf5e1a43b6 TODO: more
Change-Id: Id83f4c319340d28e500ce889cd2ed951bd3998a1
2014-03-14 09:23:18 -07:00
Brad Fitzpatrick bf01b14961 index: move seekFetcherMissTracker up a layer
In prep for missing blob dependency rescheduling in indexer.

Change-Id: I1d492e6aa64cfb658daec17e4621d1453c6d3607
2014-03-14 09:14:46 -07:00
Brad Fitzpatrick bf6031a397 search: add operator "childrenof:sha1-xxxxx" to find permanode children of a parent
Change-Id: I9e756147c70f5230aa843ab215dafe5ed6d80087
2014-03-13 19:02:26 -07:00
Brad Fitzpatrick bfa412de5d search: implement some of RelationConstraint (only the "parent" relation)
Allows searching for permanodes based on their parent(s).

Change-Id: I949f159edc43d895e1657bd6ae4c983758ac59f2
2014-03-13 18:49:28 -07:00
Brad Fitzpatrick bfeb745882 corpus: add PermanodeHasAttrValueLocked convenience accessor
Change-Id: Ie73270ce54c3125a76987daa9f50895c5acf42db
2014-03-13 18:48:00 -07:00
Brad Fitzpatrick bf6ae30368 corpus: reverse-index claims with values that look like blobrefs
Allows efficiently finding parents of permanodes, or permanodes of
file contents, etc.

Adds Corpus.ForeachClaimBackLocked.

Change-Id: I6dffb32eab4a959cd3c5922a7f47d11fdcea5f3d
2014-03-13 18:47:25 -07:00
Brad Fitzpatrick 18bd4cdc0b Merge "camtool sync: ignore diff identity with localdisk" 2014-03-13 16:33:55 +00:00
Aaron Boodman 75fb861821 Merge "Web UI: Fix for infinite loop with extremely wide aspect ratio images." 2014-03-13 15:44:21 +00:00
Aaron Boodman 12a4c07c54 Web UI: Fix for infinite loop with extremely wide aspect ratio images.
Change-Id: I567000d10adc542e5759d06a82a28db380d225ad
2014-03-13 08:43:38 -07:00
mpl dab702ea49 camtool sync: ignore diff identity with localdisk
When either the source or the destination is a localdisk, there is no
need to warn about differing identities.

Change-Id: Ia61779dd7907099c5ab58d9ca876cf4690f6fe05
2014-03-13 16:10:34 +01:00
Brad Fitzpatrick 9bd781c453 Merge "No 'and' or 'or' yet. Added description for attr and tag." 2014-03-12 21:52:24 +00:00
Steven L. Speek 6d9117f1c0 No 'and' or 'or' yet. Added description for attr and tag.
Change-Id: Ib1ab293dabc264657a970c8ef3ca54b9d59868ce
2014-03-12 22:48:19 +01:00
mpl 607a20f349 Merge "0.8 release notes" 2014-03-12 20:35:56 +00:00
mpl 2e421013f7 0.8 release notes
Change-Id: I1fb43617f7667d65239b3d264662d7ce89c76203
2014-03-12 21:34:36 +01:00