Commit Graph

6120 Commits

Author SHA1 Message Date
Brad Fitzpatrick b3ba9aa7cd sorted/sqlkv, sorted/sqlite: replace Serial bool with a *syncutil.Gate
Change-Id: I820220d49b1d316d2be0ab7fe5bf302d204108b0
2016-02-13 13:01:46 -08:00
mpl eb0db1f1ff importer/pinboard: kill World after test
Related: Issue #680

Change-Id: I10a1ffc611d159288b4dc9d52e96524bbb385ee1
2016-02-13 00:46:20 +01:00
mpl 6fd35bd4de C+A: add Attila Tajti (Individual CLA)
Change-Id: Icd62937813b95c996b5678332393e901a79295d4
2016-02-10 23:28:10 +01:00
Mathieu Lonjaret 1383f340db Merge "pkg/blobserver: fix deadlock in MergedEnumerate" 2016-02-10 22:18:51 +00:00
Attila Tajti b306c1fccd pkg/blobserver: fix deadlock in MergedEnumerate
Change-Id: I45a13d5dbc1659640c44a5898ee1068a4a335a66
2016-02-10 23:14:17 +01:00
mpl 243c7c146a deploy/gce: do not ask for password
Issue #656

Change-Id: Ida28675588b30eed786957aaaca674cadbf9c404
2016-02-10 19:43:56 +01:00
Mathieu Lonjaret 2c20c02597 Merge "ui: use authentication token instead of Basic Auth" 2016-02-10 16:00:44 +00:00
mpl e03d0af04f ui: use authentication token instead of Basic Auth
Since issue #660 showed that we could not rely on Basic Auth for some
very particular cases (of POST requests), we wanted to replace Basic
Auth with a pre-generated token based authentication mechanism for these
cases.

However, as there already is such a mechanism used to authenticate
websocket connections, we simply extended the use of that mechanism.
Therefore, the token that is initially generated for websocket
connections is also now a valid token to authenticate any other
connection. The relevant types and method names have been changed to
reflect that. The javascript code pertaining to the sensitive cases
mentioned above has been changed to use that token.

Some doc has been added to point out how security-sensitive the
OpDiscovery permission is, since it gives access to the auth token.

Fixes issue #660

Change-Id: Iafed3b6e4804364ca2559414c8d87dc4a30f6637
2016-02-10 16:58:34 +01:00
mpl 21604dbe56 misc/docker: stamp version as well to binaries
Version in binaries will now be:
"X.Y (git rev xxxxx)"
if X.Y is specified with --tarball_version,
"xxxx"
otherwise.

Also uniformized the flags names and use in other programs.

Fixes #665

Change-Id: I958094f69b279437504854a3ff10e924d42c4fb6
2016-02-09 17:31:29 +01:00
mpl 255b51a5a0 website: always remove old demoblob3179 container
Because it can happen that we end up with with an already stopped/killed
demoblob3179, in which case we need to remove it even when killing it is
unsuccessful (because it was already dead).

Change-Id: I289e96284deebeaaa775c23ef6f3fe3bffc3777c
2016-02-08 22:05:37 +01:00
mpl 1a90fe806d vendor: update go4.org
rev 40a0492aa096a3be30c750c4e2216de52a6cf2e3

adds writerutil package

Change-Id: Ia771e38283afb9750a8dad0cd462ca12ce42dd15
2016-02-08 21:30:09 +01:00
Mathieu Lonjaret 134c276cd9 Merge "website: print error for demo blobserver" 2016-02-08 13:53:13 +00:00
mpl 88b9dc826f website: print error for demo blobserver
Change-Id: Ic582b65d94dd17379feb2ac9e8f15d8e3bb4a05c
2016-02-08 14:52:18 +01:00
mpl e0d719ba21 pkg/types: remove
Most of it replaced with vendor/go4.org/types and
vendor/go4.org/readerutil

u32 went where needed in pkg/blobserver/*
invertedBool went in pkg/types/serverconfig
atomics64 went in pkg/fs

Change-Id: I230426cda35be4b45ed67e869f14e6fdae89be22
2016-02-05 18:28:47 +01:00
mpl 9afc57ced5 vendor: update go4.org
rev d7e10a7752d55a1f81e384fe49a6a5b417a9a6dd

new readerutil and types packages

Change-Id: I6a96819f7cfd67267f372cdf7a9bc1a16c206587
2016-02-05 15:10:17 +01:00
Attila Tajti 34ad44385b pkg/search: add parentof search expression
Change-Id: I74f6f5411355a0b0864739f135331ba304245ddb
2016-02-03 10:31:51 +01:00
Mathieu Lonjaret d20ad1570e Merge "server/synchandler: add exported constructor and IdleWait method" 2016-01-28 00:57:56 +00:00
mpl 1773eb0859 server/synchandler: add exported constructor and IdleWait method
This change allows (for example) to hook a blobserver and an indexer
together in a way similar as they would be in a default Camlistore setup.
This is particularly useful to setup tests that require a realistic
indexer, without having to create a heavier test with e.g. test.World.

This actual goal of this change is described below. I kept these changes
together because the one described below illustrates the need for the
synchandler changes.

pkg/index/stress/: stress test the indexer with more data

The goal of these benchmkarks is to help us compare the various sorted
implementations, so we can pick the most efficient one as a default.

A second, harder, goal would be to add tests that help us estimate the
reliability of the various kv stores, again so we can rule out the buggy
ones, or help fix them.

Change-Id: I8a1fe50398a4ba09c03786502b68b6c2599e5984
2016-01-28 01:56:00 +01:00
Brad Fitzpatrick e06e20fc14 Merge "index/Corpus: add Attrs cache to PermanodeMeta" 2016-01-27 23:28:42 +00:00
Attila Tajti 44b4e855d2 index/Corpus: add Attrs cache to PermanodeMeta
- add an Attrs map to PermanodeMap that is kept in sync with Claims
- update Attrs map on scanFromStorage and whenever a new blobRef is added
- add fast path to AppendPermanodeAttrValues(Locked),
  PermanodeAttrValueLocked and PermanodeHasAttrValueLocked
- fix bugs in and add tests for PermanodeHasAttrValueLocked and
  PermanodeAttrValueLocked slow path

Change-Id: Ib0dcdcf9355dd5fa8588c872721a029ec114cb6b
2016-01-27 17:09:18 +01:00
Mathieu Lonjaret ef2d940415 Merge "vendor: update go4.org/lock" 2016-01-27 15:07:32 +00:00
mpl c38f2a8ea2 pkg/misc/amazon/s3: fix test for Client
Because omitted in b9aec12f31

Change-Id: I2ca82360b3ff90d65be6f84eabdb785efcee99a9
2016-01-27 15:57:14 +01:00
mpl f4c5839f8c vendor: update go4.org/lock
To rev 9ba773eba85ab9e258ff516630f7f6474bc4535b

Change-Id: I32ac9fdc825e178548aaa5efc7ad219b2ec68465
2016-01-27 15:40:30 +01:00
mpl ba0126e899 blobserver/blobpacked: fix manifest doc example
Change-Id: Ibfd212d714b568565d09470ee1d6edce58731088
2016-01-26 15:25:50 +01:00
mpl b9aec12f31 blobserver/s3: limit concurrent PutObject calls
Fixes issue #675

Since camput does not seem affected by the above issue, and that camput
(pkg/client really) main difference in how it uploads is that it does
gated requests, it seemed to follow that gating the requests might fix
the problem.

With this change, I don't see the problem anymore either in
TestS3WriteFiles or by manually uploading with the web UI.

I suppose this indicates that S3 throttles incoming connections pretty
strictly, but I have no formal evidence of it.

Change-Id: I48f10923a710f4a21e44bc251e648db37ae504ec
2016-01-25 21:39:57 +01:00
mpl 39f50aa70e blobserver/s3: new test for writing files
Initially for debugging issue #675

Change-Id: I7907e7a37b197762befe6a133718ed87fc0b57d2
2016-01-22 17:16:43 +01:00
mpl 74aa3fa163 vendor: go4.org/lock
imported at rev 96a2350179b917865057073eb6dd6e4488913ba2

Change-Id: Icae1af8b0e78779e9b462f71abf779387ab695b9
2016-01-20 17:22:59 +01:00
mpl 58af581a24 dock.go: print binaries tarball path
Fixes issue #664

Change-Id: I8af4882be32c9452726a40f468b408367705a04c
2016-01-20 00:13:19 +01:00
Mathieu Lonjaret c6fa8ab12d Merge "website: fix missing return from prev commit" 2016-01-18 23:48:21 +00:00
mpl 60a2885a20 website: fix missing return from prev commit
Change-Id: Ia74013db252601a9dbd7eb7911e1dc892e1c56b4
2016-01-19 00:46:46 +01:00
Mathieu Lonjaret 35b201a71e Merge "website: make /apiget in demo authenticated too" 2016-01-18 23:41:30 +00:00
mpl 528a201827 website: make /apiget in demo authenticated too
Change-Id: Ief662a69774fcc919c4e3618df91886cb1f05b84
2016-01-19 00:39:13 +01:00
Mathieu Lonjaret 3beeb86104 Merge "website: chrome bug repro demo" 2016-01-18 23:05:01 +00:00
mpl f478c6cc67 website: chrome bug repro demo
For issue #660

Change-Id: I7b53da99b4ba50a6ec5818d291083aab72f994ab
2016-01-19 00:03:36 +01:00
Mathieu Lonjaret 2b78dca037 Merge "camtool sync: local path support on Windows" 2016-01-18 17:49:34 +00:00
Mathieu Lonjaret b77942f71a Merge "Change the diskpacked reindex test data, fix hash" 2016-01-16 14:15:36 +00:00
Brad Fitzpatrick 8e62e24ac0 Merge "adding keys to fields to make go vet happy" 2016-01-16 03:27:42 +00:00
mpl 606cfcb635 vendor: update cloud/storage
To 392f2aeac3d98ee75147c8cd8f92e4f65104b0bb

Mainly so we can use the new Reader.

Change-Id: I61d88ba2bae4e8732d3426b5ae66b7ff2d89e89b
2016-01-15 00:03:27 +01:00
Attila Tajti f0343c0fae camtool sync: local path support on Windows
Change-Id: I2d3e0c0bbd3f3ae50165d366e6a1093fceb36e5c
2016-01-11 12:00:37 +01:00
Stephen Searles 23457fb56a adding keys to fields to make go vet happy
Change-Id: I28e38da6f5499c3284e647b1c123bcfc882120f7
2016-01-09 00:34:55 -08:00
Mathieu Lonjaret ea4b5e477a Merge "vendor: update http2" 2016-01-08 15:03:24 +00:00
mpl 3b91e403e4 vendor: update http2
From golang.org/x/net/http2 at 415f1917e1dbc946ec834288a8a1e5ff6eee2900

Change-Id: If03d7d2eb3080628fc04110f927d8c7b80612508
2016-01-08 15:45:30 +01:00
Tamás Gulácsi 14d074d3d2 Change the diskpacked reindex test data, fix hash
And try to underscore that the last byte is an LF, which must be kept
and accounted for.

Andy Song raised my attention to this ambiguity.

Change-Id: Ia352980350e44b57f47394841f643a737b91dda7
2016-01-08 15:00:01 +01:00
mpl 630ed571ee pkg/client: fix transport tests for fba8b457de
Fixes issue #672

Change-Id: I36ff0bc588986582968ea478def9d67276cbea77
2016-01-07 15:50:47 +01:00
mpl 0b1fbe5b5e dock.go: fix flag doc typo for version
Change-Id: Ic390de75f579892855688a832008678c607084b7
2016-01-07 00:15:47 +01:00
mpl c88d81a2fd sorted/mysql: remove DB from pool when closing
Otherwise when openOrCachedDB is called for the second time,
it would return an actually closed DB.

Change-Id: I820834e508fed1a34ab7bd19a8a91e398badff9c
2016-01-06 20:42:52 +01:00
Mathieu Lonjaret 77dcc0559d Merge "dockertest: use official mysql image" 2016-01-06 18:39:32 +00:00
mpl 4662076934 dockertest: use official mysql image
Because orchardup/mysql is deprecated.

I had to increase the timeout, because this image takes at least twice
as long to start compared with the orchardup one. (tests went from ~5s to
~11s here).

Change-Id: I42c485d736c65ce9d93aabf02b1de73e6ac799ff
2016-01-06 19:38:13 +01:00
Mathieu Lonjaret 92568bd8b6 Merge "pkg/client: set camliType, use correct blobref" 2016-01-06 17:24:03 +00:00
mpl 00f370f38c pkg/serverinit: fix test after 7b78c50007
Change-Id: I36800bcf5f1a2e8f19890c97372297f9281999ba
2016-01-06 18:11:52 +01:00