Commit Graph

68 Commits

Author SHA1 Message Date
Mathieu Lonjaret cbec324c6a Merge "pkg/server: added help handler" 2015-05-28 13:30:25 +00:00
Mario Russo 7b1903d5b1 pkg/server: added help handler
- Available at '/help/'
- Loads and parses server config upon init and generates the necessary client config for access
- Links to 'Downloadable Tools' are not functional (task is not yet completed)
- A link was added to the default server page as well as a 'Help' option to the Piggy menu

Issue: #536

Change-Id: I10629de034c908ebc76f6feacbbfca1df7bf1c2b
2015-05-28 10:02:17 +02:00
Mario Russo 01cddcc8d5 pkg/search: fix null "files" in file search response
The "files" field of a file search response was always non-null (empty JSON array in case of error or no result).
This behaviour was broken by df7cc0b453, so this CL is restoring it.

Fixes #619

Change-Id: I8e4762d1f5a2dba4f7cf06d8ca164392c4bf6d6a
2015-05-27 19:23:39 +02:00
Mathieu Lonjaret 6570d9ea94 Merge "pkg/server: decouple sign handler from UI, replace JSON maps with structs" 2015-05-25 16:52:40 +00:00
Mathieu Lonjaret d94796d4ef Merge "pkg/search: replace JSON maps with structs in search handler responses" 2015-05-25 16:45:49 +00:00
Fabian Reinartz 76a445a956 pkg/server: decouple sign handler from UI, replace JSON maps with structs
Signhandler was moved from the ui to the root handler and genconfig
adjusted accordingly.
New discovery struct types to replace the JSON maps are defined in
pkg/types/camtypes. The changes are related to Issue #539.

Change-Id: Ib3f12feeaf6fa81107de2d468cab491723e590ae
2015-05-25 09:28:07 +02:00
Fabian Reinartz df7cc0b453 pkg/search: replace JSON maps with structs in search handler responses
This change is related to Issue #539.

Change-Id: I588cbeac8867eb565e43813b5613e476f5f9d89d
2015-05-25 09:26:35 +02:00
Fabian Reinartz a624d4271e pkg/jsonsign: replace JSON map with struct in verify response
This change is related to Issue #539.

Change-Id: I40feb09dd8a4270edd0daf6c267eed60f13280e7
2015-05-17 11:12:10 +02:00
Brad Fitzpatrick 85380193cf types: add CloseFunc and update TB's docs
Change-Id: I49dcbc5c48754e5d5074dfbbb278fb969b35aee5
2015-04-13 20:38:40 +02:00
Brad Fitzpatrick f6348948a7 Merge branch 'master' of https://camlistore.googlesource.com/camlistore 2015-02-04 21:08:08 -08:00
Brad Fitzpatrick 8229c19850 search, index: add WholeRef to pkg camtypes' FileInfo struct
So when you describe a file, you also gets its wholeref.

TODO: we'll need to migrate old indexes to this new format on
start-up.

Change-Id: I4a3fb000d68bde46474275c2070ef285a6d6ecfc
2015-02-04 21:04:39 -08:00
mpl b78ce17e13 serverinit: add genconfig for blobpacked on Google Cloud Storage.
And enable it (blobpacked) on GCE.

Change-Id: I7b51c186325e80987e48d32392d88bb4c30ee3d5
2015-02-04 19:08:43 +01:00
Brad Fitzpatrick 3026631521 serverinit: move incompatible option checks further down, into the right places
I was getting bogus errors because I replicate to s3, but the
s3+blobpacked error should only come up if s3 is primary.

Also, document the database key used for blobpack's index when mysql
or another db is in use.

Change-Id: I21a1815715f70edc3f98abf7209744703aaf6d3b
2015-01-18 19:04:31 -08:00
Brad Fitzpatrick 4155ac6a3c serverinit: high-level config support for using blobpacked
Users need to set:

     "packRelated": true,

... in their server config. This will probably become the default
value in the future.

This is currently mutually exclusive with diskpacked, which isn't good
at deleting things, and blobpacked loves to delete things (from the
loose blobs).

Updates #532

Change-Id: I8f4ea9406859b2705f26e9d1103d3acf9d1a8411
2015-01-18 18:08:18 -08:00
mpl 5c7208fb94 Merge "blobserver/cloudstorage: directory support" 2014-12-10 15:02:00 +00:00
Antonin Amand 31ebc20ba7 Add IsVideo to types/camtypes FileInfo
It is based on MIME type then falls back to extension.

Change-Id: I71d80ec7cdcf94f4fa7feffa8186943abe0279ce
2014-12-02 21:57:10 +01:00
mpl 8481c2236d blobserver/cloudstorage: directory support
http://camlistore.org/issue/528

Change-Id: I9d26b85af1cf91d1f2b102c7bfb80628af4d1e4d
2014-11-26 00:40:19 +01:00
mpl 9b68fd9f32 serverinit: syndtr/leveldb support
Change-Id: Id48a00f3896c38659a53cf131493d5a19dbfe4b7
2014-10-11 15:40:29 +02:00
mpl 95885192eb serverinit: enable memory storage in config
http://camlistore.org/issue/416

Change-Id: Idde32273ed651a5876581ad0ea06010970b92a9b
2014-09-11 01:11:07 +02:00
mpl 9ddc3988aa Merge "serverinit: add support for memory-only indexer. Enable in devcam." 2014-09-01 18:02:53 +00:00
Brad Fitzpatrick f4740e77e8 Aggregate all errors with their URLS in status.json in one place.
Change-Id: If7a540fc3d1a6caa94b5f04d44de3207f15ac4cf
2014-08-29 16:03:44 -07:00
mpl bb49185463 serverinit: add support for memory-only indexer. Enable in devcam.
To avoid confusion, the previously named "memoryIndex" configuration
parameter, which actually meant to slurp the index into the memory
corpus, has been renamed to "copyIndexToMemory". "memoryIndex" is now
the parameter to enable a memory-only indexer.

http://camlistore.org/issue/416

Change-Id: I7026e5463fda3a1887a88e0b07d4e3431c60c8b7
2014-08-26 22:46:13 +02:00
Brad Fitzpatrick a1a7e65666 camtypes: fix build. Don't call osutil path funcs during testing (in init)
Change-Id: I2f8d2966c95e29a3d6d0a0d9b117a0f022e647e7
2014-08-23 20:28:38 -07:00
mpl e8522cca36 website: allow HTML in camli errors. use website for client error.
http://camlistore.org/issue/276

Change-Id: I16f139fe2b6f1200250db668fa415ae42ff2c171
2014-08-21 23:13:56 +02:00
Brad Fitzpatrick 6e7d3e3e1e serverinit: clean up and simplify the high-to-low-level config generator
Change-Id: Ia45457718fb8832b4da32d4d6c87c8a5ba9e2301
2014-08-07 18:47:42 -07:00
Brad Fitzpatrick c3a3883395 googlecloudstorage: whren running on GCE, get access tokens automatically from the VM
Change-Id: Ibfcca0c52ff01a8f82bb0d3081119e0ac6e16ba9
2014-08-02 19:29:57 -07:00
tnt 4ac9938c8b Fixed some formatting functions that were called with invalid number/type of arguments.
Change-Id: I261c6fb732a6cf1e54e01fa3e5388c39c6deb720
2014-07-01 12:48:33 +02:00
mpl 736648ce9c publisher app: SSL support
http://camlistore.org/issue/458

Change-Id: I89013a345dbe06fd02c877422cadb8d6def1580d
2014-06-16 19:41:22 +02:00
mpl a34f9e2669 publisher app
http://camlistore.org/issue/365

Change-Id: I281fdcbbe6a2bdf15607e75a21bc93b453f82c85
2014-06-15 00:03:03 +02:00
Brad Fitzpatrick 20e952ce83 types: add NewOnceCloser, use it in camget
Change-Id: I465207031468a57f2840d5071fc27713d6184325
2014-05-31 16:08:02 +09:00
Brad Fitzpatrick bf4426e35e types: add EmptyBody
Change-Id: Ic0cc6cae6debb20fbc15bd95692e8565171eae49
2014-03-31 09:36:08 -07:00
Tamás Gulácsi 65fa5d6d3e Picasa: add importer.
Imports all images from all albums, and puts each image in its album.
Each album is linked under "Picasa (<username>)".
Tries to skip already imported files (check based on album/filename).
See Issue #391.

Change-Id: If00751671a429891c05bbe41f59c1421b2ff2de8
2014-03-17 10:52:51 +01:00
Brad Fitzpatrick bfc607fee7 index: reindex blobs when dependent blobs arrive out-of-order
Keep track of missing dependencies both in memory and in the index's
underlying sorted.KeyValue. When we see a dependent blob arrive, see
if we can reindex things.

Fixes camlistore.org/issue/102

Change-Id: I3d8cfc463e4b8c9d158be8f9656e772839b093b9
2014-03-15 08:44:09 -07:00
Brad Fitzpatrick bf96b2af24 types: add NewFakeSeeker
NewFakeSeeker returns a ReadSeeker that can pretend to Seek (based
on the provided total size of the reader's content), but any reads
will fail if the fake seek position doesn't match reality.

Change-Id: I4e63aa78c80fdbcd87b2133018009ed72bedb9f3
2014-03-14 11:09:00 -07:00
Brad Fitzpatrick 79cb116dbb Merge "pkg/types/serverconfig: json-tagged struct" 2014-02-11 00:56:11 +00:00
mpl 6707837806 pkg/types/serverconfig: json-tagged struct
Also changed the TLS/HTTPS config keys for consistency.

http://camlistore.org/issue/339

Change-Id: I704ec129f91b93ebb20bc1191816166a2f10692d
2014-02-10 21:49:11 +01:00
Tamás Gulácsi 97520583b8 Use 'uint32' instead of 'int64' for blob sizes everywhere.
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.

Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
2014-02-08 17:58:12 +01:00
Daniel Erat 5603ea8e0d pkg/index: Index audio duration.
Add pkg/media with code to calculate MPEG audio duration.
Index it in a "durationms" property.

Change-Id: Ifb6251657cadc365ef3f5667a0512fde17575560
2014-01-25 10:40:06 -08:00
mpl 20b9f5d6f1 pkg/types: add InvertedBool
Will be needed for json tagged serverconfig in subsequent commit.

http://camlistore.org/issue/339

Change-Id: I41ae6e9918b23eb9bdb471e402de0415f3499dc3
2014-01-23 22:54:18 +01:00
Bill Thiede 5c884e52ca diskpacked: keep fd open to every pack file.
Bytes read/writen per pack file, as well as per configured diskpacked
configuration are now available as expvars.

Also add reader stat helpers to pkg/types and updated the original
user in server/image.go

Change-Id: Ifc9d76c57aab329d4b947e9a4ef9eac008bc608d
2014-01-20 10:50:42 -08:00
mpl 3f13d1a043 pkg/client: fix method to find a server in the config
http://camlistore.org/issue/325

Change-Id: I4aa6d5103848b73ed169ab7d7de3a60a33723b79
2014-01-07 18:52:48 -08:00
Brad Fitzpatrick d21aa1dba2 year++
Change-Id: Ib5e17c39d3178357e1c18c4407ad45e91170ef16
2014-01-06 08:05:12 -08:00
mpl b7cee3c6ce clientconfig: forgot to git add it with bf8c463d0a
Change-Id: I098fd8c3297c48f0800f5753d220fe706601a16b
2014-01-06 08:00:35 -08:00
mpl 6e7eacea4a Merge "pkg/client: some cleanup, doc on the website" 2013-12-23 22:31:45 +00:00
mpl 1e8d46ee45 pkg/client: some cleanup, doc on the website
Changed some keys to be consistent with the server config, and removed
some others.

We're now using jsonconfig to get the keys and validate the config.

Removed some unused funcs/methods, moved some global vars as members of a client.

Also, having two possible defaults for the secret keyring
(.config/camlistore/identity-secring.gpg and .gnupg/secring.gpg)
was annoying and confusing imho, so I removed the latter.

http://camlistore.org/issue/259

Change-Id: Icf79fa8822b62b5d91843698861dd19cb553607a
2013-12-23 20:23:21 +01:00
Brad Fitzpatrick 12f87787fe types: add Time parsing test
Change-Id: Ie13a244cda76d11a39a1200895f4cd75013db087
2013-12-22 15:07:47 -08:00
Brad Fitzpatrick 95ea1dfb7d types: make Time3339 zero values be JSON null, back and forth
Change-Id: Iebf3defe041aee71e6d06b73159dbd105b545081
2013-12-20 21:35:30 -08:00
Brad Fitzpatrick 328706b8c6 index: bunch of TODOs and planning for edge tracking
Also, change to storing a pointer to claims so those pointers can be re-used elsewhere.

Change-Id: Ib0979d622ba48f1e2389575d07d7a0930264ebea
2013-12-09 17:15:34 +04:00
Brad Fitzpatrick 429fa4bf4e Fix API typo: ParseTime3339OrNil, not OrZil
Change-Id: I1255cf4560d133fe5c6128fba0be2d9d4a0c0daa
2013-12-03 21:56:09 -08:00
Brad Fitzpatrick 8c46afe490 Shrink camtypes.BlobMeta a bit. Saves 4 bytes of memory per blob.
For me,

Before:
Corpus stats: 148.859 MiB mem: 799465 blobs (89.487 GiB) (282303 schema (6478 permanode, 150962 file (112224 image), ...)

After:
Corpus stats: 146.559 MiB mem: 799465 blobs (89.487 GiB) (282303 schema (6478 permanode, 150962 file (112224 image), ...)

Change-Id: Id421dfc9d72c2a24c228418c01f9a3f05900c941
2013-11-29 13:41:39 -08:00