Commit Graph

42 Commits

Author SHA1 Message Date
tnt 4ac9938c8b Fixed some formatting functions that were called with invalid number/type of arguments.
Change-Id: I261c6fb732a6cf1e54e01fa3e5388c39c6deb720
2014-07-01 12:48:33 +02:00
mpl 736648ce9c publisher app: SSL support
http://camlistore.org/issue/458

Change-Id: I89013a345dbe06fd02c877422cadb8d6def1580d
2014-06-16 19:41:22 +02:00
mpl a34f9e2669 publisher app
http://camlistore.org/issue/365

Change-Id: I281fdcbbe6a2bdf15607e75a21bc93b453f82c85
2014-06-15 00:03:03 +02:00
Brad Fitzpatrick 20e952ce83 types: add NewOnceCloser, use it in camget
Change-Id: I465207031468a57f2840d5071fc27713d6184325
2014-05-31 16:08:02 +09:00
Brad Fitzpatrick bf4426e35e types: add EmptyBody
Change-Id: Ic0cc6cae6debb20fbc15bd95692e8565171eae49
2014-03-31 09:36:08 -07:00
Tamás Gulácsi 65fa5d6d3e Picasa: add importer.
Imports all images from all albums, and puts each image in its album.
Each album is linked under "Picasa (<username>)".
Tries to skip already imported files (check based on album/filename).
See Issue #391.

Change-Id: If00751671a429891c05bbe41f59c1421b2ff2de8
2014-03-17 10:52:51 +01:00
Brad Fitzpatrick bfc607fee7 index: reindex blobs when dependent blobs arrive out-of-order
Keep track of missing dependencies both in memory and in the index's
underlying sorted.KeyValue. When we see a dependent blob arrive, see
if we can reindex things.

Fixes camlistore.org/issue/102

Change-Id: I3d8cfc463e4b8c9d158be8f9656e772839b093b9
2014-03-15 08:44:09 -07:00
Brad Fitzpatrick bf96b2af24 types: add NewFakeSeeker
NewFakeSeeker returns a ReadSeeker that can pretend to Seek (based
on the provided total size of the reader's content), but any reads
will fail if the fake seek position doesn't match reality.

Change-Id: I4e63aa78c80fdbcd87b2133018009ed72bedb9f3
2014-03-14 11:09:00 -07:00
Brad Fitzpatrick 79cb116dbb Merge "pkg/types/serverconfig: json-tagged struct" 2014-02-11 00:56:11 +00:00
mpl 6707837806 pkg/types/serverconfig: json-tagged struct
Also changed the TLS/HTTPS config keys for consistency.

http://camlistore.org/issue/339

Change-Id: I704ec129f91b93ebb20bc1191816166a2f10692d
2014-02-10 21:49:11 +01:00
Tamás Gulácsi 97520583b8 Use 'uint32' instead of 'int64' for blob sizes everywhere.
Not just in blob.SizedRef, but in blobserver.Fetch and
blobserver.FetchStreaming, too.
Blobs have a max size of 10-32 MB anyway, and the index.Corpus is now using
uint32 to save memory.

Change-Id: I1172445c2f9463fdaee55bfe0f1218d44be4aa53
2014-02-08 17:58:12 +01:00
Daniel Erat 5603ea8e0d pkg/index: Index audio duration.
Add pkg/media with code to calculate MPEG audio duration.
Index it in a "durationms" property.

Change-Id: Ifb6251657cadc365ef3f5667a0512fde17575560
2014-01-25 10:40:06 -08:00
mpl 20b9f5d6f1 pkg/types: add InvertedBool
Will be needed for json tagged serverconfig in subsequent commit.

http://camlistore.org/issue/339

Change-Id: I41ae6e9918b23eb9bdb471e402de0415f3499dc3
2014-01-23 22:54:18 +01:00
Bill Thiede 5c884e52ca diskpacked: keep fd open to every pack file.
Bytes read/writen per pack file, as well as per configured diskpacked
configuration are now available as expvars.

Also add reader stat helpers to pkg/types and updated the original
user in server/image.go

Change-Id: Ifc9d76c57aab329d4b947e9a4ef9eac008bc608d
2014-01-20 10:50:42 -08:00
mpl 3f13d1a043 pkg/client: fix method to find a server in the config
http://camlistore.org/issue/325

Change-Id: I4aa6d5103848b73ed169ab7d7de3a60a33723b79
2014-01-07 18:52:48 -08:00
Brad Fitzpatrick d21aa1dba2 year++
Change-Id: Ib5e17c39d3178357e1c18c4407ad45e91170ef16
2014-01-06 08:05:12 -08:00
mpl b7cee3c6ce clientconfig: forgot to git add it with bf8c463d0a
Change-Id: I098fd8c3297c48f0800f5753d220fe706601a16b
2014-01-06 08:00:35 -08:00
mpl 6e7eacea4a Merge "pkg/client: some cleanup, doc on the website" 2013-12-23 22:31:45 +00:00
mpl 1e8d46ee45 pkg/client: some cleanup, doc on the website
Changed some keys to be consistent with the server config, and removed
some others.

We're now using jsonconfig to get the keys and validate the config.

Removed some unused funcs/methods, moved some global vars as members of a client.

Also, having two possible defaults for the secret keyring
(.config/camlistore/identity-secring.gpg and .gnupg/secring.gpg)
was annoying and confusing imho, so I removed the latter.

http://camlistore.org/issue/259

Change-Id: Icf79fa8822b62b5d91843698861dd19cb553607a
2013-12-23 20:23:21 +01:00
Brad Fitzpatrick 12f87787fe types: add Time parsing test
Change-Id: Ie13a244cda76d11a39a1200895f4cd75013db087
2013-12-22 15:07:47 -08:00
Brad Fitzpatrick 95ea1dfb7d types: make Time3339 zero values be JSON null, back and forth
Change-Id: Iebf3defe041aee71e6d06b73159dbd105b545081
2013-12-20 21:35:30 -08:00
Brad Fitzpatrick 328706b8c6 index: bunch of TODOs and planning for edge tracking
Also, change to storing a pointer to claims so those pointers can be re-used elsewhere.

Change-Id: Ib0979d622ba48f1e2389575d07d7a0930264ebea
2013-12-09 17:15:34 +04:00
Brad Fitzpatrick 429fa4bf4e Fix API typo: ParseTime3339OrNil, not OrZil
Change-Id: I1255cf4560d133fe5c6128fba0be2d9d4a0c0daa
2013-12-03 21:56:09 -08:00
Brad Fitzpatrick 8c46afe490 Shrink camtypes.BlobMeta a bit. Saves 4 bytes of memory per blob.
For me,

Before:
Corpus stats: 148.859 MiB mem: 799465 blobs (89.487 GiB) (282303 schema (6478 permanode, 150962 file (112224 image), ...)

After:
Corpus stats: 146.559 MiB mem: 799465 blobs (89.487 GiB) (282303 schema (6478 permanode, 150962 file (112224 image), ...)

Change-Id: Id421dfc9d72c2a24c228418c01f9a3f05900c941
2013-11-29 13:41:39 -08:00
Brad Fitzpatrick 4eaeee9ae3 Reduce in-memory size of image info dimensions.
For my personal server,

Before:
Corpus stats: 149.884 MiB mem: 787886 blobs (88.853 GiB) (279308 schema (6073 permanode, 150557 file (111819 image), ...)

After:
Corpus stats: 145.172 MiB mem: 787886 blobs (88.853 GiB) (279308 schema (6073 permanode, 150557 file (111819 image), ...)

Change-Id: I47484e46b1eb70c8c4b5ee356cdcb0ceecde94e7
2013-11-29 11:01:41 -08:00
mpl fa32346860 pkg/index: check for deletions in EdgesTo
Also introduced and used kvEdgeBackward

http://camlistore.org/issue/191

Change-Id: Ifdca9f2453ec0c7db55afc40cf0b5f1441b83158
2013-11-28 19:41:40 +01:00
mpl 15bb5f142e pkg/index: use IsDeleted in some of the search methods
1) Done in AppendClaims, PermanodeOfSignerAttrValue,
SearchPermanodesWithAttr, PathsOfSignerTarget,
and PathsLookup.

2) camtypes.Path.DateClaim is now a time.Time, and used
as such in Path search methods.

3) Added kvSignerAttrValue, kvPathBackward, and kvPathForward

4) More Delete tests. We probably want more cases, but that's
a start.

http://camlistore.org/issue/191

Change-Id: I1d42485f2c6d1eea024f70378daacaaad9197768
2013-11-28 16:08:23 +01:00
mpl 45b630f766 pkg/index: change deletes cache structure and meaning
1) The deletes cache now simply reflects all the keyDeleted index entries,
instead of trying to always hold the computed deletion state of the
blobs.

2) The IsDeleted and DeletedAt methods have been changed accordingly.
Moreover, while they previously were only taking into account the most
recent delete claim that affected the input blob, they now take
into account all delete claims that concern said blob.
In other words, a blob is considered "active" only if all of its past
deleters are deleted too (or it does not have any deleter).

http://camlistore.org/issue/191

Change-Id: I1b276dac0b3c6a3fb3beaf96828a45a85d27e4c6
2013-11-27 17:02:03 +01:00
Brad Fitzpatrick 3fa6d69405 client: be robust against servers/proxies returning compressed content
Change-Id: Id87c26a7d6e6d3f794a5421ece7cd344b10e1a8b
2013-11-24 18:27:41 -08:00
Brad Fitzpatrick e8603b1293 Put claims in memory too for in-memory search. Required index schema version bump.
Change-Id: I194d65476bddea111277cd0b1472c56b5527226b
2013-11-17 16:52:51 -08:00
Brad Fitzpatrick 0b07171c0c index: change GetOwnerClaims interface for efficiency, rename to AppendClaims
Change-Id: Icec235961ed8a09256df87c5fe1455cbad81ae3f
2013-11-17 14:54:30 -08:00
Brad Fitzpatrick 705107ad80 search/index: invert depedency. search now depends on index.
creates new package types/camtypes for misc types needed by both. might eventually go away as
search matures.

Change-Id: Ib771ead7bea39936ba478b7e5d58de997060861b
2013-11-16 15:00:30 -08:00
Brad Fitzpatrick a58dcc1e5a diskpacked: re-use fds when possible when reading blobs from pack files.
We could actually do better than this and use the fd already open
from the still-appending file, or just keep one fd open per pack
file.

But this is a start.

Change-Id: I164a9bdf44987c3f057872cc7fdee1735f9e89a9
2013-10-10 20:06:30 -10:00
Brad Fitzpatrick b0bff61d30 types: move ReadSeekCloser types from pkg/blobref to pkg/types
Change-Id: I24e2b2a068d9b09479b145409a3b7e5693dd8ac5
2013-07-28 20:08:55 -07:00
Brad Fitzpatrick 2789f65d1f types: add an Atomic64 type
Change-Id: I4677b4368b98e658b921697331d999b58de85d7f
2013-07-21 20:45:20 -07:00
Brad Fitzpatrick 9468e5ba70 More docs. Every package is documented now.
misc.CountingReader moves into readerutil.

pkg/atomics is folded into pkg/types.

pkg/test/testdep is folded into pkg/test, with better name/docs.

Old cruft from pkg/webserver is deleted.

Change-Id: I3f72d8b29804254ef944995fb085837c878f79f5
2013-07-07 21:12:30 -07:00
mpl 481bdf346a blobref: enforce Go1.1 is required,
also types: rm Go1.0 RFC3339Nanos work-around

Change-Id: I9ce8495927fe349e7882f0a7d6c4f0051693fa2c
2013-06-05 01:19:08 +02:00
Brad Fitzpatrick 1b6a0b33fc types: add more time parsing functions
Change-Id: I35ffdd030fbdc87db4821f00fa562580e8205731
2013-02-18 21:30:11 -08:00
Brad Fitzpatrick 7e559039e0 types: add ByTime sorting, renable a test
Change-Id: Ifc80b66630f86cd88e66dfe68f1133696645f1a7
2013-02-18 19:09:27 -08:00
mpl 76d6a101b2 types: make Time3339 work for Go 1 too
Fixes http://code.google.com/p/camlistore/issues/detail?id=109

Change-Id: I4c30a77465e7774a0a867e95767f3e0ef1be7ec6
2013-02-13 18:52:58 +01:00
Brad Fitzpatrick 11e07730ca types: add Time3339.String and fix bug to use RFC3339Nano
Change-Id: Ib86b7ea86d9afe7738fc9a3ee9fa18fd2abef3b1
2013-02-10 12:57:56 -08:00
Brad Fitzpatrick 3605285924 Add pkg/types with types.Time3339; use it in search/handler JSON types
Change-Id: I79bf1484ef8b3548bca2e732f189352bcfbcd6db
2013-02-08 18:10:31 -08:00