Commit Graph

4 Commits

Author SHA1 Message Date
Matthew Honnibal 14e9e6ec6c * Fix ... tokenization, and correct orth inconsistencies in specials.json 2015-07-20 12:10:56 +02:00
Matthew Honnibal b5b869366b * Adjust hyphenation rule in tokenizer 2015-06-28 06:18:58 +02:00
Matthew Honnibal 45ec92243a * Add hyphenation rule to infix.txt for tokenizer 2015-06-06 05:56:00 +02:00
Matthew Honnibal 5e27bd0c4c * Add en language data, for tokenizer etc 2015-02-25 17:10:32 -05:00