5 Commits

Author	SHA1	Message	Date
Ines Montani	35b39f53c3	Reorganise English tokenizer exceptions (as discussed in #718) ... Add logic to generate exceptions that follow a consistent pattern (like verbs and pronouns) and allow certain tokens to be excluded explicitly.	2017-01-03 18:26:09 +01:00
Ines Montani	461cbb99d8	Revert "Reorganise English tokenizer exceptions (as discussed in #718)" ... This reverts commit b19cfcc144.	2017-01-03 18:21:29 +01:00
Ines Montani	b19cfcc144	Reorganise English tokenizer exceptions (as discussed in #718) ... Add logic to generate exceptions that follow a consistent pattern (like verbs and pronouns) and allow certain tokens to be excluded explicitly.	2017-01-03 18:17:57 +01:00
Ines Montani	78e63dc7d0	Update tokenizer exceptions for English	2016-12-21 18:06:34 +01:00
Ines Montani	704c7442e0	Break language data components into their own files	2016-12-18 15:36:53 +01:00