Matthew Honnibal 2805068ca8 * Have tokens track tuples that record the start offset and pos tag as well as a lexeme pointer		2014-10-14 15:21:03 +11:00
..
__init__.py	* Basic punct tests updated and passing	2014-08-27 19:38:57 +02:00
de.pxd	* Add German tokenizer files	2014-09-25 18:29:13 +02:00
de.pyx	* Add German tokenizer files	2014-09-25 18:29:13 +02:00
en.pxd	* Refactor tokenization, splitting it into a clearer life-cycle.	2014-09-16 13:16:02 +02:00
en.pyx	* Switch to new data model, tests passing	2014-10-10 08:11:31 +11:00
lang.pxd	* Preparations in place to handle hyphenation etc	2014-10-10 20:23:23 +11:00
lang.pyx	* Preparations in place to handle hyphenation etc	2014-10-10 20:23:23 +11:00
lexeme.pxd	* Rename LexStr_casefix to LexStr_norm and LexInt_i to LexInt_id	2014-10-14 15:19:07 +11:00
lexeme.pyx	* Rename LexStr_casefix to LexStr_norm and LexInt_i to LexInt_id	2014-10-14 15:19:07 +11:00
orth.py	* Switch to new data model, tests passing	2014-10-10 08:11:31 +11:00
ptb3.pxd	* Adding PTB3 tokenizer back in, so can understand how much boilerplate is in the docs for multiple tokenizers	2014-08-29 02:30:27 +02:00
ptb3.pyx	* Switch to using a Python ref counted gateway to malloc/free, to prevent memory leaks	2014-09-17 20:02:26 +02:00
tokens.pxd	* Have tokens track tuples that record the start offset and pos tag as well as a lexeme pointer	2014-10-14 15:21:03 +11:00
tokens.pyx	* Have tokens track tuples that record the start offset and pos tag as well as a lexeme pointer	2014-10-14 15:21:03 +11:00
typedefs.pxd	* Add typedefs file	2014-09-17 23:10:32 +02:00
util.py	* Update Lexicon class to expect a list of lexeme dict descriptions	2014-10-09 14:51:35 +11:00
word.pxd	* Switch to new data model, tests passing	2014-10-10 08:11:31 +11:00
word.pyx	* Slight cleaning of tokenizer code	2014-10-10 19:17:22 +11:00