2 Commits

Author	SHA1	Message	Date
Ines Montani	40bb918a4c	Remove unicode declarations and tidy up	2020-06-21 22:34:10 +02:00
adrianeboyd	3bf111585d	Update Japanese tokenizer config and add serialization (#5562) ... * Use `config` dict for tokenizer settings * Add serialization of split mode setting * Add tests for tokenizer split modes and serialization of split mode setting Based on #5561	2020-06-08 16:29:05 +02:00