Christoph Purschke a7ee4b6f17 new tests & tokenization fixes (#4734) ... - added some tests for tokenization issues - fixed some issues with tokenization of words with hyphen infix - rewrote the "tokenizer_exceptions.py" file (stemming from the German version)		2019-12-01 23:08:21 +01:00
..
ar	…
bn	…
ca	…
da	Move lookup tables out of the core library (#4346)	2019-10-01 00:01:27 +02:00
de	Move lookup tables out of the core library (#4346)	2019-10-01 00:01:27 +02:00
el	Revert #4334	2019-09-29 17:32:12 +02:00
en	Add tokenizer explain() debugging method (#4596)	2019-11-20 13:07:25 +01:00
es	Revert #4334	2019-09-29 17:32:12 +02:00
fi	…
fr	Move lookup tables out of the core library (#4346)	2019-10-01 00:01:27 +02:00
ga	…
he	…
hu	…
id	…
it	…
ja	Revert #4334	2019-09-29 17:32:12 +02:00
ko	Revert #4334	2019-09-29 17:32:12 +02:00
lb	new tests & tokenization fixes (#4734)	2019-12-01 23:08:21 +01:00
lt	Move lookup tables out of the core library (#4346)	2019-10-01 00:01:27 +02:00
nb	…
nl	Move lookup tables out of the core library (#4346)	2019-10-01 00:01:27 +02:00
pl	…
pt	…
ro	Move lookup tables out of the core library (#4346)	2019-10-01 00:01:27 +02:00
ru	…
sr	Move lookup tables out of the core library (#4346)	2019-10-01 00:01:27 +02:00
sv	Tidy up and auto-format [ci skip]	2019-10-24 16:20:48 +02:00
th	…
tr	Move lookup tables out of the core library (#4346)	2019-10-01 00:01:27 +02:00
tt	…
uk	…
ur	…
zh	Rework Chinese language initialization and tokenization (#4619)	2019-11-11 14:23:21 +01:00
__init__.py	…
test_attrs.py	…
test_initialize.py	…