spaCy/website/docs/usage
Paul O'Leary McCann 1ee6541ab0
Moving Japanese tokenizer extra info to Token.morph (#8977)
* Use morph for extra Japanese tokenizer info

Previously Japanese tokenizer info that didn't correspond to Token
fields was put in user data. Since spaCy core should avoid touching user
data, this moves most information to the Token.morph attribute. It also
adds the normalized form, which wasn't exposed before.

The subtokens, which are a list of full tokens, are still added to user
data, except with the default tokenizer granualarity. With the default
tokenizer settings the subtokens are all None, so in this case the user
data is simply not set.

* Update tests

Also adds a new test for norm data.

* Update docs

* Add Japanese morphologizer factory

Set the default to `extend=True` so that the morphologizer does not
clobber the values set by the tokenizer.

* Use the norm_ field for normalized forms

Before this commit, normalized forms were put in the "norm" field in the
morph attributes. I am not sure why I did that instead of using the
token morph, I think I just forgot about it.

* Skip test if sudachipy is not installed

* Fix import

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
2021-10-01 19:19:26 +02:00
..
101 Update _training.md - Fix broken link on backpropagation (#7431) 2021-03-15 09:21:35 +01:00
_benchmarks-models.md final 3.0 benchmark numbers 2021-02-09 21:28:33 +01:00
embeddings-transformers.md Make docs consistent (fix #9126) 2021-09-16 15:54:12 +09:00
facts-figures.md final 3.0 benchmark numbers 2021-02-09 21:28:33 +01:00
index.md Add support for CUDA 11.2 (#7583) 2021-03-30 09:47:33 +02:00
layers-architectures.md Resizable textcat (#7862) 2021-06-16 11:45:00 +02:00
linguistic-features.md Typo fixes 2021-07-07 16:53:04 -06:00
models.md Moving Japanese tokenizer extra info to Token.morph (#8977) 2021-10-01 19:19:26 +02:00
processing-pipelines.md Minor typo fix in docs 2021-09-11 14:22:05 +09:00
projects.md Move WandB loggers into spacy-loggers (#9223) 2021-09-29 11:12:50 +02:00
rule-based-matching.md matcher doc corrections (#9115) 2021-09-02 09:26:33 +02:00
saving-loading.md Update saving/loading example 2021-03-18 16:56:10 +01:00
spacy-101.md Merge branch 'master' into develop 2020-12-11 13:44:41 +11:00
training.md Move WandB loggers into spacy-loggers (#9223) 2021-09-29 11:12:50 +02:00
v2-1.md Merge branch 'spacy.io' [ci skip] 2021-03-06 17:38:54 +11:00
v2-2.md Update v3 docs [ci skip] 2020-07-05 16:11:16 +02:00
v2-3.md Merge branch 'spacy.io' [ci skip] 2021-03-06 17:38:54 +11:00
v2.md Merge branch 'spacy.io' [ci skip] 2021-03-06 17:38:54 +11:00
v3-1.md Update v3.1 usage docs 2021-07-07 08:43:33 +02:00
v3.md Update website/docs/usage/v3.md 2021-07-29 14:56:56 +09:00
visualizers.md Adjust kb_id visualizer templating and docs 2021-09-23 11:59:02 +02:00