From 0c31f03ec5525cd33224a880b6d678c69019727d Mon Sep 17 00:00:00 2001 From: Adriane Boyd Date: Mon, 9 Mar 2020 13:41:01 +0100 Subject: [PATCH] Update docs [ci skip] --- website/docs/usage/linguistic-features.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/website/docs/usage/linguistic-features.md b/website/docs/usage/linguistic-features.md index 60a6699a9..0ceae4c4f 100644 --- a/website/docs/usage/linguistic-features.md +++ b/website/docs/usage/linguistic-features.md @@ -838,8 +838,6 @@ domain. There are five things you would need to define: hyphens etc. 5. An optional boolean function `token_match` matching strings that should never be split, overriding the infix rules. Useful for things like URLs or numbers. - Note that prefixes and suffixes will be split off before `token_match` is - applied. You shouldn't usually need to create a `Tokenizer` subclass. Standard usage is to use `re.compile()` to build a regular expression object, and pass its