spaCy/website/usage/_spacy-101/_tokenization.jade

//- 💫 DOCS > USAGE > SPACY 101 > TOKENIZATION

p
    |  During processing, spaCy first #[strong tokenizes] the text, i.e.
    |  segments it into words, punctuation and so on. This is done by applying
    |  rules specific to each language. For example, punctuation at the end of a
    |  sentence should be split off – whereas "U.K." should remain one token.
    |  Each #[code Doc] consists of individual tokens, and we can iterate
    |  over them:

+code-exec.
    import spacy

    nlp = spacy.load('en_core_web_sm')
    doc = nlp(u'Apple is looking at buying U.K. startup for $1 billion')
    for token in doc:
        print(token.text)

+table([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]).u-text-center
    +row
        for cell in ["Apple", "is", "looking", "at", "buying", "U.K.", "startup", "for", "$", "1", "billion"]
            +cell=cell

p
    |  First, the raw text is split on whitespace characters, similar to
    |  #[code text.split(' ')]. Then, the tokenizer processes the text from
    |  left to right. On each substring, it performs two checks:

+list("numbers")
    +item
        |  #[strong Does the substring match a tokenizer exception rule?] For
        |  example, "don't" does not contain whitespace, but should be split
        |  into two tokens, "do" and "n't", while "U.K." should always
        |  remain one token.
    +item
        |  #[strong Can a prefix, suffix or infix be split off?] For example
        |  punctuation like commas, periods, hyphens or quotes.

p
    |  If there's a match, the rule is applied and the tokenizer continues its
    |  loop, starting with the newly split substrings. This way, spaCy can split
    |  #[strong complex, nested tokens] like combinations of abbreviations and
    |  multiple punctuation marks.

+aside
    |  #[strong Tokenizer exception:] Special-case rule to split a string into
    |  several tokens or prevent a token from being split when punctuation rules
    |  are applied.#[br]
    |  #[strong Prefix:] Character(s) at the beginning, e.g.
    |  #[code $], #[code (], #[code “], #[code ¿].#[br]
    |  #[strong Suffix:] Character(s) at the end, e.g.
    |  #[code km], #[code )], #[code ”], #[code !].#[br]
    |  #[strong Infix:] Character(s) in between, e.g.
    |  #[code -], #[code --], #[code /], #[code …].#[br]

+graphic("/assets/img/tokenization.svg")
    include ../../assets/img/tokenization.svg

p
    |  While punctuation rules are usually pretty general, tokenizer exceptions
    |  strongly depend on the specifics of the individual language. This is
    |  why each #[+a("/usage/models#languages") available language] has its
    |  own subclass like #[code English] or #[code German], that loads in lists
    |  of hard-coded data and exception rules.