mirror of https://github.com/explosion/spaCy.git
60 lines
2.6 KiB
Plaintext
60 lines
2.6 KiB
Plaintext
|
//- 💫 DOCS > USAGE > ADDING LANGUAGES
|
|||
|
|
|
|||
|
|
include ../_includes/_mixins
|
|||
|
|
|
|||
|
|
+aside("Working on spaCy's source")
|
|||
|
|
| To add a new language to spaCy, you'll need to
|
|||
|
|
| #[strong modify the library's code]. The easiest way to do this is to
|
|||
|
|
| clone the #[+src(gh("spaCy")) repository] and #[strong build spaCy from source].
|
|||
|
|
| For more information on this, see the #[+a("/usage") installation guide].
|
|||
|
|
| Unlike spaCy's core, which is mostly written in Cython, all language
|
|||
|
|
| data is stored in regular Python files. This means that you won't have to
|
|||
|
|
| rebuild anything in between – you can simply make edits and reload spaCy
|
|||
|
|
| to test them.
|
|||
|
|
|
|||
|
|
+grid.o-no-block
|
|||
|
|
+grid-col("half")
|
|||
|
|
p
|
|||
|
|
| Obviously, there are lots of ways you can organise your code when
|
|||
|
|
| you implement your own language data. This guide will focus on
|
|||
|
|
| how it's done within spaCy. For full language support, you'll
|
|||
|
|
| need to create a #[code Language] subclass, define custom
|
|||
|
|
| #[strong language data], like a stop list and tokenizer
|
|||
|
|
| exceptions and test the new tokenizer. Once the language is set
|
|||
|
|
| up, you can #[strong build the vocabulary], including word
|
|||
|
|
| frequencies, Brown clusters and word vectors. Finally, you can
|
|||
|
|
| #[strong train the tagger and parser], and save the model to a
|
|||
|
|
| directory.
|
|||
|
|
|
|||
|
|
p
|
|||
|
|
| For some languages, you may also want to develop a solution for
|
|||
|
|
| lemmatization and morphological analysis.
|
|||
|
|
|
|||
|
|
+table-of-contents
|
|||
|
|
+item #[+a("#101") Language data 101]
|
|||
|
|
+item #[+a("#language-subclass") The Language subclass]
|
|||
|
|
+item #[+a("#stop-words") Stop words]
|
|||
|
|
+item #[+a("#tokenizer-exceptions") Tokenizer exceptions]
|
|||
|
|
+item #[+a("#norm-exceptions") Norm exceptions]
|
|||
|
|
+item #[+a("#lex-attrs") Lexical attributes]
|
|||
|
|
+item #[+a("#syntax-iterators") Syntax iterators]
|
|||
|
|
+item #[+a("#lemmatizer") Lemmatizer]
|
|||
|
|
+item #[+a("#tag-map") Tag map]
|
|||
|
|
+item #[+a("#morph-rules") Morph rules]
|
|||
|
|
+item #[+a("#testing") Testing the language]
|
|||
|
|
+item #[+a("#vocabulary") Building the vocabulary]
|
|||
|
|
+item #[+a("#training") Training]
|
|||
|
|
|
|||
|
|
+section("language-data")
|
|||
|
|
+h(2, "language-data") Language data
|
|||
|
|
include _spacy-101/_language-data
|
|||
|
|
include _adding-languages/_language-data
|
|||
|
|
|
|||
|
|
+section("testing")
|
|||
|
|
+h(2, "testing") Testing the new language
|
|||
|
|
include _adding-languages/_testing
|
|||
|
|
|
|||
|
|
+section("training")
|
|||
|
|
+h(2, "training") Training a language model
|
|||
|
|
include _adding-languages/_training
|