mirror of https://github.com/explosion/spaCy.git
105 lines
4.1 KiB
Plaintext
105 lines
4.1 KiB
Plaintext
//- 💫 DOCS > API > ANNOTATION > TRAINING
|
|
|
|
+h(3, "json-input") JSON input format for training
|
|
|
|
p
|
|
| spaCy takes training data in JSON format. The built-in
|
|
| #[+api("cli#convert") #[code convert]] command helps you convert the
|
|
| #[code .conllu] format used by the
|
|
| #[+a("https://github.com/UniversalDependencies") Universal Dependencies corpora]
|
|
| to spaCy's training format.
|
|
|
|
+aside("Annotating entities")
|
|
| Named entities are provided in the #[+a("/api/annotation#biluo") BILUO]
|
|
| notation. Tokens outside an entity are set to #[code "O"] and tokens
|
|
| that are part of an entity are set to the entity label, prefixed by the
|
|
| BILUO marker. For example #[code "B-ORG"] describes the first token of
|
|
| a multi-token #[code ORG] entity and #[code "U-PERSON"] a single
|
|
| token representing a #[code PERSON] entity. The
|
|
| #[+api("goldparse#biluo_tags_from_offsets") #[code biluo_tags_from_offsets]]
|
|
| function can help you convert entity offsets to the right format.
|
|
|
|
+code("Example structure").
|
|
[{
|
|
"id": int, # ID of the document within the corpus
|
|
"paragraphs": [{ # list of paragraphs in the corpus
|
|
"raw": string, # raw text of the paragraph
|
|
"sentences": [{ # list of sentences in the paragraph
|
|
"tokens": [{ # list of tokens in the sentence
|
|
"id": int, # index of the token in the document
|
|
"dep": string, # dependency label
|
|
"head": int, # offset of token head relative to token index
|
|
"tag": string, # part-of-speech tag
|
|
"orth": string, # verbatim text of the token
|
|
"ner": string # BILUO label, e.g. "O" or "B-ORG"
|
|
}],
|
|
"brackets": [{ # phrase structure (NOT USED by current models)
|
|
"first": int, # index of first token
|
|
"last": int, # index of last token
|
|
"label": string # phrase label
|
|
}]
|
|
}]
|
|
}]
|
|
}]
|
|
|
|
p
|
|
| Here's an example of dependencies, part-of-speech tags and names
|
|
| entities, taken from the English Wall Street Journal portion of the Penn
|
|
| Treebank:
|
|
|
|
+github("spacy", "examples/training/training-data.json", false, false, "json")
|
|
|
|
+h(3, "vocab-jsonl") Lexical data for vocabulary
|
|
+tag-new(2)
|
|
|
|
p
|
|
| The populate a model's vocabulary, you can use the
|
|
| #[+api("cli#vocab") #[code spacy vocab]] command and load in a
|
|
| #[+a("https://jsonlines.readthedocs.io/en/latest/") newline-delimited JSON]
|
|
| (JSONL) file containing one lexical entry per line. The first line
|
|
| defines the language and vocabulary settings. All other lines are
|
|
| expected to be JSON objects describing an individual lexeme. The lexical
|
|
| attributes will be then set as attributes on spaCy's
|
|
| #[+api("lexeme#attributes") #[code Lexeme]] object. The #[code vocab]
|
|
| command outputs a ready-to-use spaCy model with a #[code Vocab]
|
|
| containing the lexical data.
|
|
|
|
+code("First line").
|
|
{"lang": "en", "settings": {"oov_prob": -20.502029418945312}}
|
|
|
|
+code("Entry structure").
|
|
{
|
|
"orth": string,
|
|
"id": int,
|
|
"lower": string,
|
|
"norm": string,
|
|
"shape": string
|
|
"prefix": string,
|
|
"suffix": string,
|
|
"length": int,
|
|
"cluster": string,
|
|
"prob": float,
|
|
"is_alpha": bool,
|
|
"is_ascii": bool,
|
|
"is_digit": bool,
|
|
"is_lower": bool,
|
|
"is_punct": bool,
|
|
"is_space": bool,
|
|
"is_title": bool,
|
|
"is_upper": bool,
|
|
"like_url": bool,
|
|
"like_num": bool,
|
|
"like_email": bool,
|
|
"is_stop": bool,
|
|
"is_oov": bool,
|
|
"is_quote": bool,
|
|
"is_left_punct": bool,
|
|
"is_right_punct": bool
|
|
}
|
|
|
|
p
|
|
| Here's an example of the 100 most frequent lexemes in the English
|
|
| training data:
|
|
|
|
+github("spacy", "examples/training/vocab-data.jsonl", false, false, "json")
|