Rooba
/
spaCy
mirror of https://github.com/explosion/spaCy.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
spaCy/spacy/vocab.pxd

48 lines
1.3 KiB
Cython
Raw Normal View History

* Rename Lexicon to Vocab, and move it to its own file
2014-12-19 19:54:03 +00:00
from libcpp.vector cimport vector
from preshed.maps cimport PreshMap
from cymem.cymem cimport Pool
from murmurhash.mrmr cimport hash64
* Replace UniStr, using unicode objects instead
2015-07-22 02:49:39 +00:00
from .structs cimport LexemeC, TokenC
Use int8_t instead of char in Matcher (#6413) * Use signed char instead of char in Matcher Remove unused char* utf8_t typedef * Use int8_t instead of signed char
2020-11-23 09:26:47 +00:00
from .typedefs cimport attr_t, hash_t
* Rename Lexicon to Vocab, and move it to its own file
2014-12-19 19:54:03 +00:00
from .strings cimport StringStore
* Store Morphology class in Vocab
2015-08-26 17:21:03 +00:00
from .morphology cimport Morphology
* Rename Lexicon to Vocab, and move it to its own file
2014-12-19 19:54:03 +00:00
* Tmp commit. Refactoring to create a Python Lexeme class.
2015-01-11 23:26:22 +00:00
cdef LexemeC EMPTY_LEXEME
* Tmp
2014-12-24 06:42:00 +00:00
* Rename Lexicon to Vocab, and move it to its own file
2014-12-19 19:54:03 +00:00
cdef union LexemesOrTokens:
* Tmp commit. Refactoring to create a Python Lexeme class.
2015-01-11 23:26:22 +00:00
const LexemeC* const* lexemes
* More work on language-generic parsing
2015-08-28 00:02:33 +00:00
const TokenC* tokens
* Rename Lexicon to Vocab, and move it to its own file
2014-12-19 19:54:03 +00:00
cdef struct _Cached:
LexemesOrTokens data
bint is_lex
int length
cdef class Vocab:
cdef Pool mem
Replace cpdef variables with cdef (#7834)
2021-04-26 14:54:02 +00:00
cdef readonly StringStore strings
cdef public Morphology morphology
cdef public object vectors
cdef public object _lookups
cdef public object writing_system
cdef public object get_noun_chunks
* Index lexemes by orth, instead of a lexemes vector. Breaks the mechanism for deciding not to own LexemeC structs during parsing. Need to reinstate this.
2015-07-18 20:42:15 +00:00
cdef readonly int length
Remove two attributes marked for removal in 3.1 (#9150) * Remove two attributes marked for removal in 3.1 * Add back unused ints with changed names * Change data_dir to _unused_object This is still kept in the type definition, but I removed it from the serialization code. * Put serialization code back for now Not sure how this interacts with old serialized models yet.
2021-09-15 21:07:21 +00:00
cdef public object _unused_object # TODO remove in v4, see #9150
Refactor so that the tokenizer data is read from Python data, rather than from disk
2016-09-25 12:49:53 +00:00
cdef public object lex_attr_getters
Add Vocab.cfg attr, to hold stuff like oov probs
2017-10-30 15:08:50 +00:00
cdef public object cfg
* Rename Lexicon to Vocab, and move it to its own file
2014-12-19 19:54:03 +00:00
Update Cython string types (#9143) * Replace all basestring references with unicode `basestring` was a compatability type introduced by Cython to make dealing with utf-8 strings in Python2 easier. In Python3 it is equivalent to the unicode (or str) type. I replaced all references to basestring with unicode, since that was used elsewhere, but we could also just replace them with str, which shoudl also be equivalent. All tests pass locally. * Replace all references to unicode type with str Since we only support python3 this is simpler. * Remove all references to unicode type This removes all references to the unicode type across the codebase and replaces them with `str`, which makes it more drastic than the prior commits. In order to make this work importing `unicode_literals` had to be removed, and one explicit unicode literal also had to be removed (it is unclear why this is necessary in Cython with language level 3, but without doing it there were errors about implicit conversion). When `unicode` is used as a type in comments it was also edited to be `str`. Additionally `coding: utf8` headers were removed from a few files.
2021-09-13 15:02:17 +00:00
cdef const LexemeC* get(self, Pool mem, str string) except NULL
* Add serializer property to Vocab, and lazy-load it. Add get_by_orth method.
2015-07-22 23:18:19 +00:00
cdef const LexemeC* get_by_orth(self, Pool mem, attr_t orth) except NULL
* More work on language-generic parsing
2015-08-28 00:02:33 +00:00
cdef const TokenC* make_fused_token(self, substrings) except NULL
Work on vectors
2017-05-30 21:34:50 +00:00
Update Cython string types (#9143) * Replace all basestring references with unicode `basestring` was a compatability type introduced by Cython to make dealing with utf-8 strings in Python2 easier. In Python3 it is equivalent to the unicode (or str) type. I replaced all references to basestring with unicode, since that was used elsewhere, but we could also just replace them with str, which shoudl also be equivalent. All tests pass locally. * Replace all references to unicode type with str Since we only support python3 this is simpler. * Remove all references to unicode type This removes all references to the unicode type across the codebase and replaces them with `str`, which makes it more drastic than the prior commits. In order to make this work importing `unicode_literals` had to be removed, and one explicit unicode literal also had to be removed (it is unclear why this is necessary in Cython with language level 3, but without doing it there were errors about implicit conversion). When `unicode` is used as a type in comments it was also edited to be `str`. Additionally `coding: utf8` headers were removed from a few files.
2021-09-13 15:02:17 +00:00
cdef const LexemeC* _new_lexeme(self, Pool mem, str string) except NULL
* Tmp. Working on refactor. Compiles, must hook up lexical feats.
2015-01-13 13:03:48 +00:00
cdef int _add_lex_to_vocab(self, hash_t key, const LexemeC* lex) except -1
Update Cython string types (#9143) * Replace all basestring references with unicode `basestring` was a compatability type introduced by Cython to make dealing with utf-8 strings in Python2 easier. In Python3 it is equivalent to the unicode (or str) type. I replaced all references to basestring with unicode, since that was used elsewhere, but we could also just replace them with str, which shoudl also be equivalent. All tests pass locally. * Replace all references to unicode type with str Since we only support python3 this is simpler. * Remove all references to unicode type This removes all references to the unicode type across the codebase and replaces them with `str`, which makes it more drastic than the prior commits. In order to make this work importing `unicode_literals` had to be removed, and one explicit unicode literal also had to be removed (it is unclear why this is necessary in Cython with language level 3, but without doing it there were errors about implicit conversion). When `unicode` is used as a type in comments it was also edited to be `str`. Additionally `coding: utf8` headers were removed from a few files.
2021-09-13 15:02:17 +00:00
cdef const LexemeC* _new_lexeme(self, Pool mem, str string) except NULL
Remove trailing whitespace
2015-04-19 08:31:31 +00:00
* Index lexemes by orth, instead of a lexemes vector. Breaks the mechanism for deciding not to own LexemeC structs during parsing. Need to reinstate this.
2015-07-18 20:42:15 +00:00
cdef PreshMap _by_orth