spaCy/spacy/tokens.pxd

from cymem.cymem cimport Pool

from .lexeme cimport Lexeme
from .typedefs cimport flag_t
from .utf8string cimport StringStore
from .tagger cimport TagType

from thinc.typedefs cimport atom_t


cdef class Tokens:
    cdef Pool mem
    cdef StringStore _string_store

    cdef Lexeme** _lex_ptr
    cdef int* _idx_ptr
    cdef int* _pos_ptr
    cdef int* _ner_ptr
    cdef Lexeme** lex
    cdef int* idx
    cdef int* pos
    cdef int* ner

    cdef int length
    cdef int max_length

    cdef int extend(self, int i, Lexeme** lexemes, int n) except -1
    cdef int push_back(self, int i, Lexeme* lexeme) except -1
    cpdef int set_tag(self, int i, TagType tag_type, int tag) except -1


cdef class Token:
    cdef StringStore _string_store
    cdef public int i
    cdef public int idx
    cdef public int pos
    cdef public int ner

    cdef public atom_t id
    cdef public atom_t cluster
    cdef public atom_t length
    cdef public atom_t postype
    cdef public atom_t sensetype

    cdef public atom_t sic
    cdef public atom_t norm
    cdef public atom_t shape
    cdef public atom_t asciied
    cdef public atom_t prefix
    cdef public atom_t suffix

    cdef public float prob

    cdef public flag_t flags
* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 14:57:59 +00:00			`from cymem.cymem cimport Pool`

* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`from .lexeme cimport Lexeme`
			`from .typedefs cimport flag_t`
			`from .utf8string cimport StringStore`
* Generalize tagger code, in preparation for NER and supersense tagging. 2014-11-04 16:42:14 +00:00			`from .tagger cimport TagType`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00
* Improve array features in tokens 2014-10-22 01:55:42 +00:00			`from thinc.typedefs cimport atom_t`
* Moving tokens to use an array internally, instead of a list of Lexeme objects. 2014-09-11 14:57:08 +00:00

* Upd Tokens to use vector, with bounds checking. 2014-09-15 01:22:40 +00:00			`cdef class Tokens:`
* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 14:57:59 +00:00			`cdef Pool mem`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`cdef StringStore _string_store`
* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 14:57:59 +00:00
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`cdef Lexeme** _lex_ptr`
* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 14:57:59 +00:00			`cdef int* _idx_ptr`
			`cdef int* _pos_ptr`
* Complete refactor of Tagger features, to use a generic list of context names. 2014-11-05 09:45:29 +00:00			`cdef int* _ner_ptr`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`cdef Lexeme** lex`
* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 14:57:59 +00:00			`cdef int* idx`
			`cdef int* pos`
* Complete refactor of Tagger features, to use a generic list of context names. 2014-11-05 09:45:29 +00:00			`cdef int* ner`
* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 14:57:59 +00:00
			`cdef int length`
			`cdef int max_length`
* Have tokens track tuples that record the start offset and pos tag as well as a lexeme pointer 2014-10-14 04:21:03 +00:00
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`cdef int extend(self, int i, Lexeme** lexemes, int n) except -1`
			`cdef int push_back(self, int i, Lexeme* lexeme) except -1`
* Generalize tagger code, in preparation for NER and supersense tagging. 2014-11-04 16:42:14 +00:00			`cpdef int set_tag(self, int i, TagType tag_type, int tag) except -1`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00

			`cdef class Token:`
			`cdef StringStore _string_store`
			`cdef public int i`
			`cdef public int idx`
			`cdef public int pos`
* Complete refactor of Tagger features, to use a generic list of context names. 2014-11-05 09:45:29 +00:00			`cdef public int ner`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00
* Rename Token.lex_pos to Token.postype, and Token.lex_supersense to Token.sensetype 2014-10-31 06:44:39 +00:00			`cdef public atom_t id`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`cdef public atom_t cluster`
			`cdef public atom_t length`
* Rename Token.lex_pos to Token.postype, and Token.lex_supersense to Token.sensetype 2014-10-31 06:44:39 +00:00			`cdef public atom_t postype`
			`cdef public atom_t sensetype`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00
* Rewriting Lexeme serialization. 2014-10-29 12:19:38 +00:00			`cdef public atom_t sic`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`cdef public atom_t norm`
			`cdef public atom_t shape`
			`cdef public atom_t asciied`
			`cdef public atom_t prefix`
			`cdef public atom_t suffix`

			`cdef public float prob`

			`cdef public flag_t flags`