spaCy/spacy/tokens.pxd

import numpy as np
cimport numpy as np

from cymem.cymem cimport Pool
from thinc.typedefs cimport atom_t

from .lexeme cimport Lexeme
from .typedefs cimport flags_t
from .utf8string cimport StringStore


cdef struct TokenC:
    const Lexeme* lex
    int idx
    int pos
    int sense


cdef class Tokens:
    cdef Pool mem
    cdef StringStore _string_store

    cdef TokenC* data

    cdef int length
    cdef int max_length

    cdef int extend(self, int i, const Lexeme* const* lexemes, int n) except -1
    cdef int push_back(self, int i, const Lexeme* lexeme) except -1
    cpdef int set_tag(self, int i, int tag_type, int tag) except -1

    cpdef np.ndarray[long, ndim=2] get_array(self, list features)


cdef class Token:
    cdef StringStore _string_store
    cdef public int i
    cdef public int idx
    cdef public int pos
    cdef public int ner

    cdef public atom_t id
    cdef public atom_t cluster
    cdef public atom_t length
    cdef public atom_t postype
    cdef public atom_t sensetype

    cdef public atom_t sic
    cdef public atom_t norm
    cdef public atom_t shape
    cdef public atom_t asciied
    cdef public atom_t prefix
    cdef public atom_t suffix

    cdef public float prob

    cdef public flags_t flags
* Work on get_array method of Tokens 2014-12-02 12:48:05 +00:00			`import numpy as np`
			`cimport numpy as np`

* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 14:57:59 +00:00			`from cymem.cymem cimport Pool`
* Add as_array and count_by method 2014-12-04 09:46:55 +00:00			`from thinc.typedefs cimport atom_t`
* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 14:57:59 +00:00
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`from .lexeme cimport Lexeme`
* Hack Tokens to work without tagger.pyx 2014-12-03 00:05:15 +00:00			`from .typedefs cimport flags_t`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`from .utf8string cimport StringStore`

* Moving tokens to use an array internally, instead of a list of Lexeme objects. 2014-09-11 14:57:08 +00:00
* Introduce a TokenC struct, to handle token indices, pos tags and sense tags 2014-12-05 04:56:14 +00:00			`cdef struct TokenC:`
			`const Lexeme* lex`
			`int idx`
			`int pos`
			`int sense`


* Upd Tokens to use vector, with bounds checking. 2014-09-15 01:22:40 +00:00			`cdef class Tokens:`
* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 14:57:59 +00:00			`cdef Pool mem`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`cdef StringStore _string_store`
* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 14:57:59 +00:00
* Introduce a TokenC struct, to handle token indices, pos tags and sense tags 2014-12-05 04:56:14 +00:00			`cdef TokenC* data`
* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 14:57:59 +00:00
			`cdef int length`
			`cdef int max_length`
* Have tokens track tuples that record the start offset and pos tag as well as a lexeme pointer 2014-10-14 04:21:03 +00:00
* Introduce first attempt at const-ness 2014-12-03 04:44:25 +00:00			`cdef int extend(self, int i, const Lexeme* const* lexemes, int n) except -1`
			`cdef int push_back(self, int i, const Lexeme* lexeme) except -1`
* Hack Tokens to work without tagger.pyx 2014-12-03 00:05:15 +00:00			`cpdef int set_tag(self, int i, int tag_type, int tag) except -1`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00
* Add as_array and count_by method 2014-12-04 09:46:55 +00:00			`cpdef np.ndarray[long, ndim=2] get_array(self, list features)`
* Work on get_array method of Tokens 2014-12-02 12:48:05 +00:00
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00
			`cdef class Token:`
			`cdef StringStore _string_store`
			`cdef public int i`
			`cdef public int idx`
			`cdef public int pos`
* Complete refactor of Tagger features, to use a generic list of context names. 2014-11-05 09:45:29 +00:00			`cdef public int ner`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00
* Rename Token.lex_pos to Token.postype, and Token.lex_supersense to Token.sensetype 2014-10-31 06:44:39 +00:00			`cdef public atom_t id`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`cdef public atom_t cluster`
			`cdef public atom_t length`
* Rename Token.lex_pos to Token.postype, and Token.lex_supersense to Token.sensetype 2014-10-31 06:44:39 +00:00			`cdef public atom_t postype`
			`cdef public atom_t sensetype`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00
* Rewriting Lexeme serialization. 2014-10-29 12:19:38 +00:00			`cdef public atom_t sic`
* Large refactor, particularly to Python API 2014-10-23 13:59:17 +00:00			`cdef public atom_t norm`
			`cdef public atom_t shape`
			`cdef public atom_t asciied`
			`cdef public atom_t prefix`
			`cdef public atom_t suffix`

			`cdef public float prob`

* Hack Tokens to work without tagger.pyx 2014-12-03 00:05:15 +00:00			`cdef public flags_t flags`