spaCy/spacy/lexeme.pxd

from libc.stdint cimport uint32_t
from libc.stdint cimport uint64_t

# Put these above import to avoid circular import problem
ctypedef int ClusterID
ctypedef uint32_t StringHash
ctypedef size_t Lexeme_addr
ctypedef char Bits8
ctypedef uint64_t Bits64


cdef struct Orthography:
    StringHash last3
    StringHash shape
    StringHash norm

    size_t length
    unsigned char first
    Bits8 flags


cdef struct Distribution:
    double prob
    ClusterID cluster
    Bits64 tagdict
    Bits8 flags


cdef struct Lexeme:
    StringHash sic # Hash of the original string
    StringHash lex # Hash of the word, with punctuation and clitics split off

    Distribution* dist # Distribution info, lazy loaded
    Orthography* orth  # Extra orthographic views
    #Lexeme* tail # Lexemes are linked lists, to deal with sub-tokens


cdef Lexeme BLANK_WORD = Lexeme(0, 0, NULL, NULL, NULL)


cdef enum StringAttr:
    LEX
    NORM
    SHAPE
    LAST3
    LENGTH


cpdef StringHash attr_of(size_t lex_id, StringAttr attr) except 0

cpdef StringHash lex_of(size_t lex_id) except 0
cpdef StringHash norm_of(size_t lex_id) except 0
cpdef StringHash shape_of(size_t lex_id) except 0
cpdef StringHash last3_of(size_t lex_id) except 0
cpdef StringHash length_of(size_t lex_id)
* Switch to 32bit hash for strings 2014-08-02 20:51:52 +00:00			`from libc.stdint cimport uint32_t`
* Initial commit. Tests passing for punctuation handling. Need contractions, file transport, tokenize function, etc. 2014-07-05 18:51:42 +00:00			`from libc.stdint cimport uint64_t`

* Reorganized, moving language-independent stuff to spacy. The functions in spacy ask for the dictionaries and split function on input, but the language-specific modules are curried versions that use the globals 2014-07-07 02:21:06 +00:00			`# Put these above import to avoid circular import problem`
* Initial commit. Tests passing for punctuation handling. Need contractions, file transport, tokenize function, etc. 2014-07-05 18:51:42 +00:00			`ctypedef int ClusterID`
* Switch to 32bit hash for strings 2014-08-02 20:51:52 +00:00			`ctypedef uint32_t StringHash`
* Reorganized, moving language-independent stuff to spacy. The functions in spacy ask for the dictionaries and split function on input, but the language-specific modules are curried versions that use the globals 2014-07-07 02:21:06 +00:00			`ctypedef size_t Lexeme_addr`
* Refactor for string view features. Working on setting up flags and enums. 2014-07-07 14:58:48 +00:00			`ctypedef char Bits8`
			`ctypedef uint64_t Bits64`

* Initial commit. Tests passing for punctuation handling. Need contractions, file transport, tokenize function, etc. 2014-07-05 18:51:42 +00:00
* Refactor for string view features. Working on setting up flags and enums. 2014-07-07 14:58:48 +00:00			`cdef struct Orthography:`
			`StringHash last3`
			`StringHash shape`
			`StringHash norm`

* Add length property 2014-08-02 20:26:44 +00:00			`size_t length`
* Restore string saving to spacy 2014-08-16 14:09:24 +00:00			`unsigned char first`
* Refactor for string view features. Working on setting up flags and enums. 2014-07-07 14:58:48 +00:00			`Bits8 flags`


			`cdef struct Distribution:`
			`double prob`
			`ClusterID cluster`
			`Bits64 tagdict`
			`Bits8 flags`


* Initial commit. Tests passing for punctuation handling. Need contractions, file transport, tokenize function, etc. 2014-07-05 18:51:42 +00:00			`cdef struct Lexeme:`
			`StringHash sic # Hash of the original string`
			`StringHash lex # Hash of the word, with punctuation and clitics split off`

* Refactor for string view features. Working on setting up flags and enums. 2014-07-07 14:58:48 +00:00			`Distribution* dist # Distribution info, lazy loaded`
			`Orthography* orth # Extra orthographic views`
* Refactor spacy so that chunks return arrays of lexemes, so that there is properly one lexeme per word. 2014-08-18 17:14:00 +00:00			`#Lexeme* tail # Lexemes are linked lists, to deal with sub-tokens`
* Initial commit. Tests passing for punctuation handling. Need contractions, file transport, tokenize function, etc. 2014-07-05 18:51:42 +00:00

* Refactor for string view features. Working on setting up flags and enums. 2014-07-07 14:58:48 +00:00			`cdef Lexeme BLANK_WORD = Lexeme(0, 0, NULL, NULL, NULL)`
* Reorganized, moving language-independent stuff to spacy. The functions in spacy ask for the dictionaries and split function on input, but the language-specific modules are curried versions that use the globals 2014-07-07 02:21:06 +00:00
* Refactor for string view features. Working on setting up flags and enums. 2014-07-07 14:58:48 +00:00
Group-by seems to be working 2014-07-07 18:27:02 +00:00			`cdef enum StringAttr:`
			`LEX`
			`NORM`
			`SHAPE`
			`LAST3`
* Add length property 2014-08-02 20:26:44 +00:00			`LENGTH`
Group-by seems to be working 2014-07-07 18:27:02 +00:00

			`cpdef StringHash attr_of(size_t lex_id, StringAttr attr) except 0`

* Refactor for string view features. Working on setting up flags and enums. 2014-07-07 14:58:48 +00:00			`cpdef StringHash lex_of(size_t lex_id) except 0`
			`cpdef StringHash norm_of(size_t lex_id) except 0`
* 710k words per second for counts 2014-07-07 17:12:19 +00:00			`cpdef StringHash shape_of(size_t lex_id) except 0`
Group-by seems to be working 2014-07-07 18:27:02 +00:00			`cpdef StringHash last3_of(size_t lex_id) except 0`
* Add length property 2014-08-02 20:26:44 +00:00			`cpdef StringHash length_of(size_t lex_id)`