spaCy/spacy/lang.pxd

from libc.stdint cimport uint32_t
from libc.stdint cimport uint64_t
from spacy.word cimport Lexeme
from spacy.tokens cimport Tokens


cdef struct Flags:
    size_t is_alpha
    size_t can_noun


cdef struct ViewIDs:
    size_t canon_form


cdef class Lexicon:
    cpdef readonly size_t size

    cpdef Lexeme lookup(self, unicode string)
    
    cdef dict _dict
    
    cdef list _string_features
    cdef list _flag_features


cdef class Language:
    cdef unicode name
    cdef dict cache
    cpdef readonly Lexicon lexicon
    cpdef readonly object tokens_class

    cpdef list tokenize(self, unicode text)
    cpdef Lexeme lookup(self, unicode text)

    cdef _tokenize(self, Tokens tokens, unicode string)
    cdef list _split(self, unicode string)
    cdef int _split_one(self, unicode word)
* Switch to 32bit hash for strings 2014-08-02 20:51:52 +00:00			`from libc.stdint cimport uint32_t`
* Reorganized, moving language-independent stuff to spacy. The functions in spacy ask for the dictionaries and split function on input, but the language-specific modules are curried versions that use the globals 2014-07-07 02:21:06 +00:00			`from libc.stdint cimport uint64_t`
* More refactoring 2014-08-25 14:42:22 +00:00			`from spacy.word cimport Lexeme`
* Refactoring to use Tokens object 2014-09-10 16:11:13 +00:00			`from spacy.tokens cimport Tokens`


			`cdef struct Flags:`
			`size_t is_alpha`
			`size_t can_noun`


			`cdef struct ViewIDs:`
			`size_t canon_form`
* Refactor for string view features. Working on setting up flags and enums. 2014-07-07 14:58:48 +00:00
* Reorganized code to accomodate Tokens class. Need string views before group_by and count_by can be done well. 2014-07-07 10:47:21 +00:00
* Refactoring with Lexeme as a class now compiles. Basic design seems to work 2014-08-27 15:15:39 +00:00			`cdef class Lexicon:`
* Refactoring to use Tokens object 2014-09-10 16:11:13 +00:00			`cpdef readonly size_t size`

* Refactoring with Lexeme as a class now compiles. Basic design seems to work 2014-08-27 15:15:39 +00:00			`cpdef Lexeme lookup(self, unicode string)`
* Docs coming together 2014-08-28 23:59:23 +00:00
			`cdef dict _dict`

			`cdef list _string_features`
			`cdef list _flag_features`
* Refactoring with Lexeme as a class now compiles. Basic design seems to work 2014-08-27 15:15:39 +00:00

* Reorganized code to accomodate Tokens class. Need string views before group_by and count_by can be done well. 2014-07-07 10:47:21 +00:00			`cdef class Language:`
* Docs coming together 2014-08-28 23:59:23 +00:00			`cdef unicode name`
* Refactoring with Lexeme as a class now compiles. Basic design seems to work 2014-08-27 15:15:39 +00:00			`cdef dict cache`
			`cpdef readonly Lexicon lexicon`
* Refactoring to use Tokens object 2014-09-10 16:11:13 +00:00			`cpdef readonly object tokens_class`
* Reorganized, moving language-independent stuff to spacy. The functions in spacy ask for the dictionaries and split function on input, but the language-specific modules are curried versions that use the globals 2014-07-07 02:21:06 +00:00
* Moving to Word objects in place of the Lexeme struct. 2014-08-22 15:28:23 +00:00			`cpdef list tokenize(self, unicode text)`
* Docs coming together 2014-08-28 23:59:23 +00:00			`cpdef Lexeme lookup(self, unicode text)`
* Refactoring tokenizer 2014-08-16 01:22:03 +00:00
* Refactoring to use Tokens object 2014-09-10 16:11:13 +00:00			`cdef _tokenize(self, Tokens tokens, unicode string)`
* Docs coming together 2014-08-28 23:59:23 +00:00			`cdef list _split(self, unicode string)`
			`cdef int _split_one(self, unicode word)`