Rooba
/
spaCy
mirror of https://github.com/explosion/spaCy.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
spaCy/spacy/lexeme.pxd

62 lines
959 B
Cython
Raw Normal View History

* Fiddle with data types on Lexeme, to compress them to a much smaller size.
2014-10-30 04:42:15 +00:00
from .typedefs cimport hash_t, utf8_t, flag_t, id_t, len_t, tag_t
* Restoring Lexeme-as-struct
2014-09-10 18:41:37 +00:00
* Large refactor, particularly to Python API
2014-10-23 13:59:17 +00:00
from .utf8string cimport StringStore
* Fiddle with data types on Lexeme, to compress them to a much smaller size.
2014-10-30 04:42:15 +00:00
from libc.stdint cimport uint16_t
* Revising data model of lexeme. Compiles.
2014-10-09 08:53:30 +00:00
* Large refactor, particularly to Python API
2014-10-23 13:59:17 +00:00
cpdef flag_t OOV_DIST_FLAGS
* Revising data model of lexeme. Compiles.
2014-10-09 08:53:30 +00:00
* Large refactor, particularly to Python API
2014-10-23 13:59:17 +00:00
# Flags
cpdef enum:
IS_ALPHA
IS_ASCII
IS_DIGIT
IS_LOWER
IS_PUNCT
IS_SPACE
IS_TITLE
IS_UPPER
* Revising data model of lexeme. Compiles.
2014-10-09 08:53:30 +00:00
* Add LIKE_URL and LIKE_NUMBER flag features
2014-11-02 02:19:05 +00:00
LIKE_URL
LIKE_NUMBER
* Large refactor, particularly to Python API
2014-10-23 13:59:17 +00:00
OFT_LOWER
OFT_TITLE
OFT_UPPER
* Revising data model of lexeme. Compiles.
2014-10-09 08:53:30 +00:00
* Remove vocab10k field, and add flags for gazetteers
2014-11-02 13:13:51 +00:00
IN_MALES
IN_FEMALES
IN_SURNAMES
IN_PLACES
IN_GAMES
IN_CELEBS
IN_NAMES
* Revising data model of lexeme. Compiles.
2014-10-09 08:53:30 +00:00
* Large refactor, particularly to Python API
2014-10-23 13:59:17 +00:00
cdef struct Lexeme:
* Fiddle with data types on Lexeme, to compress them to a much smaller size.
2014-10-30 04:42:15 +00:00
flag_t flags
* Rewriting Lexeme serialization.
2014-10-29 12:19:38 +00:00
* Restore id attribute to lexeme, and rename pos field to postype, to store clustered tag dictionaries
2014-10-31 06:43:00 +00:00
id_t id
* Fiddle with data types on Lexeme, to compress them to a much smaller size.
2014-10-30 04:42:15 +00:00
id_t sic
id_t norm
id_t shape
id_t asciied
id_t prefix
id_t suffix
* Slight cleaning of tokenizer code
2014-10-10 08:17:22 +00:00
* Large refactor, particularly to Python API
2014-10-23 13:59:17 +00:00
float prob
* Fiddle with data types on Lexeme, to compress them to a much smaller size.
2014-10-30 04:42:15 +00:00
len_t length
tag_t cluster
* Restore id attribute to lexeme, and rename pos field to postype, to store clustered tag dictionaries
2014-10-31 06:43:00 +00:00
tag_t postype
* Fiddle with data types on Lexeme, to compress them to a much smaller size.
2014-10-30 04:42:15 +00:00
tag_t supersense
* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding.
2014-10-22 14:57:59 +00:00
* Large refactor, particularly to Python API
2014-10-23 13:59:17 +00:00
cdef Lexeme EMPTY_LEXEME
* Restoring Lexeme-as-struct
2014-09-10 18:41:37 +00:00
* Restore id attribute to lexeme, and rename pos field to postype, to store clustered tag dictionaries
2014-10-31 06:43:00 +00:00
cpdef Lexeme init(id_t i, unicode string, hash_t hashed,
* Rewriting Lexeme serialization.
2014-10-29 12:19:38 +00:00
StringStore store, dict props) except *
* Add serialize/deserialize functions for lexeme, transport to/from python dict.
2014-10-09 03:10:46 +00:00
* Large refactor, particularly to Python API
2014-10-23 13:59:17 +00:00
cdef inline bint check_flag(Lexeme* lexeme, size_t flag_id) nogil:
return lexeme.flags & (1 << flag_id)