spaCy/spacy/structs.pxd

from libc.stdint cimport uint8_t, uint32_t, int32_t, uint64_t

from .typedefs cimport flags_t, attr_t, hash_t
from .parts_of_speech cimport univ_pos_t


cdef struct LexemeC:
    flags_t flags

    attr_t lang

    attr_t id
    attr_t length

    attr_t orth
    attr_t lower
    attr_t norm
    attr_t shape
    attr_t prefix
    attr_t suffix

    attr_t cluster

    float prob
    float sentiment


cdef struct SerializedLexemeC:
    unsigned char[8 + 8*10 + 4 + 4] data
    #    sizeof(flags_t)  # flags
    #    + sizeof(attr_t) # lang
    #    + sizeof(attr_t) # id
    #    + sizeof(attr_t) # length
    #    + sizeof(attr_t) # orth
    #    + sizeof(attr_t) # lower
    #    + sizeof(attr_t) # norm
    #    + sizeof(attr_t) # shape
    #    + sizeof(attr_t) # prefix
    #    + sizeof(attr_t) # suffix
    #    + sizeof(attr_t) # cluster
    #    + sizeof(float)  # prob
    #    + sizeof(float)  # cluster
    #    + sizeof(float) # l2_norm


cdef struct Entity:
    hash_t id
    int start
    int end
    attr_t label


cdef struct TokenC:
    const LexemeC* lex
    uint64_t morph
    univ_pos_t pos
    bint spacy
    attr_t tag
    int idx
    attr_t lemma
    attr_t sense
    int head
    attr_t dep

    uint32_t l_kids
    uint32_t r_kids
    uint32_t l_edge
    uint32_t r_edge

    int sent_start
    int ent_iob
    attr_t ent_type # TODO: Is there a better way to do this? Multiple sources of truth..
    hash_t ent_id
* Hack on morphology structs 2015-08-26 17:18:36 +00:00			`from libc.stdint cimport uint8_t, uint32_t, int32_t, uint64_t`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00
* Fix type declarations for attr_t. Remove unused id_t. 2015-07-18 20:39:57 +00:00			`from .typedefs cimport flags_t, attr_t, hash_t`
* Move POS tag definitions to parts_of_speech.pxd 2015-01-25 05:31:07 +00:00			`from .parts_of_speech cimport univ_pos_t`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00

* Tmp commit. Refactoring to create a Python Lexeme class. 2015-01-11 23:26:22 +00:00			`cdef struct LexemeC:`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`flags_t flags`
Remove trailing whitespace 2015-04-19 08:31:31 +00:00
introduce lang field for LexemeC to hold language id put noun_chunk logic into iterators.py for each language separately 2016-03-10 12:01:34 +00:00			`attr_t lang`

* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`attr_t id`
* Tmp. Refactoring, introducing a Lexeme PyObject. 2015-01-12 00:23:44 +00:00			`attr_t length`

* Rename sic to orth 2015-01-22 15:08:25 +00:00			`attr_t orth`
* Rename NORM1 and NORM2 attrs to lower and norm 2015-01-23 19:17:03 +00:00			`attr_t lower`
			`attr_t norm`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`attr_t shape`
			`attr_t prefix`
			`attr_t suffix`
Remove trailing whitespace 2015-04-19 08:31:31 +00:00
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`attr_t cluster`

			`float prob`
			`float sentiment`


Get spaCy train command working with neural network * Integrate models into pipeline * Add basic serialization (maybe incorrect) * Fix pickle on vocab 2017-05-17 10:04:50 +00:00			`cdef struct SerializedLexemeC:`
Adjust lexeme sizing for attr_t being 64 bit 2017-05-28 10:51:09 +00:00			`unsigned char[8 + 8*10 + 4 + 4] data`
Get spaCy train command working with neural network * Integrate models into pipeline * Add basic serialization (maybe incorrect) * Fix pickle on vocab 2017-05-17 10:04:50 +00:00			`# sizeof(flags_t) # flags`
			`# + sizeof(attr_t) # lang`
			`# + sizeof(attr_t) # id`
			`# + sizeof(attr_t) # length`
			`# + sizeof(attr_t) # orth`
			`# + sizeof(attr_t) # lower`
			`# + sizeof(attr_t) # norm`
			`# + sizeof(attr_t) # shape`
			`# + sizeof(attr_t) # prefix`
			`# + sizeof(attr_t) # suffix`
			`# + sizeof(attr_t) # cluster`
			`# + sizeof(float) # prob`
			`# + sizeof(float) # cluster`
			`# + sizeof(float) # l2_norm`


* Tmp 2015-03-09 05:46:22 +00:00			`cdef struct Entity:`
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match. 2016-09-21 12:54:55 +00:00			`hash_t id`
* Tmp 2015-03-09 05:46:22 +00:00			`int start`
			`int end`
WIP on stringstore change. 27 failures 2017-05-28 12:06:40 +00:00			`attr_t label`
* NER seems to be working, scoring 69 F. Need to add decision-history features --- currently only use current word, 2 words context. Need refactoring. 2015-03-10 17:00:23 +00:00
* Tmp 2015-03-09 05:46:22 +00:00
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`cdef struct TokenC:`
* Tmp commit. Refactoring to create a Python Lexeme class. 2015-01-11 23:26:22 +00:00			`const LexemeC* lex`
* More work on language-generic parsing 2015-08-28 00:02:33 +00:00			`uint64_t morph`
* Move POS tag definitions to parts_of_speech.pxd 2015-01-25 05:31:07 +00:00			`univ_pos_t pos`
* Add TokenC.spacy attr 2015-07-13 17:48:07 +00:00			`bint spacy`
WIP on stringstore change. 27 failures 2017-05-28 12:06:40 +00:00			`attr_t tag`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`int idx`
Adjust lexeme sizing for attr_t being 64 bit 2017-05-28 10:51:09 +00:00			`attr_t lemma`
WIP on stringstore change. 27 failures 2017-05-28 12:06:40 +00:00			`attr_t sense`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`int head`
Adjust lexeme sizing for attr_t being 64 bit 2017-05-28 10:51:09 +00:00			`attr_t dep`
* Add l_edge and r_edge props in TokenC for tracking the parse-yield of the token 2015-04-29 17:14:20 +00:00
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`uint32_t l_kids`
			`uint32_t r_kids`
* Add l_edge and r_edge props in TokenC for tracking the parse-yield of the token 2015-04-29 17:14:20 +00:00			`uint32_t l_edge`
			`uint32_t r_edge`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00
Make TokenC.sent_tart an int, to allow ternary value 2017-10-08 17:58:54 +00:00			`int sent_start`
* NER seems to be working, scoring 69 F. Need to add decision-history features --- currently only use current word, 2 words context. Need refactoring. 2015-03-10 17:00:23 +00:00			`int ent_iob`
WIP on stringstore change. 27 failures 2017-05-28 12:06:40 +00:00			`attr_t ent_type # TODO: Is there a better way to do this? Multiple sources of truth..`
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match. 2016-09-21 12:54:55 +00:00			`hash_t ent_id`