spaCy/spacy/structs.pxd

from libc.stdint cimport uint8_t, uint32_t, int32_t, uint64_t

from .typedefs cimport flags_t, attr_t, hash_t
from .parts_of_speech cimport univ_pos_t
from .morphology cimport univ_morph_t


cdef struct LexemeC:
    flags_t flags

    attr_t lang

    attr_t id
    attr_t length

    attr_t orth
    attr_t lower
    attr_t norm
    attr_t shape
    attr_t prefix
    attr_t suffix

    attr_t cluster

    float prob
    float sentiment


cdef struct SerializedLexemeC:
    unsigned char[8 + 8*10 + 4 + 4] data
    #    sizeof(flags_t)  # flags
    #    + sizeof(attr_t) # lang
    #    + sizeof(attr_t) # id
    #    + sizeof(attr_t) # length
    #    + sizeof(attr_t) # orth
    #    + sizeof(attr_t) # lower
    #    + sizeof(attr_t) # norm
    #    + sizeof(attr_t) # shape
    #    + sizeof(attr_t) # prefix
    #    + sizeof(attr_t) # suffix
    #    + sizeof(attr_t) # cluster
    #    + sizeof(float)  # prob
    #    + sizeof(float)  # cluster
    #    + sizeof(float) # l2_norm


cdef struct Entity:
    hash_t id
    int start
    int end
    attr_t label


cdef struct TokenC:
    const LexemeC* lex
    uint64_t morph
    univ_pos_t pos
    bint spacy
    attr_t tag
    int idx
    attr_t lemma
    attr_t norm
    int head
    attr_t dep

    uint32_t l_kids
    uint32_t r_kids
    uint32_t l_edge
    uint32_t r_edge

    int sent_start
    int ent_iob
    attr_t ent_type # TODO: Is there a better way to do this? Multiple sources of truth..
    hash_t ent_id


cdef struct MorphAnalysisC:
    univ_pos_t pos
    
    attr_t abbr
    attr_t adp_type
    attr_t adv_type
    attr_t animacy
    attr_t aspect
    attr_t case
    attr_t conj_type
    attr_t connegative
    attr_t definite
    attr_t degree
    attr_t derivation
    attr_t echo
    attr_t foreign
    attr_t gender
    attr_t hyph
    attr_t inf_form
    attr_t mood
    attr_t negative
    attr_t number
    attr_t name_type
    attr_t noun_type
    attr_t num_form
    attr_t num_type
    attr_t num_value
    attr_t part_form
    attr_t part_type
    attr_t person
    attr_t polite
    attr_t polarity
    attr_t poss
    attr_t prefix
    attr_t prep_case
    attr_t pron_type
    attr_t punct_side
    attr_t punct_type
    attr_t reflex
    attr_t style
    attr_t style_variant
    attr_t tense
    attr_t typo
    attr_t verb_form
    attr_t voice
    attr_t verb_type
* Hack on morphology structs 2015-08-26 17:18:36 +00:00			`from libc.stdint cimport uint8_t, uint32_t, int32_t, uint64_t`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00
* Fix type declarations for attr_t. Remove unused id_t. 2015-07-18 20:39:57 +00:00			`from .typedefs cimport flags_t, attr_t, hash_t`
* Move POS tag definitions to parts_of_speech.pxd 2015-01-25 05:31:07 +00:00			`from .parts_of_speech cimport univ_pos_t`
Update structs 2018-09-24 21:58:08 +00:00			`from .morphology cimport univ_morph_t`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00

* Tmp commit. Refactoring to create a Python Lexeme class. 2015-01-11 23:26:22 +00:00			`cdef struct LexemeC:`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`flags_t flags`
Remove trailing whitespace 2015-04-19 08:31:31 +00:00
introduce lang field for LexemeC to hold language id put noun_chunk logic into iterators.py for each language separately 2016-03-10 12:01:34 +00:00			`attr_t lang`

* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`attr_t id`
* Tmp. Refactoring, introducing a Lexeme PyObject. 2015-01-12 00:23:44 +00:00			`attr_t length`

* Rename sic to orth 2015-01-22 15:08:25 +00:00			`attr_t orth`
* Rename NORM1 and NORM2 attrs to lower and norm 2015-01-23 19:17:03 +00:00			`attr_t lower`
			`attr_t norm`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`attr_t shape`
			`attr_t prefix`
			`attr_t suffix`
Remove trailing whitespace 2015-04-19 08:31:31 +00:00
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`attr_t cluster`

			`float prob`
			`float sentiment`


Get spaCy train command working with neural network * Integrate models into pipeline * Add basic serialization (maybe incorrect) * Fix pickle on vocab 2017-05-17 10:04:50 +00:00			`cdef struct SerializedLexemeC:`
Adjust lexeme sizing for attr_t being 64 bit 2017-05-28 10:51:09 +00:00			`unsigned char[8 + 8*10 + 4 + 4] data`
Get spaCy train command working with neural network * Integrate models into pipeline * Add basic serialization (maybe incorrect) * Fix pickle on vocab 2017-05-17 10:04:50 +00:00			`# sizeof(flags_t) # flags`
			`# + sizeof(attr_t) # lang`
			`# + sizeof(attr_t) # id`
			`# + sizeof(attr_t) # length`
			`# + sizeof(attr_t) # orth`
			`# + sizeof(attr_t) # lower`
			`# + sizeof(attr_t) # norm`
			`# + sizeof(attr_t) # shape`
			`# + sizeof(attr_t) # prefix`
			`# + sizeof(attr_t) # suffix`
			`# + sizeof(attr_t) # cluster`
			`# + sizeof(float) # prob`
			`# + sizeof(float) # cluster`
			`# + sizeof(float) # l2_norm`


* Tmp 2015-03-09 05:46:22 +00:00			`cdef struct Entity:`
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match. 2016-09-21 12:54:55 +00:00			`hash_t id`
* Tmp 2015-03-09 05:46:22 +00:00			`int start`
			`int end`
WIP on stringstore change. 27 failures 2017-05-28 12:06:40 +00:00			`attr_t label`
* NER seems to be working, scoring 69 F. Need to add decision-history features --- currently only use current word, 2 words context. Need refactoring. 2015-03-10 17:00:23 +00:00
* Tmp 2015-03-09 05:46:22 +00:00
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`cdef struct TokenC:`
* Tmp commit. Refactoring to create a Python Lexeme class. 2015-01-11 23:26:22 +00:00			`const LexemeC* lex`
* More work on language-generic parsing 2015-08-28 00:02:33 +00:00			`uint64_t morph`
* Move POS tag definitions to parts_of_speech.pxd 2015-01-25 05:31:07 +00:00			`univ_pos_t pos`
* Add TokenC.spacy attr 2015-07-13 17:48:07 +00:00			`bint spacy`
WIP on stringstore change. 27 failures 2017-05-28 12:06:40 +00:00			`attr_t tag`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`int idx`
Adjust lexeme sizing for attr_t being 64 bit 2017-05-28 10:51:09 +00:00			`attr_t lemma`
Make NORM a token attribute (#3029) See #3028. The solution in this patch is pretty debateable. What we do is give the TokenC struct a .norm field, by repurposing the previously idle .sense attribute. It's nice to repurpose a previous field because it means the TokenC doesn't change size, so even if someone's using the internals very deeply, nothing will break. The weird thing here is that the TokenC and the LexemeC both have an attribute named NORM. This arguably assists in backwards compatibility. On the other hand, maybe it's really bad! We're changing the semantics of the attribute subtly, so maybe it's better if someone calling lex.norm gets a breakage, and instead is told to write lex.default_norm? Overall I believe this patch makes the NORM feature work the way we sort of expected it to work. Certainly it's much more like how the docs describe it, and more in line with how we've been directing people to use the norm attribute. We'll also be able to use token.norm to do stuff like spelling correction, which is pretty cool. 2018-12-08 09:49:10 +00:00			`attr_t norm`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`int head`
Adjust lexeme sizing for attr_t being 64 bit 2017-05-28 10:51:09 +00:00			`attr_t dep`
* Add l_edge and r_edge props in TokenC for tracking the parse-yield of the token 2015-04-29 17:14:20 +00:00
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00			`uint32_t l_kids`
			`uint32_t r_kids`
* Add l_edge and r_edge props in TokenC for tracking the parse-yield of the token 2015-04-29 17:14:20 +00:00			`uint32_t l_edge`
			`uint32_t r_edge`
* Move all struct definitions to structs.pxd, to avoid circular dependencies 2014-12-19 19:51:33 +00:00
Make TokenC.sent_tart an int, to allow ternary value 2017-10-08 17:58:54 +00:00			`int sent_start`
* NER seems to be working, scoring 69 F. Need to add decision-history features --- currently only use current word, 2 words context. Need refactoring. 2015-03-10 17:00:23 +00:00			`int ent_iob`
WIP on stringstore change. 27 failures 2017-05-28 12:06:40 +00:00			`attr_t ent_type # TODO: Is there a better way to do this? Multiple sources of truth..`
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match. 2016-09-21 12:54:55 +00:00			`hash_t ent_id`
Update structs 2018-09-24 21:58:08 +00:00

Add MorphAnalysisC struct 2019-03-07 13:03:07 +00:00			`cdef struct MorphAnalysisC:`
			`univ_pos_t pos`

			`attr_t abbr`
			`attr_t adp_type`
			`attr_t adv_type`
			`attr_t animacy`
			`attr_t aspect`
			`attr_t case`
			`attr_t conj_type`
			`attr_t connegative`
			`attr_t definite`
			`attr_t degree`
			`attr_t derivation`
			`attr_t echo`
			`attr_t foreign`
			`attr_t gender`
			`attr_t hyph`
			`attr_t inf_form`
			`attr_t mood`
			`attr_t negative`
			`attr_t number`
			`attr_t name_type`
			`attr_t noun_type`
			`attr_t num_form`
			`attr_t num_type`
			`attr_t num_value`
			`attr_t part_form`
			`attr_t part_type`
			`attr_t person`
			`attr_t polite`
			`attr_t polarity`
			`attr_t poss`
			`attr_t prefix`
			`attr_t prep_case`
			`attr_t pron_type`
			`attr_t punct_side`
			`attr_t punct_type`
			`attr_t reflex`
			`attr_t style`
			`attr_t style_variant`
			`attr_t tense`
			`attr_t typo`
			`attr_t verb_form`
			`attr_t voice`
			`attr_t verb_type`
Update structs 2018-09-24 21:58:08 +00:00