spaCy/spacy/strings.pxd

from cymem.cymem cimport Pool
from preshed.maps cimport PreshMap
from murmurhash.mrmr cimport hash64
from .typedefs cimport attr_t

from libc.stdint cimport int64_t

from .typedefs cimport hash_t

DEF UINT64_MAX = 18446744073709551615

cpdef hash_t hash_string(unicode string) except 0


ctypedef union Utf8Str:
    unsigned char[8] s
    unsigned char* p


cdef class StringStore:
    cdef Pool mem
    cdef Utf8Str* c
    cdef int64_t size

    cdef public PreshMap _map
    cdef int64_t _resize_at
    cdef PreshMap oov_maps

    cpdef int remove_oov_map(self, Pool mem) except -1

    cdef hash_t intern(self, unicode py_string, Pool mem=*) except UINT64_MAX
    cdef const Utf8Str* _intern_utf8(self, const unsigned char* utf8_string,
                                     int length) except NULL
* Move StringStore class to its own file 2014-12-19 19:42:01 +00:00			`from cymem.cymem cimport Pool`
			`from preshed.maps cimport PreshMap`
* Move murmurhash to .pxd in strings file 2014-12-19 20:41:08 +00:00			`from murmurhash.mrmr cimport hash64`
* Fix type declarations for attr_t. Remove unused id_t. 2015-07-18 20:39:57 +00:00			`from .typedefs cimport attr_t`

			`from libc.stdint cimport int64_t`
* Move StringStore class to its own file 2014-12-19 19:42:01 +00:00
* Tmp commit. Refactoring to create a Python Lexeme class. 2015-01-11 23:26:22 +00:00			`from .typedefs cimport hash_t`

Work on Issue #285: intern strings into document-specific pools, to address streaming data memory growth. StringStore.__getitem__ now raises KeyError when it can't find the string. Use StringStore.intern() to get the old behaviour. Still need to hunt down all uses of StringStore.__getitem__ in library and do testing, but logic looks good. 2016-09-30 08:14:47 +00:00			`DEF UINT64_MAX = 18446744073709551615`

* Tmp commit. Refactoring to create a Python Lexeme class. 2015-01-11 23:26:22 +00:00			`cpdef hash_t hash_string(unicode string) except 0`
* Move murmurhash to .pxd in strings file 2014-12-19 20:41:08 +00:00

* Fix header for string store 2015-07-20 10:06:10 +00:00			`ctypedef union Utf8Str:`
			`unsigned char[8] s`
			`unsigned char* p`


* Move StringStore class to its own file 2014-12-19 19:42:01 +00:00			`cdef class StringStore:`
			`cdef Pool mem`
* Fix header for string store 2015-07-20 10:06:10 +00:00			`cdef Utf8Str* c`
* Fix type declarations for attr_t. Remove unused id_t. 2015-07-18 20:39:57 +00:00			`cdef int64_t size`
* Move StringStore class to its own file 2014-12-19 19:42:01 +00:00
* Use unicode in StringStore.intern, instead of unreliably casting to bytes. 2015-11-05 11:32:19 +00:00			`cdef public PreshMap _map`
			`cdef int64_t _resize_at`
Work on Issue #285: intern strings into document-specific pools, to address streaming data memory growth. StringStore.__getitem__ now raises KeyError when it can't find the string. Use StringStore.intern() to get the old behaviour. Still need to hunt down all uses of StringStore.__getitem__ in library and do testing, but logic looks good. 2016-09-30 08:14:47 +00:00			`cdef PreshMap oov_maps`

			`cpdef int remove_oov_map(self, Pool mem) except -1`
* Move StringStore class to its own file 2014-12-19 19:42:01 +00:00
Revert "Changes to strings.pyx for new StringStore scheme" This reverts commit 22d4752d64d9ced99e24f8ed20bea942fae27c5b. 2016-09-30 18:19:42 +00:00			`cdef hash_t intern(self, unicode py_string, Pool mem=*) except UINT64_MAX`
Work on Issue #285: intern strings into document-specific pools, to address streaming data memory growth. StringStore.__getitem__ now raises KeyError when it can't find the string. Use StringStore.intern() to get the old behaviour. Still need to hunt down all uses of StringStore.__getitem__ in library and do testing, but logic looks good. 2016-09-30 08:14:47 +00:00			`cdef const Utf8Str* _intern_utf8(self, const unsigned char* utf8_string,`
			`int length) except NULL`