Rooba
/
spaCy
mirror of https://github.com/explosion/spaCy.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
15,273 Commits 70 Branches 177 Tags 298 MiB
Commit Graph

869 Commits

Author SHA1 Message Date
Adriane Boyd 31528f62ed
Add / to nb infixes (#7991) 2021-05-04 11:00:10 +02:00
Sevdimali 49aed683cc
Azerbaijani language added (#7911) 2021-04-28 14:42:02 +02:00
Jacopo Farina c105ed10fd
Remove torino from stop words (#7634) 
Torino is the proper name of a city and the token has no other meaning
 2021-04-26 16:53:43 +02:00
m0canu1 921feee092
Added more exception to the italian language from https://forum.wordr… (#7246) 
* Added more exception to the italian language from https://forum.wordreference.com/threads/le-abbreviazioni-nella-lingua-italiana-abbreviations-in-italian.2464189/

* Remove unnecessary exception

Co-authored-by: Alexandru Mocanu <alexandru.mocanu@augeos.it>
Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
 2021-03-30 10:23:32 +02:00
Adriane Boyd 3bcf74aca7 Rename and update ru pymorphy2 lookup lemmatize 
* To allow default lookup lemmatization with a blank Russian model,
rename pymorphy2 lookup mode to `pymorphy2_lookup`

* Bug fix: update pymorphy2 lookup lemmatize to return list rather than
string
 2021-03-15 11:11:06 +01:00
Adriane Boyd 264862c67a
Fix Ukrainian lemmatizer init (#7127) 
Fix class variable and init for `UkrainianLemmatizer` so that it loads
the `uk` dictionaries rather than having the parent `RussianLemmatizer`
override with the `ru` settings.
 2021-02-22 11:05:08 +11:00
Boian Tzonev cca8651fc8
Bulgarian tokenizer exceptions (#7114) 
* [Bulgarian] Add tokenizer exceptions and like_num for Bulgarian

* [Bulgarian] Add tokenizer exceptions and like_num for Bulgarian
 2021-02-19 19:19:19 +01:00
Ines Montani 9ba715ed16 Tidy up and auto-format 2021-02-13 12:55:56 +11:00
Ines Montani 6c450decfc Fix punctuation settings and add to initialize tests 2021-02-13 11:51:21 +11:00
Shumi 4e514f1ea8
Update stop_words.py 
I have deleted line 1 to 5 and the statement print(STOP_WORDS)
 2021-02-11 21:30:34 +02:00
Shumi 0d57e84b7b
Update lex_attrs.py 
I have removed line 1 to 4
 2021-02-11 21:28:23 +02:00
Shumi 37ec67f868
Update examples.py 
I have removed two lines:
# coding: utf8
from __future__ import unicode_literals

And updated: >>> from spacy.lang.tn.examples import sentences
 2021-02-11 21:25:58 +02:00
Shumi 39eeba6760
Update __init__.py 
Added infixes = TOKENIZER_INFIXES
 2021-02-11 21:20:46 +02:00
Shumi ed3397727e
Delete tag_map.py 
Tag map file is deleted. I will add it later because it was failing validations
 2021-02-10 20:41:18 +02:00
Shumi 7c8721b1bd
Update tag_map.py 
Updated tag_map
 2021-02-10 20:21:22 +02:00
Shumi f6be28cfb2
Added files to Setswana Language 
Add South African Setswana Language
 2021-02-10 20:15:13 +02:00
Shumi 24046fef17
South African Setswana language 
Please accept the additional of Setswana language
 2021-02-10 20:12:33 +02:00
svlandeg 91e72c031e reformatting 2021-01-30 17:29:33 +01:00
svlandeg a8d84188f0 add stop words 
Co-authored-by: tewodrosm <tedmaam2006@gmail.com>
 2021-01-30 17:26:49 +01:00
Ines Montani e6accb3a9e Tidy up and auto-format 2021-01-30 12:52:33 +11:00
Ines Montani 817b0db521 Fix escape sequence 2021-01-30 12:39:58 +11:00
Ines Montani bbf080dfe5
Merge pull request #6645 from bittlingmayer/patch-3 2021-01-30 01:26:28 +11:00
Adriane Boyd bced6309e5
Add full exceptions with spaces 2021-01-29 14:27:22 +01:00
Ines Montani 5ed51c9dd2
Merge pull request #6828 from explosion/master-tmp 2021-01-27 23:05:46 +11:00
Adriane Boyd d17afb4826
Add Spanish rule-based lemmatizer (#6833) 
* Initial Spanish lemmatizer

* Handle merged verb+pron(s) multi-word tokens

* Use VERB for AUX rule lookup

* Add morph to lemma cache key

* Fix aux lookups, minor refactoring

* Improve verb+pron handling

* Move verb+pron handling into its own method
* Check for exceptions (primarily for se)
* Collect pronouns in the same (not reversed) order

* Only add modified possible lemmas
 2021-01-27 19:21:35 +08:00
Ines Montani 615dba9d99 Fix tokenizer exceptions 2021-01-27 22:11:42 +11:00
Ines Montani e3f8be9a94 Update language data 2021-01-27 13:29:22 +11:00
Ines Montani 230e651ad6 Merge branch 'develop' into master-tmp 2021-01-27 13:26:29 +11:00
Adriane Boyd 71a6350744
Implement overwrite param for all custom lemmatizers (#6794) 2021-01-26 14:53:43 +11:00
muratjumashev 2b19ebad59 Remove Kyrgyz chars fr. char_classes since Tatar ones already cover 2021-01-25 00:46:45 +06:00
muratjumashev 53abf759ad Fix punctuation 2021-01-24 20:54:22 +06:00
muratjumashev 2a2646362b Fix language subclass 2021-01-23 22:00:50 +06:00
muratjumashev fe3b5b8ff5 Add kyrgyz to char_classes 2021-01-23 21:53:41 +06:00
muratjumashev e30bbf5432 Add examples 2021-01-23 21:49:08 +06:00
muratjumashev 2f385385a9 Remove comment 2021-01-23 21:36:28 +06:00
muratjumashev d53724ba1d Add lex_attrs 2021-01-23 21:35:25 +06:00
muratjumashev 4418ec2eee Add punctuation 2021-01-23 21:31:31 +06:00
muratjumashev 101d265778 Add stopwords 2021-01-23 21:25:28 +06:00
muratjumashev 28d06ab860 Add tokenizer_exceptions 2021-01-22 23:08:41 +06:00
Sofie Van Landeghem fed8f48965
raise NotImplementedError when noun_chunks iterator is not implemented (#6711) 
* raise NotImplementedError when noun_chunks iterator is not implemented

* bring back, fix and document span.noun_chunks

* formatting

Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>
 2021-01-17 19:56:05 +08:00
Adriane Boyd 185fc62f4d
Remove unused is_base_form for mk lemmatizer (#6743) 
Remove unimplemented/incorrect is_base_form for Macedonian lemmatizer.
 2021-01-17 09:41:35 +01:00
Ines Montani b0b743597c Tidy up and auto-format 2021-01-15 11:57:36 +11:00
Adriane Boyd 0c936004d1 Merge remote-tracking branch 'upstream/master' into chore/update-develop-from-master-rc3 2021-01-14 11:49:58 +01:00
Adriane Boyd e649242927
Prevent overlapping noun chunks for Spanish (#6712) 
* Prevent overlapping noun chunks in Spanish noun chunk iterator
* Clean up similar code in Danish noun chunk iterator
 2021-01-14 17:33:31 +11:00
Adriane Boyd 54e8e3c208
Update model-related dependencies (#6725) 
* Update pymorphy2 error messages for Russian and Ukrainian
* Add pymorphy2 to pex
* Update spacy-pkuseg version for pex
 2021-01-14 17:29:44 +11:00
Alex Combessie 9cc880014c
Remove questionable French stopwords (#6310) 
* Remove questionable French stopwords

* Create alexcombessie.md
 2021-01-08 11:36:22 +11:00
Cristiana S Parada 7a0222f260
Update stop_words.py in Portuguese (a,o,e) (#6345) 
* Update stop_words.py

Added three aditional stopwords: "a" and "o" that means "the", and "e" that means "and"

* Create cristianasp.md

* zero edit to push CI

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
 2021-01-08 11:35:38 +11:00
Lorena Ciutacu f11002f1f1
add new Romanian stopwords (#6621) 
* add contributor agreement

* update ro stopwords list

* add new stopwords
 2021-01-08 11:34:47 +11:00
ophelielacroix e3222fdec9
Add (noun chunks) syntax iterators for Danish (#6246) 
* add syntax iterators for danish

* add test noun chunks for danish syntax iterators

* add contributor agreement

* update da syntax iterators to remove nested chunks

* add tests for da noun chunks

* Fix test

* add missing import
* fix example

* Prevent overlapping noun chunks

Prevent overlapping noun chunks by tracking the end index of the
previous noun chunk span.

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
 2021-01-07 16:33:00 +11:00
Sofie Van Landeghem 6f7e7d88b9
remove cause without apostrophe from norm exceptions (#6636) 2021-01-06 12:30:30 +08:00
Ines Montani 991669c934 Tidy up and auto-format 2021-01-05 13:41:53 +11:00
Adam Bittlingmayer f2fe60bacf
Update tokenizer_exceptions.py 
See https://github.com/explosion/spaCy/pull/6643
 2020-12-29 16:05:11 +04:00
Yosi cf52510631
Add Amharic አማርኛ Language support (#6583) 
* Add Amharic to space

* clean up

* Add some PRON_LEMMA

* add Tigrinya support

* remove text_noun_chunks

* Tigrinya Support

* added some more details for ti

* fix unit test

* add amharic char range

* changes from review

* amharic and tigrinya share same unicode block

* get rid of _amharic/_tigrinya in char_classes

Co-authored-by: Josiah Solomon <jsolomon@meteorcomm.com>
 2020-12-22 16:50:34 +01:00
Ines Montani 1da1568110 Remove tag map 2020-12-09 11:13:49 +11:00
Adriane Boyd 724831b066 Merge remote-tracking branch 'upstream/master' into chore/update-develop-from-master 
* Update Macedonian for v3
* Update Turkish for v3
 2020-11-25 11:49:34 +01:00
Daniel Vasic 20d72de986
Added Multext-East V5 tagset for Croatian language (#6248) 
* Added Multext-East V5 tagset for Croatian language

* Create danielvasic.md

* Update danielvasic.md

* Update danielvasic.md

* Add tag map to CroatianDefaults

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
 2020-11-05 12:19:22 +01:00
Robert Šípek 6069efe57d
Add tag map to cs language (#6284) 2020-11-05 10:13:11 +01:00
Vu Ha 6d465ec52c
add oprd to the list of accepted deps for noun chunking (#6302) 
* add oprd to the list of accepted deps for noun chunking

* add SCA
 2020-11-05 09:17:35 +01:00
Duygu Altinok 0e55f806dd
Turkish tokenization improvements (#6268) 
* added single and paired orth variants

* added token match

* added long text tokenization test

* inverted init

* normalized lemmas to lowercase

* more abbrevs

* tests for ordinals and abbrevs

* separated period abbvrevs to another list

* fiex typo

* added ordinal and abbrev tests

* added number tests for dates

* minor refinement

* added inflected abbrevs regex

* added percentage and inflection

* cosmetics

* added token match

* added url inflection tests

* excluded url tokens from custom pattern

* removed url match import
 2020-10-29 09:43:17 +01:00
Adriane Boyd 4299a7f654 Setup / install / quickstart updates 
* Add `cuda110` to setup.cfg and quickstart dropdown
* Switch to `pip` for pip-only packages in conda quickstart instructions
* Update zh pkuseg install message with version range and conda
* Remove `zh` from `extras_require` because the default doesn't require
additional packages
 2020-10-23 11:27:54 +02:00
Borijan Georgievski 2311192ba1
Include Macedonian language (#6230) 
* Include Macedonian language

* Fix indentation at char_classes.py

* Fix indentation at char_classes.py

* Add Macedonian tests, update lex_attrs and char_classes

* Import unicode literals for python 2
 2020-10-15 15:55:01 +02:00
Ines Montani d165af26be Auto-format [ci skip] 2020-10-15 10:08:53 +02:00
Ines Montani 5665a21517 Tidy up 2020-10-15 09:30:32 +02:00
Ines Montani 178760855f Merge branch 'develop' into master-tmp 2020-10-15 09:06:03 +02:00
Ines Montani 7f92a5ee6a
Update spacy/lang/ta/examples.py 2020-10-13 11:03:35 +02:00
Ines Montani 539b0c10da Tidy up and auto-format 2020-10-10 19:14:48 +02:00
Duygu Altinok 80fb1bffc9 Ordinal numbers for Turkish (#6142) 
* minor ordinal number addition

* fixed typo

* added corresponding lexical test
 2020-10-09 10:13:15 +02:00
Duygu Altinok 2fad279a44 Turkish language syntax iterators (#6191) 
* added tr_vocab to config

* basic test

* added syntax iterator to Turkish lang class

* first version for Turkish syntax iter, without flat

* added simple tests with nmod, amod, det

* more tests to amod and nmod

* separated noun chunks and parser test

* rearrangement after nchunk parser separation

* added recursive NPs

* tests with complicated recursive NPs

* tests with conjed NPs

* additional tests for conj NP

* small modification for shaving off conj from NP

* added tests with flat

* more tests with flat

* added examples with flats conjed

* added inner func for flat trick

* corrected parse

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
 2020-10-09 10:10:22 +02:00
Baranitharan d6037c1860
added sentence 2020-10-08 08:22:58 +05:30
Baranitharan 81afe9b19d
Update examples.py 2020-10-08 08:17:25 +05:30
Wannaphong Phatthiyaphaibun 9fc8392b38
Add Thai tag map (LST20 Corpus) (#6163) 
* Add Thai tag map (LST20 Corpus)

By @korakot

* Update tag_map.py

* Update tag_map.py

* Update tag_map.py
 2020-10-07 11:12:01 +02:00
Duygu Altinok 7e821c2776
Turkish language syntax iterators (#6191) 
* added tr_vocab to config

* basic test

* added syntax iterator to Turkish lang class

* first version for Turkish syntax iter, without flat

* added simple tests with nmod, amod, det

* more tests to amod and nmod

* separated noun chunks and parser test

* rearrangement after nchunk parser separation

* added recursive NPs

* tests with complicated recursive NPs

* tests with conjed NPs

* additional tests for conj NP

* small modification for shaving off conj from NP

* added tests with flat

* more tests with flat

* added examples with flats conjed

* added inner func for flat trick

* corrected parse

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
 2020-10-07 11:07:52 +02:00
Duygu Altinok 2ce6fc2611
Turkish tag map and morph rules addition (#6141) 
* feat: added turkish tag map

* feat: morph rules cconj and sconj

* feat: more conjuncts

* feat: added popular postpositions

* feat: added adverbs

* feat: added personal pronouns

* feat: added reflexive pronouns

* minor: corrected case capital

* minor: fixed comma typo

* feat: added indef pronouns

* feat: added dict iter

* fixed comma typo

* updated language class with tag map and morph

* use default tag map instead

* removed tag map
 2020-10-07 10:27:36 +02:00
Duygu Altinok b95a11dd95
Ordinal numbers for Turkish (#6142) 
* minor ordinal number addition

* fixed typo

* added corresponding lexical test
 2020-10-07 10:25:37 +02:00
Rahul Gupta 1a00bff06d
Hindi: Adds tests for lexical attributes (norm and like_num) (#5829) 
* Hindi: Adds tests for lexical attributes (norm and like_num)

* Signs and sdds the contributor agreement

* Add ordinal numbers to be tagged as like_num

* Adds alternate pronunciation for 31 and 39
 2020-10-07 10:23:32 +02:00
Nuccy90 c809b2c8e7
Update morph_rules.py (#6102) 
* Update morph_rules.py

Added "dig" and "dej" ("you" in accusative form)

* Create Nuccy90.md

* Update Nuccy90.md
 2020-10-06 15:14:47 +02:00
Ines Montani 126268ce50 Auto-format [ci skip] 2020-10-05 21:58:18 +02:00
Ines Montani 2d0c0134bc Adjust message [ci skip] 2020-10-05 21:38:23 +02:00
Ines Montani 6abfc2911d
Merge pull request #6203 from adrianeboyd/feature/zh-spacy-pkuseg 2020-10-05 21:35:57 +02:00
Adriane Boyd f102ef6b54 Read features.msgpack instead of features.pkl 2020-10-05 17:47:39 +02:00
Adriane Boyd 187234648c Revert back to "default" as default for pkuseg_user_dict 2020-10-05 16:24:28 +02:00
Adriane Boyd 5d19dfc9d3 Update Chinese tokenizer for spacy-pkuseg fork 2020-10-05 14:21:53 +02:00
Adriane Boyd b0b93854cb Update ru/uk lemmatizers for new nlp.initialize 2020-10-05 09:27:16 +02:00
Ines Montani 59deeb7da6 Merge branch 'develop' into master-tmp 2020-10-04 14:52:20 +02:00
Ines Montani 3bc3c05fcc Tidy up and auto-format 2020-10-03 17:20:18 +02:00
Ines Montani 7c4ab7e82c Fix Lemmatizer.get_lookups_config 2020-10-03 17:16:10 +02:00
Ines Montani f0b30aedad
Make lemmatizers use initialize logic (#6182) 
* Make lemmatizer use initialize logic and tidy up

* Fix typo

* Raise for uninitialized tables
 2020-10-02 15:42:36 +02:00
Ines Montani d48ddd6c9a Remove default initialize lookups 2020-10-01 21:54:33 +02:00
Ines Montani 381258b75b
Merge pull request #6165 from explosion/feature/update-tokenizers-initialize 2020-10-01 09:49:47 +02:00
Ines Montani 4b6afd3611 Remove English [initialize] default block for now to get tests to pass 2020-09-30 23:49:29 +02:00
Ines Montani 6f29f68f69 Update errors and make Tokenizer.initialize args less strict 2020-09-30 23:48:47 +02:00
Adriane Boyd 6b7bb32834 Refactor Chinese initialization 2020-09-30 11:46:45 +02:00
Ines Montani 34f9c26c62 Add lexeme norm defaults 2020-09-30 10:20:14 +02:00
Ines Montani fa47f87924 Tidy up and auto-format 2020-09-29 21:39:28 +02:00
Ines Montani 6467a560e3 WIP: Test updating Chinese tokenizer 2020-09-29 21:10:22 +02:00
Ines Montani 4f3102d09c Auto-format 2020-09-29 21:09:10 +02:00
Ines Montani c3f8c09d7d
Merge pull request #6154 from adrianeboyd/bugfix/chinese-tokenizer-pickle 2020-09-29 20:54:59 +02:00
Adriane Boyd 013b66de05
Add tokenizer scoring to ja / ko / zh (#6152) 2020-09-27 22:20:45 +02:00
Adriane Boyd 8393dbedad Minor fixes 
* Put `cfg` back in serialization
* Add `pickle5` to pytest conf
 2020-09-27 15:15:53 +02:00
Adriane Boyd 54fe871935 Fix formatting, refactor pickle5 exceptions 2020-09-27 14:37:28 +02:00