Rooba
/
spaCy
mirror of https://github.com/explosion/spaCy.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,643 Commits 70 Branches 177 Tags 298 MiB
Commit Graph

73 Commits

Author SHA1 Message Date
Jani Monoses ec62cadf4c Updates to Romanian support (#2354) 
* Add back Romanian in conftest

* Romanian lex_attr

* More tokenizer exceptions for Romanian

* Add tests for some Romanian tokenizer exceptions
 2018-05-24 11:40:00 +02:00
Matthew Honnibal 581d318971 Fix conftest 2018-05-15 00:54:45 +02:00
Tahar Zanouda 00417794d3 Add Arabic language (#2314) 
* added support for Arabic lang

* added Arabic language support

* updated conftest
 2018-05-15 00:27:19 +02:00
Jani Monoses 0e08e49e87 Lemmatizer ro (#2319) 
* Add Romanian lemmatizer lookup table.

Adapted from http://www.lexiconista.com/datasets/lemmatization/
by replacing cedillas with commas (ș and ț).

The original dataset is licensed under the Open Database License.

* Fix one blatant issue in the Romanian lemmatizer

* Romanian examples file

* Add ro_tokenizer in conftest

* Add Romanian lemmatizer test
 2018-05-12 15:20:04 +02:00
Paul O'Leary McCann bd72fbf09c Port Japanese mecab tokenizer from v1 (#2036) 
* Port Japanese mecab tokenizer from v1

This brings the Mecab-based Japanese tokenization introduced in #1246 to
spaCy v2. There isn't a JapaneseTagger implementation yet, but POS tag
information from Mecab is stored in a token extension. A tag map is also
included.

As a reminder, Mecab is required because Universal Dependencies are
based on Unidic tags, and Janome doesn't support Unidic.

Things to check:

1. Is this the right way to use a token extension?

2. What's the right way to implement a JapaneseTagger? The approach in
 #1246 relied on `tag_from_strings` which is just gone now. I guess the
best thing is to just try training spaCy's default Tagger?

-POLM

* Add tagging/make_doc and tests
 2018-05-03 18:38:26 +02:00
Matthew Honnibal 95a9615221 Fix loading of multiple pre-trained vectors 
This patch addresses #1660, which was caused by keying all pre-trained
vectors with the same ID when telling Thinc how to refer to them. This
meant that if multiple models were loaded that had pre-trained vectors,
errors or incorrect behaviour resulted.

The vectors class now includes a .name attribute, which defaults to:
{nlp.meta['lang']_nlp.meta['name']}.vectors
The vectors name is set in the cfg of the pipeline components under the
key pretrained_vectors. This replaces the previous cfg key
pretrained_dims.

In order to make existing models compatible with this change, we check
for the pretrained_dims key when loading models in from_disk and
from_bytes, and add the cfg key pretrained_vectors if we find it.
 2018-03-28 16:02:59 +02:00
Canbey Bilgili abe098b255 Adds Turkish Lemmatization 2017-12-01 17:04:32 +03:00
Vadim Mazaev 4ba7ddf651 Bugfixies 2017-11-30 12:29:38 +03:00
Vadim Mazaev 53e7c38637 Fixed tests depends on pymorphy2 2017-11-26 21:04:44 +03:00
Vadim Mazaev cacd859dcd Added tag map, fixed tests fails, added more exceptions 2017-11-26 20:54:48 +03:00
Vadim Mazaev 81314f8659 Fixed tokenizer: added char classes; added first lemmatizer and 
tokenizer tests
 2017-11-21 22:23:59 +03:00
ines 3af281a334 Update test model name 2017-11-01 23:02:00 +01:00
Jim O'Regan 34ca59691b no idea what is wrong here 2017-10-31 14:50:13 +00:00
Jim O'Regan 41dd29e48e merge 2017-10-31 14:07:45 +00:00
Ines Montani facf77e541 Merge branch 'develop' into support-danish 2017-10-24 11:53:19 +02:00
ines 612224c10d Port over changes from #1157 2017-10-14 13:11:39 +02:00
ines 9b3f8f9ec3 Fix formatting and add comment on languages 2017-10-14 13:11:18 +02:00
ines 61a503a611 Fix parser test 2017-10-07 00:38:51 +02:00
Wannaphong Phatthiyaphaibun 7b5263ffa4 fix thai test 2017-09-26 23:54:15 +07:00
Wannaphong Phatthiyaphaibun 5cba67146c add thai in spacy2 2017-09-26 21:36:27 +07:00
Jim O'Regan 7de709483b missed adding here 2017-09-11 10:51:21 +01:00
Jim O'Regan b1b6123867 add ga_tokenizer 2017-09-11 10:31:41 +01:00
Matthew Honnibal cb4839033c Fix loader for EN tests 2017-09-04 15:19:18 +02:00
Jim Geovedi 713d7c0aa0 added indonesian lang test 2017-08-20 12:17:14 +07:00
mollerhoj e840077601 Add some basic tests for Danish 2017-07-03 15:49:51 +02:00
ines a0f4592f0a Update tests 2017-06-05 02:26:13 +02:00
ines 3e105bcd36 Update tests 2017-06-05 02:09:27 +02:00
ines 078232932c Fix tokenizer fixture scope 2017-06-05 01:06:34 +02:00
Matthew Honnibal 55d0621532 Merge branch 'develop' of https://github.com/explosion/spaCy into develop 2017-06-04 15:53:25 -05:00
Matthew Honnibal 5b9f116aca Update tests 2017-06-04 15:53:17 -05:00
ines f432bb4b48 Fix fixture scopes 2017-06-04 22:34:31 +02:00
ines 20a7003c0d Update model fixtures and reorganise tests 2017-05-29 22:14:31 +02:00
ines 6e3937efc5 Check for arguments of model markers to specify models to test 
Lets user set --models --en for only English models
 2017-05-29 22:10:16 +02:00
ines b462076d80 Merge load_lang_class and get_lang_class 2017-05-14 01:31:10 +02:00
ines 5858857a78 Update languages list in conftest 2017-05-13 15:37:54 +02:00
ines bd57b611cc Update conftest to lazy load languages 2017-05-09 00:02:21 +02:00
Gregory Howard c0afcd22bb Merge remote-tracking branch 'remotes/upstream/master' 2017-04-27 14:42:54 +02:00
Gregory Howard 8ff4682255 correcting tokenizer exception. 
Adding tests for lemmatization
 2017-04-27 11:52:14 +02:00
luvogels d12a0b6431 Hooked up tokenizer tests 2017-04-26 23:21:41 +02:00
oeg c693d40791 feature(model): Add support for creating the Spanish model, including rich tagset, configuration, and basich tests 2017-04-06 18:48:45 +02:00
Ines Montani 97cb4d5e3c Merge branch 'master' into master 2017-03-25 10:03:47 +01:00
Iddo Berger da135bd823 add hebrew tokenizer 2017-03-24 18:27:44 +03:00
Matthew Honnibal a630726b13 Fix typo in tests 2017-03-16 20:50:36 -05:00
Matthew Honnibal f98b30583f Fix tests 2017-03-16 19:48:00 -05:00
Matthew Honnibal db51abf685 Fix tests 2017-03-16 18:53:47 -05:00
Aniruddha Adhikary 696215a3fb add tests for Bengali 2017-03-05 11:25:12 +06:00
ines 21f09d10d7 Revert "Revert "Merge pull request #818 from raphael0202/tokenizer_exceptions"" 
This reverts commit f02a2f9322.
 2017-02-10 13:17:05 +01:00
ines f02a2f9322 Revert "Merge pull request #818 from raphael0202/tokenizer_exceptions" 
This reverts commit b95afdf39c, reversing
changes made to b0ccf32378.
 2017-02-09 17:07:21 +01:00
Raphaël Bournhonesque 309da78bf0 Merge branch 'master' into tokenizer_exceptions 2017-02-09 16:32:12 +01:00
Michael Wallin 35100c8bdd [issue 805] Add regression test and the required fixture 2017-02-04 16:21:34 +02:00