Rooba
/
spaCy
mirror of https://github.com/explosion/spaCy.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,053 Commits 68 Branches 175 Tags 298 MiB
Commit Graph

74 Commits

Author SHA1 Message Date
Matthew Honnibal 4336397ecb Update develop from master 2018-08-14 03:04:28 +02:00
Emil Stenström 1914c488d3 Swedish: Exceptions for single letter words ending sentence (#2615) 
* Exceptions for single letter words ending sentence

Sentences ending in "i." (as in "... peka i."), "m." (as in "...än 2000 m."), should be tokenized as two separate tokens.

* Add test
 2018-08-05 14:14:30 +02:00
Ines Montani 75f3234404
💫 Refactor test suite (#2568) 
## Description

Related issues: #2379 (should be fixed by separating model tests)

* **total execution time down from > 300 seconds to under 60 seconds** 🎉
* removed all model-specific tests that could only really be run manually anyway – those will now live in a separate test suite in the [`spacy-models`](https://github.com/explosion/spacy-models) repository and are already integrated into our new model training infrastructure
* changed all relative imports to absolute imports to prepare for moving the test suite from `/spacy/tests` to `/tests` (it'll now always test against the installed version)
* merged old regression tests into collections, e.g. `test_issue1001-1500.py` (about 90% of the regression tests are very short anyways)
* tidied up and rewrote existing tests wherever possible

### Todo

- [ ] move tests to `/tests` and adjust CI commands accordingly
- [x] move model test suite from internal repo to `spacy-models`
- [x] ~~investigate why `pipeline/test_textcat.py` is flakey~~
- [x] review old regression tests (leftover files) and see if they can be merged, simplified or deleted
- [ ] update documentation on how to run tests


### Types of change
enhancement, tests

## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I have submitted the spaCy Contributor Agreement.
- [x] I ran the tests, and all new and existing tests passed.
- [ ] My changes don't require a change to the documentation, or if they do, I've added all required information.
 2018-07-24 23:38:44 +02:00
Matthew Honnibal 6303ce3d0e Try to fix memory error by moving fr_tokenizer to module scope 2018-07-24 20:09:06 +02:00
ines 80e7485630 Merge branch 'master' into develop 2018-07-18 17:28:47 +02:00
Paul O'Leary McCann 1987f3f784 Add Japanese lemmas (#2543) 
This info was already available from Mecab, forgot to add it before.
 2018-07-13 10:55:14 +02:00
ines 3a321e79ac Merge branch 'master' into develop 2018-07-10 13:49:08 +02:00
Eleni170 6042723535 Add support for Greek language (#2535) 
* Add contributor agreement

* Support for Greek language

* Fix missing el_tokenizer
 2018-07-10 13:48:38 +02:00
ines fd6207426a Merge branch 'master' into develop 2018-07-09 18:05:10 +02:00
Duygu Altinok 00b9a58558 German lemmatizer additions (#2529) 
* lemma of was-> was

* added new pairs issue @2486

* added article tests
 2018-07-09 11:10:15 +02:00
ines 63666af328 Merge branch 'master' into develop 2018-07-04 14:52:25 +02:00
ines 8feb7cfe2d Remove model dependency from French lemmatizer tests 2018-07-04 14:46:45 +02:00
Muhammad Irfan f33c703066 Add Urdu Language Support (#2430) 
* added Urdu language support.

* added Urdu language tests.

* modified conftest.py for Urdu language support.

* added spacy contributor agreement.
 2018-06-22 11:14:03 +02:00
Aliia E 428bae66b5 Add Tatar Language Support (#2444) 
* add Tatar lang support

* add Tatar letters

* add Tatar tests

* sign contributor agreement

* sign contributor agreement [x]

* remove comments from Language class

* remove all template comments
 2018-06-19 10:17:53 +02:00
Jani Monoses ec62cadf4c Updates to Romanian support (#2354) 
* Add back Romanian in conftest

* Romanian lex_attr

* More tokenizer exceptions for Romanian

* Add tests for some Romanian tokenizer exceptions
 2018-05-24 11:40:00 +02:00
Tahar Zanouda 00417794d3 Add Arabic language (#2314) 
* added support for Arabic lang

* added Arabic language support

* updated conftest
 2018-05-15 00:27:19 +02:00
Jani Monoses 0e08e49e87 Lemmatizer ro (#2319) 
* Add Romanian lemmatizer lookup table.

Adapted from http://www.lexiconista.com/datasets/lemmatization/
by replacing cedillas with commas (ș and ț).

The original dataset is licensed under the Open Database License.

* Fix one blatant issue in the Romanian lemmatizer

* Romanian examples file

* Add ro_tokenizer in conftest

* Add Romanian lemmatizer test
 2018-05-12 15:20:04 +02:00
Paul O'Leary McCann bd72fbf09c Port Japanese mecab tokenizer from v1 (#2036) 
* Port Japanese mecab tokenizer from v1

This brings the Mecab-based Japanese tokenization introduced in #1246 to
spaCy v2. There isn't a JapaneseTagger implementation yet, but POS tag
information from Mecab is stored in a token extension. A tag map is also
included.

As a reminder, Mecab is required because Universal Dependencies are
based on Unidic tags, and Janome doesn't support Unidic.

Things to check:

1. Is this the right way to use a token extension?

2. What's the right way to implement a JapaneseTagger? The approach in
 #1246 relied on `tag_from_strings` which is just gone now. I guess the
best thing is to just try training spaCy's default Tagger?

-POLM

* Add tagging/make_doc and tests
 2018-05-03 18:38:26 +02:00
Jens Dahl Møllerhøj e5055e3cf6 Add Danish lemmatizer (#2184) 
* add danish lemmatizer

* fill contributor agreement
 2018-04-07 19:07:28 +02:00
ines 6d2c85f428 Drop six and related hacks as a dependency 2018-03-28 10:45:25 +02:00
4altinok 471d3c9e23 added lex test for is_currency 2018-02-11 18:50:50 +01:00
Ines Montani a3dd167d7f
Merge branch 'master' into da_ud_tokenization 2017-12-20 21:05:34 +00:00
Søren Lind Kristiansen 15d13efafd Tune Danish tokenizer to more closely match tokenization in Universal Dependencies. 2017-12-20 17:36:52 +01:00
Canbey Bilgili abe098b255 Adds Turkish Lemmatization 2017-12-01 17:04:32 +03:00
Matthew Honnibal f9ed9ea529
Merge pull request #1624 from GreenRiverRUS/russian 
Add support for Russian
 2017-11-29 23:10:01 +01:00
Søren Lind Kristiansen 0ffd27b0f6 Add several Danish alternative spellings 2017-11-27 13:35:41 +01:00
Vadim Mazaev cacd859dcd Added tag map, fixed tests fails, added more exceptions 2017-11-26 20:54:48 +03:00
Søren Lind Kristiansen 6aa241bcec Add day of month tokenizer exceptions for Danish. 2017-11-24 15:03:24 +01:00
Søren Lind Kristiansen 0c276ed020 Add weekday abbreviations and remove abiguous month abbreviations for Danish. 2017-11-24 14:43:29 +01:00
Søren Lind Kristiansen 056547e989 Add multiple tokenizer exceptions for Danish. 2017-11-24 11:51:26 +01:00
Søren Lind Kristiansen 8dc265ac0c Add test for tokenization of 'i.' for Danish. 2017-11-24 11:29:37 +01:00
Vadim Mazaev 81314f8659 Fixed tokenizer: added char classes; added first lemmatizer and 
tokenizer tests
 2017-11-21 22:23:59 +03:00
ines 17849dee4b Fix French test (see #1617) 2017-11-20 13:59:59 +01:00
Matthew Honnibal 63c6ae4191 Fix lemmatizer test 2017-11-06 11:57:06 +01:00
Matthew Honnibal 144a93c2a5 Back-off to tensor for similarity if no vectors 2017-11-03 20:56:33 +01:00
Matthew Honnibal d6e831bf89 Fix lemmatizer tests 2017-11-03 19:46:34 +01:00
Jim O'Regan 08b0bfd153 merge 2017-10-31 22:55:59 +00:00
Jim O'Regan 00ecfa5417 Ó, not O 2017-10-31 22:54:42 +00:00
Ines Montani 25b1d6cd91
Fix syntax error 2017-10-31 22:36:03 +01:00
Jim O'Regan fe4b10346a replace example sentence until I get around to adding a punctuation.py 2017-10-31 20:24:53 +00:00
Jim O'Regan d4a8160c36 change quotes 2017-10-31 15:15:44 +00:00
Jim O'Regan 41dd29e48e merge 2017-10-31 14:07:45 +00:00
Ines Montani facf77e541 Merge branch 'develop' into support-danish 2017-10-24 11:53:19 +02:00
ines cd6a29dce7 Port over changes from #1294 2017-10-14 13:28:46 +02:00
ines 38c756fd85 Port over changes from #1287 2017-10-14 13:16:21 +02:00
ines 612224c10d Port over changes from #1157 2017-10-14 13:11:39 +02:00
Matthew Honnibal cf6da9301a Update lemmatizer test 2017-10-12 22:50:52 +02:00
ines 453c47ca24 Add German lemmatizer tests 2017-10-11 13:27:26 +02:00
Matthew Honnibal c6cd81f192 Wrap try/except around model saving 2017-10-05 08:14:24 -05:00
Matthew Honnibal fd4baff475 Update tests 2017-10-05 08:12:27 -05:00