spaCy/spacy/lang/fr/lemmatizer/_dets_irreg.py

50 lines
1.1 KiB
Python
Raw Normal View History

Rule-based French Lemmatizer (#2818) <!--- Provide a general summary of your changes in the title. --> ## Description <!--- Use this section to describe your changes. If your changes required testing, include information about the testing environment and the tests you ran. If your test fixes a bug reported in an issue, don't forget to include the issue number. If your PR is still a work in progress, that's totally fine – just include a note to let us know. --> Add a rule-based French Lemmatizer following the english one and the excellent PR for [greek language optimizations](https://github.com/explosion/spaCy/pull/2558) to adapt the Lemmatizer class. ### Types of change <!-- What type of change does your PR cover? Is it a bug fix, an enhancement or new feature, or a change to the documentation? --> - Lemma dictionary used can be found [here](http://infolingu.univ-mlv.fr/DonneesLinguistiques/Dictionnaires/telechargement.html), I used the XML version. - Add several files containing exhaustive list of words for each part of speech - Add some lemma rules - Add POS that are not checked in the standard Lemmatizer, i.e PRON, DET, ADV and AUX - Modify the Lemmatizer class to check in lookup table as a last resort if POS not mentionned - Modify the lemmatize function to check in lookup table as a last resort - Init files are updated so the model can support all the functionalities mentioned above - Add words to tokenizer_exceptions_list.py in respect to regex used in tokenizer_exceptions.py ## Checklist <!--- Before you submit the PR, go over this checklist and make sure you can tick off all the boxes. [] -> [x] --> - [X] I have submitted the spaCy Contributor Agreement. - [X] I ran the tests, and all new and existing tests passed. - [X] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-10-13 14:38:21 +00:00
# coding: utf8
from __future__ import unicode_literals
DETS_IRREG = {
"aucune": ("aucun",),
"ces": ("ce",),
"cet": ("ce",),
"cette": ("ce",),
"cents": ("cent",),
"certaines": ("certains",),
"différentes": ("différents",),
"diverses": ("divers",),
"la": ("le",),
"les": ("le",),
"l'": ("le",),
"laquelle": ("lequel",),
"lesquelles": ("lequel",),
"lesquels": ("lequel",),
"leurs": ("leur",),
"mainte": ("maint",),
"maintes": ("maint",),
"maints": ("maint",),
"ma": ("mon",),
"mes": ("mon",),
"nos": ("notre",),
"nulle": ("nul",),
"nulles": ("nul",),
"nuls": ("nul",),
"quelle": ("quel",),
"quelles": ("quel",),
"quels": ("quel",),
"quelqu'": ("quelque",),
"quelques": ("quelque",),
"sa": ("son",),
"ses": ("son",),
"telle": ("tel",),
"telles": ("tel",),
"tels": ("tel",),
"ta": ("ton",),
"tes": ("ton",),
"tous": ("tout",),
"toute": ("tout",),
"toutes": ("tout",),
"des": ("un",),
"une": ("un",),
"vingts": ("vingt",),
"vos": ("votre",)
}