mirror of https://github.com/explosion/spaCy.git
Merge pull request #688 from nlesc-sherlock/dutch
Support for Dutch in SpaCy
This commit is contained in:
Matthew Honnibal
GitHub
commit
ff03ade08f
|
|
@ -0,0 +1,107 @@
|
|||
# spaCy contributor agreement
|
||||
|
||||
This spaCy Contributor Agreement (**"SCA"**) is based on the
|
||||
[Oracle Contributor Agreement](http://www.oracle.com/technetwork/oca-405177.pdf).
|
||||
The SCA applies to any contribution that you make to any product or project
|
||||
managed by us (the **"project"**), and sets out the intellectual property rights
|
||||
you grant to us in the contributed materials. The term **"us"** shall mean
|
||||
[ExplosionAI UG (haftungsbeschränkt)](https://explosion.ai/legal). The term
|
||||
**"you"** shall mean the person or entity identified below.
|
||||
|
||||
If you agree to be bound by these terms, fill in the information requested
|
||||
below and include the filled-in version with your first pull request, under the
|
||||
folder [`.github/contributors/`](/.github/contributors/). The name of the file
|
||||
should be your GitHub username, with the extension `.md`. For example, the user
|
||||
example_user would create the file `.github/contributors/example_user.md`.
|
||||
|
||||
Read this agreement carefully before signing. These terms and conditions
|
||||
constitute a binding legal agreement.
|
||||
|
||||
## Contributor Agreement
|
||||
|
||||
1. The term "contribution" or "contributed materials" means any source code,
|
||||
object code, patch, tool, sample, graphic, specification, manual,
|
||||
documentation, or any other material posted or submitted by you to the project.
|
||||
|
||||
2. With respect to any worldwide copyrights, or copyright applications and
|
||||
registrations, in your contribution:
|
||||
|
||||
* you hereby assign to us joint ownership, and to the extent that such
|
||||
assignment is or becomes invalid, ineffective or unenforceable, you hereby
|
||||
grant to us a perpetual, irrevocable, non-exclusive, worldwide, no-charge,
|
||||
royalty-free, unrestricted license to exercise all rights under those
|
||||
copyrights. This includes, at our option, the right to sublicense these same
|
||||
rights to third parties through multiple levels of sublicensees or other
|
||||
licensing arrangements;
|
||||
|
||||
* you agree that each of us can do all things in relation to your
|
||||
contribution as if each of us were the sole owners, and if one of us makes
|
||||
a derivative work of your contribution, the one who makes the derivative
|
||||
work (or has it made will be the sole owner of that derivative work;
|
||||
|
||||
* you agree that you will not assert any moral rights in your contribution
|
||||
against us, our licensees or transferees;
|
||||
|
||||
* you agree that we may register a copyright in your contribution and
|
||||
exercise all ownership rights associated with it; and
|
||||
|
||||
* you agree that neither of us has any duty to consult with, obtain the
|
||||
consent of, pay or render an accounting to the other for any use or
|
||||
distribution of your contribution.
|
||||
|
||||
3. With respect to any patents you own, or that you can license without payment
|
||||
to any third party, you hereby grant to us a perpetual, irrevocable,
|
||||
non-exclusive, worldwide, no-charge, royalty-free license to:
|
||||
|
||||
* make, have made, use, sell, offer to sell, import, and otherwise transfer
|
||||
your contribution in whole or in part, alone or in combination with or
|
||||
included in any product, work or materials arising out of the project to
|
||||
which your contribution was submitted, and
|
||||
|
||||
* at our option, to sublicense these same rights to third parties through
|
||||
multiple levels of sublicensees or other licensing arrangements.
|
||||
|
||||
4. Except as set out above, you keep all right, title, and interest in your
|
||||
contribution. The rights that you grant to us under these terms are effective
|
||||
on the date you first submitted a contribution to us, even if your submission
|
||||
took place before the date you sign these terms.
|
||||
|
||||
5. You covenant, represent, warrant and agree that:
|
||||
|
||||
* Each contribution that you submit is and shall be an original work of
|
||||
authorship and you can legally grant the rights set out in this SCA;
|
||||
|
||||
* to the best of your knowledge, each contribution will not violate any
|
||||
third party's copyrights, trademarks, patents, or other intellectual
|
||||
property rights; and
|
||||
|
||||
* each contribution shall be in compliance with U.S. export control laws and
|
||||
other applicable export and import laws. You agree to notify us if you
|
||||
become aware of any circumstance which would make any of the foregoing
|
||||
representations inaccurate in any respect. We may publicly disclose your
|
||||
participation in the project, including the fact that you have signed the SCA.
|
||||
|
||||
6. This SCA is governed by the laws of the State of California and applicable
|
||||
U.S. Federal law. Any choice of law rules will not apply.
|
||||
|
||||
7. Please place an “x” on one of the applicable statement below. Please do NOT
|
||||
mark both statements:
|
||||
|
||||
* [ ] I am signing on behalf of myself as an individual and no other person
|
||||
or entity, including my employer, has or will have rights with respect my
|
||||
contributions.
|
||||
|
||||
* [x] I am signing on behalf of my employer or a legal entity and I have the
|
||||
actual authority to contractually bind that entity.
|
||||
|
||||
## Contributor Details
|
||||
|
||||
| Field | Entry |
|
||||
|------------------------------- | -------------------------------- |
|
||||
| Name | Rob van Nieuwpoort |
|
||||
| Signing on behalf of | Dafne van Kuppevelt, Janneke van der Zwaan, Willem van Hage |
|
||||
| Company name (if applicable) | Netherlands eScience center |
|
||||
| Title or role (if applicable) | Director of technology |
|
||||
| Date | 14-12-2016 |
|
||||
| GitHub username | RvanNieuwpoort |
|
||||
| Website (optional) | https://www.esciencecenter.nl/ |
|
||||
|
|
@ -151,10 +151,10 @@ def _read_senses(loc):
|
|||
def setup_vocab(lex_attr_getters, tag_map, src_dir, dst_dir):
|
||||
if not dst_dir.exists():
|
||||
dst_dir.mkdir()
|
||||
|
||||
print('Reading vocab from ', src_dir)
|
||||
vectors_src = src_dir / 'vectors.bz2'
|
||||
if vectors_src.exists():
|
||||
write_binary_vectors(vectors_src.as_posix, (dst_dir / 'vec.bin').as_posix())
|
||||
write_binary_vectors(vectors_src.as_posix(), (dst_dir / 'vec.bin').as_posix())
|
||||
else:
|
||||
print("Warning: Word vectors file not found")
|
||||
vocab = Vocab(lex_attr_getters=lex_attr_getters, tag_map=tag_map)
|
||||
|
|
|
|||
1
setup.py
1
setup.py
|
|
@ -28,6 +28,7 @@ PACKAGES = [
|
|||
'spacy.fr',
|
||||
'spacy.it',
|
||||
'spacy.pt',
|
||||
'spacy.nl',
|
||||
'spacy.serialize',
|
||||
'spacy.syntax',
|
||||
'spacy.munge',
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ from . import es
|
|||
from . import it
|
||||
from . import fr
|
||||
from . import pt
|
||||
from . import nl
|
||||
|
||||
|
||||
try:
|
||||
|
|
@ -25,6 +26,7 @@ set_lang_class(pt.Portuguese.lang, pt.Portuguese)
|
|||
set_lang_class(fr.French.lang, fr.French)
|
||||
set_lang_class(it.Italian.lang, it.Italian)
|
||||
set_lang_class(zh.Chinese.lang, zh.Chinese)
|
||||
set_lang_class(nl.Dutch.lang, nl.Dutch)
|
||||
|
||||
|
||||
def load(name, **overrides):
|
||||
|
|
@ -33,7 +35,7 @@ def load(name, **overrides):
|
|||
if target_name == 'en' and 'add_vectors' not in overrides:
|
||||
if 'vectors' in overrides:
|
||||
vec_path = util.match_best_version(overrides['vectors'], None, data_path)
|
||||
if vec_path is None:
|
||||
if vec_path is None:
|
||||
raise IOError(
|
||||
'Could not load data pack %s from %s' % (overrides['vectors'], data_path))
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,26 @@
|
|||
from __future__ import unicode_literals, print_function
|
||||
|
||||
from os import path
|
||||
|
||||
from ..language import Language
|
||||
from ..attrs import LANG
|
||||
from . import language_data
|
||||
|
||||
|
||||
class Dutch(Language):
|
||||
lang = 'nl'
|
||||
|
||||
class Defaults(Language.Defaults):
|
||||
tokenizer_exceptions = dict(language_data.TOKENIZER_EXCEPTIONS)
|
||||
lex_attr_getters = dict(Language.Defaults.lex_attr_getters)
|
||||
lex_attr_getters[LANG] = lambda text: 'nl'
|
||||
|
||||
prefixes = tuple(language_data.TOKENIZER_PREFIXES)
|
||||
|
||||
suffixes = tuple(language_data.TOKENIZER_SUFFIXES)
|
||||
|
||||
infixes = tuple(language_data.TOKENIZER_INFIXES)
|
||||
|
||||
tag_map = dict(language_data.TAG_MAP)
|
||||
|
||||
stop_words = set(language_data.STOP_WORDS)
|
||||
|
|
@ -0,0 +1,285 @@
|
|||
# encoding: utf8
|
||||
from __future__ import unicode_literals
|
||||
import re
|
||||
|
||||
# Stop words are retrieved from http://www.damienvanholten.com/downloads/dutch-stop-words.txt
|
||||
STOP_WORDS = set("""
|
||||
aan
|
||||
af
|
||||
al
|
||||
alles
|
||||
als
|
||||
altijd
|
||||
andere
|
||||
ben
|
||||
bij
|
||||
daar
|
||||
dan
|
||||
dat
|
||||
de
|
||||
der
|
||||
deze
|
||||
die
|
||||
dit
|
||||
doch
|
||||
doen
|
||||
door
|
||||
dus
|
||||
een
|
||||
eens
|
||||
en
|
||||
er
|
||||
ge
|
||||
geen
|
||||
geweest
|
||||
haar
|
||||
had
|
||||
heb
|
||||
hebben
|
||||
heeft
|
||||
hem
|
||||
het
|
||||
hier
|
||||
hij
|
||||
hoe
|
||||
hun
|
||||
iemand
|
||||
iets
|
||||
ik
|
||||
in
|
||||
is
|
||||
ja
|
||||
je
|
||||
kan
|
||||
kon
|
||||
kunnen
|
||||
maar
|
||||
me
|
||||
meer
|
||||
men
|
||||
met
|
||||
mij
|
||||
mijn
|
||||
moet
|
||||
na
|
||||
naar
|
||||
niet
|
||||
niets
|
||||
nog
|
||||
nu
|
||||
of
|
||||
om
|
||||
omdat
|
||||
ons
|
||||
ook
|
||||
op
|
||||
over
|
||||
reeds
|
||||
te
|
||||
tegen
|
||||
toch
|
||||
toen
|
||||
tot
|
||||
u
|
||||
uit
|
||||
uw
|
||||
van
|
||||
veel
|
||||
voor
|
||||
want
|
||||
waren
|
||||
was
|
||||
wat
|
||||
we
|
||||
wel
|
||||
werd
|
||||
wezen
|
||||
wie
|
||||
wij
|
||||
wil
|
||||
worden
|
||||
zal
|
||||
ze
|
||||
zei
|
||||
zelf
|
||||
zich
|
||||
zij
|
||||
zijn
|
||||
zo
|
||||
zonder
|
||||
zou
|
||||
""".split())
|
||||
|
||||
|
||||
TOKENIZER_PREFIXES = map(re.escape, r'''
|
||||
,
|
||||
"
|
||||
(
|
||||
[
|
||||
{
|
||||
*
|
||||
<
|
||||
>
|
||||
$
|
||||
£
|
||||
„
|
||||
“
|
||||
'
|
||||
``
|
||||
`
|
||||
#
|
||||
US$
|
||||
C$
|
||||
A$
|
||||
a-
|
||||
‘
|
||||
....
|
||||
...
|
||||
‚
|
||||
»
|
||||
_
|
||||
§
|
||||
'''.strip().split('\n'))
|
||||
|
||||
|
||||
TOKENIZER_SUFFIXES = r'''
|
||||
,
|
||||
\"
|
||||
\)
|
||||
\]
|
||||
\}
|
||||
\*
|
||||
\!
|
||||
\?
|
||||
%
|
||||
\$
|
||||
>
|
||||
:
|
||||
;
|
||||
'
|
||||
”
|
||||
“
|
||||
«
|
||||
_
|
||||
''
|
||||
's
|
||||
'S
|
||||
’s
|
||||
’S
|
||||
’
|
||||
‘
|
||||
°
|
||||
€
|
||||
\.\.
|
||||
\.\.\.
|
||||
\.\.\.\.
|
||||
(?<=[a-zäöüßÖÄÜ)\]"'´«‘’%\)²“”])\.
|
||||
\-\-
|
||||
´
|
||||
(?<=[0-9])km²
|
||||
(?<=[0-9])m²
|
||||
(?<=[0-9])cm²
|
||||
(?<=[0-9])mm²
|
||||
(?<=[0-9])km³
|
||||
(?<=[0-9])m³
|
||||
(?<=[0-9])cm³
|
||||
(?<=[0-9])mm³
|
||||
(?<=[0-9])ha
|
||||
(?<=[0-9])km
|
||||
(?<=[0-9])m
|
||||
(?<=[0-9])cm
|
||||
(?<=[0-9])mm
|
||||
(?<=[0-9])µm
|
||||
(?<=[0-9])nm
|
||||
(?<=[0-9])yd
|
||||
(?<=[0-9])in
|
||||
(?<=[0-9])ft
|
||||
(?<=[0-9])kg
|
||||
(?<=[0-9])g
|
||||
(?<=[0-9])mg
|
||||
(?<=[0-9])µg
|
||||
(?<=[0-9])t
|
||||
(?<=[0-9])lb
|
||||
(?<=[0-9])oz
|
||||
(?<=[0-9])m/s
|
||||
(?<=[0-9])km/h
|
||||
(?<=[0-9])mph
|
||||
(?<=[0-9])°C
|
||||
(?<=[0-9])°K
|
||||
(?<=[0-9])°F
|
||||
(?<=[0-9])hPa
|
||||
(?<=[0-9])Pa
|
||||
(?<=[0-9])mbar
|
||||
(?<=[0-9])mb
|
||||
(?<=[0-9])T
|
||||
(?<=[0-9])G
|
||||
(?<=[0-9])M
|
||||
(?<=[0-9])K
|
||||
(?<=[0-9])kb
|
||||
'''.strip().split('\n')
|
||||
|
||||
|
||||
TOKENIZER_INFIXES = r'''
|
||||
\.\.\.
|
||||
(?<=[a-z])\.(?=[A-Z])
|
||||
(?<=[a-zöäüßA-ZÖÄÜ"]):(?=[a-zöäüßA-ZÖÄÜ])
|
||||
(?<=[a-zöäüßA-ZÖÄÜ"])>(?=[a-zöäüßA-ZÖÄÜ])
|
||||
(?<=[a-zöäüßA-ZÖÄÜ"])<(?=[a-zöäüßA-ZÖÄÜ])
|
||||
(?<=[a-zöäüßA-ZÖÄÜ"])=(?=[a-zöäüßA-ZÖÄÜ])
|
||||
'''.strip().split('\n')
|
||||
|
||||
|
||||
#TODO Make tokenizer excpetions for Dutch
|
||||
TOKENIZER_EXCEPTIONS = {}
|
||||
|
||||
#TODO insert TAG_MAP for Dutch
|
||||
TAG_MAP = {
|
||||
"ADV": {
|
||||
"pos": "ADV"
|
||||
},
|
||||
"NOUN": {
|
||||
"pos": "NOUN"
|
||||
},
|
||||
"ADP": {
|
||||
"pos": "ADP"
|
||||
},
|
||||
"PRON": {
|
||||
"pos": "PRON"
|
||||
},
|
||||
"SCONJ": {
|
||||
"pos": "SCONJ"
|
||||
},
|
||||
"PROPN": {
|
||||
"pos": "PROPN"
|
||||
},
|
||||
"DET": {
|
||||
"pos": "DET"
|
||||
},
|
||||
"SYM": {
|
||||
"pos": "SYM"
|
||||
},
|
||||
"INTJ": {
|
||||
"pos": "INTJ"
|
||||
},
|
||||
"PUNCT": {
|
||||
"pos": "PUNCT"
|
||||
},
|
||||
"NUM": {
|
||||
"pos": "NUM"
|
||||
},
|
||||
"AUX": {
|
||||
"pos": "AUX"
|
||||
},
|
||||
"X": {
|
||||
"pos": "X"
|
||||
},
|
||||
"CONJ": {
|
||||
"pos": "CONJ"
|
||||
},
|
||||
"ADJ": {
|
||||
"pos": "ADJ"
|
||||
},
|
||||
"VERB": {
|
||||
"pos": "VERB"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,6 +1,7 @@
|
|||
# encoding: utf8
|
||||
from __future__ import unicode_literals
|
||||
from ...fr import French
|
||||
from ...nl import Dutch
|
||||
|
||||
def test_load_french():
|
||||
nlp = French()
|
||||
|
|
@ -10,3 +11,11 @@ def test_load_french():
|
|||
assert doc[2].text == u'vous'
|
||||
assert doc[3].text == u'français'
|
||||
assert doc[4].text == u'?'
|
||||
|
||||
def test_load_dutch():
|
||||
nlp = Dutch()
|
||||
doc = nlp(u'Is dit Nederlands?')
|
||||
assert doc[0].text == u'Is'
|
||||
assert doc[1].text == u'dit'
|
||||
assert doc[2].text == u'Nederlands'
|
||||
assert doc[3].text == u'?'
|
||||
Loading…
Reference in New Issue