Commit Graph

12330 Commits

Author SHA1 Message Date
Ines Montani 10b84e1e27 Add flag to toggle sdist creation on package [ci skip] 2020-07-27 16:52:23 +02:00
Ines Montani 7dd53d0964 Fix typo [ci skip] 2020-07-27 00:34:00 +02:00
Ines Montani 7adbaf9a5b Update docs [ci skip] 2020-07-27 00:29:45 +02:00
Ines Montani 3d56a3f286 Make more args keyword-only 2020-07-27 00:27:53 +02:00
Matthew Honnibal 80271ac0ba Update default config 2020-07-26 15:27:39 +02:00
Ines Montani ed61fb10fc Rename default textcat arch to TextCatEnsemble 2020-07-26 15:11:43 +02:00
Ines Montani 53d37da29a Make sure @factories is removed from config 2020-07-26 15:11:24 +02:00
Matthew Honnibal ac5901d076 Merge branch 'develop' of https://github.com/explosion/spaCy into develop 2020-07-26 14:20:27 +02:00
Matthew Honnibal fb5dbe30b5 Trim training 101 2020-07-26 13:43:22 +02:00
Matthew Honnibal e6a7deb7cc Edits to the training 101 section 2020-07-26 13:42:08 +02:00
Ines Montani 787d066e22 Remove pipes.pyx
Probably accidentally re-added in a merge?
2020-07-26 13:08:52 +02:00
Matthew Honnibal 520d25cb50
Add smart_open dependency to fetch project assets (#5812)
* Use smart_open for project assets

* Fix assets.py

* Update pyproject.toml
2020-07-26 12:15:00 +02:00
Ines Montani c288dba8e7 Update docs [ci skip] 2020-07-25 18:51:12 +02:00
Ines Montani 1346ee06d4
Merge pull request #5813 from explosion/chore/tidy-autoformat-types
Tidy up, autoformat, add types
2020-07-25 18:44:08 +02:00
Ines Montani eb9acae34d
Merge pull request #5791 from adrianeboyd/docs/morphology 2020-07-25 15:10:21 +02:00
Ines Montani e92df281ce Tidy up, autoformat, add types 2020-07-25 15:01:15 +02:00
Matthew Honnibal 71242327b2 Set version to v3.0.0a5 2020-07-25 14:06:01 +02:00
Matthew Honnibal afd504f8c0 Update config 2020-07-25 14:04:25 +02:00
Ines Montani cdbd6ba912
Merge pull request #5798 from explosion/feature/language-data-config 2020-07-25 13:34:49 +02:00
Matthew Honnibal 44a0b072e0 Merge branch 'feature/language-data-config' of https://github.com/explosion/spaCy into feature/language-data-config 2020-07-25 13:34:07 +02:00
Matthew Honnibal 17f39eebdc Update PTB config 2020-07-25 13:33:40 +02:00
Ines Montani 49f27a2a7b Tidy up [ci skip] 2020-07-25 13:00:49 +02:00
Ines Montani 4a0a692875 Add missing lex_attr_getters (resolves #5806 ) 2020-07-25 12:55:18 +02:00
Adriane Boyd 2bcceb80c4
Refactor the Scorer to improve flexibility (#5731)
* Refactor the Scorer to improve flexibility

Refactor the `Scorer` to improve flexibility for arbitrary pipeline
components.

* Individual pipeline components provide their own `evaluate` methods
that score a list of `Example`s and return a dictionary of scores
* `Scorer` is initialized either:
  * with a provided pipeline containing components to be scored
  * with a default pipeline containing the built-in statistical
    components (senter, tagger, morphologizer, parser, ner)
* `Scorer.score` evaluates a list of `Example`s and returns a dictionary
of scores referring to the scores provided by the components in the
pipeline

Significant differences:

* `tags_acc` is renamed to `tag_acc` to be consistent with `token_acc`
and the new `morph_acc`, `pos_acc`, and `lemma_acc`
* Scoring is no longer cumulative: `Scorer.score` scores a list of
examples rather than a single example and does not retain any state
about previously scored examples
* PRF values in the returned scores are no longer multiplied by 100

* Add kwargs to Morphologizer.evaluate

* Create generalized scoring methods in Scorer

* Generalized static scoring methods are added to `Scorer`
  * Methods require an attribute (either on Token or Doc) that is
used to key the returned scores

Naming differences:

* `uas`, `las`, and `las_per_type` in the scores dict are renamed to
`dep_uas`, `dep_las`, and `dep_las_per_type`

Scoring differences:

* `Doc.sents` is now scored as spans rather than on sentence-initial
token positions so that `Doc.sents` and `Doc.ents` can be scored with
the same method (this lowers scores since a single incorrect sentence
start results in two incorrect spans)

* Simplify / extend hasattr check for eval method

* Add hasattr check to tokenizer scoring
* Simplify to hasattr check for component scoring

* Reset Example alignment if docs are set

Reset the Example alignment if either doc is set in case the
tokenization has changed.

* Add PRF tokenization scoring for tokens as spans

Add PRF scores for tokens as character spans. The scores are:

* token_acc: # correct tokens / # gold tokens
* token_p/r/f: PRF for (token.idx, token.idx + len(token))

* Add docstring to Scorer.score_tokenization

* Rename component.evaluate() to component.score()

* Update Scorer API docs

* Update scoring for positive_label in textcat

* Fix TextCategorizer.score kwargs

* Update Language.evaluate docs

* Update score names in default config
2020-07-25 12:53:02 +02:00
Ines Montani c003d26b94 Tidy up 2020-07-25 12:21:37 +02:00
Ines Montani a063a82c40 Tidy up __init__.py 2020-07-25 12:14:37 +02:00
Ines Montani 8d9d28eb8b Re-add setting for vocab data and tidy up 2020-07-25 12:14:28 +02:00
Ines Montani b9aaa4e457 Improve vocab data integration and warning 2020-07-25 11:51:30 +02:00
Ines Montani 38f6ea7a78 Simplify language data and revert detailed configs 2020-07-24 14:50:26 +02:00
Adriane Boyd 656574a01a
Update Japanese tests (#5807)
* Update POS tests to reflect current behavior (it is not entirely clear
whether the AUX/VERB mapping is indeed the desired behavior?)
* Switch to `from_config` initialization in subtoken test
2020-07-24 12:45:14 +02:00
Adriane Boyd fdb8815ef5
Minor refactor for Morphology and MorphAnalysis (#5804)
* `MorphAnalysis.get` returns only the field values
* Move `_normalize_props` inside `Morphology` as
`Morphology.normalize_attrs` and simplify
  * Simplify POS field detection/conversion
  * Convert all non-POS features to strings
* `Morphology` returns an empty string for a missing morph to align
with the FEATS string returned for an existing morph
* Remove unused `list_to_feats`
2020-07-24 09:28:06 +02:00
Adriane Boyd 41525901ef Move MorphAnalysis to Other section 2020-07-23 08:58:22 +02:00
Adriane Boyd 8f44584bef Update MorphAnalysis.get and related examples 2020-07-23 08:51:31 +02:00
Ines Montani 87737a5a60 Tidy up 2020-07-23 00:16:23 +02:00
Ines Montani a624ae0675 Remove POS, TAG and LEMMA from tokenizer exceptions 2020-07-22 23:09:01 +02:00
Ines Montani 14d7d46f89 Merge branch 'develop' into feature/language-data-config 2020-07-22 22:18:53 +02:00
Ines Montani b507f61629 Tidy up and move noun_chunks, token_match, url_match 2020-07-22 22:18:46 +02:00
Ines Montani 7fc4dadd22 Fix typo 2020-07-22 20:27:22 +02:00
Adriane Boyd 941b9e33f7 Add Token.morph_ 2020-07-22 17:59:45 +02:00
Ines Montani d0c6d1efc5
@factories -> factory (#5801) 2020-07-22 17:29:31 +02:00
Ines Montani be476e495e
Merge pull request #5787 from adrianeboyd/docs/morphologizer
Initial draft of Morphologizer API docs
2020-07-22 17:16:57 +02:00
Ines Montani 2c5bb59909 Use consistent --gpu-id option name 2020-07-22 16:53:41 +02:00
Ines Montani 0fcd352179 Remove omit_extra_lookups 2020-07-22 16:01:17 +02:00
Ines Montani 945f795a3e WIP: move more language data to config 2020-07-22 15:59:37 +02:00
Adriane Boyd b84fd70cc3
Fix exceptions for Morphology.__reduce__ (#5792)
Pickle exceptions in the MORPH_RULES format instead of the internal
format after the recent `Morphology.__init__` changes.
2020-07-22 15:00:25 +02:00
Ines Montani 43b960c01b
Refactor pipeline components, config and language data (#5759)
* Update with WIP

* Update with WIP

* Update with pipeline serialization

* Update types and pipe factories

* Add deep merge, tidy up and add tests

* Fix pipe creation from config

* Don't validate default configs on load

* Update spacy/language.py

Co-authored-by: Ines Montani <ines@ines.io>

* Adjust factory/component meta error

* Clean up factory args and remove defaults

* Add test for failing empty dict defaults

* Update pipeline handling and methods

* provide KB as registry function instead of as object

* small change in test to make functionality more clear

* update example script for EL configuration

* Fix typo

* Simplify test

* Simplify test

* splitting pipes.pyx into separate files

* moving default configs to each component file

* fix batch_size type

* removing default values from component constructors where possible (TODO: test 4725)

* skip instead of xfail

* Add test for config -> nlp with multiple instances

* pipeline.pipes -> pipeline.pipe

* Tidy up, document, remove kwargs

* small cleanup/generalization for Tok2VecListener

* use DEFAULT_UPSTREAM field

* revert to avoid circular imports

* Fix tests

* Replace deprecated arg

* Make model dirs require config

* fix pickling of keyword-only arguments in constructor

* WIP: clean up and integrate full config

* Add helper to handle function args more reliably

Now also includes keyword-only args

* Fix config composition and serialization

* Improve config debugging and add visual diff

* Remove unused defaults and fix type

* Remove pipeline and factories from meta

* Update spacy/default_config.cfg

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update spacy/default_config.cfg

* small UX edits

* avoid printing stack trace for debug CLI commands

* Add support for language-specific factories

* specify the section of the config which holds the model to debug

* WIP: add Language.from_config

* Update with language data refactor WIP

* Auto-format

* Add backwards-compat handling for Language.factories

* Update morphologizer.pyx

* Fix morphologizer

* Update and simplify lemmatizers

* Fix Japanese tests

* Port over tagger changes

* Fix Chinese and tests

* Update to latest Thinc

* WIP: xfail first Russian lemmatizer test

* Fix component-specific overrides

* fix nO for output layers in debug_model

* Fix default value

* Fix tests and don't pass objects in config

* Fix deep merging

* Fix lemma lookup data registry

Only load the lookups if an entry is available in the registry (and if spacy-lookups-data is installed)

* Add types

* Add Vocab.from_config

* Fix typo

* Fix tests

* Make config copying more elegant

* Fix pipe analysis

* Fix lemmatizers and is_base_form

* WIP: move language defaults to config

* Fix morphology type

* Fix vocab

* Remove comment

* Update to latest Thinc

* Add morph rules to config

* Tidy up

* Remove set_morphology option from tagger factory

* Hack use_gpu

* Move [pipeline] to top-level block and make [nlp.pipeline] list

Allows separating component blocks from component order – otherwise, ordering the config would mean a changed component order, which is bad. Also allows initial config to define more components and not use all of them

* Fix use_gpu and resume in CLI

* Auto-format

* Remove resume from config

* Fix formatting and error

* [pipeline] -> [components]

* Fix types

* Fix tagger test: requires set_morphology?

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
Co-authored-by: svlandeg <sofie.vanlandeghem@gmail.com>
Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>
2020-07-22 13:42:59 +02:00
Adriane Boyd d3385f4be2 Add Morphology and MorphAnalysis to overview 2020-07-21 13:06:22 +02:00
Adriane Boyd fcd3a4abe3 Add morph to Token API docs 2020-07-21 13:05:58 +02:00
Adriane Boyd 14df00ae98 Add Morphology and MorphAnalsysis API docs
Add initial draft of `Morphology` and `MorphAnalysis` API docs.
2020-07-21 10:33:46 +02:00
Ines Montani 311d0bde29
Merge pull request #5788 from explosion/master-tmp 2020-07-20 15:39:24 +02:00