c0fd8a2e71
find-threshold: CLI command for multi-label classifier threshold tuning ( #11280 )
...
* Add foundation for find-threshold CLI functionality.
* Finish first draft for find-threshold.
* Add tests.
* Revert adjusted import statements.
* Fix mypy errors.
* Fix imports.
* Harmonize arguments with spacy evaluate command.
* Generalize component and threshold handling. Harmonize arguments with 'spacy evaluate' CLI.
* Fix Spancat test.
* Add beta parameter to Scorer and PRFScore.
* Make beta a component scorer setting.
* Remove beta.
* Update nlp.config (workaround).
* Reload pipeline on threshold change. Adjust tests. Remove confection reference.
* Remove assumption of component being a Pipe object or having a .cfg attribute.
* Adjust test output and reference values.
* Remove beta references. Delete universe.json.
* Reverting unnecessary changes. Removing unused default values. Renaming variables in find-cli tests.
* Update spacy/cli/find_threshold.py
Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
* Remove adding labels in tests.
* Remove unused error
* Undo changes to PRFScorer
* Change default value for n_trials. Log table iteratively.
* Add warnings for pointless applications of find_threshold().
* Fix imports.
* Adjust type check of TextCategorizer to exclude subclasses.
* Change check of if there's only one unique value in scores.
* Update spacy/cli/find_threshold.py
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
* Incorporate feedback.
* Fix test issue. Update docstring.
* Update docs & docstring.
* Update spacy/tests/test_cli.py
Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
* Add examples to docs. Rename _nlp to nlp in tests.
* Update spacy/cli/find_threshold.py
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
* Update spacy/cli/find_threshold.py
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
2022-11-25 11:44:55 +01:00
Daniël de Kok