Rooba
/
genienlp
mirror of https://github.com/stanford-oval/genienlp.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update README.md

This commit is contained in:
s-jse 2022-02-25 12:42:44 -08:00 committed by GitHub
parent 1e1e132ba8
commit 5cec08df1b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 3 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
README.md
View File

@ -85,18 +85,19 @@ use standard input/output instead of TCP.
### Calibrating a trained model
Calibrate the confidence scores of a trained model:
Calibrate the confidence scores of a trained model. This is usually done on the validation set. After calibration, you can use the confidence scores `genienlp predict` outputs to identifying how confident the model is about each one of its predictions.
1. Calculate and save confidence features of the evaluation set in a pickle file:
```bash
genienlp predict --tasks almond --data <datadir> --path <model_dir> --save_confidence_features --confidence_feature_path <confidence_feature_file> --mc_dropout_num 1
genienlp predict --tasks almond --data <datadir> --path <model_dir> --evaluate valid --eval_dir <output> --save_confidence_features --confidence_feature_path <confidence_feature_file> --mc_dropout_num 1
```
2. Train a boosted tree to map confidence features to a score between 0 and 1:
```bash
genienlp calibrate --confidence_path <confidence_feature_file> --save <calibrator_directory> --name_prefix <calibrator_name>
````
Optionally, you can add `--plot` to this command to get 3 plots descirbing the quality of the calibrator. Note that you need to install the `matplotlib` package (version `>3`) first.
3. Now if you provide `--calibrator_paths` during prediction, it will output confidence scores for each output:
```bash