diff --git a/README.md b/README.md index 99f4f13c..a69af422 100644 --- a/README.md +++ b/README.md @@ -85,18 +85,19 @@ use standard input/output instead of TCP. ### Calibrating a trained model -Calibrate the confidence scores of a trained model: +Calibrate the confidence scores of a trained model. This is usually done on the validation set. After calibration, you can use the confidence scores `genienlp predict` outputs to identifying how confident the model is about each one of its predictions. 1. Calculate and save confidence features of the evaluation set in a pickle file: ```bash - genienlp predict --tasks almond --data --path --save_confidence_features --confidence_feature_path --mc_dropout_num 1 + genienlp predict --tasks almond --data --path --evaluate valid --eval_dir --save_confidence_features --confidence_feature_path --mc_dropout_num 1 ``` 2. Train a boosted tree to map confidence features to a score between 0 and 1: ```bash genienlp calibrate --confidence_path --save --name_prefix ```` + Optionally, you can add `--plot` to this command to get 3 plots descirbing the quality of the calibrator. Note that you need to install the `matplotlib` package (version `>3`) first. 3. Now if you provide `--calibrator_paths` during prediction, it will output confidence scores for each output: ```bash