From 5cec08df1bb8150f368f8e4dccdef8ae2123e71f Mon Sep 17 00:00:00 2001 From: s-jse <60150701+s-jse@users.noreply.github.com> Date: Fri, 25 Feb 2022 12:42:44 -0800 Subject: [PATCH] Update README.md --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 99f4f13c..a69af422 100644 --- a/README.md +++ b/README.md @@ -85,18 +85,19 @@ use standard input/output instead of TCP. ### Calibrating a trained model -Calibrate the confidence scores of a trained model: +Calibrate the confidence scores of a trained model. This is usually done on the validation set. After calibration, you can use the confidence scores `genienlp predict` outputs to identifying how confident the model is about each one of its predictions. 1. Calculate and save confidence features of the evaluation set in a pickle file: ```bash - genienlp predict --tasks almond --data --path --save_confidence_features --confidence_feature_path --mc_dropout_num 1 + genienlp predict --tasks almond --data --path --evaluate valid --eval_dir --save_confidence_features --confidence_feature_path --mc_dropout_num 1 ``` 2. Train a boosted tree to map confidence features to a score between 0 and 1: ```bash genienlp calibrate --confidence_path --save --name_prefix ```` + Optionally, you can add `--plot` to this command to get 3 plots descirbing the quality of the calibrator. Note that you need to install the `matplotlib` package (version `>3`) first. 3. Now if you provide `--calibrator_paths` during prediction, it will output confidence scores for each output: ```bash