GenieNLP

GenieNLP is suitable for all NLP tasks, including text generation (e.g. translation, paraphasing), token classification (e.g. named entity recognition) and sequence classification (e.g. NLI, sentiment analysis). This library contains the code to run NLP models for the [Genie Toolkit](https://github.com/stanford-oval/genie-toolkit) and the [Genie Virtual Assistant](https://genie.stanford.edu/). Genie primarily uses this library for semantic parsing, paraphrasing, translation, and dialogue state tracking. Therefore, GenieNLP has a lot of extra features for these tasks. Works with [🤗 models](https://huggingface.co/models) and [🤗 Datasets](https://huggingface.co/datasets). ## Table of Contents - [Installation](#installation) - [Usage](#usage) - [Training a semantic parser](#training-a-semantic-parser) - [Inference on a semantic parser](#inference-on-a-semantic-parser) - [Calibrating a trained model](#calibrating-a-trained-model) - [Paraphrasing](#paraphrasing) - [Translation](#translation) - [Named Entity Disambiguation](#named-entity-disambiguation) - [Citation](#citation) ## Installation GenieNLP is tested with Python 3.8. You can install the latest release with pip from PyPI: ```bash pip install genienlp ``` Or from source: ```bash git clone https://github.com/stanford-oval/genienlp.git cd genienlp pip install -e . # -e means your changes to the code will automatically take effect without the need to reinstall ``` After installation, `genienlp` command becomes available. Some GenieNLP commands have additional dependencies for plotting and entity detection. If you are using those commands, you can obtain their dependencies by running the following: ``` pip install matplotlib~=3.0 seaborn~=0.9 python -m spacy download en_core_web_sm ``` ## Usage ### Training a semantic parser The general form is: ```bash genienlp train --train_tasks almond --train_iterations 50000 --data --save ``` The `` should contain a single folder called "almond" (the name of the task). That folder should contain the files "train.tsv" and "eval.tsv" for train and dev set respectively. To train a BERT-LSTM (or other MLM-based models) use: ```bash genienlp train --train_tasks almond --train_iterations 50000 --data --save \ --model TransformerLSTM --pretrained_model bert-base-cased --trainable_decoder_embedding 50 ``` To train a BART or other Seq2Seq model, use: ```bash genienlp train --train_tasks almond --train_iterations 50000 --data --save \ --model TransformerSeq2Seq --pretrained_model facebook/bart-large --gradient_accumulation_steps 20 ``` The default batch sizes are tuned for training on a single V100 GPU. Use `--train_batch_tokens` and `--val_batch_size` to control the batch sizes. See `genienlp train --help` for the full list of options. **NOTE**: the BERT-LSTM model used by the current version of the library is not comparable with the one used in our published paper (cited below), because the input preprocessing is different. If you wish to compare with published results you should use genienlp <= 0.5.0. ### Inference on a semantic parser In batch mode: ```bash genienlp predict --tasks almond --data --path --eval_dir ``` The `` should contain a single folder called "almond" (the name of the task). That folder should contain the files "train.tsv" and "eval.tsv" for train and dev set respectively. The result of batch prediction will be saved in `/almond/valid.tsv`, as a TSV file containing ID and prediction. In interactive mode: ```bash genienlp server --path ``` Opens a TCP server that listens to requests, formatted as JSON objects containing `id` (the ID of the request), `task` (the name of the task), `context`, and `question`. The server writes out JSON objects containing `id` and `answer`. The server listens to port 8401 by default. Use `--port` to specify a different port or `--stdin` to use standard input/output instead of TCP. ### Calibrating a trained model Calibrate the confidence scores of a trained model. This is usually done on the validation set. After calibration, you can use the confidence scores `genienlp predict` outputs to identifying how confident the model is about each one of its predictions. 1. Calculate and save confidence features of the evaluation set in a pickle file: ```bash genienlp predict --tasks almond --data --path --evaluate valid --eval_dir --save_confidence_features --confidence_feature_path --mc_dropout_num 1 ``` 2. Train a boosted tree to map confidence features to a score between 0 and 1: ```bash genienlp calibrate --confidence_path --save --name_prefix ```` Optionally, you can add `--plot` to this command to get 3 plots descirbing the quality of the calibrator. Note that you need to install the `matplotlib` package (version `>3`) first. 3. Now if you provide `--calibrator_paths` during prediction, it will output confidence scores for each output: ```bash genienlp predict --tasks almond --data --path --calibrator_paths /.calib ``` `--mc_dropout_num` specifies the number of additional forward passes of the model. For example, `--mc_dropout_num 5` makes subsequent inferences using this calibrator 6 times slower. If inference speed is important to you, you should use `--mc_dropout_num 0` in the first command and add `--fast` to the second command so that only fast calibration methods are used. This way, you gain a lot of inference speed, and lose a little bit of calibration quality. ### Paraphrasing Generate paraphrases: ```bash genienlp run-paraphrase --model_name_or_path --temperature 0.3 --repetition_penalty 1.0 --num_samples 4 --batch_size 32 --input_file --input_column 1 ``` ### Translation Use the following command for training/ finetuning an NMT model: ```bash genienlp train --train_tasks almond_translate --data --train_languages --eval_languages --no_commit --train_iterations --preserve_case --save --exist_ok --model TransformerSeq2Seq --pretrained_model ``` We currently support MarianMT, MBART, MT5, and M2M100 models.
To save a pretrained model in genienlp format without any finetuning, set train_iterations to 0. You can then use this model to do inference. To produce translations for an eval/ test set run the following command: ```bash genienlp predict --tasks almond_translate --data --pred_languages --pred_tgt_languages --path --eval_dir --val_batch_size 4000 --evaluate --overwrite --silent ``` If your dataset is a document or contains long examples, pass `--translate_example_split` to break the examples down into individual sentences before translation for better results.
To use [alignment](https://aclanthology.org/2020.emnlp-main.481.pdf), pass `--do_alignment` which ensures the tokens between quotations marks in the sentence are preserved during translation. ### Named Entity Disambiguation First run a bootleg model to extract mentions, entity candidates, and contextual embeddings for the mentions. ```bash genienlp bootleg-dump-features --train_tasks --save --preserve_case --data --train_batch_tokens 1200 --val_batch_size 2000 --database_type json --database_dir --min_entity_len 1 --max_entity_len 4 --bootleg_model ``` This command generates several output files. In `` you should see a `prep` dir which contains preprocessed data (e.g. data converted to memory-mapped format, several arrays to facilitate embedding lookup, etc.) If your dataset doesn't change you can reuse the same files. It will also generate several files in folder. In `eval_bootleg/[train|eval]//bootleg_lables.jsonl` you can see the examples, mentions, predicted candidates and their probabilities according to bootleg. Now you can use the extracted features from bootleg in downstream tasks such as semantic parsing to improve named entity understanding and consequently generation: ```bash genienlp train --train_tasks --train_iterations --preserve_case --save --data --model TransformerSeq2Seq --pretrained_model facebook/bart-base --train_batch_tokens 1000 --val_batch_size 1000 --do_ned --database_dir --ned_retrieve_method bootleg --entity_attributes type_id type_prob --add_entities_to_text append --bootleg_model ``` See `genienlp --help` and `genienlp --help` for more details about each argument. ## Citation If you use multiTask training in your work, please cite [*The Natural Language Decathlon: Multitask Learning as Question Answering*](https://arxiv.org/abs/1806.08730). ```bibtex @article{McCann2018decaNLP, title={The Natural Language Decathlon: Multitask Learning as Question Answering}, author={Bryan McCann and Nitish Shirish Keskar and Caiming Xiong and Richard Socher}, journal={arXiv preprint arXiv:1806.08730}, year={2018} } ``` If you use the BERT-LSTM model (Identity encoder + MQAN decoder), please cite [Schema2QA: High-Quality and Low-Cost Q&A Agents for the Structured Web](https://arxiv.org/abs/2001.05609) ```bibtex @InProceedings{xu2020schema2qa, title={{Schema2QA}: High-Quality and Low-Cost {Q\&A} Agents for the Structured Web}, author={Silei Xu and Giovanni Campagna and Jian Li and Monica S. Lam}, booktitle={Proceedings of the 29th ACM International Conference on Information and Knowledge Management}, year={2020}, doi={https://doi.org/10.1145/3340531.3411974} } ``` If you use the paraphrasing model (BART or GPT-2 fine-tuned on a paraphrasing dataset), please cite [AutoQA: From Databases to QA Semantic Parsers with Only Synthetic Training Data](https://arxiv.org/abs/2010.04806) ```bibtex @inproceedings{xu-etal-2020-autoqa, title = "{A}uto{QA}: From Databases to {QA} Semantic Parsers with Only Synthetic Training Data", author = "Xu, Silei and Semnani, Sina and Campagna, Giovanni and Lam, Monica", booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)", month = nov, year = "2020", address = "Online", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/2020.emnlp-main.31", pages = "422--434", } ``` If you use multilingual models such as MarianMT, MBART, MT5, or XLMR-LSTM for Seq2Seq tasks, please cite [Localizing Open-Ontology QA Semantic Parsers in a Day Using Machine Translation](https://aclanthology.org/2020.emnlp-main.481/), [Contextual Semantic Parsing for Multilingual Task-Oriented Dialogues](https://arxiv.org/abs/2111.02574), and the original paper that introduced the model. ```bibtex @inproceedings{moradshahi-etal-2020-localizing, title = "Localizing Open-Ontology {QA} Semantic Parsers in a Day Using Machine Translation", author = "Moradshahi, Mehrad and Campagna, Giovanni and Semnani, Sina and Xu, Silei and Lam, Monica", booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)", month = November, year = "2020", address = "Online", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/2020.emnlp-main.481", pages = "5970--5983", } ``` ```bibtex @article{moradshahi2021contextual, title={Contextual Semantic Parsing for Multilingual Task-Oriented Dialogues}, author={Moradshahi, Mehrad and Tsai, Victoria and Campagna, Giovanni and Lam, Monica S}, journal={arXiv preprint arXiv:2111.02574}, year={2021} } ``` If you use English models such as BART for Seq2Seq tasks, please cite [A Few-Shot Semantic Parser for Wizard-of-Oz Dialogues with the Precise ThingTalk Representation](https://aclanthology.org/2022.findings-acl.317/), and the original paper that introduced the model. ```bibtex @inproceedings{campagna-etal-2022-shot, title = "A Few-Shot Semantic Parser for {W}izard-of-{O}z Dialogues with the Precise {T}hing{T}alk Representation", author = "Campagna, Giovanni and Semnani, Sina and Kearns, Ryan and Koba Sato, Lucas Jun and Xu, Silei and Lam, Monica", booktitle = "Findings of the Association for Computational Linguistics: ACL 2022", month = may, year = "2022", address = "Dublin, Ireland", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2022.findings-acl.317", doi = "10.18653/v1/2022.findings-acl.317", pages = "4021--4034", } ```