spaCy/website/docs/api/large-language-models.mdx

---
title: Large Language Models
teaser: Integrating LLMs into structured NLP pipelines
menu:
  - ['Config and implementation', 'config']
  - ['Tasks', 'tasks']
  - ['Models', 'models']
  - ['Cache', 'cache']
  - ['Various Functions', 'various-functions']
---

[The `spacy-llm` package](https://github.com/explosion/spacy-llm) integrates
Large Language Models (LLMs) into spaCy, featuring a modular system for **fast
prototyping** and **prompting**, and turning unstructured responses into
**robust outputs** for various NLP tasks, **no training data** required.

## Config and implementation {id="config"}

An LLM component is implemented through the `LLMWrapper` class. It is accessible
through a generic `llm`
[component factory](https://spacy.io/usage/processing-pipelines#custom-components-factories)
as well as through task-specific component factories: `llm_ner`, `llm_spancat`,
`llm_rel`, `llm_textcat`, `llm_sentiment`, `llm_summarization`,
`llm_entity_linker`, `llm_raw` and `llm_translation`. For these factories, the
GPT-3-5 model from OpenAI is used by default, but this can be customized.

> #### Example
>
> ```python
> # Construction via add_pipe with the default GPT 3.5 model and an explicitly defined task
> config = {"task": {"@llm_tasks": "spacy.NER.v3", "labels": ["PERSON", "ORGANISATION", "LOCATION"]}}
> llm = nlp.add_pipe("llm", config=config)
>
> # Construction via add_pipe with a task-specific factory and default GPT3.5 model
> llm = nlp.add_pipe("llm_ner")
>
> # Construction via add_pipe with a task-specific factory and custom model
> llm = nlp.add_pipe("llm_ner", config={"model": {"@llm_models": "spacy.Dolly.v1", "name": "dolly-v2-12b"}})
>
> # Construction from class
> from spacy_llm.pipeline import LLMWrapper
> llm = LLMWrapper(vocab=nlp.vocab, task=task, model=model, cache=cache, save_io=True)
> ```

### LLMWrapper.\_\_init\_\_ {id="init",tag="method"}

Create a new pipeline instance. In your application, you would normally use a
shortcut for this and instantiate the component using its string name and
[`nlp.add_pipe`](/api/language#add_pipe).

| Name           | Description                                                                                        |
| -------------- | -------------------------------------------------------------------------------------------------- |
| `name`         | String name of the component instance. `llm` by default. ~~str~~                                   |
| _keyword-only_ |                                                                                                    |
| `vocab`        | The shared vocabulary. ~~Vocab~~                                                                   |
| `task`         | An [LLM Task](#tasks) can generate prompts and parse LLM responses. ~~LLMTask~~                    |
| `model`        | The [LLM Model](#models) queries a specific LLM API.. ~~Callable[[Iterable[Any]], Iterable[Any]]~~ |
| `cache`        | [Cache](#cache) to use for caching prompts and responses per doc. ~~Cache~~                        |
| `save_io`      | Whether to save LLM I/O (prompts and responses) in the `Doc._.llm_io` custom attribute. ~~bool~~   |

### LLMWrapper.\_\_call\_\_ {id="call",tag="method"}

Apply the pipe to one document. The document is modified in place and returned.
This usually happens under the hood when the `nlp` object is called on a text
and all pipeline components are applied to the `Doc` in order.

> #### Example
>
> ```python
> doc = nlp("Ingrid visited Paris.")
> llm_ner = nlp.add_pipe("llm_ner")
> # This usually happens under the hood
> processed = llm_ner(doc)
> ```

| Name        | Description                      |
| ----------- | -------------------------------- |
| `doc`       | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~  |

### LLMWrapper.pipe {id="pipe",tag="method"}

Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
applied to the `Doc` in order.

> #### Example
>
> ```python
> llm_ner = nlp.add_pipe("llm_ner")
> for doc in llm_ner.pipe(docs, batch_size=50):
>     pass
> ```

| Name           | Description                                                   |
| -------------- | ------------------------------------------------------------- |
| `docs`         | A stream of documents. ~~Iterable[Doc]~~                      |
| _keyword-only_ |                                                               |
| `batch_size`   | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS**     | The processed documents in order. ~~Doc~~                     |

### LLMWrapper.add_label {id="add_label",tag="method"}

Add a new label to the pipe's task. Alternatively, provide the labels upon the
[task](#task) definition, or through the `[initialize]` block of the
[config](#config).

> #### Example
>
> ```python
> llm_ner = nlp.add_pipe("llm_ner")
> llm_ner.add_label("MY_LABEL")
> ```

| Name        | Description                                                 |
| ----------- | ----------------------------------------------------------- |
| `label`     | The label to add. ~~str~~                                   |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |

### LLMWrapper.to_disk {id="to_disk",tag="method"}

Serialize the pipe to disk.

> #### Example
>
> ```python
> llm_ner = nlp.add_pipe("llm_ner")
> llm_ner.to_disk("/path/to/llm_ner")
> ```

| Name           | Description                                                                                                                                |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `path`         | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| _keyword-only_ |                                                                                                                                            |
| `exclude`      | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~                                                |

### LLMWrapper.from_disk {id="from_disk",tag="method"}

Load the pipe from disk. Modifies the object in place and returns it.

> #### Example
>
> ```python
> llm_ner = nlp.add_pipe("llm_ner")
> llm_ner.from_disk("/path/to/llm_ner")
> ```

| Name           | Description                                                                                     |
| -------------- | ----------------------------------------------------------------------------------------------- |
| `path`         | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| _keyword-only_ |                                                                                                 |
| `exclude`      | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~     |
| **RETURNS**    | The modified `LLMWrapper` object. ~~LLMWrapper~~                                                |

### LLMWrapper.to_bytes {id="to_bytes",tag="method"}

> #### Example
>
> ```python
> llm_ner = nlp.add_pipe("llm_ner")
> ner_bytes = llm_ner.to_bytes()
> ```

Serialize the pipe to a bytestring.

| Name           | Description                                                                                 |
| -------------- | ------------------------------------------------------------------------------------------- |
| _keyword-only_ |                                                                                             |
| `exclude`      | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS**    | The serialized form of the `LLMWrapper` object. ~~bytes~~                                   |

### LLMWrapper.from_bytes {id="from_bytes",tag="method"}

Load the pipe from a bytestring. Modifies the object in place and returns it.

> #### Example
>
> ```python
> ner_bytes = llm_ner.to_bytes()
> llm_ner = nlp.add_pipe("llm_ner")
> llm_ner.from_bytes(ner_bytes)
> ```

| Name           | Description                                                                                 |
| -------------- | ------------------------------------------------------------------------------------------- |
| `bytes_data`   | The data to load from. ~~bytes~~                                                            |
| _keyword-only_ |                                                                                             |
| `exclude`      | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS**    | The `LLMWrapper` object. ~~LLMWrapper~~                                                     |

### LLMWrapper.labels {id="labels",tag="property"}

The labels currently added to the component. Empty tuple if the LLM's task does
not require labels.

> #### Example
>
> ```python
> llm_ner.add_label("MY_LABEL")
> assert "MY_LABEL" in llm_ner.labels
> ```

| Name        | Description                                            |
| ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |

## Tasks {id="tasks"}

In `spacy-llm`, a _task_ defines an NLP problem or question and its solution
using an LLM. It does so by implementing the following responsibilities:

1. Loading a prompt template and injecting documents' data into the prompt.
   Optionally, include fewshot examples in the prompt.
2. Splitting the prompt into several pieces following a map-reduce paradigm,
   _if_ the prompt is too long to fit into the model's context and the task
   supports sharding prompts.
3. Parsing the LLM's responses back into structured information and validating
   the parsed output.

Two different task interfaces are supported: `ShardingLLMTask` and
`NonShardingLLMTask`. Only the former supports the sharding of documents, i. e.
splitting up prompts if they are too long.

All tasks are registered in the `llm_tasks` registry.

### On Sharding {id="task-sharding"}

"Sharding" describes, generally speaking, the process of distributing parts of a
dataset across multiple storage units for easier processing and lookups. In
`spacy-llm` we use this term (synonymously: "mapping") to describe the splitting
up of prompts if they are too long for a model to handle, and "fusing"
(synonymously: "reducing") to describe how the model responses for several
shards are merged back together into a single document.

Prompts are broken up in a manner that _always_ keeps the prompt in the template
intact, meaning that the instructions to the LLM will always stay complete. The
document content however will be split, if the length of the fully rendered
prompt exceeds a model context length.

A toy example: let's assume a model has a context window of 25 tokens and the
prompt template for our fictional, sharding-supporting task looks like this:

```
Estimate the sentiment of this text:
"{text}"
Estimated sentiment:
```

Depending on how tokens are counted exactly (this is a config setting), we might
come up with `n = 12` tokens for the number of tokens in the prompt
instructions. Furthermore let's assume that our `text` is "This has been
amazing - I can't remember the last time I left the cinema so impressed." -
which has roughly 19 tokens.

Considering we only have 13 tokens to add to our prompt before we hit the
context limit, we'll have to split our prompt into two parts. Thus `spacy-llm`,
assuming the task used supports sharding, will split the prompt into two (the
default splitting strategy splits by tokens, but alternative splitting
strategies splitting e. g. by sentences can be configured):

_(Prompt 1/2)_

```
Estimate the sentiment of this text:
"This has been amazing - I can't remember "
Estimated sentiment:
```

_(Prompt 2/2)_

```
Estimate the sentiment of this text:
"the last time I left the cinema so impressed."
Estimated sentiment:
```

The reduction step is task-specific - a sentiment estimation task might e. g. do
a weighted average of the sentiment scores. Note that prompt sharding introduces
potential inaccuracies, as the LLM won't have access to the entire document at
once. Depending on your use case this might or might not be problematic.

### `NonShardingLLMTask` {id="task-nonsharding"}

#### task.generate_prompts {id="task-nonsharding-generate-prompts"}

Takes a collection of documents, and returns a collection of "prompts", which
can be of type `Any`. Often, prompts are of type `str` - but this is not
enforced to allow for maximum flexibility in the framework.

| Argument    | Description                              |
| ----------- | ---------------------------------------- |
| `docs`      | The input documents. ~~Iterable[Doc]~~   |
| **RETURNS** | The generated prompts. ~~Iterable[Any]~~ |

#### task.parse_responses {id="task-non-sharding-parse-responses"}

Takes a collection of LLM responses and the original documents, parses the
responses into structured information, and sets the annotations on the
documents. The `parse_responses` function is free to set the annotations in any
way, including `Doc` fields like `ents`, `spans` or `cats`, or using custom
defined fields.

The `responses` are of type `Iterable[Any]`, though they will often be `str`
objects. This depends on the return type of the [model](#models).

| Argument    | Description                                            |
| ----------- | ------------------------------------------------------ |
| `docs`      | The input documents. ~~Iterable[Doc]~~                 |
| `responses` | The responses received from the LLM. ~~Iterable[Any]~~ |
| **RETURNS** | The annotated documents. ~~Iterable[Doc]~~             |

### `ShardingLLMTask` {id="task-sharding"}

#### task.generate_prompts {id="task-sharding-generate-prompts"}

Takes a collection of documents, breaks them up into shards if necessary to fit
all content into the model's context, and returns a collection of collections of
"prompts" (i. e. each doc can have multiple shards, each of which have exactly
one prompt), which can be of type `Any`. Often, prompts are of type `str` - but
this is not enforced to allow for maximum flexibility in the framework.

| Argument    | Description                                        |
| ----------- | -------------------------------------------------- |
| `docs`      | The input documents. ~~Iterable[Doc]~~             |
| **RETURNS** | The generated prompts. ~~Iterable[Iterable[Any]]~~ |

#### task.parse_responses {id="task-sharding-parse-responses"}

Receives a collection of collections of LLM responses (i. e. each doc can have
multiple shards, each of which have exactly one prompt / prompt response) and
the original shards, parses the responses into structured information, sets the
annotations on the shards, and merges back doc shards into single docs. The
`parse_responses` function is free to set the annotations in any way, including
`Doc` fields like `ents`, `spans` or `cats`, or using custom defined fields.

The `responses` are of type `Iterable[Iterable[Any]]`, though they will often be
`str` objects. This depends on the return type of the [model](#models).

| Argument    | Description                                                      |
| ----------- | ---------------------------------------------------------------- |
| `shards`    | The input document shards. ~~Iterable[Iterable[Doc]]~~           |
| `responses` | The responses received from the LLM. ~~Iterable[Iterable[Any]]~~ |
| **RETURNS** | The annotated documents. ~~Iterable[Doc]~~                       |

### Translation {id="translation"}

The translation task translates texts from a defined or inferred source to a
defined target language.

#### spacy.Translation.v1 {id="translation-v1"}

`spacy.Translation.v1` supports both zero-shot and few-shot prompting.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.Translation.v1"
> examples = null
> target_lang = "Spanish"
> ```

| Argument                    | Description                                                                                                                                                                               |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template`                  | Custom prompt template to send to LLM model. Defaults to [translation.v1.jinja](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/translation.v1.jinja). ~~str~~ |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                            |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[TranslationTask]]~~                                |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `TranslationExample`. ~~Optional[Type[FewshotExample]]~~                                                                                    |
| `source_lang`               | Language to translate from. Doesn't have to be set. ~~Optional[str]~~                                                                                                                     |
| `target_lang`               | Language to translate to. No default value, has to be set. ~~str~~                                                                                                                        |
| `field`                     | Name of extension attribute to store translation in (i. e. the translation will be available in `doc._.{field}`). Defaults to `translation`. ~~str~~                                      |

To perform [few-shot learning](/usage/large-language-models#few-shot-prompts),
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader `spacy.FewShotReader.v1`
supports `.yml`, `.yaml`, `.json` and `.jsonl`.

```yaml
- text: 'Top of the morning to you!'
  translation: '¡Muy buenos días!'
- text: 'The weather is great today.'
  translation: 'El clima está fantástico hoy.'
- text: 'Do you know what will happen tomorrow?'
  translation: '¿Sabes qué pasará mañana?'
```

```ini
[components.llm.task]
@llm_tasks = "spacy.Translation.v1"
target_lang = "Spanish"
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "translation_examples.yml"
```

### Raw prompting {id="raw"}

Different to all other tasks `spacy.Raw.vX` doesn't provide a specific prompt,
wrapping doc data, to the model. Instead it instructs the model to reply to the
doc content. This is handy for use cases like question answering (where each doc
contains one question) or if you want to include customized prompts for each
doc.

#### spacy.Raw.v1 {id="raw-v1"}

Note that since this task may request arbitrary information, it doesn't do any
parsing per se - the model response is stored in a custom `Doc` attribute (i. e.
can be accessed via `doc._.{field}`).

It supports both zero-shot and few-shot prompting.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.Raw.v1"
> examples = null
> ```

| Argument              | Description                                                                                                                                                               |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template`            | Custom prompt template to send to LLM model. Defaults to [raw.v1.jinja](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/raw.v1.jinja). ~~str~~ |
| `examples`            | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                            |
| `parse_responses`     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[RawTask]]~~                        |
| `prompt_example_type` | Type to use for fewshot examples. Defaults to `RawExample`. ~~Optional[Type[FewshotExample]]~~                                                                            |
| `field`               | Name of extension attribute to store model reply in (i. e. the reply will be available in `doc._.{field}`). Defaults to `reply`. ~~str~~                                  |

To perform [few-shot learning](/usage/large-language-models#few-shot-prompts),
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader `spacy.FewShotReader.v1`
supports `.yml`, `.yaml`, `.json` and `.jsonl`.

```yaml
# Each example can follow an arbitrary pattern. It might help the prompt performance though if the examples resemble
# the actual docs' content.
- text: "3 + 5 = x. What's x?"
  reply: '8'

- text: 'Write me a limerick.'
  reply:
    "There was an Old Man with a beard, Who said, 'It is just as I feared! Two
    Owls and a Hen, Four Larks and a Wren, Have all built their nests in my
    beard!"

- text: "Analyse the sentiment of the text 'This is great'."
  reply: "'This is great' expresses a very positive sentiment."
```

```ini
[components.llm.task]
@llm_tasks = "spacy.Raw.v1"
field = "llm_reply"
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "raw_examples.yml"
```

### Summarization {id="summarization"}

A summarization task takes a document as input and generates a summary that is
stored in an extension attribute.

#### spacy.Summarization.v1 {id="summarization-v1"}

The `spacy.Summarization.v1` task supports both zero-shot and few-shot
prompting.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.Summarization.v1"
> examples = null
> max_n_words = null
> ```

| Argument                    | Description                                                                                                                                                                                   |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template`                  | Custom prompt template to send to LLM model. Defaults to [summarization.v1.jinja](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/summarization.v1.jinja). ~~str~~ |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                                |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[SummarizationTask]]~~                                  |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `SummarizationExample`. ~~Optional[Type[FewshotExample]]~~                                                                                      |
| `max_n_words`               | Maximum number of words to be used in summary. Note that this should not expected to work exactly. Defaults to `None`. ~~Optional[int]~~                                                      |
| `field`                     | Name of extension attribute to store summary in (i. e. the summary will be available in `doc._.{field}`). Defaults to `summary`. ~~str~~                                                      |

The summarization task prompts the model for a concise summary of the provided
text. It optionally allows to limit the response to a certain number of tokens -
note that this requirement will be included in the prompt, but the task doesn't
perform a hard cut-off. It's hence possible that your summary exceeds
`max_n_words`.

To perform [few-shot learning](/usage/large-language-models#few-shot-prompts),
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader `spacy.FewShotReader.v1`
supports `.yml`, `.yaml`, `.json` and `.jsonl`.

```yaml
- text: >
    The United Nations, referred to informally as the UN, is an
    intergovernmental organization whose stated purposes are  to maintain
    international peace and security, develop friendly relations among nations,
    achieve international cooperation, and serve as a centre for harmonizing the
    actions of nations. It is the world's largest international organization.
    The UN is headquartered on international territory in New York City, and the
    organization has other offices in Geneva, Nairobi, Vienna, and The Hague,
    where the International Court of Justice is headquartered.\n\n The UN was
    established after World War II with the aim of preventing future world wars,
    and succeeded the League of  Nations, which was characterized as
    ineffective.
  summary:
    'The UN is an international organization that promotes global peace,
    cooperation, and harmony. Established after WWII, its purpose is to prevent
    future world wars.'
```

```ini
[components.llm.task]
@llm_tasks = "spacy.Summarization.v1"
max_n_words = 20
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "summarization_examples.yml"
```

### EL (Entity Linking) {id="nel"}

The EL links recognized entities (see [NER](#ner)) to those in a knowledge base
(KB). The EL task prompts the LLM to select the most likely candidate from the
KB, whose structure can be arbitrary.

Note that the documents processed by the entity linking task are expected to
have recognized entities in their `.ents` attribute. This can be achieved by
either running the [NER task](#ner), using a trained spaCy NER model or setting
the entities manually prior to running the EL task.

In order to be able to pull data from the KB, an object implementing the
`CandidateSelector` protocol has to be provided. This requires two functions:
(1) `__call__()` to fetch candidate entities for entity mentions in the text
(assumed to be available in `Doc.ents`) and (2) `get_entity_description()` to
fetch descriptions for any given entity ID. Descriptions can be empty, but
ideally provide more context for entities stored in the KB.

`spacy-llm` provides a `CandidateSelector` implementation
(`spacy.CandidateSelector.v1`) that leverages a spaCy knowledge base - as used
in an `entity_linking` component - to select candidates. This knowledge base can
be loaded from an existing spaCy pipeline (note that the pipeline's EL component
doesn't have to be trained) or from a separate .yaml file.

#### spacy.EntityLinker.v1 {id="el-v1"}

Supports zero- and few-shot prompting. Relies on a configurable component
suggesting viable entities before letting the LLM pick the most likely
candidate.

> #### Example config for spacy.EntityLinker.v1
>
> ```ini
> [paths]
> el_nlp = null
>
> ...
>
> [components.llm.task]
> @llm_tasks = "spacy.EntityLinker.v1"
>
> [initialize]
> [initialize.components]
> [initialize.components.llm]
> [initialize.components.llm.candidate_selector]
> @llm_misc = "spacy.CandidateSelector.v1"
>
> # Load a KB from a KB file. For loading KBs from spaCy pipelines see spacy.KBObjectLoader.v1.
> [initialize.components.llm.candidate_selector.kb_loader]
> @llm_misc = "spacy.KBFileLoader.v1"
> # Path to knowledge base .yaml file.
> path = ${paths.el_kb}
> ```

| Argument              | Description                                                                                                                                                                                   |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template`            | Custom prompt template to send to LLM model. Defaults to [entity_linker.v1.jinja](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/entity_linker.v1.jinja). ~~str~~ |
| `parse_responses`     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[EntityLinkerTask]]~~                                   |
| `prompt_example_type` | Type to use for fewshot examples. Defaults to `ELExample`. ~~Optional[Type[FewshotExample]]~~                                                                                                 |
| `examples`            | Optional callable that reads a file containing task examples for few-shot learning. If `None` is passed, zero-shot learning will be used. Defaults to `None`. ~~ExamplesConfigType~~          |
| `scorer`              | Scorer function. Defaults to the metric used by spaCy to evaluate entity linking performance. ~~Optional[Scorer]~~                                                                            |

##### spacy.CandidateSelector.v1 {id="candidate-selector-v1"}

`spacy.CandidateSelector.v1` is an implementation of the `CandidateSelector`
protocol required by [`spacy.EntityLinker.v1`](#el-v1). The built-in candidate
selector method allows loading existing knowledge bases in several ways, e. g.
loading from a spaCy pipeline with a (not necessarily trained) entity linking
component, and loading from a file describing the knowlege base as a .yaml file.
Either way the loaded data will be converted to a spaCy `InMemoryLookupKB`
instance. The KB's selection capabilities are used to select the most likely
entity candidates for the specified mentions.

> #### Example config for spacy.CandidateSelector.v1
>
> ```ini
> [initialize]
> [initialize.components]
> [initialize.components.llm]
> [initialize.components.llm.candidate_selector]
> @llm_misc = "spacy.CandidateSelector.v1"
>
> # Load a KB from a KB file. For loading KBs from spaCy pipelines see spacy.KBObjectLoader.v1.
> [initialize.components.llm.candidate_selector.kb_loader]
> @llm_misc = "spacy.KBFileLoader.v1"
> # Path to knowledge base .yaml file.
> path = ${paths.el_kb}
> ```

| Argument    | Description                                                       |
| ----------- | ----------------------------------------------------------------- |
| `kb_loader` | KB loader object. ~~InMemoryLookupKBLoader~~                      |
| `top_n`     | Top-n candidates to include in the prompt. Defaults to 5. ~~int~~ |

##### spacy.KBObjectLoader.v1 {id="kb-object-loader-v1"}

Adheres to the `InMemoryLookupKBLoader` interface required by
[`spacy.CandidateSelector.v1`](#candidate-selector-v1). Loads a knowledge base
from an existing spaCy pipeline.

> #### Example config for spacy.KBObjectLoader.v1
>
> ```ini
> [initialize.components.llm.candidate_selector.kb_loader]
> @llm_misc = "spacy.KBObjectLoader.v1"
> # Path to knowledge base directory in serialized spaCy pipeline.
> path = ${paths.el_kb}
> # Path to spaCy pipeline. If this is not specified, spacy-llm tries to determine this automatically (but may fail).
> nlp_path = ${paths.el_nlp}
> # Path to file with descriptions for entity.
> desc_path = ${paths.el_desc}
> ```

| Argument          | Description                                                                                                                                                                                                                         |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `path`            | Path to KB file. ~~Union[str, Path]~~                                                                                                                                                                                               |
| `nlp_path`        | Path to serialized NLP pipeline. If None, path will be guessed. ~~Optional[Union[Path, str]]~~                                                                                                                                      |
| `desc_path`       | Path to file with descriptions for entities. ~~int~~                                                                                                                                                                                |
| `ent_desc_reader` | Entity description reader. Defaults to an internal method expecting a CSV file without header row, with ";" as delimiters, and with two columns - one for the entitys' IDs, one for their descriptions. ~~Optional[EntDescReader]~~ |

##### spacy.KBFileLoader.v1 {id="kb-file-loader-v1"}

Adheres to the `InMemoryLookupKBLoader` interface required by
[`spacy.CandidateSelector.v1`](#candidate-selector-v1). Loads a knowledge base
from a knowledge base file. The KB .yaml file has to stick to the following
format:

```yaml
entities:
  # The key should be whatever ID identifies this entity uniquely in your knowledge base.
  ID1:
      name: "..."
      desc: "..."
  ID2:
      ...
# Data on aliases in your knowledge base - e. g. "Apple" for the entity "Apple Inc.".
aliases:
  - alias: "..."
    # List of all entities that this alias refers to.
    entities: ["ID1", "ID2", ...]
    # Optional: prior probabilities that this alias refers to the n-th entity in the "entities" attribute.
    probabilities: [0.5, 0.2, ...]
  - alias: "..."
    entities: [...]
    probabilities: [...]
  ...
```

See
[here](https://github.com/explosion/spacy-llm/blob/main/usage_examples/el_openai/el_kb_data.yml)
for a toy example of how such a KB file might look like.

> #### Example config for spacy.KBFileLoader.v1
>
> ```ini
> [initialize.components.llm.candidate_selector.kb_loader]
> @llm_misc = "spacy.KBFileLoader.v1"
> # Path to knowledge base file.
> path = ${paths.el_kb}
> ```

| Argument | Description                           |
| -------- | ------------------------------------- |
| `path`   | Path to KB file. ~~Union[str, Path]~~ |

### NER {id="ner"}

The NER task identifies non-overlapping entities in text.

#### spacy.NER.v3 {id="ner-v3"}

Version 3 is fundamentally different to v1 and v2, as it implements
Chain-of-Thought prompting, based on the
[PromptNER paper](https://arxiv.org/pdf/2305.15444.pdf) by Ashok and Lipton
(2023). On an internal use-case, we have found this implementation to obtain
significant better accuracy - with an increase of F-score of up to 15 percentage
points.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.NER.v3"
> labels = ["PERSON", "ORGANISATION", "LOCATION"]
> ```

When no examples are [specified](/usage/large-language-models#few-shot-prompts),
the v3 implementation will use a dummy example in the prompt. Technically this
means that the task will always perform few-shot prompting under the hood.

| Argument                    | Description                                                                                                                                                                                            |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `template`                  | Custom prompt template to send to LLM model. Defaults to [ner.v3.jinja](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/ner.v3.jinja). ~~str~~                              |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                                         |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[NERTask]]~~                                                     |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `NERExample`. ~~Optional[Type[FewshotExample]]~~                                                                                                         |
| `scorer`                    | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                                                                   |
| `labels`                    | List of labels or str of comma-separated list of labels. ~~Union[List[str], str]~~                                                                                                                     |
| `label_definitions`         | Optional dict mapping a label to a description of that label. These descriptions are added to the prompt to help instruct the LLM on what to extract. Defaults to `None`. ~~Optional[Dict[str, str]]~~ |
| `description` (NEW)         | A description of what to recognize or not recognize as entities. ~~str~~                                                                                                                               |
| `normalizer`                | Function that normalizes the labels as returned by the LLM. If `None`, defaults to `spacy.LowercaseNormalizer.v1`. Defaults to `None`. ~~Optional[Callable[[str], str]]~~                              |
| `alignment_mode`            | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are `"strict"`, `"contract"` or `"expand"`. Defaults to `"contract"`. ~~str~~                         |
| `case_sensitive_matching`   | Whether to search without case sensitivity. Defaults to `False`. ~~bool~~                                                                                                                              |

Note that the `single_match` parameter, used in v1 and v2, is not supported
anymore, as the CoT parsing algorithm takes care of this automatically.

New to v3 is the fact that you can provide an explicit description of what
entities should look like. You can use this feature in addition to
`label_definitions`.

```ini
[components.llm.task]
@llm_tasks = "spacy.NER.v3"
labels = ["DISH", "INGREDIENT", "EQUIPMENT"]
description = Entities are the names food dishes,
    ingredients, and any kind of cooking equipment.
    Adjectives, verbs, adverbs are not entities.
    Pronouns are not entities.

[components.llm.task.label_definitions]
DISH = "Known food dishes, e.g. Lobster Ravioli, garlic bread"
INGREDIENT = "Individual parts of a food dish, including herbs and spices."
EQUIPMENT = "Any kind of cooking equipment. e.g. oven, cooking pot, grill"
```

To perform [few-shot learning](/usage/large-language-models#few-shot-prompts),
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader `spacy.FewShotReader.v1`
supports `.yml`, `.yaml`, `.json` and `.jsonl`.

While not required, this task works best when both positive and negative
examples are provided. The format is different than the files required for v1
and v2, as additional fields such as `is_entity` and `reason` should now be
provided.

```json
[
  {
    "text": "You can't get a great chocolate flavor with carob.",
    "spans": [
      {
        "text": "chocolate",
        "is_entity": false,
        "label": "==NONE==",
        "reason": "is a flavor in this context, not an ingredient"
      },
      {
        "text": "carob",
        "is_entity": true,
        "label": "INGREDIENT",
        "reason": "is an ingredient to add chocolate flavor"
      }
    ]
  },
  ...
]
```

```ini
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "${paths.examples}"
```

For a fully working example, see this
[usage example](https://github.com/explosion/spacy-llm/tree/main/usage_examples/ner_v3_openai).

#### spacy.NER.v2 {id="ner-v2"}

This version supports explicitly defining the provided labels with custom
descriptions, and further supports zero-shot and few-shot prompting just like
v1.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.NER.v2"
> labels = ["PERSON", "ORGANISATION", "LOCATION"]
> examples = null
> ```

| Argument                    | Description                                                                                                                                                                                            |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `template` (NEW)            | Custom prompt template to send to LLM model. Defaults to [ner.v2.jinja](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/ner.v2.jinja). ~~str~~                              |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                                         |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[NERTask]]~~                                                     |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `NERExample`. ~~Optional[Type[FewshotExample]]~~                                                                                                         |
| `scorer` (NEW)              | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                                                                   |
| `labels`                    | List of labels or str of comma-separated list of labels. ~~Union[List[str], str]~~                                                                                                                     |
| `label_definitions` (NEW)   | Optional dict mapping a label to a description of that label. These descriptions are added to the prompt to help instruct the LLM on what to extract. Defaults to `None`. ~~Optional[Dict[str, str]]~~ |
| `normalizer`                | Function that normalizes the labels as returned by the LLM. If `None`, defaults to `spacy.LowercaseNormalizer.v1`. Defaults to `None`. ~~Optional[Callable[[str], str]]~~                              |
| `alignment_mode`            | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are `"strict"`, `"contract"` or `"expand"`. Defaults to `"contract"`. ~~str~~                         |
| `case_sensitive_matching`   | Whether to search without case sensitivity. Defaults to `False`. ~~bool~~                                                                                                                              |
| `single_match`              | Whether to match an entity in the LLM's response only once (the first hit) or multiple times. Defaults to `False`. ~~bool~~                                                                            |

The parameters `alignment_mode`, `case_sensitive_matching` and `single_match`
are identical to the [v1](#ner-v1) implementation. The format of few-shot
examples are also the same.

> Label descriptions can also be used with explicit examples to give as much
> info to the LLM model as possible.

New to v2 is the fact that you can write definitions for each label and provide
them via the `label_definitions` argument. This lets you tell the LLM exactly
what you're looking for rather than relying on the LLM to interpret its task
given just the label name. Label descriptions are freeform so you can write
whatever you want here, but a brief description along with some examples and
counter examples seems to work quite well.

```ini
[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = PERSON,SPORTS_TEAM

[components.llm.task.label_definitions]
PERSON = "Extract any named individual in the text."
SPORTS_TEAM = "Extract the names of any professional sports team. e.g. Golden State Warriors, LA Lakers, Man City, Real Madrid"
```

For a fully working example, see this
[usage example](https://github.com/explosion/spacy-llm/tree/main/usage_examples/ner_dolly).

#### spacy.NER.v1 {id="ner-v1"}

The original version of the built-in NER task supports both zero-shot and
few-shot prompting.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.NER.v1"
> labels = PERSON,ORGANISATION,LOCATION
> examples = null
> ```

| Argument                    | Description                                                                                                                                                                    |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                 |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[NERTask]]~~                             |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `NERExample`. ~~Optional[Type[FewshotExample]]~~                                                                                 |
| `scorer` (NEW)              | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                                           |
| `labels`                    | Comma-separated list of labels. ~~str~~                                                                                                                                        |
| `normalizer`                | Function that normalizes the labels as returned by the LLM. If `None`, defaults to `spacy.LowercaseNormalizer.v1`. ~~Optional[Callable[[str], str]]~~                          |
| `alignment_mode`            | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are `"strict"`, `"contract"` or `"expand"`. Defaults to `"contract"`. ~~str~~ |
| `case_sensitive_matching`   | Whether to search without case sensitivity. Defaults to `False`. ~~bool~~                                                                                                      |
| `single_match`              | Whether to match an entity in the LLM's response only once (the first hit) or multiple times. Defaults to `False`. ~~bool~~                                                    |

The NER task implementation doesn't currently ask the LLM for specific offsets,
but simply expects a list of strings that represent the enties in the document.
This means that a form of string matching is required. This can be configured by
the following parameters:

- The `single_match` parameter is typically set to `False` to allow for multiple
  matches. For instance, the response from the LLM might only mention the entity
  "Paris" once, but you'd still want to mark it every time it occurs in the
  document.
- The case-sensitive matching is typically set to `False` to be robust against
  case variances in the LLM's output.
- The `alignment_mode` argument is used to match entities as returned by the LLM
  to the tokens from the original `Doc` - specifically it's used as argument in
  the call to [`doc.char_span()`](/api/doc#char_span). The `"strict"` mode will
  only keep spans that strictly adhere to the given token boundaries.
  `"contract"` will only keep those tokens that are fully within the given
  range, e.g. reducing `"New Y"` to `"New"`. Finally, `"expand"` will expand the
  span to the next token boundaries, e.g. expanding `"New Y"` out to
  `"New York"`.

To perform [few-shot learning](/usage/large-language-models#few-shot-prompts),
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader `spacy.FewShotReader.v1`
supports `.yml`, `.yaml`, `.json` and `.jsonl`.

```yaml
- text: Jack and Jill went up the hill.
  entities:
    PERSON:
      - Jack
      - Jill
    LOCATION:
      - hill
- text: Jack fell down and broke his crown.
  entities:
    PERSON:
      - Jack
```

```ini
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "ner_examples.yml"
```

### SpanCat {id="spancat"}

The SpanCat task identifies potentially overlapping entities in text.

#### spacy.SpanCat.v3 {id="spancat-v3"}

The built-in SpanCat v3 task is a simple adaptation of the NER v3 task to
support overlapping entities and store its annotations in `doc.spans`.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.SpanCat.v3"
> labels = ["PERSON", "ORGANISATION", "LOCATION"]
> examples = null
> ```

| Argument                    | Description                                                                                                                                                                                            |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `template`                  | Custom prompt template to send to LLM model. Defaults to [`spancat.v3.jinja`](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/spancat.v3.jinja). ~~str~~                    |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                                         |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[SpanCatTask]]~~                                                 |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `SpanCatExample`. ~~Optional[Type[FewshotExample]]~~                                                                                                     |
| `scorer` (NEW)              | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                                                                   |
| `labels`                    | List of labels or str of comma-separated list of labels. ~~Union[List[str], str]~~                                                                                                                     |
| `label_definitions`         | Optional dict mapping a label to a description of that label. These descriptions are added to the prompt to help instruct the LLM on what to extract. Defaults to `None`. ~~Optional[Dict[str, str]]~~ |
| `description` (NEW)         | A description of what to recognize or not recognize as entities. ~~str~~                                                                                                                               |
| `spans_key`                 | Key of the `Doc.spans` dict to save the spans under. Defaults to `"sc"`. ~~str~~                                                                                                                       |
| `normalizer`                | Function that normalizes the labels as returned by the LLM. If `None`, defaults to `spacy.LowercaseNormalizer.v1`. ~~Optional[Callable[[str], str]]~~                                                  |
| `alignment_mode`            | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are `"strict"`, `"contract"` or `"expand"`. Defaults to `"contract"`. ~~str~~                         |
| `case_sensitive_matching`   | Whether to search without case sensitivity. Defaults to `False`. ~~bool~~                                                                                                                              |

Note that the `single_match` parameter, used in v1 and v2, is not supported
anymore, as the CoT parsing algorithm takes care of this automatically.

#### spacy.SpanCat.v2 {id="spancat-v2"}

The built-in SpanCat v2 task is a simple adaptation of the NER v2 task to
support overlapping entities and store its annotations in `doc.spans`.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.SpanCat.v2"
> labels = ["PERSON", "ORGANISATION", "LOCATION"]
> examples = null
> ```

| Argument                    | Description                                                                                                                                                                                            |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `template` (NEW)            | Custom prompt template to send to LLM model. Defaults to [`spancat.v2.jinja`](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/spancat.v2.jinja). ~~str~~                    |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                                         |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[SpanCatTask]]~~                                                 |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `SpanCatExample`. ~~Optional[Type[FewshotExample]]~~                                                                                                     |
| `scorer` (NEW)              | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                                                                   |
| `labels`                    | List of labels or str of comma-separated list of labels. ~~Union[List[str], str]~~                                                                                                                     |
| `label_definitions` (NEW)   | Optional dict mapping a label to a description of that label. These descriptions are added to the prompt to help instruct the LLM on what to extract. Defaults to `None`. ~~Optional[Dict[str, str]]~~ |
| `spans_key`                 | Key of the `Doc.spans` dict to save the spans under. Defaults to `"sc"`. ~~str~~                                                                                                                       |
| `normalizer`                | Function that normalizes the labels as returned by the LLM. If `None`, defaults to `spacy.LowercaseNormalizer.v1`. ~~Optional[Callable[[str], str]]~~                                                  |
| `alignment_mode`            | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are `"strict"`, `"contract"` or `"expand"`. Defaults to `"contract"`. ~~str~~                         |
| `case_sensitive_matching`   | Whether to search without case sensitivity. Defaults to `False`. ~~bool~~                                                                                                                              |
| `single_match`              | Whether to match an entity in the LLM's response only once (the first hit) or multiple times. Defaults to `False`. ~~bool~~                                                                            |

Except for the `spans_key` parameter, the SpanCat v2 task reuses the
configuration from the NER v2 task. Refer to [its documentation](#ner-v2) for
more insight.

#### spacy.SpanCat.v1 {id="spancat-v1"}

The original version of the built-in SpanCat task is a simple adaptation of the
v1 NER task to support overlapping entities and store its annotations in
`doc.spans`.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.SpanCat.v1"
> labels = PERSON,ORGANISATION,LOCATION
> examples = null
> ```

| Argument                    | Description                                                                                                                                                                    |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                 |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[SpanCatTask]]~~                         |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `SpanCatExample`. ~~Optional[Type[FewshotExample]]~~                                                                             |
| `scorer` (NEW)              | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                                           |
| `labels`                    | Comma-separated list of labels. ~~str~~                                                                                                                                        |
| `spans_key`                 | Key of the `Doc.spans` dict to save the spans under. Defaults to `"sc"`. ~~str~~                                                                                               |
| `normalizer`                | Function that normalizes the labels as returned by the LLM. If `None`, defaults to `spacy.LowercaseNormalizer.v1`. ~~Optional[Callable[[str], str]]~~                          |
| `alignment_mode`            | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are `"strict"`, `"contract"` or `"expand"`. Defaults to `"contract"`. ~~str~~ |
| `case_sensitive_matching`   | Whether to search without case sensitivity. Defaults to `False`. ~~bool~~                                                                                                      |
| `single_match`              | Whether to match an entity in the LLM's response only once (the first hit) or multiple times. Defaults to `False`. ~~bool~~                                                    |

Except for the `spans_key` parameter, the SpanCat v1 task reuses the
configuration from the NER v1 task. Refer to [its documentation](#ner-v1) for
more insight.

### TextCat {id="textcat"}

The TextCat task labels documents with relevant categories.

#### spacy.TextCat.v3 {id="textcat-v3"}

On top of the functionality from v2, version 3 of the built-in TextCat tasks
allows setting definitions of labels. Those definitions are included in the
prompt.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.TextCat.v3"
> labels = ["COMPLIMENT", "INSULT"]
>
> [components.llm.task.label_definitions]
> "COMPLIMENT" = "a polite expression of praise or admiration.",
> "INSULT" = "a disrespectful or scornfully abusive remark or act."
> examples = null
> ```

| Argument                    | Description                                                                                                                                                                         |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template`                  | Custom prompt template to send to LLM model. Defaults to [`textcat.v3.jinja`](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/textcat.v3.jinja). ~~str~~ |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                      |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[SpanCatTask]]~~                              |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `TextCatExample`. ~~Optional[Type[FewshotExample]]~~                                                                                  |
| `scorer` (NEW)              | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                                                |
| `labels`                    | List of labels or str of comma-separated list of labels. ~~Union[List[str], str]~~                                                                                                  |
| `label_definitions` (NEW)   | Dictionary of label definitions. Included in the prompt, if set. Defaults to `None`. ~~Optional[Dict[str, str]]~~                                                                   |
| `normalizer`                | Function that normalizes the labels as returned by the LLM. If `None`, falls back to `spacy.LowercaseNormalizer.v1`. Defaults to `None`. ~~Optional[Callable[[str], str]]~~         |
| `exclusive_classes`         | If set to `True`, only one label per document should be valid. If set to `False`, one document can have multiple labels. Defaults to `False`. ~~bool~~                              |
| `allow_none`                | When set to `True`, allows the LLM to not return any of the given label. The resulting dict in `doc.cats` will have `0.0` scores for all labels. Defaults to `True`. ~~bool~~       |
| `verbose`                   | If set to `True`, warnings will be generated when the LLM returns invalid responses. Defaults to `False`. ~~bool~~                                                                  |

The formatting of few-shot examples is the same as those for the
[v1](#textcat-v1) implementation.

#### spacy.TextCat.v2 {id="textcat-v2"}

V2 includes all v1 functionality, with an improved prompt template.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.TextCat.v2"
> labels = ["COMPLIMENT", "INSULT"]
> examples = null
> ```

| Argument                    | Description                                                                                                                                                                         |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template` (NEW)            | Custom prompt template to send to LLM model. Defaults to [`textcat.v2.jinja`](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/textcat.v2.jinja). ~~str~~ |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                      |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[SpanCatTask]]~~                              |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `TextCatExample`. ~~Optional[Type[FewshotExample]]~~                                                                                  |
| `scorer` (NEW)              | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                                                |
| `labels`                    | List of labels or str of comma-separated list of labels. ~~Union[List[str], str]~~                                                                                                  |
| `normalizer`                | Function that normalizes the labels as returned by the LLM. If `None`, falls back to `spacy.LowercaseNormalizer.v1`. ~~Optional[Callable[[str], str]]~~                             |
| `exclusive_classes`         | If set to `True`, only one label per document should be valid. If set to `False`, one document can have multiple labels. Defaults to `False`. ~~bool~~                              |
| `allow_none`                | When set to `True`, allows the LLM to not return any of the given label. The resulting dict in `doc.cats` will have `0.0` scores for all labels. Defaults to `True`. ~~bool~~       |
| `verbose`                   | If set to `True`, warnings will be generated when the LLM returns invalid responses. Defaults to `False`. ~~bool~~                                                                  |

The formatting of few-shot examples is the same as those for the
[v1](#textcat-v1) implementation.

#### spacy.TextCat.v1 {id="textcat-v1"}

Version 1 of the built-in TextCat task supports both zero-shot and few-shot
prompting.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.TextCat.v1"
> labels = COMPLIMENT,INSULT
> examples = null
> ```

| Argument                    | Description                                                                                                                                                                   |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `examples`                  | Optional function that generates examples for few-shot learning. Deafults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[SpanCatTask]]~~                        |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `TextCatExample`. ~~Optional[Type[FewshotExample]]~~                                                                            |
| `scorer` (NEW)              | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                                          |
| `labels`                    | Comma-separated list of labels. ~~str~~                                                                                                                                       |
| `normalizer`                | Function that normalizes the labels as returned by the LLM. If `None`, falls back to `spacy.LowercaseNormalizer.v1`. ~~Optional[Callable[[str], str]]~~                       |
| `exclusive_classes`         | If set to `True`, only one label per document should be valid. If set to `False`, one document can have multiple labels. Defaults to `False`. ~~bool~~                        |
| `allow_none`                | When set to `True`, allows the LLM to not return any of the given label. The resulting dict in `doc.cats` will have `0.0` scores for all labels. Defaults to `True`. ~~bool~~ |
| `verbose`                   | If set to `True`, warnings will be generated when the LLM returns invalid responses. Defaults to `False`. ~~bool~~                                                            |

To perform [few-shot learning](/usage/large-language-models#few-shot-prompts),
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader `spacy.FewShotReader.v1`
supports `.yml`, `.yaml`, `.json` and `.jsonl`.

```json
[
  {
    "text": "You look great!",
    "answer": "Compliment"
  },
  {
    "text": "You are not very clever at all.",
    "answer": "Insult"
  }
]
```

```ini
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "textcat_examples.json"
```

If you want to perform few-shot learning with a binary classifier (i. e. a text
either should or should not be assigned to a given class), you can provide
positive and negative examples with answers of "POS" or "NEG". "POS" means that
this example should be assigned the class label defined in the configuration,
"NEG" means it shouldn't. E. g. for spam classification:

```json
[
  {
    "text": "You won the lottery! Wire a fee of 200$ to be able to withdraw your winnings.",
    "answer": "POS"
  },
  {
    "text": "Your order #123456789 has arrived",
    "answer": "NEG"
  }
]
```

### REL {id="rel"}

The REL task extracts relations between named entities.

#### spacy.REL.v1 {id="rel-v1"}

The built-in REL task supports both zero-shot and few-shot prompting. It relies
on an upstream NER component for entities extraction.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.REL.v1"
> labels = ["LivesIn", "Visits"]
> ```

| Argument                    | Description                                                                                                                                                                 |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template`                  | Custom prompt template to send to LLM model. Defaults to [`rel.v3.jinja`](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/rel.v1.jinja). ~~str~~ |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                              |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[RELTask]]~~                          |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `RELExample`. ~~Optional[Type[FewshotExample]]~~                                                                              |
| `scorer` (NEW)              | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                                        |
| `labels`                    | List of labels or str of comma-separated list of labels. ~~Union[List[str], str]~~                                                                                          |
| `label_definitions`         | Dictionary providing a description for each relation label. Defaults to `None`. ~~Optional[Dict[str, str]]~~                                                                |
| `normalizer`                | Function that normalizes the labels as returned by the LLM. If `None`, falls back to `spacy.LowercaseNormalizer.v1`. Defaults to `None`. ~~Optional[Callable[[str], str]]~~ |
| `verbose`                   | If set to `True`, warnings will be generated when the LLM returns invalid responses. Defaults to `False`. ~~bool~~                                                          |

To perform [few-shot learning](/usage/large-language-models#few-shot-prompts),
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader `spacy.FewShotReader.v1`
supports `.yml`, `.yaml`, `.json` and `.jsonl`.

```json
{"text": "Laura bought a house in Boston with her husband Mark.", "ents": [{"start_char": 0, "end_char": 5, "label": "PERSON"}, {"start_char": 24, "end_char": 30, "label": "GPE"}, {"start_char": 48, "end_char": 52, "label": "PERSON"}], "relations": [{"dep": 0, "dest": 1, "relation": "LivesIn"}, {"dep": 2, "dest": 1, "relation": "LivesIn"}]}
{"text": "Michael travelled through South America by bike.", "ents": [{"start_char": 0, "end_char": 7, "label": "PERSON"}, {"start_char": 26, "end_char": 39, "label": "LOC"}], "relations": [{"dep": 0, "dest": 1, "relation": "Visits"}]}
```

```ini
[components.llm.task]
@llm_tasks = "spacy.REL.v1"
labels = ["LivesIn", "Visits"]

[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "rel_examples.jsonl"
```

Note: the REL task relies on pre-extracted entities to make its prediction.
Hence, you'll need to add a component that populates `doc.ents` with recognized
spans to your spaCy pipeline and put it _before_ the REL component.

For a fully working example, see this
[usage example](https://github.com/explosion/spacy-llm/tree/main/usage_examples/rel_openai).

### Lemma {id="lemma"}

The Lemma task lemmatizes the provided text and updates the `lemma_` attribute
in the doc's tokens accordingly.

#### spacy.Lemma.v1 {id="lemma-v1"}

This task supports both zero-shot and few-shot prompting.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.Lemma.v1"
> examples = null
> ```

| Argument                    | Description                                                                                                                                                                   |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template`                  | Custom prompt template to send to LLM model. Defaults to [lemma.v1.jinja](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/templates/lemma.v1.jinja). ~~str~~ |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                                                |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[LemmaTask]]~~                          |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `LemmaExample`. ~~Optional[Type[FewshotExample]]~~                                                                              |
| `scorer` (NEW)              | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                                          |

The task prompts the LLM to lemmatize the passed text and return the lemmatized
version as a list of tokens and their corresponding lemma. E. g. the text
`I'm buying ice cream for my friends` should invoke the response

```
I: I
'm: be
buying: buy
ice: ice
cream: cream
for: for
my: my
friends: friend
.: .
```

If for any given text/doc instance the number of lemmas returned by the LLM
doesn't match the number of tokens from the pipeline's tokenizer, no lemmas are
stored in the corresponding doc's tokens. Otherwise the tokens `.lemma_`
property is updated with the lemma suggested by the LLM.

To perform [few-shot learning](/usage/large-language-models#few-shot-prompts),
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader `spacy.FewShotReader.v1`
supports `.yml`, `.yaml`, `.json` and `.jsonl`.

```yaml
- text: I'm buying ice cream.
  lemmas:
    - 'I': 'I'
    - "'m": 'be'
    - 'buying': 'buy'
    - 'ice': 'ice'
    - 'cream': 'cream'
    - '.': '.'

- text: I've watered the plants.
  lemmas:
    - 'I': 'I'
    - "'ve": 'have'
    - 'watered': 'water'
    - 'the': 'the'
    - 'plants': 'plant'
    - '.': '.'
```

```ini
[components.llm.task]
@llm_tasks = "spacy.Lemma.v1"
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "lemma_examples.yml"
```

### Sentiment {id="sentiment"}

Performs sentiment analysis on provided texts. Scores between 0 and 1 are stored
in `Doc._.sentiment` - the higher, the more positive. Note in cases of parsing
issues (e. g. in case of unexpected LLM responses) the value might be `None`.

#### spacy.Sentiment.v1 {id="sentiment-v1"}

This task supports both zero-shot and few-shot prompting.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.Sentiment.v1"
> examples = null
> ```

| Argument                    | Description                                                                                                                                              |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template`                  | Custom prompt template to send to LLM model. Defaults to [sentiment.v1.jinja](./spacy_llm/tasks/templates/sentiment.v1.jinja). ~~str~~                   |
| `examples`                  | Optional function that generates examples for few-shot learning. Defaults to `None`. ~~Optional[Callable[[], Iterable[Any]]]~~                           |
| `parse_responses` (NEW)     | Callable for parsing LLM responses for this task. Defaults to the internal parsing method for this task. ~~Optional[TaskResponseParser[SentimentTask]]~~ |
| `prompt_example_type` (NEW) | Type to use for fewshot examples. Defaults to `SentimentExample`. ~~Optional[Type[FewshotExample]]~~                                                     |
| `scorer` (NEW)              | Scorer function that evaluates the task performance on provided examples. Defaults to the metric used by spaCy. ~~Optional[Scorer]~~                     |
| `field`                     | Name of extension attribute to store summary in (i. e. the summary will be available in `doc._.{field}`). Defaults to `sentiment`. ~~str~~               |

To perform [few-shot learning](/usage/large-language-models#few-shot-prompts),
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader `spacy.FewShotReader.v1`
supports `.yml`, `.yaml`, `.json` and `.jsonl`.

```yaml
- text: 'This is horrifying.'
  score: 0
- text: 'This is underwhelming.'
  score: 0.25
- text: 'This is ok.'
  score: 0.5
- text: "I'm looking forward to this!"
  score: 1.0
```

```ini
[components.llm.task]
@llm_tasks = "spacy.Sentiment.v1"
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "sentiment_examples.yml"
```

### NoOp {id="noop"}

This task is only useful for testing - it tells the LLM to do nothing, and does
not set any fields on the `docs`.

> #### Example config
>
> ```ini
> [components.llm.task]
> @llm_tasks = "spacy.NoOp.v1"
> ```

#### spacy.NoOp.v1 {id="noop-v1"}

This task needs no further configuration.

## Models {id="models"}

A _model_ defines which LLM model to query, and how to query it. It can be a
simple function taking a collection of prompts (consistent with the output type
of `task.generate_prompts()`) and returning a collection of responses
(consistent with the expected input of `parse_responses`). Generally speaking,
it's a function of type
`Callable[[Iterable[Iterable[Any]]], Iterable[Iterable[Any]]]`, but specific
implementations can have other signatures, like
`Callable[[Iterable[Iterable[str]]], Iterable[Iterable[str]]]`.

Note: the model signature expects a nested iterable so it's able to deal with
sharded docs. Unsharded docs (i. e. those produced by (nonsharding
tasks)[/api/large-language-models#task-nonsharding]) are reshaped to fit the
expected data structure.

### Models via REST API {id="models-rest"}

These models all take the same parameters, but note that the `config` should
contain provider-specific keys and values, as it will be passed onwards to the
provider's API.

| Argument           | Description                                                                                                                                                                    |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `name`             | Model name, i. e. any supported variant for this particular model. Default depends on the specific model (cf. below) ~~str~~                                                   |
| `config`           | Further configuration passed on to the model. Default depends on the specific model (cf. below). ~~Dict[Any, Any]~~                                                            |
| `strict`           | If `True`, raises an error if the LLM API returns a malformed response. Otherwise, return the error responses as is. Defaults to `True`. ~~bool~~                              |
| `max_tries`        | Max. number of tries for API request. Defaults to `5`. ~~int~~                                                                                                                 |
| `max_request_time` | Max. time (in seconds) to wait for request to terminate before raising an exception. Defaults to `30.0`. ~~float~~                                                             |
| `interval`         | Time interval (in seconds) for API retries in seconds. Defaults to `1.0`. ~~float~~                                                                                            |
| `endpoint`         | Endpoint URL. Defaults to the provider's standard URL, if available (which is not the case for providers with exclusively custom deployments, such as Azure) ~~Optional[str]~~ |

> #### Example config:
>
> ```ini
> [components.llm.model]
> @llm_models = "spacy.GPT-4.v1"
> name = "gpt-4"
> config = {"temperature": 0.0}
> ```

Currently, these models are provided as part of the core library:

| Model                         | Provider          | Supported names                                                                                                    | Default name           | Default config                       |
| ----------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------ | ---------------------- | ------------------------------------ |
| `spacy.GPT-4.v1`              | OpenAI            | `["gpt-4", "gpt-4-0314", "gpt-4-32k", "gpt-4-32k-0314"]`                                                           | `"gpt-4"`              | `{}`                                 |
| `spacy.GPT-4.v2`              | OpenAI            | `["gpt-4", "gpt-4-0314", "gpt-4-32k", "gpt-4-32k-0314"]`                                                           | `"gpt-4"`              | `{temperature=0.0}`                  |
| `spacy.GPT-4.v3`              | OpenAI            | All names of [GPT-4 models](https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo) offered by OpenAI       | `"gpt-4"`              | `{temperature=0.0}`                  |
| `spacy.GPT-3-5.v1`            | OpenAI            | `["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-3.5-turbo-0613", "gpt-3.5-turbo-0613-16k", "gpt-3.5-turbo-instruct"]` | `"gpt-3.5-turbo"`      | `{}`                                 |
| `spacy.GPT-3-5.v2`            | OpenAI            | `["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-3.5-turbo-0613", "gpt-3.5-turbo-0613-16k", "gpt-3.5-turbo-instruct"]` | `"gpt-3.5-turbo"`      | `{temperature=0.0}`                  |
| `spacy.GPT-3-5.v3`            | OpenAI            | All names of [GPT-3.5 models](https://platform.openai.com/docs/models/gpt-3-5) offered by OpenAI                   | `"gpt-3.5-turbo"`      | `{temperature=0.0}`                  |
| `spacy.Davinci.v1`            | OpenAI            | `["davinci"]`                                                                                                      | `"davinci"`            | `{}`                                 |
| `spacy.Davinci.v2`            | OpenAI            | `["davinci"]`                                                                                                      | `"davinci"`            | `{temperature=0.0, max_tokens=500}`  |
| `spacy.Text-Davinci.v1`       | OpenAI            | `["text-davinci-003", "text-davinci-002"]`                                                                         | `"text-davinci-003"`   | `{}`                                 |
| `spacy.Text-Davinci.v2`       | OpenAI            | `["text-davinci-003", "text-davinci-002"]`                                                                         | `"text-davinci-003"`   | `{temperature=0.0, max_tokens=1000}` |
| `spacy.Code-Davinci.v1`       | OpenAI            | `["code-davinci-002"]`                                                                                             | `"code-davinci-002"`   | `{}`                                 |
| `spacy.Code-Davinci.v2`       | OpenAI            | `["code-davinci-002"]`                                                                                             | `"code-davinci-002"`   | `{temperature=0.0, max_tokens=500}`  |
| `spacy.Curie.v1`              | OpenAI            | `["curie"]`                                                                                                        | `"curie"`              | `{}`                                 |
| `spacy.Curie.v2`              | OpenAI            | `["curie"]`                                                                                                        | `"curie"`              | `{temperature=0.0, max_tokens=500}`  |
| `spacy.Text-Curie.v1`         | OpenAI            | `["text-curie-001"]`                                                                                               | `"text-curie-001"`     | `{}`                                 |
| `spacy.Text-Curie.v2`         | OpenAI            | `["text-curie-001"]`                                                                                               | `"text-curie-001"`     | `{temperature=0.0, max_tokens=500}`  |
| `spacy.Babbage.v1`            | OpenAI            | `["babbage"]`                                                                                                      | `"babbage"`            | `{}`                                 |
| `spacy.Babbage.v2`            | OpenAI            | `["babbage"]`                                                                                                      | `"babbage"`            | `{temperature=0.0, max_tokens=500}`  |
| `spacy.Text-Babbage.v1`       | OpenAI            | `["text-babbage-001"]`                                                                                             | `"text-babbage-001"`   | `{}`                                 |
| `spacy.Text-Babbage.v2`       | OpenAI            | `["text-babbage-001"]`                                                                                             | `"text-babbage-001"`   | `{temperature=0.0, max_tokens=500}`  |
| `spacy.Ada.v1`                | OpenAI            | `["ada"]`                                                                                                          | `"ada"`                | `{}`                                 |
| `spacy.Ada.v2`                | OpenAI            | `["ada"]`                                                                                                          | `"ada"`                | `{temperature=0.0, max_tokens=500}`  |
| `spacy.Text-Ada.v1`           | OpenAI            | `["text-ada-001"]`                                                                                                 | `"text-ada-001"`       | `{}`                                 |
| `spacy.Text-Ada.v2`           | OpenAI            | `["text-ada-001"]`                                                                                                 | `"text-ada-001"`       | `{temperature=0.0, max_tokens=500}`  |
| `spacy.Azure.v1`              | Microsoft, OpenAI | Arbitrary values                                                                                                   | No default             | `{temperature=0.0}`                  |
| `spacy.Command.v1`            | Cohere            | `["command", "command-light", "command-light-nightly", "command-nightly"]`                                         | `"command"`            | `{}`                                 |
| `spacy.Claude-2-1.v1`         | Anthropic         | `["claude-2-1"]`                                                                                                   | `"claude-2-1"`         | `{}`                                 |
| `spacy.Claude-2.v1`           | Anthropic         | `["claude-2", "claude-2-100k"]`                                                                                    | `"claude-2"`           | `{}`                                 |
| `spacy.Claude-1.v1`           | Anthropic         | `["claude-1", "claude-1-100k"]`                                                                                    | `"claude-1"`           | `{}`                                 |
| `spacy.Claude-1-0.v1`         | Anthropic         | `["claude-1.0"]`                                                                                                   | `"claude-1.0"`         | `{}`                                 |
| `spacy.Claude-1-2.v1`         | Anthropic         | `["claude-1.2"]`                                                                                                   | `"claude-1.2"`         | `{}`                                 |
| `spacy.Claude-1-3.v1`         | Anthropic         | `["claude-1.3", "claude-1.3-100k"]`                                                                                | `"claude-1.3"`         | `{}`                                 |
| `spacy.Claude-instant-1.v1`   | Anthropic         | `["claude-instant-1", "claude-instant-1-100k"]`                                                                    | `"claude-instant-1"`   | `{}`                                 |
| `spacy.Claude-instant-1-1.v1` | Anthropic         | `["claude-instant-1.1", "claude-instant-1.1-100k"]`                                                                | `"claude-instant-1.1"` | `{}`                                 |
| `spacy.PaLM.v1`               | Google            | `["chat-bison-001", "text-bison-001"]`                                                                             | `"text-bison-001"`     | `{temperature=0.0}`                  |

To use these models, make sure that you've [set the relevant API](#api-keys)
keys as environment variables.

**⚠️ A note on `spacy.Azure.v1`.** Working with Azure OpenAI is slightly
different than working with models from other providers:

- In Azure LLMs have to be made available by creating a _deployment_ of a given
  model (e. g. GPT-3.5). This deployment can have an arbitrary name. The `name`
  argument, which everywhere else denotes the model name (e. g. `claude-1.0`,
  `gpt-3.5`), here refers to the _deployment name_.
- Deployed Azure OpenAI models are reachable via a resource-specific base URL,
  usually of the form `https://{resource}.openai.azure.com`. Hence the URL has
  to be specified via the `base_url` argument.
- Azure further expects the _API version_ to be specified. The default value for
  this, via the `api_version` argument, is currently `2023-05-15` but may be
  updated in the future.
- Finally, since we can't infer information about the model from the deployment
  name, `spacy-llm` requires the `model_type` to be set to either
  `"completions"` or `"chat"`, depending on whether the deployed model is a
  completion or chat model.

#### API Keys {id="api-keys"}

Note that when using hosted services, you have to ensure that the proper API
keys are set as environment variables as described by the corresponding
provider's documentation.

E. g. when using OpenAI, you have to get an API key from openai.com, and ensure
that the keys are set as environmental variables:

```shell
export OPENAI_API_KEY="sk-..."
export OPENAI_API_ORG="org-..."
```

For Cohere:

```shell
export CO_API_KEY="..."
```

For Anthropic:

```shell
export ANTHROPIC_API_KEY="..."
```

For PaLM:

```shell
export PALM_API_KEY="..."
```

### Models via HuggingFace {id="models-hf"}

These models all take the same parameters:

| Argument      | Description                                                                                                                           |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `name`        | Model name, i. e. any supported variant for this particular model. ~~str~~                                                            |
| `config_init` | Further configuration passed on to the construction of the model with `transformers.pipeline()`. Defaults to `{}`. ~~Dict[str, Any]~~ |
| `config_run`  | Further configuration used during model inference. Defaults to `{}`. ~~Dict[str, Any]~~                                               |

> #### Example config
>
> ```ini
> [components.llm.model]
> @llm_models = "spacy.Llama2.v1"
> name = "Llama-2-7b-hf"
> ```

Currently, these models are provided as part of the core library:

| Model                | Provider        | Supported names                                                                                              | HF directory                           |
| -------------------- | --------------- | ------------------------------------------------------------------------------------------------------------ | -------------------------------------- |
| `spacy.Dolly.v1`     | Databricks      | `["dolly-v2-3b", "dolly-v2-7b", "dolly-v2-12b"]`                                                             | https://huggingface.co/databricks      |
| `spacy.Falcon.v1`    | TII             | `["falcon-rw-1b", "falcon-7b", "falcon-7b-instruct", "falcon-40b-instruct"]`                                 | https://huggingface.co/tiiuae          |
| `spacy.Llama2.v1`    | Meta AI         | `["Llama-2-7b-hf", "Llama-2-13b-hf", "Llama-2-70b-hf"]`                                                      | https://huggingface.co/meta-llama      |
| `spacy.Mistral.v1`   | Mistral AI      | `["Mistral-7B-v0.1", "Mistral-7B-Instruct-v0.1"]`                                                            | https://huggingface.co/mistralai       |
| `spacy.StableLM.v1`  | Stability AI    | `["stablelm-base-alpha-3b", "stablelm-base-alpha-7b", "stablelm-tuned-alpha-3b", "stablelm-tuned-alpha-7b"]` | https://huggingface.co/stabilityai     |
| `spacy.OpenLLaMA.v1` | OpenLM Research | `["open_llama_3b", "open_llama_7b", "open_llama_7b_v2", "open_llama_13b"]`                                   | https://huggingface.co/openlm-research |

<Infobox variant="warning" title="Gated models on Hugging Face" id="hf_licensing">

Some models available on Hugging Face (HF), such as Llama 2, are _gated models_.
That means that users have to fulfill certain requirements to be allowed access
to these models. In the case of Llama 2 you'll need to request agree to Meta's
Terms of Service while logged in with your HF account. After Meta grants you
permission to use Llama 2, you'll be able to download and use the model.

This requires that you are logged in with your HF account on your local
machine - check out the HF quick start documentation. In a nutshell, you'll need
to create an access token on HF and log in to HF using your access token, e. g.
with `huggingface-cli login`.

</Infobox>

Note that Hugging Face will download the model the first time you use it - you
can
[define the cached directory](https://huggingface.co/docs/huggingface_hub/main/en/guides/manage-cache)
by setting the environmental variable `HF_HOME`.

#### Installation with HuggingFace {id="install-hf"}

To use models from HuggingFace, ideally you have a GPU enabled and have
installed `transformers`, `torch` and CUDA in your virtual environment. This
allows you to have the setting `device=cuda:0` in your config, which ensures
that the model is loaded entirely on the GPU (and fails otherwise).

You can do so with

```shell
python -m pip install "spacy-llm[transformers]" "transformers[sentencepiece]"
```

If you don't have access to a GPU, you can install `accelerate` and
set`device_map=auto` instead, but be aware that this may result in some layers
getting distributed to the CPU or even the hard drive, which may ultimately
result in extremely slow queries.

```shell
python -m pip install "accelerate>=0.16.0,<1.0"
```

### LangChain models {id="langchain-models"}

To use [LangChain](https://github.com/hwchase17/langchain) for the API retrieval
part, make sure you have installed it first:

```shell
python -m pip install "langchain==0.0.191"
# Or install with spacy-llm directly
python -m pip install "spacy-llm[extras]"
```

Note that LangChain currently only supports Python 3.9 and beyond.

LangChain models in `spacy-llm` work slightly differently. `langchain`'s models
are parsed automatically, each LLM class in `langchain` has one entry in
`spacy-llm`'s registry. As `langchain`'s design has one class per API and not
per model, this results in registry entries like `langchain.OpenAI.v1` - i. e.
there is one registry entry per API and not per model (family), as for the REST-
and HuggingFace-based entries.

The name of the model to be used has to be passed in via the `name` attribute.

> #### Example config
>
> ```ini
> [components.llm.model]
> @llm_models = "langchain.OpenAI.v1"
> name = "gpt-3.5-turbo"
> query = {"@llm_queries": "spacy.CallLangChain.v1"}
> config = {"temperature": 0.0}
> ```

| Argument | Description                                                                                                                                                           |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`   | The name of a model supported by LangChain for this API. ~~str~~                                                                                                     |
| `config` | Configuration passed on to the LangChain model. Defaults to `{}`. ~~Dict[Any, Any]~~                                                                                  |
| `query`  | Function that executes the prompts. If `None`, defaults to `spacy.CallLangChain.v1`. ~~Optional[Callable[["langchain.llms.BaseLLM", Iterable[Any]], Iterable[Any]]]~~ |

The default `query` (`spacy.CallLangChain.v1`) executes the prompts by running
`model(text)` for each given textual prompt.

## Cache {id="cache"}

Interacting with LLMs, either through an external API or a local instance, is
costly. Since developing an NLP pipeline generally means a lot of exploration
and prototyping, `spacy-llm` implements a built-in cache to avoid reprocessing
the same documents at each run that keeps batches of documents stored on disk.

> #### Example config
>
> ```ini
> [components.llm.cache]
> @llm_misc = "spacy.BatchCache.v1"
> path = "path/to/cache"
> batch_size = 64
> max_batches_in_mem = 4
> ```

| Argument             | Description                                                                                                                                      |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `path`               | Cache directory. If `None`, no caching is performed, and this component will act as a NoOp. Defaults to `None`. ~~Optional[Union[str, Path]]~~   |
| `batch_size`         | Number of docs in one batch (file). Once a batch is full, it will be peristed to disk. Defaults to 64. ~~int~~                                   |
| `max_batches_in_mem` | Max. number of batches to hold in memory. Allows you to limit the effect on your memory if you're handling a lot of docs. Defaults to 4. ~~int~~ |

When retrieving a document, the `BatchCache` will first figure out what batch
the document belongs to. If the batch isn't in memory it will try to load the
batch from disk and then move it into memory.

Note that since the cache is generated by a registered function, you can also
provide your own registered function returning your own cache implementation. If
you wish to do so, ensure that your cache object adheres to the `Protocol`
defined in `spacy_llm.ty.Cache`.

## Various functions {id="various-functions"}

### spacy.FewShotReader.v1 {id="fewshotreader-v1"}

This function is registered in spaCy's `misc` registry, and reads in examples
from a `.yml`, `.yaml`, `.json` or `.jsonl` file. It uses
[`srsly`](https://github.com/explosion/srsly) to read in these files and parses
them depending on the file extension.

> #### Example config
>
> ```ini
> [components.llm.task.examples]
> @misc = "spacy.FewShotReader.v1"
> path = "ner_examples.yml"
> ```

| Argument | Description                                                                                     |
| -------- | ----------------------------------------------------------------------------------------------- |
| `path`   | Path to an examples file with suffix `.yml`, `.yaml`, `.json` or `.jsonl`. ~~Union[str, Path]~~ |

### spacy.FileReader.v1 {id="filereader-v1"}

This function is registered in spaCy's `misc` registry, and reads a file
provided to the `path` to return a `str` representation of its contents. This
function is typically used to read
[Jinja](https://jinja.palletsprojects.com/en/3.1.x/) files containing the prompt
template.

> #### Example config
>
> ```ini
> [components.llm.task.template]
> @misc = "spacy.FileReader.v1"
> path = "ner_template.jinja2"
> ```

| Argument | Description                                       |
| -------- | ------------------------------------------------- |
| `path`   | Path to the file to be read. ~~Union[str, Path]~~ |

### Normalizer functions {id="normalizer-functions"}

These functions provide simple normalizations for string comparisons, e.g.
between a list of specified labels and a label given in the raw text of the LLM
response. They are registered in spaCy's `misc` registry and have the signature
`Callable[[str], str]`.

- `spacy.StripNormalizer.v1`: only apply `text.strip()`
- `spacy.LowercaseNormalizer.v1`: applies `text.strip().lower()` to compare
  strings in a case-insensitive way.