# Go AI Client Library - Complete Documentation

## Table of Contents

- [Home](#home)
- [Modules](#modules)
- [Examples](#examples)
- [Getting Started > Installation](#getting-started-installation)
- [Getting Started > Quick Start](#getting-started-quick-start)
- [Providers > Overview](#providers-overview)
- [Providers > LLM](#providers-llm)
- [Providers > Embeddings](#providers-embeddings)
- [Providers > Image Generation](#providers-image-generation)
- [Providers > Audio](#providers-audio)
- [Providers > Speech-to-Text](#providers-speech-to-text)
- [Providers > Rerankers](#providers-rerankers)
- [Providers > Decision Models](#providers-decision-models)
- [Providers > Fill-in-the-Middle](#providers-fill-in-the-middle)
- [Providers > Vision](#providers-vision)
- [Agent Framework > Overview](#agent-framework-overview)
- [Agent Framework > Session Management](#agent-framework-session-management)
- [Agent Framework > Persistent Memory](#agent-framework-persistent-memory)
- [Agent Framework > Streaming](#agent-framework-streaming)
- [Agent Framework > Hooks](#agent-framework-hooks)
- [Agent Framework > Tool Confirmation](#agent-framework-tool-confirmation)
- [Agent Framework > Loop Continuation](#agent-framework-loop-continuation)
- [Agent Framework > Sub-Agents](#agent-framework-sub-agents)
- [Agent Framework > Background Agents](#agent-framework-background-agents)
- [Agent Framework > Handoffs](#agent-framework-handoffs)
- [Agent Framework > Fan-Out](#agent-framework-fan-out)
- [Agent Framework > Team Coordination](#agent-framework-team-coordination)
- [Agent Framework > Continue/Resume](#agent-framework-continueresume)
- [Agent Framework > Context Strategies](#agent-framework-context-strategies)
- [Agent Framework > Toolsets](#agent-framework-toolsets)
- [Agent Framework > Instruction Templates](#agent-framework-instruction-templates)
- [Integrations > PostgreSQL](#integrations-postgresql)
- [Integrations > SQLite](#integrations-sqlite)
- [Integrations > pgvector](#integrations-pgvector)
- [Advanced > Batch Processing](#advanced-batch-processing)
- [Advanced > BYOM](#advanced-byom)
- [Advanced > MCP Integration](#advanced-mcp-integration)
- [Advanced > Tool Calling](#advanced-tool-calling)
- [Advanced > Structured Output](#advanced-structured-output)
- [Advanced > Reasoning](#advanced-reasoning)
- [Advanced > Cost Tracking](#advanced-cost-tracking)
- [Advanced > Prompt Templates](#advanced-prompt-templates)
- [Advanced > OpenTelemetry Tracing](#advanced-opentelemetry-tracing)
- [Advanced > Configuration](#advanced-configuration)

---

## Home

> Source: index.md

# Go AI Client Library

[![CI](https://github.com/joakimcarlsson/ai/actions/workflows/ci.yml/badge.svg)](https://github.com/joakimcarlsson/ai/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Go Version](https://img.shields.io/badge/go-1.25%2B-00ADD8?logo=go)](https://go.dev/)

A multi-provider Go library for interacting with AI models through unified
interfaces. Each capability — LLMs, embeddings, TTS, STT, image generation,
rerankers, decision models, fill-in-the-middle — is published as its own
modality module, and each vendor implementation is published as its own sub-module. You only pull
the SDK you actually use.

Supported providers: Anthropic, OpenAI, Google (Gemini & Vertex AI), AWS
Bedrock, Azure OpenAI, Voyage AI, Cohere, Mistral, DeepSeek, Groq, OpenRouter,
xAI, ElevenLabs, Deepgram, AssemblyAI, Google Cloud Speech, plus any
OpenAI-compatible endpoint via [BYOM](advanced/byom.md).

## Module structure

The library is published as ~50 independent Go modules. The core split:

- **Modality interfaces** define the contract for each capability:
  `llm`, `embeddings`, `tts`, `stt`, `image`, `rerankers`, `fim`, `decision`. These pull
  no vendor SDKs.
- **Vendor sub-modules** under each modality carry the SDK and the
  implementation: `llm/anthropic`, `llm/openai`, `embeddings/voyage`,
  `tts/elevenlabs`, etc. You import only the vendors you use.
- **Tier 0 leaves** (`model`, `message`, `tool`, `schema`, `tracing`,
  `prompt`, `types`) are dependency-free building blocks shared across the
  rest.
- **Agent runtimes** (`agent` for text, `voice` for voice-first conversations)
  build on top of `memory`, `session`, and the modality interfaces. Persistence
  integrations live under `memory/{pgvector,postgres,sqlite}`.

## Install

You install only the modules you use. For an OpenAI chat client:

```bash
go get github.com/joakimcarlsson/ai/llm
go get github.com/joakimcarlsson/ai/llm/openai
go get github.com/joakimcarlsson/ai/message
```

## Quick example

```go
package main

import (
    "context"
    "fmt"
    "log"
    "os"

    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
    "github.com/joakimcarlsson/ai/message"
)

func main() {
    ctx := context.Background()

    client := llmopenai.NewLLM(
        llmopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
        llmopenai.WithModel(llmopenai.Models[llmopenai.GPT4o]),
        llmopenai.WithMaxTokens(1000),
    )

    response, err := client.SendMessages(ctx, []message.Message{
        message.NewUserMessage("Hello, how are you?"),
    }, nil)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(response.Content)
}
```

## Next steps

- [Modules](modules.md) — full list of every package, its purpose, and the vendor SDK it carries
- [Installation & Quick Start](getting-started/installation.md) — get up and running
- [Provider Overview](providers/overview.md) — every supported provider with capability matrix
- [LLM module](providers/llm.md) — chat, streaming, tools, structured output
- [Agent Framework](agent/overview.md) — multi-agent runtime
- [Voice Agent](voice/overview.md) — streaming STT → LLM → TTS with tool calls
- [Bring Your Own Model (BYOM)](advanced/byom.md) — Ollama, LocalAI, custom OpenAI-compatible endpoints


---

## Modules

> Source: modules.md

# Modules

The library is published as ~50 independent Go modules. Every module lives at
`github.com/joakimcarlsson/ai/<path>` and is versioned independently with
path-prefixed tags (e.g. `llm/openai/v0.1.0`).

You install only what you use. A typical OpenAI chat client needs `llm`,
`llm/openai`, `message`, and `model`.

## Tier 0 — Leaves

Dependency-free building blocks. No vendor SDKs.

| Module | Purpose |
|---|---|
| `model` | Model catalog, capability flags, custom-model builder |
| `message` | Conversation message types and constructors |
| `tool` | Tool interface, MCP integration, function-tool helpers |
| `hostedtool` | Price type for tools a provider runs and bills per call (web search, file search); unrelated to `tool` |
| `schema` | JSON Schema generation for tool inputs and structured output |
| `tracing` | OpenTelemetry setup helper for traces, metrics, and logs |
| `prompt` | Prompt template rendering |
| `types` | Shared event types (`EventContentDelta`, `EventThinkingDelta`, etc.) |

## Tier 1 — Modality interfaces

Define the contract for each capability. No vendor SDKs; vendor modules
implement these.

| Module | Purpose |
|---|---|
| `llm` | Chat completion interface, retry config, tracing wrapper |
| `embeddings` | Text / multimodal / contextualized embedding interface |
| `tts` | Text-to-speech interface (with optional `ForcedAlignmentProvider`) |
| `stt` | Speech-to-text interface (transcribe + translate, streaming) |
| `image` | Image generation interface |
| `rerankers` | Document reranking interface |
| `decision` | Decision model interface (typed choice / score / yes-no answers) |
| `fim` | Fill-in-the-middle code completion interface |

## Tier 2 — Vendor implementations

Each carries exactly one vendor SDK.

### LLM

| Module | Vendor SDK |
|---|---|
| `llm/openai` | `openai-go` — Responses API via `NewLLM`, including server-side built-in tools; `NewChatCompletionsLLM` for OpenAI-compatible endpoints (Groq, Mistral, Ollama and others via `WithBaseURL`) |
| `llm/anthropic` | `anthropic-sdk-go` (also Bedrock backend); server-side `web_search` via `WithWebSearch` |
| `llm/gemini` | `google.golang.org/genai`; server-side `google_search`, `code_execution`, `url_context` via dedicated options |
| `llm/groq` | `openai-go` — fast OpenAI-compatible chat via `NewLLM`; compound-model built-ins (`browser_search`, `code_execution`, `visit_website`) via `NewCompoundLLM` |
| `llm/xai` | `openai-go` — Responses API via `NewLLM`, with built-ins (`web_search`, `x_search`, `code_execution`) |
| `llm/vertexai` | `google.golang.org/genai` (Vertex AI backend) |
| `llm/azure` | `openai-go` against Azure OpenAI |
| `llm/bedrock` | `aws-sdk-go-v2` Bedrock Runtime |

### Embeddings

| Module | Vendor SDK |
|---|---|
| `embeddings/openai` | `openai-go` |
| `embeddings/voyage` | `net/http` |
| `embeddings/cohere` | `net/http` |
| `embeddings/gemini` | `google.golang.org/genai` |
| `embeddings/mistral` | `net/http` |
| `embeddings/bedrock` | `aws-sdk-go-v2` Bedrock Runtime |
| `embeddings/berget` | `openai-go` (thin wrapper over `embeddings/openai`) |

### TTS

| Module | Vendor SDK |
|---|---|
| `tts/openai` | `openai-go` |
| `tts/elevenlabs` | `net/http` |
| `tts/google` | `cloud.google.com/go/texttospeech` |
| `tts/azure` | `net/http` (Azure Cognitive Services Speech) |
| `tts/deepgram` | `net/http` |
| `tts/openrouter` | `openai-go` (thin wrapper over `tts/openai`) |

### STT

| Module | Vendor SDK |
|---|---|
| `stt/openai` | `openai-go` |
| `stt/elevenlabs` | `net/http` + `gorilla/websocket` for streaming |
| `stt/deepgram` | `deepgram-go-sdk` |
| `stt/assemblyai` | `assemblyai-go-sdk` |
| `stt/google` | `cloud.google.com/go/speech` |
| `stt/berget` | `openai-go` (thin wrapper over `stt/openai`) |
| `stt/openrouter` | `openai-go` (thin wrapper over `stt/openai`) |

### Image generation

| Module | Vendor SDK |
|---|---|
| `image/openai` | `openai-go` (also xAI via `WithBaseURL`) |
| `image/azure` | `openai-go` (Azure OpenAI; wraps `image/openai`) |
| `image/gemini` | `google.golang.org/genai` (also Vertex AI) |
| `image/openrouter` | `net/http` (OpenRouter's own `/images` endpoint) |

### Rerankers

| Module | Vendor SDK |
|---|---|
| `rerankers/voyage` | `net/http` |
| `rerankers/cohere` | `net/http` |
| `rerankers/berget` | `net/http` |

### Decision models

| Module | Vendor SDK |
|---|---|
| `decision/jev` | `net/http` |
| `decision/openrouter` | `net/http` (thin wrapper over `decision/jev`) |

### Fill-in-the-middle

| Module | Vendor SDK |
|---|---|
| `fim/mistral` | `net/http` |
| `fim/deepseek` | `net/http` |

### Hosted tool pricing

Generated price catalogs, not clients: they carry no vendor SDK.

| Module | Contents |
|---|---|
| `hostedtool/openai` | Per-1k-call rates for OpenAI's hosted tools (`WebSearch`, `FileSearch`, ...), generated by `make models` |

## Tier 3 — Utilities

| Module | Purpose |
|---|---|
| `tokens/sliding` | Sliding-window context strategy |
| `tokens/truncate` | Hard-truncate context strategy |
| `tokens/summarize` | Summarization-based context strategy |
| `batch` | Batch request / progress / event types |
| `batch/openai` | OpenAI native batch API processor |
| `batch/anthropic` | Anthropic native batch API processor |
| `batch/gemini` | Gemini / Vertex AI native batch API processor |
| `batch/concurrent` | Bounded-concurrency runner over any LLM / embedding client |

## Tier 4 — Agent runtimes and conversation primitives

| Module | Purpose |
|---|---|
| `agent` | Agent runtime: chat, streaming, hooks, tools, sub-agents, handoffs, fan-out |
| `voice` | Voice-first agent: streaming STT → LLM → TTS pipeline with tool calls |
| `memory` | Persistent memory interface, dedup + extraction helpers |
| `session` | Conversation session storage interfaces and implementations |

`agent/team` is a sub-package of `agent` (same module).

## Tier 5 — Persistence

| Module | Purpose |
|---|---|
| `memory/pgvector` | PostgreSQL + pgvector backend with HNSW vector search |
| `memory/postgres` | PostgreSQL session + memory store |
| `memory/sqlite` | SQLite session + memory store |

## Adding new modules

The repo is a Go workspace; each module has its own `go.mod`. To add a new
vendor implementation:

1. Create the directory and `go.mod` (declare module path
   `github.com/joakimcarlsson/ai/<modality>/<vendor>`).
2. Add a `replace` directive for any internal modules it imports.
3. Add the path to `go.work.example`.
4. Tag releases as `<modality>/<vendor>/vX.Y.Z`.

See [the release process](https://github.com/joakimcarlsson/ai#release-process)
for details on tagging and publishing.


---

## Examples

> Source: examples.md

# Examples

The repository includes runnable examples under [`/examples`](https://github.com/joakimcarlsson/ai/tree/main/examples).
Each example is its own Go module so it shows exactly which packages a downstream
application imports after the multi-module refactor.

Run examples from their own directory:

```bash
cd examples/llm/openai-chat
go run .
```

If you are working inside a local checkout with `go.work` enabled, either add
the example module to the workspace or disable workspace mode for the command:

```bash
GOWORK=off go run .
```

PowerShell:

```powershell
$env:GOWORK = "off"; go run .
```

The example `go.mod` files use local `replace` directives so they build against
the checkout. In your own application, remove those `replace` directives and
install the modules you import with `go get`.

## Package Examples

- `llm/openai-chat` — basic chat completion with `llm/openai`
- `llm/anthropic-stream` — streaming chat with `llm/anthropic`
- `llm/builtin-tools` — server-side built-in tools across `anthropic`, `gemini`, `openai-responses`, `groq-compound`
- `embeddings/voyage` — text embeddings with `embeddings/voyage`
- `image/gemini` — image generation with `image/gemini`
- `image/azure` — image generation against Azure OpenAI with `image/azure`
- `image/openrouter` — image generation, progressive streaming, and two ways to use an OpenRouter model the `model` registry does not define
- `tts/elevenlabs` — text-to-speech with `tts/elevenlabs`
- `stt/deepgram` — speech-to-text with `stt/deepgram`
- `rerankers/cohere` — document reranking with `rerankers/cohere`
- `fim/mistral` — fill-in-the-middle code completion with `fim/mistral`
- `batch/concurrent` — concurrent batch processing around an LLM client
- `agent/basic` — a minimal agent using an LLM client
- `tokens/truncate` — local context truncation without an API key

## Provider Switching

Provider-switch examples show the main point of the modality interfaces: the
business logic stays typed against the shared interface, while only construction
changes per vendor.

- `llm/provider-switch` — `openai`, `anthropic`, `gemini`
- `embeddings/provider-switch` — `openai`, `voyage`, `cohere`
- `image/provider-switch` — `openai`, `gemini`
- `tts/provider-switch` — `openai`, `elevenlabs`
- `stt/provider-switch` — `openai`, `deepgram`
- `rerankers/provider-switch` — `cohere`, `voyage`
- `fim/provider-switch` — `mistral`, `deepseek`
- `batch/provider-switch` — `openai`, `anthropic`, `gemini`
- `agent/provider-switch` — `openai`, `anthropic`, `gemini`

Set `AI_PROVIDER` to choose the implementation:

```bash
AI_PROVIDER=gemini go run .
```

PowerShell:

```powershell
$env:AI_PROVIDER = "gemini"; go run .
```

## Observability And Pricing

- `tracing/otel` — configures OpenTelemetry with `tracing.New`, attaches a stdout
  span exporter, and runs a traced LLM request.
- `model/pricing` — estimates chat, embedding, image, and TTS costs from the
  public model registry fields.

The tracing example prints spans locally by default. For collector-based export,
configure `tracing.New` with OTLP settings such as `OTEL_EXPORTER_OTLP_ENDPOINT`.

## Environment Variables

Set the provider key used by the example you run:

- `OPENAI_API_KEY` for OpenAI LLM, embedding, image, TTS, STT, batch, agent, and tracing examples
- `ANTHROPIC_API_KEY` for Anthropic LLM, batch, and agent examples
- `VOYAGE_API_KEY` for Voyage embedding and reranker examples
- `GEMINI_API_KEY` for Gemini LLM, image, batch, and agent examples
- `GROQ_API_KEY` for the `groq-compound` provider in `llm/builtin-tools`
- `XAI_API_KEY` for the `xai-responses` provider in `llm/builtin-tools`
- `ELEVENLABS_API_KEY` for ElevenLabs TTS examples
- `DEEPGRAM_API_KEY` for Deepgram STT examples
- `COHERE_API_KEY` for Cohere embedding and reranker examples
- `MISTRAL_API_KEY` for Mistral FIM examples
- `DEEPSEEK_API_KEY` for DeepSeek FIM examples
- `OPENROUTER_API_KEY` for the `image/openrouter` example

`model/pricing` and `tokens/truncate` run locally and do not require credentials.

## Generated Files

Audio and image examples may write generated artifacts next to the example
program:

- `image/gemini` writes `gemini-image.png`
- `image/azure` writes `azure-image.png`
- `image/openrouter` writes `openrouter-image.png`, `openrouter-image-custom.png`, `openrouter-image-custom-described.png` and `openrouter-image-streamed.png`
- `image/provider-switch` writes `<provider>-image.png`
- `tts/elevenlabs` writes `elevenlabs-speech.mp3`
- `tts/provider-switch` writes `<provider>-speech.mp3`
- `stt/deepgram` expects a local audio file path as its only argument
- `stt/provider-switch` expects a local audio file path as its only argument

Do not commit generated audio or image outputs.


---

## Getting Started > Installation

> Source: getting-started/installation.md

# Installation

## Requirements

- Go 1.25 or later

## Install only what you use

Each modality and vendor is a separate Go module. Pull just the ones you need.

### Common combinations

**LLM with OpenAI:**

```bash
go get github.com/joakimcarlsson/ai/llm
go get github.com/joakimcarlsson/ai/llm/openai
go get github.com/joakimcarlsson/ai/message
```

**LLM with Anthropic:**

```bash
go get github.com/joakimcarlsson/ai/llm
go get github.com/joakimcarlsson/ai/llm/anthropic
go get github.com/joakimcarlsson/ai/message
```

**Embeddings (Voyage) + LLM (OpenAI) + agent runtime:**

```bash
go get github.com/joakimcarlsson/ai/agent
go get github.com/joakimcarlsson/ai/llm
go get github.com/joakimcarlsson/ai/llm/openai
go get github.com/joakimcarlsson/ai/embeddings
go get github.com/joakimcarlsson/ai/embeddings/voyage
go get github.com/joakimcarlsson/ai/message
```

**Persistent memory with pgvector:**

```bash
go get github.com/joakimcarlsson/ai/agent
go get github.com/joakimcarlsson/ai/memory
go get github.com/joakimcarlsson/ai/memory/pgvector
go get github.com/joakimcarlsson/ai/embeddings
go get github.com/joakimcarlsson/ai/embeddings/openai
```

## Import shape

The Go convention is to alias the vendor package with a short name to avoid
clashing with the modality package:

```go
import (
    "github.com/joakimcarlsson/ai/message"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
)
```

For agent-runtime code that wires multiple modalities:

```go
import (
    "github.com/joakimcarlsson/ai/agent"
    "github.com/joakimcarlsson/ai/memory"
    pgvectormem "github.com/joakimcarlsson/ai/memory/pgvector"
    embopenai "github.com/joakimcarlsson/ai/embeddings/openai"
    llmanthropic "github.com/joakimcarlsson/ai/llm/anthropic"
)
```

## API keys

Each vendor needs its own credential. Common environment variables:

```bash
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="..."
export VOYAGE_API_KEY="pa-..."
export COHERE_API_KEY="..."
export ELEVENLABS_API_KEY="..."
export DEEPGRAM_API_KEY="..."
export ASSEMBLYAI_API_KEY="..."
```

You can also pass the key directly to the constructor:

```go
client := llmopenai.NewLLM(
    llmopenai.WithAPIKey("sk-..."),
    llmopenai.WithModel(openai.Models[openai.GPT4o]),
)
```

## OpenAI-compatible providers

Groq, OpenRouter, xAI, Mistral, DeepSeek, Perplexity, and any
OpenAI-compatible endpoint use `llm/openai` with a custom base URL — no
separate vendor module:

```go
client := llmopenai.NewLLM(
    llmopenai.WithAPIKey(os.Getenv("GROQ_API_KEY")),
    llmopenai.WithBaseURL("https://api.groq.com/openai/v1"),
    llmopenai.WithModel(groq.Models[groq.Llama3_3_70BVersatile]),
)
```

See [BYOM](../advanced/byom.md) for the registry helper that organises these.


---

## Getting Started > Quick Start

> Source: getting-started/quick-start.md

# Quick Start

## Basic chat

```go
package main

import (
    "context"
    "fmt"
    "log"
    "os"

    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
    "github.com/joakimcarlsson/ai/message"
)

func main() {
    ctx := context.Background()

    client := llmopenai.NewLLM(
        llmopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
        llmopenai.WithModel(llmopenai.Models[llmopenai.GPT4o]),
        llmopenai.WithMaxTokens(1000),
    )

    response, err := client.SendMessages(ctx, []message.Message{
        message.NewUserMessage("Hello, how are you?"),
    }, nil)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(response.Content)
}
```

The `client` value satisfies the `llm.LLM` interface, so swapping vendors is
just a matter of changing which package you construct from. The rest of your
code (`SendMessages`, `StreamResponse`, etc.) stays the same.

## Streaming

```go
import "github.com/joakimcarlsson/ai/types"

stream := client.StreamResponse(ctx, messages, nil)

for event := range stream {
    switch event.Type {
    case types.EventContentDelta:
        fmt.Print(event.Content)
    case types.EventComplete:
        fmt.Printf("\nTokens: %d in / %d out\n",
            event.Response.Usage.InputTokens,
            event.Response.Usage.OutputTokens)
    case types.EventError:
        log.Fatal(event.Error)
    }
}
```

## Multimodal (images)

```go
imageData, err := os.ReadFile("image.png")
if err != nil {
    log.Fatal(err)
}

msg := message.NewUserMessage("What's in this image?")
msg.AddBinary("image/png", imageData)

response, err := client.SendMessages(ctx, []message.Message{msg}, nil)
```

## Switching vendors

Anthropic instead of OpenAI:

```go
import llmanthropic "github.com/joakimcarlsson/ai/llm/anthropic"

client := llmanthropic.NewLLM(
    llmanthropic.WithAPIKey(os.Getenv("ANTHROPIC_API_KEY")),
    llmanthropic.WithModel(anthropic.Models[anthropic.Claude45Sonnet]),
    llmanthropic.WithMaxTokens(1000),
)
```

The `client` is still a `llm.LLM`, so the rest of the program is unchanged.

## Your first agent

The `agent` module wires an LLM client, tools, session storage, and optional
memory into a runtime:

```go
import (
    "github.com/joakimcarlsson/ai/agent"
    "github.com/joakimcarlsson/ai/session"
)

myAgent := agent.New(client,
    agent.WithSystemPrompt("You are a helpful assistant."),
    agent.WithTools(&weatherTool{}),
    agent.WithSession("user-123", session.FileStore("./sessions")),
)

response, _ := myAgent.Chat(ctx, "What's the weather in Tokyo?")
fmt.Println(response.Content)
```

See the [Agent Framework](../agent/overview.md) section for the full guide.


---

## Providers > Overview

> Source: providers/overview.md

# Supported Providers

Each modality has its own Go module. Vendor implementations are sub-modules.
Pull only the ones you use.

## LLM Providers

Each native LLM vendor is its own sub-module under `llm/`:

| Module | Provider | Streaming | Tools | Structured Output | Attachments |
|---|---|---|---|---|---|
| `llm/anthropic` | Anthropic (Claude) | ✅ | ✅ | ✅ | ✅ |
| `llm/openai` | OpenAI (GPT) | ✅ | ✅ | ✅ | ✅ |
| `llm/gemini` | Google Gemini | ✅ | ✅ | ✅ | ✅ |
| `llm/bedrock` | AWS Bedrock (wraps `llm/anthropic` for Claude on Bedrock) | ✅ | ✅ | ✅ | ✅ |
| `llm/azure` | Azure OpenAI (wraps `llm/openai`) | ✅ | ✅ | ✅ | ✅ |
| `llm/vertexai` | Google Vertex AI (wraps `llm/gemini`) | ✅ | ✅ | ✅ | ✅ |

### OpenAI-compatible vendors

These are thin wrappers over `llm/openai` that hardcode the vendor's base URL.
They expose only the OpenAI-compatible subset; vendor-unique features (xAI Live
Search, Perplexity citations, etc.) are not wired up. Pass any vendor-supported
model id via `openai.WithModel` even without an entry in the `model` package.

| Module | Provider | Default base URL |
|---|---|---|
| `llm/xai` | xAI (Grok) | `https://api.x.ai/v1` |
| `llm/openrouter` | OpenRouter | `https://openrouter.ai/api/v1` |
| `llm/groq` | Groq | `https://api.groq.com/openai/v1` |
| `llm/deepseek` | DeepSeek | `https://api.deepseek.com/v1` |
| `llm/perplexity` | Perplexity Sonar | `https://api.perplexity.ai` |
| `llm/mistral` | Mistral AI | `https://api.mistral.ai/v1` |
| `llm/cerebras` | Cerebras Inference | `https://api.cerebras.ai/v1` |
| `llm/fireworks` | Fireworks AI | `https://api.fireworks.ai/inference/v1` |
| `llm/together` | Together AI | `https://api.together.xyz/v1` |
| `llm/ollama` | Ollama (local) | `http://localhost:11434/v1` |
| `llm/berget` | Berget AI (EU-hosted, EUR pricing) | `https://api.berget.ai/v1` |

For any other OpenAI-compatible endpoint, use `llm/openai` directly with
`WithBaseURL(...)`. See [BYOM](../advanced/byom.md).

## Embedding Providers

Each native embedding vendor is its own sub-module under `embeddings/`:

| Module | Provider | Text | Multimodal | Contextualized |
|---|---|---|---|---|
| `embeddings/voyage` | Voyage AI | ✅ | ✅ | ✅ |
| `embeddings/openai` | OpenAI | ✅ | ❌ | ❌ |
| `embeddings/cohere` | Cohere | ✅ | ❌ | ❌ |
| `embeddings/gemini` | Google Gemini | ✅ | ❌ | ❌ |
| `embeddings/mistral` | Mistral | ✅ | ❌ | ❌ |
| `embeddings/bedrock` | AWS Bedrock (Titan + Cohere) | ✅ | ❌ | ❌ |
| `embeddings/berget` | Berget AI (multilingual E5) | ✅ | ❌ | ❌ |

## Reranker Providers

Each native reranker vendor under `rerankers/`:

| Module | Provider |
|---|---|
| `rerankers/voyage` | Voyage AI |
| `rerankers/cohere` | Cohere |
| `rerankers/berget` | Berget AI (EU-hosted, EUR pricing) |

## Decision Providers

Each native decision-model vendor under `decision/`:

| Module | Provider |
|---|---|
| `decision/jev` | TypeSafe AI (Jev, System One) |
| `decision/openrouter` | OpenRouter (wraps `decision/jev`) — Jev via an OpenRouter key |

## Image Generation Providers

Under `image/`:

| Module | Provider | Models | Streaming |
|---|---|---|---|
| `image/openai` | OpenAI | DALL-E 2, DALL-E 3, GPT Image 1 / 1-mini / 1.5 / 2 | ✅ (gpt-image-*) |
| `image/azure` | Azure OpenAI | GPT Image 1.5 / 2 (Azure-hosted; api-key or Entra ID) | ✅ (gpt-image-*) |
| `image/gemini` | Google Gemini | Gemini 2.5 Flash Image, Gemini 3 Pro Image, Imagen 4 / 4 Ultra / 4 Fast | ❌ |
| `image/xai` | xAI | Grok 2 Image, Grok Imagine, Grok Imagine Pro | ❌ |
| `image/openrouter` | OpenRouter | 26 registered (GPT Image / GPT-5 Image, Gemini 2.5–3.1 Image, Seedream 4.5, FLUX.2, Recraft V4/V4.1, MAI-Image, Riverflow, Grok Imagine) + any routed id | ✅ (gpt-image-*, gpt-5-image*) |

## TTS (Text-to-Speech) Providers

Under `tts/`:

| Module | Provider | Streaming | Forced Alignment |
|---|---|---|---|
| `tts/openai` | OpenAI | (buffered) | ❌ |
| `tts/elevenlabs` | ElevenLabs | ✅ | ✅ |
| `tts/google` | Google Cloud | (buffered) | ❌ |
| `tts/azure` | Azure Speech | (buffered) | ❌ |
| `tts/deepgram` | Deepgram Aura | ✅ | ❌ |
| `tts/openrouter` | OpenRouter (wraps `tts/openai`) — 18 registered + any routed id | (buffered) | ❌ |

## STT (Speech-to-Text) Providers

Under `stt/`:

| Module | Provider | Streaming | Translation | Timestamps |
|---|---|---|---|---|
| `stt/openai` | OpenAI Whisper / GPT-4o Transcribe | ❌ | ✅ | ✅ |
| `stt/deepgram` | Deepgram | ✅ | ❌ | ✅ |
| `stt/assemblyai` | AssemblyAI v3 Universal-Streaming | ✅ | ❌ | ✅ |
| `stt/google` | Google Cloud Speech | ❌ | ❌ | ✅ |
| `stt/elevenlabs` | ElevenLabs Scribe v2 Realtime | ✅ | ❌ | ✅ |
| `stt/berget` | Berget AI (Whisper: Swedish/Norwegian/v3) | ❌ | ✅ | ✅ |
| `stt/openrouter` | OpenRouter (wraps `stt/openai`) — 13 registered + any routed id | ❌ | ❌ | ✅ (OpenAI-compatible upstreams) |

## Fill-in-the-Middle (FIM) Providers

Under `fim/`:

| Module | Provider | Streaming |
|---|---|---|
| `fim/mistral` | Mistral Codestral | ✅ |
| `fim/deepseek` | DeepSeek FIM | ✅ |

## Modality interfaces

Each modality also publishes a thin interface module so consumers can write
generic code:

- `llm` — `llm.LLM` interface, request/response types, retry helpers
- `embeddings` — `embeddings.Embedding` interface
- `rerankers` — `rerankers.Reranker` interface
- `decision` — `decision.Decider` interface
- `image` — `image.ImageGeneration` interface
- `tts` — `tts.Generation` interface (+ optional `tts.ForcedAlignmentProvider`)
- `stt` — `stt.SpeechToText` interface
- `fim` — `fim.FIM` interface

These interface modules carry no vendor SDKs.


---

## Providers > LLM

> Source: providers/llm.md

# LLM Providers

Each native LLM vendor is published as its own Go module under `llm/`. The
client returned from a vendor's `NewLLM(...)` satisfies the `llm.LLM`
interface, so once you've constructed it the rest of your code is
vendor-agnostic.

## Creating a client

```go
import (
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
)

client := llmopenai.NewLLM(
    llmopenai.WithAPIKey("your-api-key"),
    llmopenai.WithModel(llmopenai.Models[llmopenai.GPT4o]),
    llmopenai.WithMaxTokens(1000),
)
```

`llmopenai.NewLLM` uses the Responses API, which accepts function tools on every
OpenAI model. Options that exist only on Chat Completions (`WithStopSequences`,
`WithSeed`, `WithFrequencyPenalty`, `WithPresencePenalty`, `WithLogitBias`,
`WithN`, `WithLogprobs`) are rejected with an error rather than ignored; use
`llmopenai.NewChatCompletionsLLM` for OpenAI-compatible endpoints that need them.

For Anthropic instead:

```go
import llmanthropic "github.com/joakimcarlsson/ai/llm/anthropic"

client := llmanthropic.NewLLM(
    llmanthropic.WithAPIKey("..."),
    llmanthropic.WithModel(anthropic.Models[anthropic.Claude45Sonnet]),
    llmanthropic.WithMaxTokens(1000),
)
```

## Sending messages

```go
import "github.com/joakimcarlsson/ai/message"

response, err := client.SendMessages(ctx, []message.Message{
    message.NewUserMessage("Hello, how are you?"),
}, nil)
fmt.Println(response.Content)
```

## Streaming

```go
import "github.com/joakimcarlsson/ai/types"

stream := client.StreamResponse(ctx, messages, nil)

for event := range stream {
    switch event.Type {
    case types.EventContentDelta:
        fmt.Print(event.Content)
    case types.EventComplete:
        fmt.Printf("\nTokens: %d in / %d out\n",
            event.Response.Usage.InputTokens,
            event.Response.Usage.OutputTokens)
    case types.EventError:
        log.Fatal(event.Error)
    }
}
```

## Multimodal (images)

```go
imageData, _ := os.ReadFile("image.png")

msg := message.NewUserMessage("What's in this image?")
msg.AddBinary("image/png", imageData)

response, err := client.SendMessages(ctx, []message.Message{msg}, nil)
```

## Common options

Every vendor exports the standard set:

```go
llmopenai.WithAPIKey("...")
llmopenai.WithModel(openai.Models[openai.GPT4o])
llmopenai.WithMaxTokens(2000)
llmopenai.WithTemperature(0.7)
llmopenai.WithTopP(0.9)
llmopenai.WithTopK(40)
llmopenai.WithStopSequences("STOP", "END")
llmopenai.WithTimeout(30 * time.Second)
llmopenai.WithToolChoice(llm.ToolChoice{Mode: llm.ToolChoiceRequired})
```

> **`WithToolChoice`:**
> `WithToolChoice` is shared by the OpenAI, Anthropic, and Gemini modules
> (OpenAI-compatible providers inherit it through `llm/openai`). It takes the
> vendor-neutral `llm.ToolChoice` type: `Mode` is `ToolChoiceAuto` (default),
> `ToolChoiceNone`, `ToolChoiceRequired`, or `ToolChoiceSpecific` with a `Name`.
> It maps to each provider's native field (`tool_choice` for OpenAI/Anthropic,
> `toolConfig.functionCallingConfig` for Gemini) and is emitted only when tools
> are supplied. `ToolChoiceSpecific` with an empty `Name` is rejected before the
> request is sent.
>
> **`WithTopK` on the OpenAI client:**
> OpenAI's and Azure's own APIs reject `top_k` (HTTP 400), so `llmopenai.WithTopK`
> is sent only when a custom base URL points at an OpenAI-compatible provider that
> accepts it (Together, OpenRouter, Fireworks, ...); against OpenAI or Azure proper
> it has no effect. Native providers (Anthropic, Gemini, Bedrock) honor `WithTopK`
> directly. `WithStopSequences` sends every sequence provided (the OpenAI client
> caps at the API's limit of 4).
>
## Vendor-specific options

OpenAI:

```go
llmopenai.WithBaseURL("https://custom-endpoint")
llmopenai.WithExtraHeaders(map[string]string{"X-My-Header": "value"})
llmopenai.WithReasoningEffort(llmopenai.ReasoningEffortHigh)
llmopenai.WithFrequencyPenalty(0.5)
llmopenai.WithPresencePenalty(0.5)
llmopenai.WithSeed(42)
llmopenai.WithParallelToolCalls(false)
llmopenai.WithLogitBias(map[string]int{"1212": 5, "50256": -100})  // bias/ban tokens by id
llmopenai.WithLogprobs(3)                                          // logprobs:true + top_logprobs:3
llmopenai.WithN(3)                                                 // n completions per request
```

> **Sampling knobs that change the response shape:**
> `WithLogitBias`, `WithLogprobs`, and `WithN` live on the OpenAI client and so
> also cover every OpenAI-compatible provider (Groq, OpenRouter, xAI, Together,
> Fireworks, DeepSeek, Mistral, Ollama). They are emitted only when set, and
> are **OpenAI-only**: Anthropic supports none of them, and Gemini's
> `candidateCount` (the `n` equivalent) is out of scope — those providers never
> receive the fields.
>
> - `WithLogitBias` maps token IDs (tokenizer ids, OpenAI's wire shape) to a
>   bias from -100 (ban) to 100 (force).
> - `WithLogprobs(n)` requests per-token log probabilities with up to `n`
>   alternatives per position; the result lands on `Response.LogProbs`
>   (`[]llm.TokenLogProb`), nil when not requested.
> - `WithN(n)` requests `n` completions; all land on `Response.Choices`
>   (`[]llm.Choice`). The top-level `Content` / `FinishReason` / `ToolCalls` /
>   `LogProbs` mirror choice 0, so single-completion callers are unaffected
>   (`Choices` is empty when `n` is unset or `1`). When streaming with `n > 1`,
>   content and thinking deltas are emitted for choice 0 only; every choice is
>   delivered on `Response.Choices` of the final `EventComplete` response.
>
> `logit_bias` is rejected by reasoning-tier models (the gpt-5 family) with an
> HTTP 400; use a classic chat model such as `gpt-4o-mini` when you need it.
>
Anthropic:

```go
llmanthropic.WithBedrock(true)              // route through AWS Bedrock
llmanthropic.WithDisableCache()
llmanthropic.WithReasoningEffort(llmanthropic.ReasoningEffortHigh)
```

Gemini:

```go
import llmgemini "github.com/joakimcarlsson/ai/llm/gemini"

llmgemini.WithThinkingLevel(llmgemini.ThinkingLevelHigh)
llmgemini.WithFrequencyPenalty(0.5)
llmgemini.WithSeed(42)
```

Gemini context caching stores a system instruction, tools and history once
and bills later requests that reference it at the cached rate. Create the
cache through `llm.ContextCacheProvider`, which the clients from
`llmgemini.NewLLM` and `llmvertexai.NewLLM` implement, then attach it to a
client. Requests that use a cache must not pass system messages or tools of
their own; they fail with `llmgemini.ErrCachedContentConflict`.

```go
creator := llmgemini.NewLLM(
    llmgemini.WithAPIKey(os.Getenv("GEMINI_API_KEY")),
    llmgemini.WithModel(model),
    llmgemini.WithCacheTTL(time.Hour),
    llmgemini.WithToolChoice(llm.ToolChoice{Mode: llm.ToolChoiceAuto}),
)
name, err := creator.(llm.ContextCacheProvider).CreateCache(
    ctx, []message.Message{message.NewSystemMessage(longPrompt)}, tools,
)

client := llmgemini.NewLLM(
    llmgemini.WithAPIKey(os.Getenv("GEMINI_API_KEY")),
    llmgemini.WithModel(model),
    llmgemini.WithCachedContent(name),
)
resp, err := client.SendMessages(ctx, userMessages, nil)
```

## Provider built-in tools

Server-side built-in tools (web search, code execution, file search) run
inside the provider's infrastructure. They're opt-in per-client; results land
inline in `Response.Content`, with structured metadata under
`Response.ProviderMetadata`. See [Tool Calling](../advanced/tools.md#provider-built-in-tools)
for the full picture; below is the per-provider surface.

Anthropic — `web_search`:

```go
llmanthropic.WithWebSearch(llmanthropic.WebSearchConfig{
    MaxUses:        5,
    AllowedDomains: []string{"go.dev"},
    BlockedDomains: nil,
    UserLocation: &llmanthropic.WebSearchUserLocation{
        City: "Stockholm", Country: "SE",
    },
})
```

Gemini — `google_search`, `code_execution`, `url_context`:

```go
llmgemini.WithGoogleSearch()
llmgemini.WithCodeExecution()
llmgemini.WithURLContext()
```

OpenAI — `web_search`, `file_search`, `code_interpreter`. `NewLLM` uses the
Responses API, which is where OpenAI exposes server-side built-in tools:

```go
client := llmopenai.NewLLM(
    llmopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
    llmopenai.WithModel(openai.Models[openai.GPT5]),
    llmopenai.WithMaxTokens(1024),
    llmopenai.WithWebSearch(llmopenai.WebSearchOpts{
        SearchContextSize: llmopenai.SearchContextMedium,
    }),
    llmopenai.WithFileSearch("vs_abc123"),
    llmopenai.WithCodeInterpreter(),
)
```

`WithWebSearchPreview` is also available for models that don't yet support
the newer `web_search` tool.

Groq — `browser_search`, `code_execution`, `visit_website` (requires a
`groq/compound*` model via the dedicated `NewCompoundLLM`):

```go
import llmgroq "github.com/joakimcarlsson/ai/llm/groq"

client := llmgroq.NewCompoundLLM(
    llmgroq.WithCompoundAPIKey(os.Getenv("GROQ_API_KEY")),
    llmgroq.WithCompoundModel(llm.Model{APIModel: "groq/compound"}),
    llmgroq.WithBrowserSearch(llmgroq.BrowserSearchOpts{
        Country:       "us",
        IncludeImages: true,
    }),
    llmgroq.WithCodeExecution(),
    llmgroq.WithVisitWebsite(),
)
```

The regular `llmgroq.NewLLM` wrapper stays available for OpenAI-compatible
chat without built-ins.

### Stateless Responses and Azure

Set `llmopenai.WithStore(false)` on an OpenAI client to
request encrypted reasoning for reasoning models. Storage is omitted by default.
Keep `response.ReasoningItems` in the assistant message's parts when resending
history, including after tool results; streaming exposes the same items on the
completion event's response. Items without encrypted content are dropped when
replaying history, and no `previous_response_id` is sent. The agent preserves
these items automatically in both chat and streaming tool loops.

Azure uses the same Responses implementation with Azure authentication and
model metadata resolved from the deployment name:

```go
import (
    llmazure "github.com/joakimcarlsson/ai/llm/azure"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
)

client := llmazure.NewLLM(
    llmazure.WithEndpoint(os.Getenv("AZURE_OPENAI_ENDPOINT")),
    llmazure.WithAPIKey(os.Getenv("AZURE_OPENAI_API_KEY")),
    llmazure.WithDeployment("gpt-6.1-sol-prod"),
    llmazure.WithStore(false),
    llmazure.WithMaxTokens(1024),
    llmazure.WithOpenAIOptions(
        llmopenai.WithReasoningEffort(llmopenai.ReasoningEffortMedium),
    ),
)
```

Omit the API key to use `DefaultAzureCredential`. A resource endpoint defaults to
`/openai/v1/responses`; an endpoint already ending in `/openai/v1` also works.
Encrypted reasoning requires **v1**, without an `api-version` query parameter;
dated previews through `2025-04-01-preview` lack it, as documented in the
[Azure API version lifecycle](https://learn.microsoft.com/en-us/azure/foundry/openai/api-version-lifecycle).
An explicit dated `WithAPIVersion` uses `/openai/responses` with that version.

For direct LLM calls, retain the returned items alongside the output:

```go
assistant := message.NewAssistantMessage()
for _, item := range response.ReasoningItems {
    assistant.Parts = append(assistant.Parts, item)
}
assistant.AppendContent(response.Content)
assistant.AppendToolCalls(response.ToolCalls)
messages = append(messages, assistant)
```

### xAI built-in tools

xAI — `web_search`, `x_search`, `code_execution` via the Responses API:

```go
import llmxai "github.com/joakimcarlsson/ai/llm/xai"

client := llmxai.NewLLM(
    llmxai.WithAPIKey(os.Getenv("XAI_API_KEY")),
    llmxai.WithModel(xai.Models[xai.Grok4]),
    llmxai.WithWebSearch(llmxai.WebSearchOpts{
        SearchContextSize: llmxai.SearchContextMedium,
    }),
    llmxai.WithXSearch(llmxai.XSearchOpts{
        AllowedXHandles: []string{"xai"},
        FromDate:        "2026-01-01",
    }),
    llmxai.WithCodeExecution(),
)
```

## Cross-vendor wrappers

`llm/azure` (Azure OpenAI), `llm/vertexai` (Gemini on Vertex), and
`llm/bedrock` (Claude on Bedrock) are thin wrappers that delegate to their
underlying vendor module:

```go
import llmazure "github.com/joakimcarlsson/ai/llm/azure"

client := llmazure.NewLLM(
    llmazure.WithAPIKey(os.Getenv("AZURE_OPENAI_KEY")),
    llmazure.WithEndpoint("https://my-resource.openai.azure.com"),
    llmazure.WithDeployment("my-chat-deployment"),
)
```

```go
import llmbedrock "github.com/joakimcarlsson/ai/llm/bedrock"

// Region is read from $AWS_REGION (or $AWS_DEFAULT_REGION).
client := llmbedrock.NewLLM(
    llmbedrock.WithModel(anthropic.Models[anthropic.Claude45Sonnet]),
    llmbedrock.WithMaxTokens(2000),
)
```

Prompt caching is on by default on Bedrock: the underlying Anthropic client's
`cache_control` breakpoints reach Bedrock and populate `CacheReadTokens` /
`CacheCreationTokens` in the response usage. Pass `llmbedrock.WithDisableCache()`
to opt out. (Newer Claude models require at least 4096 cached tokens per
checkpoint before a cache hit is recorded.)

```go
import llmvertex "github.com/joakimcarlsson/ai/llm/vertexai"

client := llmvertex.NewLLM(
    llmvertex.WithProject(os.Getenv("VERTEXAI_PROJECT")),
    llmvertex.WithLocation(os.Getenv("VERTEXAI_LOCATION")),
    llmvertex.WithModel(llmvertex.Models[llmvertex.Gemini31Pro]),
)
```

## OpenAI-compatible providers (BYOM)

Mistral, Ollama, LocalAI and any other endpoint that implements Chat Completions
but not the Responses API — point `NewChatCompletionsLLM` at the right base URL.
`NewLLM` is for OpenAI itself and uses the Responses API:

```go
compatible := llmopenai.NewChatCompletionsLLM(
    llmopenai.WithAPIKey(os.Getenv("OPENROUTER_API_KEY")),
    llmopenai.WithBaseURL("https://openrouter.ai/api/v1"),
    llmopenai.WithModel(openai.Models[openai.GPT5]),
)
```

Groq and xAI are published as their own modules (`llm/groq`, `llm/xai`) so
they can expose vendor-specific built-in tools on top of the OpenAI-compatible
surface. Groq's `NewLLM` is a thin Chat Completions wrapper and `NewCompoundLLM`
exposes its built-in tools; xAI's `NewLLM` targets the Responses API with its built-in tools.

Berget AI (Swedish, EU-hosted; open-weight models) ships as `llm/berget`, a
thin wrapper pinned to `https://api.berget.ai/v1`. Pricing in the `model`
catalog is in EUR (`BergetModels`, `BergetEmbeddingModels`,
`BergetRerankerModels`, `BergetTranscriptionModels`):

Like the other OpenAI-compatible wrappers, `llm/berget` aliases `Option` but
does not re-export the option constructors; pass the standard ones from
`llm/openai`:

```go
import (
    llmberget "github.com/joakimcarlsson/ai/llm/berget"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
)

client := llmberget.NewLLM(
    llmopenai.WithAPIKey(os.Getenv("BERGET_API_KEY")),
    llmopenai.WithModel(llmberget.Models[llmberget.GPTOSS120B]),
    llmopenai.WithMaxTokens(1000),
)
```

For a managed registry of these, see [BYOM](../advanced/byom.md).

## Tracing

Every vendor's `NewLLM(...)` returns a tracing-wrapped client. Spans + metrics
are emitted automatically via OpenTelemetry. See [Tracing](../advanced/tracing.md)
for setup.

## Reasoning history across providers

Reasoning state travels in the `message.Message` history: keep
`response.ReasoningItems` in the assistant message's parts (the agent does this
in chat and streaming tool loops). Items serialize to JSON unchanged, and the
client keeps no hidden conversation state. Encrypted Responses replay and
plaintext Chat replay are separate capabilities.

| Provider | Mechanism | Configuration |
| --- | --- | --- |
| xAI Responses | Encrypted items, `store: false` plus `include: ["reasoning.encrypted_content"]`, replayed before the matching output and function calls | `xai.WithStore(false)`; built-in tools are unaffected |
| OpenRouter Responses | Same encrypted replay on the shared OpenAI Responses client | `openrouter.NewLLM`; storage defaults to false and `previous_response_id` is never sent. The API is stateless-only and rejects `store: true` |
| DeepSeek | `reasoning_content` on every assistant turn of tool-enabled thinking conversations, with content and tool calls | On for models with `CanReason` |
| Fireworks | `reasoning_content` for interleaved or preserved thinking; encrypted reasoning is never sent | On for models with `CanReason`; `fireworks.WithReasoningHistory(fireworks.ReasoningHistoryPreserved)` |
| Together | Reasoning under the field the model returned it in, `reasoning` or `reasoning_content` | On for models with `CanReason`; `together.WithPreservedThinking(true)` sends `chat_template_kwargs.clear_thinking=false` |
| Cerebras | The assistant `reasoning` field, for Qwen models only | On for Qwen models; `cerebras.WithClearThinking(bool)` sets `clear_thinking` and is ignored for other families |

`openai.WithReasoningContentReplay` always overrides a provider's default.
Groq, Ollama, Perplexity, Mistral and Berget receive no reasoning-history
fields.

## Reasoning effort and usage

Leaving effort and thinking options unset sends no reasoning parameter; the
provider default applies. Options are sent only for models with `CanReason`.
Accepted values depend on the model, including whether thinking can be disabled.
OpenAI-compatible wrappers accept `openai.Option`, so their effort control is
`openai.WithReasoningEffort` unless a provider option is listed below.

`Usage.ReasoningTokens` is part of `Usage.OutputTokens`, including during
streaming. Add output tokens to a total only once. Gemini and Vertex AI return
candidate and thought counts separately; the client combines them into output
usage. Perplexity Sonar Deep Research also reports a separate reasoning count,
which is included once in output usage, but charges it at a different rate.
Its exact bill requires the provider's cost breakdown rather than pricing all
output tokens at the model's one output rate.

| Provider | Control and lowest/off setting | Reasoning-token count |
| --- | --- | --- |
| [OpenAI](https://github.com/openai/openai-go/blob/main/responses/response.go) | `WithReasoningEffort`; `ReasoningEffortNone` disables reasoning where supported | Chat `completion_tokens_details.reasoning_tokens`; Responses `output_tokens_details.reasoning_tokens` |
| [Azure](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/reasoning) | `WithReasoningEffort(openai.ReasoningEffortNone)`; inferred deployment capabilities or `WithReasoning(true)` gate it | Inherits OpenAI Chat usage |
| [xAI](https://docs.x.ai/developers/model-capabilities/text/reasoning) | `WithReasoningEffort(xai.ReasoningEffortLow)`. Supports medium/high/xhigh where accepted; reasoning cannot be disabled | Inherits Chat details; Responses `output_tokens_details.reasoning_tokens` |
| [Anthropic](https://github.com/anthropics/anthropic-sdk-go/blob/main/message.go) | `WithReasoningEffort(ReasoningEffortLow)` minimizes adaptive thinking; `WithThinkingBudget(0)` explicitly disables it. Positive budgets enable manual thinking on supported models and override adaptive effort | `output_tokens_details.thinking_tokens`, including the final streaming message delta |
| [Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/claude-messages-extended-thinking.html) | Claude forwards `WithReasoningEffort` and `WithThinkingBudget` to Anthropic; budget zero disables thinking | Inherits Claude's thinking count when Bedrock supplies it. Other catalog vendors are not implemented by this client |
| [Gemini](https://ai.google.dev/gemini-api/docs/thinking) | `WithThinkingLevel(ThinkingLevelMinimal)` is the lowest level; `WithThinkingBudget(0)` disables thinking on supported models (not 2.5 Pro). Budget `-1` is dynamic | `thoughtsTokenCount`; output combines candidates and thoughts |
| [Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/thinking) | Forwards `WithThinkingLevel` and `WithThinkingBudget` to Gemini | Inherits Gemini usage |
| [DeepSeek](https://api-docs.deepseek.com/guides/thinking_mode/) | `WithReasoningEffort(ReasoningEffortNone)` / `WithThinking(false)` disables thinking; low/high/max set depth on models that accept them | Inherits Chat details; top-level `reasoning_tokens` is also recognized when details are absent |
| [Groq](https://console.groq.com/docs/api-reference) | OpenAI effort; none disables supported Qwen models, low is the lowest GPT-OSS level | `completion_tokens_details.reasoning_tokens` through the OpenAI client |
| [Cerebras](https://inference-docs.cerebras.ai/capabilities/reasoning) | OpenAI effort; none disables supported models, low minimizes GPT-OSS reasoning; always-thinking models can ignore the control | `completion_tokens_details.reasoning_tokens` through the OpenAI client |
| [OpenRouter](https://openrouter.ai/docs/api/api-reference/chat/send-chat-completion-request) | OpenAI effort shorthand maps to `reasoning.effort`; none requests no reasoning where the routed model supports it | `completion_tokens_details.reasoning_tokens` through the OpenAI client |
| [Perplexity](https://github.com/perplexityai/perplexity-py/blob/main/src/perplexity/types/shared/usage_info.py) | OpenAI effort; minimal/low minimize reasoning where supported; no documented off value for Sonar Deep Research | Top-level `reasoning_tokens`; [Sonar Deep Research bills it separately](https://docs.perplexity.ai/docs/getting-started/pricing) |
| [Together](https://docs.together.ai/reference/chat-completions) | OpenAI effort low/medium/high; `WithThinking(false)` sends `reasoning.enabled=false` on models supporting the toggle | No separate reasoning count in the documented completion usage |
| [Fireworks](https://docs.fireworks.ai/api-reference/post-chatcompletions) | OpenAI effort; none disables reasoning on supported models, low minimizes always-thinking models | No separate reasoning count in the documented completion usage |
| [Mistral](https://docs.mistral.ai/studio/conversations/reasoning) | OpenAI effort; none minimizes thinking and omits the trace on supported Mistral models; hosted models may require low instead | No separate reasoning count in the documented completion usage |
| [Ollama](https://github.com/ollama/ollama/blob/main/docs/api/openai-compatibility.mdx) | OpenAI effort; none maps to disabled thinking where supported, low is the lowest GPT-OSS level | The compatible endpoint reports completion totals without a separate thinking count |
| [Berget](https://api.berget.ai/) | The documented compatible API has no effort/thinking control; model defaults apply | No documented separate reasoning count |

Providers without a documented breakdown leave `ReasoningTokens` at zero.
The compatible client still maps standard reasoning details if an endpoint
supplies them; it does not estimate counts from reasoning text.

Azure Responses effort forwarding depends on the Azure Responses constructor
in [#294](https://github.com/JoakimCarlsson/ai/issues/294), which is not present
in this checkout. OpenAI Responses usage and effort handling are ready for
that constructor to reuse.


---

## Providers > Embeddings

> Source: providers/embeddings.md

# Embeddings

Each native embedding vendor is its own Go module under `embeddings/`.

## Text embeddings

OpenAI:

```go
import (
    "github.com/joakimcarlsson/ai/embeddings"
    embopenai "github.com/joakimcarlsson/ai/embeddings/openai"
)

embedder := embopenai.NewEmbedding(
    embopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
    embopenai.WithModel(embopenai.Models[embopenai.TextEmbedding3Small]),
)

resp, err := embedder.GenerateEmbeddings(ctx, []string{
    "Hello, world",
    "How are you?",
})
fmt.Printf("Generated %d embeddings\n", len(resp.Embeddings))
```

Voyage:

```go
import embvoyage "github.com/joakimcarlsson/ai/embeddings/voyage"

embedder := embvoyage.NewEmbedding(
    embvoyage.WithAPIKey(os.Getenv("VOYAGE_API_KEY")),
    embvoyage.WithModel(voyage.Models[voyage.Voyage35]),
    embvoyage.WithInputType("document"),  // or "query"
)
```

Cohere:

```go
import embcohere "github.com/joakimcarlsson/ai/embeddings/cohere"

embedder := embcohere.NewEmbedding(
    embcohere.WithAPIKey(os.Getenv("COHERE_API_KEY")),
    embcohere.WithModel(cohere.Models[cohere.EmbedEnV3]),
    embcohere.WithInputType("search_document"),
)
```

Gemini, Mistral, Bedrock follow the same shape — `embeddings/gemini`,
`embeddings/mistral`, `embeddings/bedrock`.

Berget AI (EU-hosted, OpenAI-compatible; pricing in EUR):

`embeddings/berget` aliases `Option` but does not re-export the option
constructors; pass the standard ones from `embeddings/openai`:

```go
import (
    embberget "github.com/joakimcarlsson/ai/embeddings/berget"
    embopenai "github.com/joakimcarlsson/ai/embeddings/openai"
)

embedder := embberget.NewEmbedding(
    embopenai.WithAPIKey(os.Getenv("BERGET_API_KEY")),
    embopenai.WithModel(embberget.Models[embberget.E5Large]),
)
```

## Multimodal embeddings (Voyage)

```go
inputs := []embeddings.MultimodalInput{
    {
        Content: []embeddings.MultimodalContent{
            {Type: "text", Text: "A photo of a cat"},
            {Type: "image_url", ImageURL: "https://example.com/cat.jpg"},
        },
    },
}

resp, err := embedder.GenerateMultimodalEmbeddings(ctx, inputs)
```

## Contextualized embeddings (Voyage)

For document chunks where each chunk should be embedded with awareness of
its surrounding context:

```go
documents := [][]string{
    {"chunk 1 of doc 1", "chunk 2 of doc 1", "chunk 3 of doc 1"},
    {"chunk 1 of doc 2", "chunk 2 of doc 2"},
}

resp, err := embedder.GenerateContextualizedEmbeddings(ctx, documents)
// resp.DocumentEmbeddings[i][j] is the embedding of doc i, chunk j
```

## Per-call input type

The optional `inputType` variadic argument overrides the constructor default:

```go
resp, err := embedder.GenerateEmbeddings(ctx, texts, "query")
```

## Bedrock (Titan + Cohere on Bedrock)

```go
import embbedrock "github.com/joakimcarlsson/ai/embeddings/bedrock"

embedder := embbedrock.NewEmbedding(
    embbedrock.WithModel(bedrock.Models[bedrock.TitanEmbedV2]),
    embbedrock.WithRegion("us-east-1"),
    // or embbedrock.WithProfile("my-aws-profile"),
)
```

The vendor is detected from the model's API ID (`cohere.*` vs everything-else
defaults to Titan).


---

## Providers > Image Generation

> Source: providers/image-generation.md

# Image Generation

The `image` modality. Vendors live under `image/`.

`image.GenerateImage(ctx, prompt)` takes only a prompt — every vendor knob
(size, aspect ratio, quality, response format, style, seed, safety, …) lives
on the vendor's `Options` and is set at construction. Image generation is
"configure once, prompt many" and vendor request bodies don't share enough
common shape to support a portable per-call surface.

## OpenAI

```go
import (
    "github.com/joakimcarlsson/ai/image"
    imageopenai "github.com/joakimcarlsson/ai/image/openai"
)

client := imageopenai.NewGeneration(
    imageopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
    imageopenai.WithModel(imageopenai.Models[imageopenai.GPTImage15]),
    imageopenai.WithSize(imageopenai.Size1024x1024),
    imageopenai.WithQuality(imageopenai.QualityHigh),
    imageopenai.WithBackground(imageopenai.BackgroundTransparent),
    imageopenai.WithOutputFormat(imageopenai.OutputFormatPNG),
)

resp, err := client.GenerateImage(ctx, "A serene mountain landscape at sunset")
if err != nil {
    log.Fatal(err)
}

data, _ := image.DecodeBase64Image(resp.Images[0].ImageBase64)
os.WriteFile("output.png", data, 0644)
```

Full option set (typed enums — see the package's exported `Size`, `Quality`,
`Background`, `Moderation`, `OutputFormat` types):

```go
imageopenai.WithN(int)                                  // 1–10
imageopenai.WithSize(imageopenai.Size1024x1024)         // 1024x1024 | 1024x1536 | 1536x1024 | auto
imageopenai.WithQuality(imageopenai.QualityHigh)        // low | medium | high | auto
imageopenai.WithBackground(imageopenai.BackgroundAuto)  // transparent | opaque | auto — gpt-image-1.5 only (gpt-image-2 rejects)
imageopenai.WithModeration(imageopenai.ModerationAuto)  // auto | low
imageopenai.WithOutputFormat(imageopenai.OutputFormatPNG) // png | jpeg | webp
imageopenai.WithOutputCompression(int)                  // 0–100 — jpeg/webp only
imageopenai.WithUser(string)                            // end-user identifier
imageopenai.WithStreamingOptions(...)                   // partial-image count for streaming
```

Supported models: `gpt-image-1.5` and `gpt-image-2`. DALL-E 2/3 and gpt-image-1
(plus mini) are removed; pricing-registry entries dropped along with the
matching package code paths.

## Azure OpenAI

`image/azure` is to `image/openai` what `llm/azure` is to `llm/openai` — a thin
wrapper that reuses the OpenAI request building and overrides only endpoint and
auth. It resolves in three branches, mirroring `llm/azure`:

- no endpoint set → plain `image/openai` (optionally with `WithAPIKey`);
- an endpoint containing `/openai/v1` (the OpenAI-compatible surface) → routed
  through `image/openai` with `WithBaseURL`;
- a classic `https://<resource>.openai.azure.com` endpoint → the `api-key`
  header plus the `?api-version=` query param.

```go
import imageazure "github.com/joakimcarlsson/ai/image/azure"

client := imageazure.NewGeneration(
    imageazure.WithEndpoint("https://my-resource.openai.azure.com"),
    imageazure.WithAPIVersion("2025-04-01-preview"),
    imageazure.WithAPIKey(os.Getenv("AZURE_OPENAI_API_KEY")),
    imageazure.WithModel(openai.Models[openai.GPTImage2]),
    imageazure.WithSize(imageazure.Size1024x1024),
    imageazure.WithOutputFormat(imageazure.OutputFormatPNG),
)

resp, err := client.GenerateImage(ctx, "A serene mountain landscape at sunset")
```

Auth resolution matches `llm/azure`: a static `api-key` is used when
`WithAPIKey` is set; otherwise the client falls back to
`DefaultAzureCredential` (Entra ID / managed identity) automatically — omit
`WithAPIKey` and ensure `az login` or a managed identity is available.

The `image/openai` enum types **and their values** are re-exported
(`imageazure.Size1024x1024`, `imageazure.QualityHigh`,
`imageazure.OutputFormatPNG`, …), and the full option set is forwarded:
`WithSize`, `WithQuality`, `WithN`, `WithBackground`, `WithModeration`,
`WithOutputFormat`, `WithOutputCompression`, `WithUser`, `WithExtraHeaders`,
`WithStreamingOptions`, `WithTimeout`. Returned clients are tracing-wrapped like
`image/openai`.

## Gemini / Imagen

```go
import imagegemini "github.com/joakimcarlsson/ai/image/gemini"

client := imagegemini.NewGeneration(
    imagegemini.WithAPIKey(os.Getenv("GEMINI_API_KEY")),
    imagegemini.WithModel(gemini.Models[gemini.Imagen4]),
    imagegemini.WithAspectRatio(imagegemini.AspectRatio16x9),
    imagegemini.WithN(2),
)

resp, err := client.GenerateImage(ctx, "A cyberpunk cityscape")
for i, img := range resp.Images {
    data, _ := image.DecodeBase64Image(img.ImageBase64)
    os.WriteFile(fmt.Sprintf("image_%d.png", i), data, 0644)
}
```

Full option set (Imagen-only fields are ignored when the active model is a
Gemini Image variant):

```go
import "google.golang.org/genai"

imagegemini.WithN(int32)                                          // Imagen: 1–4
imagegemini.WithAspectRatio(imagegemini.AspectRatio16x9)          // see imagegemini.AspectRatio*
imagegemini.WithNegativePrompt(string)                            // Imagen only
imagegemini.WithSeed(int32)                                       // Imagen only (requires AddWatermark=false)
imagegemini.WithPersonGeneration(genai.PersonGenerationAllowAdult) // both paths
imagegemini.WithSafetyFilterLevel(genai.SafetyFilterLevelBlockOnlyHigh) // Imagen only
imagegemini.WithLanguage(genai.ImagePromptLanguageEn)             // Imagen only
imagegemini.WithEnhancePrompt(bool)                               // Imagen only
imagegemini.WithImageSize(imagegemini.ImageSize2K)                // 1K | 2K | 4K — model-dependent
imagegemini.WithIncludeRAIReason(bool)                            // Imagen only
imagegemini.WithOutputMIMEType(imagegemini.OutputMIMETypePNG)     // image/png | image/jpeg — Imagen only
imagegemini.WithOutputCompressionQuality(int32)                   // 0–100 — Imagen jpeg only
```

## xAI Grok Imagine

```go
import imagexai "github.com/joakimcarlsson/ai/image/xai"

client := imagexai.NewGeneration(
    imagexai.WithAPIKey(os.Getenv("XAI_API_KEY")),
    imagexai.WithModel(xai.Models[xai.GrokImagineImage]),
    imagexai.WithAspectRatio(imagexai.AspectRatio16x9),
    imagexai.WithResolution(imagexai.Resolution2K),
    imagexai.WithResponseFormat(imagexai.ResponseFormatBase64),
)

resp, err := client.GenerateImage(ctx, "A neon-lit street market")
```

Full option set:

```go
imagexai.WithN(int)                                       // 1–10
imagexai.WithAspectRatio(imagexai.AspectRatio16x9)        // 14 values — see imagexai.AspectRatio*
imagexai.WithResolution(imagexai.Resolution2K)            // 1K | 2K
imagexai.WithResponseFormat(imagexai.ResponseFormatBase64) // url | b64_json
imagexai.WithUser(string)                                  // end-user identifier
```

## OpenRouter

Unlike the TTS and STT OpenRouter packages, `image/openrouter` is a full
implementation rather than a base-URL wrapper: OpenRouter's image endpoint is
`POST /api/v1/images`, a different path and body from OpenAI's
`/v1/images/generations`.

```go
import imageopenrouter "github.com/joakimcarlsson/ai/image/openrouter"

client := imageopenrouter.NewGeneration(
    imageopenrouter.WithAPIKey(os.Getenv("OPENROUTER_API_KEY")),
    imageopenrouter.WithModel(openrouter.Models[openrouter.Seedream45]),
    imageopenrouter.WithAspectRatio(imageopenrouter.AspectRatio16x9),
)

resp, err := client.GenerateImage(ctx, "A red panda astronaut")
data, _ := image.DecodeBase64Image(resp.Images[0].ImageBase64)
fmt.Println(resp.Images[0].MediaType) // image/png
fmt.Println(resp.Usage.Cost)         // 0.04 — dollars, as OpenRouter reported them
```

Full option set:

```go
imageopenrouter.WithN(int)                                          // 1–10, per-model ceiling
imageopenrouter.WithSize("2048x2048")                               // tier or explicit pixels
imageopenrouter.WithAspectRatio(imageopenrouter.AspectRatio16x9)    // 18 values
imageopenrouter.WithResolution(imageopenrouter.Resolution4K)        // 1K | 2K | 4K
imageopenrouter.WithQuality(imageopenrouter.QualityHigh)            // auto | low | medium | high
imageopenrouter.WithBackground(imageopenrouter.BackgroundTransparent) // auto | transparent | opaque
imageopenrouter.WithOutputFormat(imageopenrouter.OutputFormatWebP)  // png | jpeg | webp | svg
imageopenrouter.WithOutputCompression(int)                          // 0–100, webp/jpeg
imageopenrouter.WithSeed(int64)                                     // where supported
imageopenrouter.WithInputReferences(urls ...string)                 // image-to-image
imageopenrouter.WithProviderRouting(order []string, allowFallbacks bool)
imageopenrouter.WithRequestJSONField(key string, value any)         // escape hatch
imageopenrouter.WithHTTPClient(*http.Client)
imageopenrouter.WithExtraHeaders(map[string]string)                 // HTTP-Referer, X-Title
imageopenrouter.WithTimeout(time.Duration)
```

Not every model accepts every knob, and a model advertising a value does not mean
it accepts it in every combination. `seedream-4.5` lists `1K`, `2K` and `4K` but
enforces a floor of 3,686,400 output pixels, so `1K` and `2K` both draw an HTTP
400 at 16:9. Omitting `WithResolution` lets OpenRouter apply the model's own
default, which is why the examples above do. Query
`GET /api/v1/images/models` for a model's `supported_parameters` first.

There is no `WithModelFallbacks`. OpenRouter documents its `models` fallback
array for chat completions and `/api/v1/messages` only; on `/images` a
nonexistent primary answers `404 No model found` rather than falling through,
and the request schema silently drops fields it does not know. Fall back in
caller code with a second client instead. `WithProviderRouting` *is* documented
for this endpoint and is wired up.

### Using a model the registry does not define

`openrouter.Models` carries 26 known-good defaults, but
OpenRouter routes more than that and adds new models weekly. You never have to
wait for a release to use one. `WithModelID` takes any raw OpenRouter id:

```go
client := imageopenrouter.NewGeneration(
    imageopenrouter.WithAPIKey(os.Getenv("OPENROUTER_API_KEY")),
    imageopenrouter.WithModelID("krea/krea-2-large"),
    imageopenrouter.WithAspectRatio(imageopenrouter.AspectRatio4x5),
)
```

That is shorthand for `WithModel(image.GenerationModel{APIModel: id,
Provider: "openrouter"})`, so `Model()` reports the id and provider
and nothing else — no pricing, no supported-value lists. Generation works fine;
what breaks is anything that reads that metadata, such as a cost estimator or a
UI offering the caller a list of aspect ratios.

When you need those, describe the model yourself and it behaves exactly like a
registered entry, `DefaultAspectRatio` included:

```go
riverflowFast := image.GenerationModel{
    ID:       "openrouter.riverflow-v2.5-fast",
    Name:     "OpenRouter – Riverflow V2.5 Fast",
    Provider: "openrouter",
    APIModel: "sourceful/riverflow-v2.5-fast",
    Pricing: map[string]map[string]float64{
        "default": {"default": 0.019},
    },
    MaxPromptTokens:       4000,
    SupportedAspectRatios: []string{"1:1", "4:3", "16:9", "21:9", "auto"},
    DefaultAspectRatio:    "16:9",
    SupportedSizes:        []string{"1K", "2K"},
    DefaultSize:           "1K",
    SupportedQualities:    []string{"default"},
    DefaultQuality:        "default",
}

client := imageopenrouter.NewGeneration(
    imageopenrouter.WithAPIKey(os.Getenv("OPENROUTER_API_KEY")),
    imageopenrouter.WithModel(riverflowFast),
)
```

The real capability and pricing data for any id lives at
`GET /api/v1/images/models/<id>/endpoints`. `stt/openrouter` and
`tts/openrouter` expose the same `WithModelID` shorthand over
`stt.TranscriptionModel` and `tts.AudioModel`.

`examples/image/openrouter` runs all of this, including both custom-model forms.

`GenerateImageStreaming` is real here, not `ErrStreamingNotSupported`: models
whose descriptor reports `supports_streaming` (the `gpt-image-*` family today)
deliver `EventPartialImage` frames before the final `EventCompleted`.

Per-model capability data — including `SupportedAspectRatios` — lives on
`image.GenerationModel`. Inspect it to know what a given model accepts:

```go
m := gemini.Models[gemini.Imagen4]
fmt.Println(m.SupportedAspectRatios) // [1:1 3:4 4:3 9:16 16:9]
```

## Streaming partial images (OpenAI gpt-image-*)

```go
client := imageopenai.NewGeneration(
    imageopenai.WithAPIKey(...),
    imageopenai.WithModel(openai.Models[openai.GPTImage15]),
    imageopenai.WithStreamingOptions(imageopenai.StreamingOptions{PartialImages: 3}),
)

err := client.GenerateImageStreaming(ctx, prompt,
    func(event image.StreamEvent) error {
        switch event.Type {
        case image.EventPartialImage:
            data, _ := image.DecodeBase64Image(event.ImageBase64)
            os.WriteFile(fmt.Sprintf("partial_%d.png", event.PartialImageIndex), data, 0644)
        case image.EventCompleted:
            data, _ := image.DecodeBase64Image(event.ImageBase64)
            os.WriteFile("final.png", data, 0644)
        }
        return nil
    },
)
```

Returns `image.ErrStreamingNotSupported` if the model can't stream.

## Helpers

```go
// Download from URL
data, err := image.DownloadImage(resp.Images[0].ImageURL)

// Decode base64 payload
data, err := image.DecodeBase64Image(resp.Images[0].ImageBase64)
```


---

## Providers > Audio

> Source: providers/audio.md

# Text-to-Speech (TTS)

The `tts` modality (formerly "audio generation"). Each native vendor lives
under `tts/`.

## Basic usage

ElevenLabs:

```go
import (
    ttselevenlabs "github.com/joakimcarlsson/ai/tts/elevenlabs"
    "github.com/joakimcarlsson/ai/tts"
)

client := ttselevenlabs.NewGeneration(
    ttselevenlabs.WithAPIKey(os.Getenv("ELEVENLABS_API_KEY")),
    ttselevenlabs.WithModel(ttselevenlabs.Models[ttselevenlabs.TurboV2_5]),
    ttselevenlabs.WithVoiceID("EXAVITQu4vr4xnSDxMaL"),  // Rachel
)

resp, err := client.GenerateAudio(ctx, "Hello, how are you today?")
os.WriteFile("output.mp3", resp.AudioData, 0644)
```

OpenAI:

```go
import ttsopenai "github.com/joakimcarlsson/ai/tts/openai"

client := ttsopenai.NewGeneration(
    ttsopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
    ttsopenai.WithModel(openai.Models[openai.TTS1HD]),
    ttsopenai.WithVoice("nova"),
    ttsopenai.WithOutputFormat("mp3"),
)
```

Google Cloud, Azure Speech, Deepgram Aura follow the same shape.

OpenRouter is a thin wrapper over `tts/openai` that fixes the base URL, so one
key reaches OpenAI, Google, Mistral, Microsoft and Deepgram voices:

```go
import ttsopenrouter "github.com/joakimcarlsson/ai/tts/openrouter"

client := ttsopenrouter.NewGeneration(
    ttsopenai.WithAPIKey(os.Getenv("OPENROUTER_API_KEY")),
    ttsopenai.WithModel(openrouter.Models[openrouter.MAIVoice2]),
    ttsopenai.WithVoice("en-US-Harper:MAI-Voice-2"),
    ttsopenai.WithOutputFormat("mp3"),
)
```

There is no model-fallback option: OpenRouter documents its `models` fallback
array for chat completions only, and the audio endpoints ignore request fields
they do not recognise, so sending it would look like it worked while doing
nothing. `WithProviderRouting` is documented for `/audio/speech` and is wired up.

Two OpenRouter specifics: `response_format` defaults to `pcm` rather than
OpenAI's `mp3` and the only documented values are `mp3` and `pcm`; and `speed`
is honored only by upstreams that support it and silently ignored by the rest.
`openrouter.Models` carries 18 known-good defaults. OpenRouter routes
more than that, so for anything it does not define, `WithModelID` takes a raw
id:

```go
client := ttsopenrouter.NewGeneration(
    ttsopenai.WithAPIKey(os.Getenv("OPENROUTER_API_KEY")),
    ttsopenrouter.WithModelID("minimax/speech-2.8-hd"),
    ttsopenai.WithOutputFormat("mp3"),
)
```

`Model()` then reports only the id and provider — no per-character cost, no
format list. Pass a hand-built `tts.AudioModel` to `ttsopenai.WithModel`
instead when something downstream reads those fields.

## Streaming

ElevenLabs and Deepgram stream chunked audio:

```go
chunks, err := client.StreamAudio(ctx, "Hello world",
    tts.WithOptimizeStreamingLatency(3),
)

for chunk := range chunks {
    if chunk.Error != nil {
        log.Fatal(chunk.Error)
    }
    if chunk.Done {
        break
    }
    output.Write(chunk.Data)
}
```

The other vendors (`tts/openai`, `tts/google`, `tts/azure`) buffer the
non-streaming response into a single chunk for API parity.

## Voice listing

```go
voices, err := client.ListVoices(ctx)
for _, v := range voices {
    fmt.Printf("%s — %s (%s)\n", v.VoiceID, v.Name, v.Category)
}
```

## ElevenLabs voice settings

```go
resp, err := client.GenerateAudio(ctx, "Expressive line",
    tts.WithStability(0.75),
    tts.WithSimilarityBoost(0.85),
    tts.WithStyle(0.5),
    tts.WithSpeakerBoost(true),
)
```

## ElevenLabs alignment

`tts/elevenlabs.Client` also implements `tts.ForcedAlignmentProvider`. The
canonical alignment-enabled call:

```go
resp, err := client.GenerateAudio(ctx, "Hello world",
    tts.WithAlignmentEnabled(true),
)

for i, ch := range resp.Alignment.Characters {
    fmt.Printf("%s: %.2fs - %.2fs\n",
        ch,
        resp.Alignment.CharacterStartTimesSeconds[i],
        resp.Alignment.CharacterEndTimesSeconds[i],
    )
}
```

For aligning existing audio against a transcript:

```go
if fap, ok := client.(tts.ForcedAlignmentProvider); ok {
    audio, _ := os.ReadFile("recording.mp3")
    align, err := fap.GenerateForcedAlignment(ctx, audio,
        "the spoken transcript")

    for _, w := range align.Words {
        fmt.Printf("%s: %.2fs - %.2fs (loss=%.4f)\n",
            w.Text, w.Start, w.End, w.Loss)
    }
}
```

The type assertion succeeds against the wrapper returned from
`ttselevenlabs.NewGeneration` because the wrapper preserves the optional
sub-interface when the inner concrete client implements it.

## Common per-call options

```go
tts.WithOutputFormat("mp3_44100_128")   // ElevenLabs
tts.WithOutputFormat("LINEAR16")        // Google Cloud
tts.WithStability(0.75)
tts.WithSimilarityBoost(0.85)
tts.WithStyle(0.5)
tts.WithSpeakerBoost(true)
tts.WithOptimizeStreamingLatency(3)
tts.WithAlignmentEnabled(true)
```


---

## Providers > Speech-to-Text

> Source: providers/speech-to-text.md

# Speech-to-Text (STT)

The `stt` modality. Each native vendor lives under `stt/`.

## Basic transcription

OpenAI Whisper:

```go
import (
    sttopenai "github.com/joakimcarlsson/ai/stt/openai"
    "github.com/joakimcarlsson/ai/stt"
)

client := sttopenai.NewSpeechToText(
    sttopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
    sttopenai.WithModel(sttopenai.Models[sttopenai.Whisper1]),
)

audio, _ := os.ReadFile("audio.mp3")
resp, err := client.Transcribe(ctx, audio,
    stt.WithLanguage("en"),
)
fmt.Println(resp.Text)
```

Deepgram, AssemblyAI, Google Cloud Speech, ElevenLabs Scribe follow the same
shape via their respective vendor packages.

Berget AI (EU-hosted, OpenAI-compatible Whisper; pricing in EUR) is the same
shape via `stt/berget`, with Swedish (`BergetKBWhisperLarge`) and Norwegian
(`BergetNBWhisperLarge`) fine-tunes alongside `BergetFasterWhisperLargeV3`:

```go
import sttberget "github.com/joakimcarlsson/ai/stt/berget"

client := sttberget.NewSpeechToText(
    sttberget.WithAPIKey(os.Getenv("BERGET_API_KEY")),
    sttberget.WithModel(berget.Models[berget.KBWhisperLarge]),
)
```

OpenRouter is the same shape via `stt/openrouter`, reaching Whisper, GPT-4o
Transcribe, Voxtral, Fish Audio and Grok STT on one key:

```go
import sttopenrouter "github.com/joakimcarlsson/ai/stt/openrouter"

client := sttopenrouter.NewSpeechToText(
    sttopenai.WithAPIKey(os.Getenv("OPENROUTER_API_KEY")),
    sttopenai.WithModel(openrouter.Models[openrouter.Whisper1]),
)
```

Two OpenRouter specifics. `response_format="verbose_json"` — and with it the
`Segments` and `Words` arrays — is only accepted by the OpenAI-compatible
upstreams (OpenAI, Groq, Together); the rest answer HTTP 400, so pass
`stt.WithResponseFormat("json")` when routing elsewhere. And OpenRouter
publishes no `/audio/translations` route, so `Translate` returns
`sttopenrouter.ErrTranslationNotSupported` instead of issuing a doomed request.
`openrouter.Models` carries 13 known-good defaults. OpenRouter
routes more than that, so for anything it does not define, `WithModelID` takes a
raw id:

```go
client := sttopenrouter.NewSpeechToText(
    sttopenai.WithAPIKey(os.Getenv("OPENROUTER_API_KEY")),
    sttopenrouter.WithModelID("nvidia/parakeet-tdt-0.6b-v3"),
)

resp, err := client.Transcribe(ctx, audio, stt.WithResponseFormat("json"))
```

`Model()` then reports only the id and provider — no cost, no capability flags.
Pass a hand-built `stt.TranscriptionModel` to `sttopenai.WithModel` instead
when something downstream reads those fields.

## Translation (OpenAI only)

```go
resp, err := client.Translate(ctx, audio)  // returns English translation
```

## Streaming transcription

Deepgram, AssemblyAI, and ElevenLabs support real-time streaming over
WebSocket. The `stt.SpeechToText` interface exposes `StreamTranscribe`:

```go
import sttdeepgram "github.com/joakimcarlsson/ai/stt/deepgram"

client := sttdeepgram.NewSpeechToText(
    sttdeepgram.WithAPIKey(os.Getenv("DEEPGRAM_API_KEY")),
    sttdeepgram.WithModel(deepgram.Models[deepgram.Nova3]),
    sttdeepgram.WithSmartFormat(true),
    sttdeepgram.WithStreamInterimResults(true),
)

audioCh := make(chan []byte, 16)
go feedAudio(audioCh)  // your PCM frame source

results, err := client.StreamTranscribe(ctx, audioCh,
    stt.WithSampleRate(16000),
    stt.WithChannels(1),
)
if err != nil {
    log.Fatal(err)
}

for r := range results {
    if r.Error != nil {
        log.Fatal(r.Error)
    }
    if r.IsFinal {
        fmt.Println("FINAL:", r.Text)
    } else {
        fmt.Println("interim:", r.Text)
    }
}
```

## SupportsStreaming

For vendors that don't stream:

```go
if !client.SupportsStreaming() {
    // fall back to batch Transcribe
    resp, _ := client.Transcribe(ctx, audio)
}
```

`stt/openai` and `stt/google` return `false`. The streaming-capable vendors
(`stt/deepgram`, `stt/assemblyai`, `stt/elevenlabs`) return `true`.

## Per-call options

```go
stt.WithLanguage("en")
stt.WithPrompt("Domain-specific words: Claude, Anthropic, ...")
stt.WithResponseFormat("verbose_json")  // OpenAI
stt.WithTimestampGranularities("word", "segment")
stt.WithFilename("audio.wav")           // for format detection
stt.WithSampleRate(16000)               // streaming only
stt.WithChannels(1)                     // streaming only
```

## Vendor-specific options

Deepgram:

```go
sttdeepgram.WithPunctuate(true)
sttdeepgram.WithDiarize(true)
sttdeepgram.WithSmartFormat(true)
sttdeepgram.WithKeyterms("Claude", "Anthropic")
sttdeepgram.WithStreamEndpointingMs(300)
```

AssemblyAI:

```go
sttassemblyai.WithSpeakerLabels(true)
sttassemblyai.WithStreamEndOfTurnSilenceMs(700)
sttassemblyai.WithStreamFormatTurns(true)
```

ElevenLabs Scribe:

```go
sttelevenlabs.WithDiarize(true)
sttelevenlabs.WithNumSpeakers(2)
sttelevenlabs.WithStreamLanguageCode("en")
sttelevenlabs.WithStreamIncludeTimestamps(true)
```


---

## Providers > Rerankers

> Source: providers/rerankers.md

# Document Reranking

The `rerankers` modality. Vendors under `rerankers/`.

## Voyage

```go
import (
    rrvoyage "github.com/joakimcarlsson/ai/rerankers/voyage"
)

reranker := rrvoyage.NewReranker(
    rrvoyage.WithAPIKey(os.Getenv("VOYAGE_API_KEY")),
    rrvoyage.WithModel(rrvoyage.Models[rrvoyage.Rerank25Lite]),
    rrvoyage.WithTopK(5),
    rrvoyage.WithReturnDocuments(true),
)

query := "What is machine learning?"
documents := []string{
    "Machine learning is a subset of artificial intelligence.",
    "The weather today is sunny.",
    "Deep learning uses neural networks.",
}

resp, err := reranker.Rerank(ctx, query, documents)
for i, r := range resp.Results {
    fmt.Printf("Rank %d (score=%.4f): %s\n", i+1, r.RelevanceScore, r.Document)
}
```

## Cohere

```go
import rrcohere "github.com/joakimcarlsson/ai/rerankers/cohere"

reranker := rrcohere.NewReranker(
    rrcohere.WithAPIKey(os.Getenv("COHERE_API_KEY")),
    rrcohere.WithModel(cohere.Models[cohere.Rerank35]),
    rrcohere.WithTopK(5),
    rrcohere.WithReturnDocuments(true),
)

resp, err := reranker.Rerank(ctx, query, documents)
```

## Berget AI

EU-hosted, Cohere/Jina-compatible `/v1/rerank` (e.g. BGE Reranker v2 m3);
pricing in EUR.

```go
import rrberget "github.com/joakimcarlsson/ai/rerankers/berget"

reranker := rrberget.NewReranker(
    rrberget.WithAPIKey(os.Getenv("BERGET_API_KEY")),
    rrberget.WithModel(berget.Models[berget.BGERerankerV2M3]),
    rrberget.WithTopK(5),
    rrberget.WithReturnDocuments(true),
)

resp, err := reranker.Rerank(ctx, query, documents)
```

## Vendor-specific options

Voyage:

```go
rrvoyage.WithTruncation(true)
```

Cohere:

```go
rrcohere.WithMaxChunksPerDoc(8)
```


---

## Providers > Decision Models

> Source: providers/decision.md

# Decision Models

The `decision` modality. Vendors under `decision/`.

A decision model does not generate text. It reads a state and returns typed
answers — a choice, a score or a probability — in a single pass, which suits
routing, tool-call gating, classification and guardrails inside agent loops.
Several questions are answered over one state in a single call.

## Questions

| Constructor | Answer | Fields |
|---|---|---|
| `decision.Choice(instructions, criteria)` | `decision.ChoiceAnswer` | `Choice`, `Probabilities`, `Confidence` |
| `decision.Score(instructions, levels...)` | `decision.ScoreAnswer` | `Score` (may fall between levels), `Probabilities`, `Legend`, `Confidence` |
| `decision.Noul(instructions)` | `decision.NoulAnswer` | `Probability` of yes, 0 to 1 |
| `decision.NoulWithCriteria(instructions, yes, no)` | `decision.NoulAnswer` | as `Noul`, with explicit yes/no definitions |
| `decision.Box(instructions)` | `decision.BoxAnswer` | `Box` (`Top`, `Bottom`, `Left`, `Right` as 0 to 1 fractions of the image), `Confidence` (nil if unreported) |

Box needs an image and a model that supports it (Jev 1.14).

## Jev (TypeSafe AI)

```go
import (
    "github.com/joakimcarlsson/ai/decision"
    "github.com/joakimcarlsson/ai/decision/jev"
)

decider := jev.NewDecider(
    jev.WithAPIKey(os.Getenv("TYPESAFE_API_KEY")), // default: TYPESAFE_API_KEY
    jev.WithModel(jev.Models[jev.JevLatest]),
)

resp, err := decider.Decide(ctx,
    "My invoice is wrong and I need it fixed today.",
    map[string]decision.Question{
        "department": decision.Choice("Route the ticket.", map[string]string{
            "billing":   "Payments and invoices",
            "technical": "Bugs and outages",
        }),
        "frustration": decision.Score(
            "How frustrated is the customer?", "calm", "annoyed", "furious",
        ),
        "urgent": decision.Noul("Is the customer asking for urgent help?"),
    },
)

dept, _ := resp.Choice("department")
fmt.Println(dept.Choice, dept.Confidence)
urgent, _ := resp.Noul("urgent")
fmt.Println(urgent.Probability)
fmt.Println(resp.Usage.Cost(decider.Model()))
```

Options: `WithAPIKey`, `WithModel`, `WithBaseURL`, `WithHTTPClient`,
`WithTimeout`, `WithLongContext`. Input is billed at $0.042 per 1M tokens;
output is free.

### Images and Box

Wrap image bytes with `decision.NewImage`, write its `Ref` wherever the image
belongs in the state, instructions or criteria, and pass it with
`decision.WithImages`. The images header is set automatically, and images are
sent as base64. Images are PNG or JPEG, 280 to 1568 px per side, and an early
feature of the provider.

```go
png, _ := os.ReadFile("photo.png")
img := decision.NewImage("png", png)

resp, err := decider.Decide(ctx, "photo: "+img.Ref,
    map[string]decision.Question{
        "rect": decision.Box("Locate the red rectangle."),
        "has":  decision.Noul("Is there a red rectangle in the photo?"),
    },
    decision.WithImages(img),
)
box, _ := resp.Box("rect")
fmt.Println(box.Box.Top, box.Box.Left)
```

Box always returns coordinates, even when the target is absent; ask a `Noul`
alongside to check presence. Text over about 32,000 bytes and every image
request use the provider's long-context mode, which is slower and less
accurate; `WithLongContext` opts every request in.

### Listing models

The decider also implements `decision.ModelLister`:

```go
if l, ok := decider.(decision.ModelLister); ok {
    models, err := l.ListModels(ctx)
}
```

## OpenRouter

OpenRouter serves Jev over the same System One wire format, billed to an
OpenRouter key. `decision/openrouter` wraps `decision/jev` with that base URL.

```go
import "github.com/joakimcarlsson/ai/decision/openrouter"

decider := openrouter.NewDecider() // default key: OPENROUTER_API_KEY
```

It takes the same options as `jev.NewDecider`, and Box, images and
`ListModels` work the same way. Models are
`openrouter.Models[openrouter.JevLatest]` (default), `openrouter.Jev114` and
`openrouter.Jev113`; `openrouter.WithModelID("...")` selects any other
OpenRouter id. Pricing is the same as TypeSafe's, and OpenRouter's billed cost
is returned in `Usage.ReportedCost`.

`ListModels` returns every decision model OpenRouter serves, which includes
other vendors' System One-style models besides Jev. Only Jev has been tested
with this package.

Failed requests are returned as `*decision.APIError`, which satisfies
`llm.RetryableError`. `llm.DefaultRetryConfig` retries 429 and 5xx, but not
TypeSafe's 529 (overloaded); add it to `RetryStatusCodes` to retry those too.


---

## Providers > Fill-in-the-Middle

> Source: providers/fim.md

# Fill-in-the-Middle (FIM)

Code completion that takes a `Prompt` (code before the cursor) and an optional
`Suffix` (code after the cursor) and fills in the middle. Useful for code
editors and IDE integrations. The `fim` modality lives at `fim/`; vendors
under `fim/<name>/`.

## Mistral Codestral

```go
import (
    "github.com/joakimcarlsson/ai/fim"
    fimmistral "github.com/joakimcarlsson/ai/fim/mistral"
    "github.com/joakimcarlsson/ai/llm/mistral"
)

client := fimmistral.NewFIM(
    fimmistral.WithAPIKey(os.Getenv("MISTRAL_API_KEY")),
    fimmistral.WithModel(mistral.Models[mistral.Codestral]),
)

resp, err := client.Complete(ctx, fim.Request{
    Prompt: "def add(a, b):\n    ",
    Suffix: "\n    return result",
})
fmt.Println(resp.Content)
```

## DeepSeek

```go
import fimdeepseek "github.com/joakimcarlsson/ai/fim/deepseek"

client := fimdeepseek.NewFIM(
    fimdeepseek.WithAPIKey(os.Getenv("DEEPSEEK_API_KEY")),
    fimdeepseek.WithModel(fimdeepseek.Models[fimdeepseek.V32]),
)
```

## Streaming

```go
events := client.CompleteStream(ctx, fim.Request{
    Prompt: prompt,
    Suffix: suffix,
})

for event := range events {
    switch event.Type {
    case fim.EventContentDelta:
        fmt.Print(event.Content)
    case fim.EventComplete:
        fmt.Printf("\nFinish: %s\n", event.Response.FinishReason)
    case fim.EventError:
        log.Fatal(event.Error)
    }
}
```

## Per-request overrides

The `fim.Request` struct lets you override constructor defaults per call:

```go
maxTokens := int64(500)
temp := 0.2

resp, err := client.Complete(ctx, fim.Request{
    Prompt:      prompt,
    Suffix:      suffix,
    MaxTokens:   &maxTokens,
    Temperature: &temp,
    Stop:        []string{"\n\n"},
})
```

## Vendor-specific options

Mistral:

```go
fimmistral.WithMinTokens(20)
```

DeepSeek:

```go
fimdeepseek.WithFrequencyPenalty(0.3)
fimdeepseek.WithPresencePenalty(0.3)
fimdeepseek.WithEcho(true)
```


---

## Providers > Vision

> Source: providers/vision.md

# Vision (Multimodal Images)

Send images to LLMs for analysis using URL references or raw binary data. Works with any provider that supports multimodal input (Anthropic, OpenAI, Gemini).

## Image from URL

```go
import "github.com/joakimcarlsson/ai/message"

msg := message.NewUserMessage("What do you see in this image?")
msg.AddImageURL("https://example.com/photo.jpg", "")

response, err := client.SendMessages(ctx, []message.Message{msg}, nil)
fmt.Println(response.Content)
```

The second argument to `AddImageURL` is an optional detail level (`"low"`, `"high"`, or `""` for auto).

## Image from Binary Data

```go
imageData, _ := os.ReadFile("photo.jpg")

msg := message.NewUserMessage("Describe this image.")
msg.AddBinary("image/jpeg", imageData)

response, err := client.SendMessages(ctx, []message.Message{msg}, nil)
fmt.Println(response.Content)
```

## Multiple Images

```go
msg := message.NewUserMessage("Compare these two images.")
msg.AddImageURL("https://example.com/before.jpg", "")
msg.AddImageURL("https://example.com/after.jpg", "")

response, err := client.SendMessages(ctx, []message.Message{msg}, nil)
```

## Supported Formats

Most providers accept JPEG, PNG, GIF, and WebP. Check your provider's documentation for size limits.


---

## Agent Framework > Overview

> Source: agent/overview.md

# Agent Framework

The agent package provides multi-agent orchestration with automatic tool execution, session management, persistent memory, sub-agents, handoffs, fan-out, team coordination, and context strategies.

## Basic Agent

```go
import (
    "github.com/joakimcarlsson/ai/agent"
    "github.com/joakimcarlsson/ai/session"
)

myAgent := agent.New(llmClient,
    agent.WithSystemPrompt("You are a helpful assistant."),
    agent.WithTools(&weatherTool{}),
    agent.WithSession("user-123", session.FileStore("./sessions")),
)

response, _ := myAgent.Chat(ctx, "What's the weather in Tokyo?")
fmt.Println(response.Content)
```

## How It Works

When you call `Chat()`, the agent:

1. Builds the message history (system prompt + session messages + user message)
2. Sends messages to the LLM
3. If the LLM requests tool calls, executes them automatically
4. Loops back to step 2 with tool results until the LLM responds with text
5. Persists the conversation to the session store

## Configuration Options

| Option | Description | Default |
|--------|-------------|---------|
| `WithSystemPrompt(prompt)` | Sets the agent's behavior | none |
| `WithTools(tools...)` | Adds tools the agent can use | none |
| `WithSession(id, store)` | Enables conversation persistence | none |
| `WithMemory(id, store, opts...)` | Enables long-term memory | none |
| `WithMaxIterations(n)` | Max tool execution loops | 10 |
| `WithAutoExecute(bool)` | Auto-execute tool calls | true |
| `WithContextStrategy(strategy, maxTokens)` | Context window management | none |
| `WithSequentialToolExecution()` | Disable parallel tool execution | parallel |
| `WithMaxParallelTools(n)` | Limit concurrent tool execution | unlimited |
| `WithState(map)` | Template variables for system prompt | none |
| `WithInstructionProvider(fn)` | Dynamic system prompt generation | none |
| `WithHooks(hooks...)` | Add hook interceptors for observation/interception | none |
| `WithConfirmationProvider(fn)` | Require human approval for sensitive tools | none |
| `WithSubAgents(configs...)` | Register child agents | none |
| `WithHandoffs(configs...)` | Register peer agents for transfer | none |
| `WithFanOut(configs...)` | Register parallel task distribution | none |
| `WithTeam(config)` | Enable team coordination with peer-to-peer messaging | none |
| `WithCoordinatorMode()` | Restrict lead agent to team-only tools | disabled |
| `WithTeammateTemplates(map)` | Pre-configured agent templates for teammates | none |

## ChatResponse

```go
type ChatResponse struct {
    Content        string
    ToolCalls      []message.ToolCall
    ToolResults    []ToolExecutionResult
    Usage          llm.TokenUsage
    FinishReason   message.FinishReason
    AgentName      string         // Set when a handoff occurred
    TotalToolCalls int
    TotalDuration  time.Duration
    TotalTurns     int
}
```

All metrics are aggregated across the full agent loop, not just the final LLM call:

| Field | Description |
|-------|-------------|
| `TotalTurns` | Number of LLM round-trips (API calls) made |
| `TotalDuration` | Wall-clock time from `Chat()` entry to return |
| `TotalToolCalls` | Total tool invocations across all iterations |
| `ToolResults` | Results of every tool execution during the conversation |
| `FinishReason` | Why the agent stopped: `FinishReasonStop` (natural completion), `FinishReasonMaxTokens` (hit context limit), `FinishReasonToolUse` (tool required), or `FinishReasonMaxIterations` (hit tool loop limit) |

### Finish Reasons and Max Iterations

The `FinishReason` indicates why the agent loop terminated:
- **`FinishReasonStop`**: The agent naturally completed its task and produced a final text response.
- **`FinishReasonMaxTokens`**: The LLM hit its configured token limit during generation.
- **`FinishReasonToolUse`**: The agent returned pending tool calls. This occurs either because `WithAutoExecute(false)` is set (see [Continue/Resume](continue.md)), or because a tool requires human confirmation (see [Confirmation](confirmation.md)).
- **`FinishReasonMaxIterations`**: The agent hit the loop limit configured by `WithMaxIterations(n)` (default 10) before completing its task. When this occurs, the agent can no longer execute tools automatically. You can intercept this behavior by registering a [Continuation Provider](continuation.md).

## Debug APIs

Inspect the messages that would be sent to the LLM after applying context strategies:

```go
// Non-destructive — does not modify the session
messages, err := myAgent.PeekContextMessages(ctx, "Hello")

// Modifying — adds the user message to the session
messages, err := myAgent.BuildContextMessages(ctx, "Hello")
```

Use `PeekContextMessages` to debug context window management without side effects.


---

## Agent Framework > Session Management

> Source: agent/sessions.md

# Session Management

Sessions persist conversation history across multiple `Chat()` calls.

## Setup

```go
import "github.com/joakimcarlsson/ai/session"

myAgent := agent.New(llmClient,
    agent.WithSystemPrompt("You are a helpful assistant."),
    agent.WithSession("conversation-id", session.FileStore("./sessions")),
)
```

## Built-in Stores

```go
// Persistent JSON files
store := session.FileStore("./sessions")

// In-memory (ephemeral, lost on restart)
store := session.MemoryStore()
```

## Database Stores

Ready-to-use stores for production backends:

- [PostgreSQL](../integrations/postgres.md) — `postgres.SessionStore(ctx, connString)`
- [SQLite](../integrations/sqlite.md) — `sqlite.SessionStore(ctx, db)`

## Store Interface

Implement this interface to use any backend:

```go
type Store interface {
    Exists(ctx context.Context, id string) (bool, error)
    Create(ctx context.Context, id string) (Session, error)
    Load(ctx context.Context, id string) (Session, error)
    Delete(ctx context.Context, id string) error
}
```

## Session Interface

```go
type Session interface {
    ID() string
    GetMessages(ctx context.Context, limit *int) ([]message.Message, error)
    AddMessages(ctx context.Context, msgs []message.Message) error
    PopMessage(ctx context.Context) (*message.Message, error)
    Clear(ctx context.Context) error
}
```


---

## Agent Framework > Persistent Memory

> Source: agent/memory.md

# Persistent Memory

Memory enables cross-conversation fact storage and retrieval using vector-based semantic search.

## Setup

```go
import "github.com/joakimcarlsson/ai/memory"

store := memory.NewStore(embedder)

myAgent := agent.New(llmClient,
    agent.WithSystemPrompt("You are a personal assistant."),
    agent.WithMemory("user-123", store,
        memory.AutoExtract(),  // Auto-extract facts from conversations
        memory.AutoDedup(),    // LLM-based memory deduplication
    ),
)

response, _ := myAgent.Chat(ctx, "My name is Alice and I'm allergic to peanuts.")
// Agent automatically stores this fact and recalls it in future conversations
```

## Built-in Stores

```go
// In-memory vector store
store := memory.NewStore(embedder)

// File-persisted vector store
store := memory.FileStore("./memories", embedder)
```

## Memory Options

| Option | Description |
|--------|-------------|
| `memory.AutoExtract()` | Automatically extract facts from conversations after each response |
| `memory.AutoDedup()` | Use LLM to deduplicate similar memories before storing |
| `memory.LLM(l)` | Use a separate (cheaper) LLM for extraction and deduplication |

## Database Stores

Ready-to-use stores for production backends:

- [pgvector](../integrations/pgvector.md) — `pgvector.MemoryStore(ctx, connString, embedder)` — PostgreSQL with HNSW vector search

## Store Interface

Implement for any vector database backend:

```go
type Store interface {
    Store(ctx context.Context, id string, fact string, metadata map[string]any) error
    Search(ctx context.Context, id string, query string, limit int) ([]Entry, error)
    GetAll(ctx context.Context, id string, limit int) ([]Entry, error)
    Delete(ctx context.Context, memoryID string) error
    Update(ctx context.Context, memoryID string, fact string, metadata map[string]any) error
}
```

## How It Works

When `AutoExtract` is enabled:

1. After the agent responds, it reviews the conversation
2. An LLM extracts factual information worth remembering
3. If `AutoDedup` is enabled, the LLM checks for existing similar memories
4. New facts are stored, duplicates are merged or skipped

## Manual Memory Tools

When `AutoExtract` is disabled, the agent gets four memory tools that the LLM can call directly:

### store_memory

Store a fact about the user for future conversations.

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `fact` | string | yes | The fact to remember |
| `category` | string | no | One of: `preference`, `personal`, `health`, `professional`, `other` |

### recall_memories

Search for relevant memories. Returns memory IDs for use with replace/delete.

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `query` | string | yes | What to search for |

### replace_memory

Update an existing memory with corrected or updated information.

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `memory_id` | string | yes | ID from `recall_memories` results |
| `fact` | string | yes | The updated fact |
| `category` | string | no | One of: `preference`, `personal`, `health`, `professional`, `other` |

### delete_memory

Remove a memory that is no longer accurate or relevant.

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `memory_id` | string | yes | ID from `recall_memories` results |
| `reason` | string | no | Why the memory is being deleted |


---

## Agent Framework > Streaming

> Source: agent/streaming.md

# Streaming

`ChatStream` returns a channel of events for real-time response handling.

## Basic Usage

```go
for event := range myAgent.ChatStream(ctx, "Tell me a story") {
    switch event.Type {
    case types.EventContentDelta:
        fmt.Print(event.Content)
    case types.EventThinkingDelta:
        fmt.Print(event.Thinking)
    case types.EventToolUseStart:
        fmt.Printf("\nUsing tool: %s\n", event.ToolCall.Name)
    case types.EventToolUseStop:
        if event.ToolResult != nil {
            fmt.Printf("Tool result: %s\n", event.ToolResult.Output)
        }
    case types.EventHandoff:
        fmt.Printf("Handed off to: %s\n", event.AgentName)
    case types.EventComplete:
        fmt.Printf("\nDone! Tokens: %d\n", event.Response.Usage.InputTokens)
    case types.EventError:
        log.Fatal(event.Error)
    }
}
```

## ContinueStream

The streaming variant of `Continue()`:

```go
for event := range myAgent.ContinueStream(ctx, toolResults) {
    switch event.Type {
    case types.EventContentDelta:
        fmt.Print(event.Content)
    case types.EventComplete:
        fmt.Println("\nDone!")
    }
}
```

## Event Types

| Event | Field | Description |
|-------|-------|-------------|
| `EventContentStart` | — | Content generation is beginning |
| `EventContentDelta` | `Content` | Partial text token |
| `EventContentStop` | — | Content generation finished |
| `EventToolUseStart` | `ToolCall` | Tool invocation starting (name, ID) |
| `EventToolUseDelta` | `ToolCall` | Partial tool input JSON |
| `EventToolUseStop` | `ToolResult` | Tool execution completed with result |
| `EventThinkingDelta` | `Thinking` | Chain-of-thought reasoning (if model supports it) |
| `EventHandoff` | `AgentName` | Control transferred to another agent |
| `EventConfirmationRequired` | `ConfirmationRequest` | Tool awaiting human approval ([details](confirmation.md)) |
| `EventTeammateSpawned` | `AgentName` | A new teammate was spawned in a [team](team-coordination.md) |
| `EventTeamMessage` | `AgentName` | A message was sent between team members |
| `EventTeammateComplete` | `AgentName` | A teammate finished its task successfully |
| `EventTeammateError` | `AgentName`, `Error` | A teammate encountered an error |
| `EventComplete` | `Response` | Streaming finished — contains the full `ChatResponse` |
| `EventError` | `Error` | An error occurred during streaming |
| `EventWarning` | `Error` | A non-fatal warning |

## ChatEvent

```go
type ChatEvent struct {
    Type       types.EventType
    Content    string              // EventContentDelta
    Thinking   string              // EventThinkingDelta
    ToolCall   *message.ToolCall   // EventToolUseStart/Delta
    ToolResult *ToolExecutionResult // EventToolUseStop
    Response   *ChatResponse       // EventComplete
    Error               error                    // EventError, EventWarning
    AgentName           string                   // EventHandoff
    ConfirmationRequest *tool.ConfirmationRequest // EventConfirmationRequired
}
```


---

## Agent Framework > Hooks

> Source: agent/hooks.md

# Hooks

Hooks let you observe, modify, or block agent behavior at key points in the execution pipeline. They cover tool calls, model interactions, error recovery, agent lifecycle, input validation, and cross-cutting event observation.

## Setup

```go
myAgent := agent.New(llmClient,
    agent.WithHooks(agent.Hooks{
        PreToolUse: func(ctx context.Context, tc agent.ToolUseContext) (agent.PreToolUseResult, error) {
            log.Printf("Tool call: %s (branch: %s)", tc.ToolName, tc.Branch)
            return agent.PreToolUseResult{Action: agent.HookAllow}, nil
        },
    }),
)
```

## Hook Types

| Hook | Fires | Can |
|------|-------|-----|
| `PreToolUse` | Before a tool executes | Allow, Deny, or Modify input |
| `PostToolUse` | After a tool executes | Allow or Modify output |
| `PreModelCall` | Before an LLM request | Allow or Modify messages/tools |
| `PostModelCall` | After an LLM response | Allow or Modify response |
| `OnSubagentStart` | When a background sub-agent launches | Observe only |
| `OnSubagentStop` | When a background sub-agent finishes | Observe only |
| `OnToolError` | When a tool returns an error | Allow (re-raise) or Modify (recover) |
| `OnModelError` | When an LLM call fails | Allow (re-raise) or Modify (recover) |
| `BeforeAgent` | Before an agent starts its run | Allow, Deny, or Modify (short-circuit) |
| `AfterAgent` | After an agent completes its run | Allow or Modify response |
| `BeforeRun` | At the start of Chat/ChatStream | Observe only |
| `AfterRun` | At the end of Chat/ChatStream | Observe only |
| `OnUserMessage` | When a user message arrives | Allow, Deny, or Modify message |
| `OnEvent` | On every hook event emitted | Observe only |
| `OnTeammateJoin` | When a teammate is spawned | Observe only |
| `OnTeammateLeave` | When a teammate leaves (stopped) | Observe only |
| `OnTeammateComplete` | When a teammate finishes successfully | Observe only |
| `OnTeammateError` | When a teammate encounters an error | Observe only |
| `OnTeamMessage` | When a message is sent between members | Observe only |

## HookAction

Every hook returns a `HookAction` that controls what happens next:

| Action | Behavior |
|--------|----------|
| `HookAllow` | Continue normally (default) |
| `HookDeny` | Block execution (PreToolUse, BeforeAgent, OnUserMessage) |
| `HookModify` | Replace input, output, messages, response, or recover from errors |

## Denying a Tool Call

Return `HookDeny` from `PreToolUse` to block a tool before it runs:

```go
agent.Hooks{
    PreToolUse: func(_ context.Context, tc agent.ToolUseContext) (agent.PreToolUseResult, error) {
        if tc.ToolName == "dangerous_tool" {
            return agent.PreToolUseResult{
                Action:     agent.HookDeny,
                DenyReason: "this tool is not allowed",
            }, nil
        }
        return agent.PreToolUseResult{Action: agent.HookAllow}, nil
    },
}
```

The agent receives a tool error result with the deny reason.

## Modifying Tool Input

Return `HookModify` from `PreToolUse` to rewrite the input before execution:

```go
agent.Hooks{
    PreToolUse: func(_ context.Context, tc agent.ToolUseContext) (agent.PreToolUseResult, error) {
        modified := strings.ReplaceAll(tc.Input, "SECRET", "[REDACTED]")
        return agent.PreToolUseResult{
            Action: agent.HookModify,
            Input:  modified,
        }, nil
    },
}
```

## Modifying Model Messages

Return `HookModify` from `PreModelCall` to inject or filter messages before they reach the LLM:

```go
agent.Hooks{
    PreModelCall: func(_ context.Context, mc agent.ModelCallContext) (agent.ModelCallResult, error) {
        extra := message.NewUserMessage("Remember: always respond in JSON.")
        return agent.ModelCallResult{
            Action:   agent.HookModify,
            Messages: append(mc.Messages, extra),
            Tools:    mc.Tools,
        }, nil
    },
}
```

## Error Recovery

### Tool Error Recovery

`OnToolError` fires when a tool returns an error, before the error reaches `PostToolUse`. Return `HookModify` with replacement output to recover:

```go
agent.Hooks{
    OnToolError: func(_ context.Context, tc agent.ToolErrorContext) (agent.ToolErrorResult, error) {
        if tc.ToolName == "flaky_api" {
            return agent.ToolErrorResult{
                Action: agent.HookModify,
                Output: "API temporarily unavailable, using cached data",
            }, nil
        }
        return agent.ToolErrorResult{Action: agent.HookAllow}, nil
    },
}
```

When recovery succeeds, the error flag is cleared and `PostToolUse` sees a non-error result. Multiple error callbacks chain — the first recovery wins.

### Model Error Recovery

`OnModelError` fires when an LLM call fails. Return `HookModify` with a replacement response to recover:

```go
agent.Hooks{
    OnModelError: func(_ context.Context, mc agent.ModelErrorContext) (agent.ModelErrorResult, error) {
        return agent.ModelErrorResult{
            Action: agent.HookModify,
            Response: &llm.Response{
                Content: "Service temporarily unavailable. Please try again.",
            },
        }, nil
    },
}
```

This works in both `Chat()` and `ChatStream()` paths.

## Agent Lifecycle

### Short-Circuiting with BeforeAgent

`BeforeAgent` fires before an agent starts its run. Return `HookModify` with a response to skip the agent entirely:

```go
agent.Hooks{
    BeforeAgent: func(_ context.Context, ac agent.LifecycleContext) (agent.LifecycleResult, error) {
        if cached, ok := cache.Get(ac.Input); ok {
            return agent.LifecycleResult{
                Action:   agent.HookModify,
                Response: &agent.ChatResponse{Content: cached},
            }, nil
        }
        return agent.LifecycleResult{Action: agent.HookAllow}, nil
    },
}
```

Return `HookDeny` to block the agent run with a nil response.

### Modifying with AfterAgent

`AfterAgent` fires after an agent completes. Modify the response before it reaches the caller:

```go
agent.Hooks{
    AfterAgent: func(_ context.Context, ac agent.LifecycleContext) (agent.LifecycleResult, error) {
        modified := *ac.Response
        modified.Content = sanitize(modified.Content)
        return agent.LifecycleResult{
            Action:   agent.HookModify,
            Response: &modified,
        }, nil
    },
}
```

## Run Lifecycle

`BeforeRun` and `AfterRun` are observation-only hooks that fire at the very start and end of `Chat()`/`ChatStream()`:

```go
agent.Hooks{
    BeforeRun: func(_ context.Context, rc agent.RunContext) {
        metrics.StartTimer(rc.AgentName)
    },
    AfterRun: func(_ context.Context, rc agent.RunContext) {
        metrics.RecordDuration(rc.AgentName, rc.Duration)
        if rc.Error != nil {
            metrics.RecordError(rc.AgentName, rc.Error)
        }
    },
}
```

`AfterRun` receives the final response, any error, and the total duration.

## Input Validation

`OnUserMessage` fires when a user message arrives, before it reaches any agent logic. Use it to preprocess, validate, or reject messages:

```go
agent.Hooks{
    OnUserMessage: func(_ context.Context, uc agent.UserMessageContext) (agent.UserMessageResult, error) {
        if containsPII(uc.Message) {
            return agent.UserMessageResult{
                Action:     agent.HookDeny,
                DenyReason: "message contains PII",
            }, nil
        }
        return agent.UserMessageResult{
            Action:  agent.HookModify,
            Message: sanitizeInput(uc.Message),
        }, nil
    },
}
```

`OnUserMessage` does not fire for `Continue()`/`ContinueStream()` since those resume with tool results, not user messages.

## Cross-Cutting Event Observation

`OnEvent` fires on every hook event emitted during execution. Use it for logging, analytics, or event transformation:

```go
agent.Hooks{
    OnEvent: func(_ context.Context, evt agent.HookEvent) {
        log.Printf("[%s] agent=%s tool=%s", evt.Type, evt.AgentName, evt.ToolName)
    },
}
```

`OnEvent` fires once per hook-point invocation (after all hooks in the chain have run), not once per registered hook. It covers all event types except itself.

## Chaining Multiple Hooks

Pass multiple `Hooks` to `WithHooks`, or call `WithHooks` multiple times. Hooks run in registration order.

```go
myAgent := agent.New(llmClient,
    agent.WithHooks(loggingHooks, guardRailHooks, metricsHooks),
)
```

Chain rules:

- **Deny wins immediately** — if any hook returns `HookDeny`, later hooks are skipped
- **Last Modify wins** — if multiple hooks return `HookModify`, the last one's value is used
- **First recovery wins** — for error callbacks (`OnToolError`, `OnModelError`), the first `HookModify` response is used
- **nil fields are skipped** — you only need to set the hooks you care about

## Observation with NewObservingHooks

For pure observation (logging, metrics, streaming to a UI), use the `NewObservingHooks` helper. It wires all hooks to emit structured `HookEvent` values to a single callback:

```go
myAgent := agent.New(llmClient,
    agent.WithHooks(agent.NewObservingHooks(func(evt agent.HookEvent) {
        log.Printf("[%s] agent=%s branch=%s tool=%s",
            evt.Type, evt.AgentName, evt.Branch, evt.ToolName)
    })),
)
```

All observing hooks return `HookAllow` — they never block or modify execution. `OnEvent` is left nil in observing hooks to avoid double-emission.

### HookEvent

| Field | Type | Description |
|-------|------|-------------|
| `Type` | `HookEventType` | Event type (see below) |
| `Timestamp` | `time.Time` | When the event fired |
| `AgentName` | `string` | Name of the agent |
| `TaskID` | `string` | Background task ID (if applicable) |
| `Branch` | `string` | Agent hierarchy path (e.g. `"orchestrator/researcher"`) |
| `ToolCallID` | `string` | Tool call ID (tool events only) |
| `ToolName` | `string` | Tool name (tool events only) |
| `Input` | `string` | Tool input, sub-agent task, or user message |
| `Output` | `string` | Tool output or sub-agent result |
| `IsError` | `bool` | Whether an error occurred |
| `Duration` | `time.Duration` | Execution duration (post-events only) |
| `Usage` | `llm.TokenUsage` | Token usage (post model call only) |
| `Error` | `string` | Error message (if `IsError` is true) |

### Event Types

| Constant | Value | When |
|----------|-------|------|
| `HookEventPreToolUse` | `"pre_tool_use"` | Before tool execution |
| `HookEventPostToolUse` | `"post_tool_use"` | After tool execution |
| `HookEventPreModelCall` | `"pre_model_call"` | Before LLM request |
| `HookEventPostModelCall` | `"post_model_call"` | After LLM response |
| `HookEventSubagentStart` | `"subagent_start"` | Background sub-agent launched |
| `HookEventSubagentStop` | `"subagent_stop"` | Background sub-agent finished |
| `HookEventToolError` | `"tool_error"` | Tool returned an error |
| `HookEventModelError` | `"model_error"` | LLM call failed |
| `HookEventBeforeAgent` | `"before_agent"` | Before agent starts |
| `HookEventAfterAgent` | `"after_agent"` | After agent completes |
| `HookEventBeforeRun` | `"before_run"` | Start of Chat/ChatStream |
| `HookEventAfterRun` | `"after_run"` | End of Chat/ChatStream |
| `HookEventUserMessage` | `"user_message"` | User message received |
| `HookEventTeammateJoin` | `"teammate_join"` | Teammate spawned in a team |
| `HookEventTeammateLeave` | `"teammate_leave"` | Teammate left (stopped) |
| `HookEventTeamMessage` | `"team_message"` | Message sent between team members |
| `HookEventTeammateComplete` | `"teammate_complete"` | Teammate finished successfully |
| `HookEventTeammateError` | `"teammate_error"` | Teammate encountered an error |

## Branch

The `Branch` field on all hook contexts gives you the agent hierarchy as a `/`-separated path. For a nested setup where an orchestrator delegates to a researcher which delegates to a scraper:

```
Branch: "orchestrator/researcher/scraper"
```

This lets you immediately see which agent in the hierarchy produced an event, without cross-referencing task IDs.

## Hook Propagation

Hooks set on a parent agent automatically propagate to sub-agents that don't have their own hooks:

```go
orchestrator := agent.New(llmClient,
    agent.WithHooks(myHooks),
    agent.WithSubAgents(
        agent.SubAgentConfig{Name: "worker", Agent: worker},
    ),
)
// worker inherits myHooks since it has none of its own
```

If a sub-agent already has hooks configured, the parent's hooks are not applied.

## Context Structs

### ToolUseContext

Passed to `PreToolUse` and embedded in `PostToolUseContext` and `ToolErrorContext`:

```go
type ToolUseContext struct {
    ToolCallID string
    ToolName   string
    Input      string
    AgentName  string
    TaskID     string
    Branch     string
}
```

### PostToolUseContext

Passed to `PostToolUse`:

```go
type PostToolUseContext struct {
    ToolUseContext        // Embeds all fields from ToolUseContext
    Output   string
    IsError  bool
    Duration time.Duration
}
```

### ToolErrorContext

Passed to `OnToolError`:

```go
type ToolErrorContext struct {
    ToolUseContext        // Embeds all fields from ToolUseContext
    Error    error
    Output   string
    Duration time.Duration
}
```

### ModelCallContext

Passed to `PreModelCall`:

```go
type ModelCallContext struct {
    Messages  []message.Message
    Tools     []tool.BaseTool
    AgentName string
    TaskID    string
    Branch    string
}
```

### ModelResponseContext

Passed to `PostModelCall`:

```go
type ModelResponseContext struct {
    Response  *llm.Response
    Duration  time.Duration
    AgentName string
    TaskID    string
    Branch    string
    Error     error
}
```

### ModelErrorContext

Passed to `OnModelError`:

```go
type ModelErrorContext struct {
    Messages  []message.Message
    Tools     []tool.BaseTool
    Error     error
    AgentName string
    TaskID    string
    Branch    string
}
```

### SubagentEventContext

Passed to `OnSubagentStart` and `OnSubagentStop`:

```go
type SubagentEventContext struct {
    TaskID    string
    AgentName string
    Task      string
    Branch    string
    Result    string
    Error     error
    Duration  time.Duration
}
```

### TeammateEventContext

Passed to `OnTeammateJoin`, `OnTeammateLeave`, `OnTeammateComplete`, and `OnTeammateError`:

```go
type TeammateEventContext struct {
    TeamName   string
    MemberID   string
    MemberName string
    Task       string
    Result     string
    Error      error
    Duration   time.Duration
}
```

### TeamMessageContext

Passed to `OnTeamMessage`:

```go
type TeamMessageContext struct {
    TeamName string
    Message  team.Message
}
```

See [Team Coordination](team-coordination.md) for full details on team hooks and messaging.

### LifecycleContext

Passed to `BeforeAgent` and `AfterAgent`:

```go
type LifecycleContext struct {
    AgentName string
    TaskID    string
    Branch    string
    Input     string
    Response  *ChatResponse   // nil for BeforeAgent, set for AfterAgent
}
```

### RunContext

Passed to `BeforeRun` and `AfterRun`:

```go
type RunContext struct {
    AgentName string
    TaskID    string
    Branch    string
    Input     string
    Response  *ChatResponse   // nil for BeforeRun, set for AfterRun
    Error     error           // nil for BeforeRun, set for AfterRun if failed
    Duration  time.Duration   // zero for BeforeRun
}
```

### UserMessageContext

Passed to `OnUserMessage`:

```go
type UserMessageContext struct {
    Message   string
    AgentName string
    TaskID    string
    Branch    string
}
```

## Streaming to a UI

A common use case is forwarding hook events to a frontend over WebSocket or SSE:

```go
agent.NewObservingHooks(func(evt agent.HookEvent) {
    data, _ := json.Marshal(evt)
    websocket.Send(data)
})
```

This gives the UI real-time visibility into tool calls, model interactions, error recovery, and agent lifecycle — including nested agent hierarchies via `Branch`.


---

## Agent Framework > Tool Confirmation

> Source: agent/confirmation.md

# Tool Confirmation

The confirmation protocol lets tools require human approval before executing. The framework provides the mechanism — consumers provide the UI/interaction layer.

## Setup

Register a `ConfirmationProvider` on the agent. The provider is called whenever a tool requires confirmation and blocks until the consumer provides a decision.

```go
myAgent := agent.New(llmClient,
    agent.WithTools(&DeleteTool{}),
    agent.WithConfirmationProvider(
        func(ctx context.Context, req tool.ConfirmationRequest) (bool, error) {
            // Present req to the user, wait for their decision
            return askUser(req.ToolName, req.Input, req.Hint), nil
        },
    ),
)
```

Return `true` to approve, `false` to reject. If the provider returns an error, the tool call fails with that error.

## Declarative Confirmation

Set `RequireConfirmation` on a tool's `Info` to require approval before `Run()` is called:

```go
func (t *DeleteTool) Info() tool.Info {
    info := tool.NewInfo("delete_records", "Delete database records", DeleteParams{})
    info.RequireConfirmation = true
    return info
}
```

When the agent encounters this tool, it calls the `ConfirmationProvider` before executing. If no provider is configured, the tool runs normally — confirmation is opt-in.

## Dynamic Confirmation

Tools can request confirmation from within `Run()` for conditional approval:

```go
func (t *TransferTool) Run(ctx context.Context, params tool.Call) (tool.Response, error) {
    var input TransferParams
    json.Unmarshal([]byte(params.Input), &input)

    if input.Amount > 10000 {
        err := tool.RequestConfirmation(ctx, "Large transfer exceeding $10,000", input)
        if err != nil {
            return tool.Response{}, err
        }
    }

    // Proceed with transfer
    return tool.NewTextResponse("Transfer complete"), nil
}
```

`RequestConfirmation` blocks until the consumer decides. If rejected, it returns `tool.ErrConfirmationRejected` — propagate this error to halt execution. If no `ConfirmationProvider` is configured, `RequestConfirmation` is a no-op (auto-approve).

## ConfirmationRequest

The provider receives a `ConfirmationRequest` with context about the tool call:

```go
type ConfirmationRequest struct {
    ToolCallID string // Unique ID of this tool call
    ToolName   string // Name of the tool
    Input      string // JSON-encoded arguments
    Hint       string // Human-readable description (dynamic confirmation only)
    Payload    any    // Arbitrary structured data (dynamic confirmation only)
}
```

For declarative confirmation (`RequireConfirmation` flag), `Hint` and `Payload` are empty. For dynamic confirmation (`RequestConfirmation`), they carry the values passed by the tool.

## Toolset-Level Confirmation

Use `tool.WithConfirmation` to mark all tools in a toolset as requiring confirmation:

```go
dangerousTools := tool.NewToolset("dangerous",
    &DeleteTool{},
    &DropTableTool{},
    &FormatDiskTool{},
)

confirmed := tool.WithConfirmation(dangerousTools)

myAgent := agent.New(llmClient,
    agent.WithToolsets(confirmed),
    agent.WithConfirmationProvider(myProvider),
)
```

This sets `RequireConfirmation = true` on every tool in the toolset without modifying the originals.

## Streaming

In the streaming path (`ChatStream`), an `EventConfirmationRequired` event is emitted before the provider blocks. This allows the consumer to present a UI and then unblock the provider:

```go
for event := range myAgent.ChatStream(ctx, "Delete old records") {
    switch event.Type {
    case types.EventConfirmationRequired:
        req := event.ConfirmationRequest
        fmt.Printf("Tool %q wants to run with input: %s\n", req.ToolName, req.Input)
        // The provider is blocking — respond via whatever mechanism it uses
    case types.EventContentDelta:
        fmt.Print(event.Content)
    case types.EventComplete:
        fmt.Println("\nDone!")
    }
}
```

A common pattern is to use a channel-based provider that the streaming consumer unblocks:

```go
type approval struct {
    approved bool
    ch       chan struct{}
}

pending := make(map[string]*approval)
var mu sync.Mutex

provider := func(ctx context.Context, req tool.ConfirmationRequest) (bool, error) {
    a := &approval{ch: make(chan struct{})}
    mu.Lock()
    pending[req.ToolCallID] = a
    mu.Unlock()
    <-a.ch // Block until consumer decides
    return a.approved, nil
}

// In the stream consumer, when EventConfirmationRequired arrives:
// mu.Lock()
// a := pending[req.ToolCallID]
// mu.Unlock()
// a.approved = userClickedApprove
// close(a.ch)
```

## Interaction with Hooks

`PreToolUse` hooks run before confirmation. If a hook denies the tool, the confirmation provider is never called:

```
PreToolUse hooks → Confirmation check → tool.Run()
```

This means hooks enforce policy (rate limits, blocklists), while confirmation handles human approval.

## Handoffs

Each agent has its own `ConfirmationProvider`. When a handoff occurs, the new agent's provider is used. If the target agent has no provider, its tools run without confirmation.

## Auto-Approve Patterns

The provider is a regular function — implement any approval logic:

```go
// Always approve (useful for testing)
agent.WithConfirmationProvider(
    func(_ context.Context, _ tool.ConfirmationRequest) (bool, error) {
        return true, nil
    },
)

// Check a database of pre-approved tools
agent.WithConfirmationProvider(
    func(ctx context.Context, req tool.ConfirmationRequest) (bool, error) {
        return db.IsToolPreApproved(ctx, userID, req.ToolName)
    },
)

// Approve safe tools, prompt for dangerous ones
agent.WithConfirmationProvider(
    func(ctx context.Context, req tool.ConfirmationRequest) (bool, error) {
        if req.ToolName == "read_file" {
            return true, nil
        }
        return promptUser(ctx, req)
    },
)
```


---

## Agent Framework > Loop Continuation

> Source: agent/continuation.md

# Loop Continuation

The continuation protocol lets you control agent execution when it hits its tool iteration limit. The framework provides the mechanism — consumers provide the logic to approve, decline, or timeout the continuation.

## Setup

Register a `ContinuationProvider` on the agent. The provider is called whenever the agent reaches its `MaxIterations` limit while trying to execute tools. It blocks until the consumer provides a decision.

```go
myAgent := agent.New(llmClient,
    agent.WithMaxIterations(5),
    agent.WithContinuationProvider(
        func(ctx context.Context, req agent.ContinuationRequest) (agent.ContinuationResponse, error) {
            // Present req to the user or apply automated logic
            return agent.ContinuationResponse{
                Decision: agent.ContinuationApprove,
            }, nil
        },
    ),
)
```

Return `agent.ContinuationApprove` inside the response to reset the iteration counter and continue the loop, `agent.ContinuationDecline` to stop execution gracefully, or `agent.ContinuationTimeout` if the wait period expires. If the provider returns `ContinuationDecline` or `ContinuationTimeout`, the agent gracefully cancels any pending tool calls, makes a final LLM call to summarize the interruption, and successfully returns with `FinishReasonMaxIterations`. If the provider function itself returns a Go `error`, the agent halts with an error.

## ContinuationResponse

The provider must return a `ContinuationResponse` struct:

```go
type ContinuationResponse struct {
    Decision         ContinuationDecision // Approve, Decline, or Timeout
    Message          string               // Optional "steering message" to append to the conversation
    DiscardToolCalls bool                 // If true, pending tool calls are skipped
    ToolMessage      string               // Custom error message for skipped tools
}
```

- **Message:** Supply an optional steering message. If the decision is `ContinuationApprove`, this message acts as a steering instruction for the next iteration (e.g., "Please try using the search tool instead"). If `ContinuationDecline` or `ContinuationTimeout`, it overrides the default system halt notification to explain why it was stopped. *Note: Regardless of the decision or whether tools were discarded, the steering message is always appended to the context strictly after the tool execution results (or synthetic error results) to preserve expected LLM request/response sequences.*
- **DiscardToolCalls & ToolMessage:** When approving, you can set `DiscardToolCalls: true` to prevent the pending tools from executing. The agent injects synthetic error results for those tools, optionally using `ToolMessage` (defaults to "Tool execution canceled by user during continuation.").

## ContinuationRequest

The provider receives a `ContinuationRequest` with context about the agent's current state:

```go
type ContinuationRequest struct {
    MaxIterations   int                  // The limit that was reached
    TotalIterations int                  // Total iterations across all continuations
    ToolCalls       []message.ToolCall   // The tools the agent wants to execute next
}
```

## Streaming

In the streaming path (`ChatStream`), an `EventContinuationRequired` event is emitted before the provider blocks. This allows the consumer to present a UI and then unblock the provider:

```go
for event := range myAgent.ChatStream(ctx, "Analyze this large dataset") {
    switch event.Type {
    case types.EventContinuationRequired:
        req := event.ContinuationRequest
        fmt.Printf("Agent reached %d iterations. It wants to call %d tools.\n", req.MaxIterations, len(req.ToolCalls))
        // The provider is blocking — respond via whatever mechanism it uses
    case types.EventContentDelta:
        fmt.Print(event.Content)
    case types.EventComplete:
        fmt.Println("\nDone!")
    }
}
```

A common pattern is to use a channel-based provider that the streaming consumer unblocks, similar to tool confirmation:

```go
type continuationApproval struct {
    response agent.ContinuationResponse
    ch       chan struct{}
}

// In the provider:
// ... wait on channel, return decision ...

// In the stream consumer:
// ... unblock channel with decision ...
```

## Auto-Approve Patterns

The provider is a regular function — implement any continuation logic:

```go
// Always approve (effectively disables max iterations, use with caution!)
agent.WithContinuationProvider(
    func(_ context.Context, _ agent.ContinuationRequest) (agent.ContinuationResponse, error) {
        return agent.ContinuationResponse{Decision: agent.ContinuationApprove}, nil
    },
)

// Approve up to a hard limit
agent.WithContinuationProvider(
    func(ctx context.Context, req agent.ContinuationRequest) (agent.ContinuationResponse, error) {
        if req.TotalIterations >= 20 {
            return agent.ContinuationResponse{Decision: agent.ContinuationDecline}, nil
        }
        return agent.ContinuationResponse{Decision: agent.ContinuationApprove}, nil
    },
)
```


---

## Agent Framework > Sub-Agents

> Source: agent/sub-agents.md

# Sub-Agents

Sub-agents let an orchestrator delegate tasks to specialized child agents. Each sub-agent becomes a callable tool.

## Setup

```go
researcher := agent.New(llmClient,
    agent.WithSystemPrompt("You are a research specialist."),
    agent.WithTools(&webSearchTool{}),
)

writer := agent.New(llmClient,
    agent.WithSystemPrompt("You are a content writer."),
)

orchestrator := agent.New(llmClient,
    agent.WithSystemPrompt("You coordinate research and writing tasks."),
    agent.WithSubAgents(
        agent.SubAgentConfig{Name: "researcher", Description: "Researches topics", Agent: researcher},
        agent.SubAgentConfig{Name: "writer", Description: "Writes content", Agent: writer},
    ),
)

response, _ := orchestrator.Chat(ctx, "Research and write about quantum computing")
```

## How It Works

1. Each `SubAgentConfig` registers a tool named after the sub-agent
2. The orchestrator LLM decides when to delegate a task
3. The sub-agent runs to completion and returns its response
4. The orchestrator continues with the sub-agent's output

## SubAgentConfig

```go
type SubAgentConfig struct {
    Name        string  // Tool name the orchestrator calls
    Description string  // Describes when to use this sub-agent
    Agent       *Agent  // The sub-agent instance
}
```

## Background Execution

Sub-agents can run asynchronously by passing `background: true`. The orchestrator gets a `task_id` immediately and can check status or wait for results later.

```go
orchestrator := agent.New(llmClient,
    agent.WithSystemPrompt(`Launch background tasks, then collect results.`),
    agent.WithSubAgents(
        agent.SubAgentConfig{
            Name:        "researcher",
            Description: "Research a topic. Supports background: true for async execution.",
            Agent:       researcher,
        },
    ),
)
```

When the LLM calls the sub-agent with `background: true`:

1. The task launches in a goroutine and returns `{"task_id": "task-1", "status": "launched"}`
2. Three task management tools are automatically available: `get_task_result`, `stop_task`, `list_tasks`
3. The orchestrator uses `get_task_result` with `wait: true` to collect results

See [Background Agents](background-agents.md) for the full tool reference and examples.


---

## Agent Framework > Background Agents

> Source: agent/background-agents.md

# Background Agents

Background agents let the orchestrator launch sub-agents asynchronously. Tasks run in goroutines and the orchestrator can continue working, check status, or wait for results.

## Setup

```go
researcher := agent.New(llmClient,
    agent.WithSystemPrompt("You are a concise research assistant."),
)

orchestrator := agent.New(llmClient,
    agent.WithSystemPrompt(`You coordinate research tasks.
1. Launch background tasks with background: true
2. Collect results with get_task_result (wait: true)
3. Synthesize the results`),
    agent.WithSubAgents(
        agent.SubAgentConfig{
            Name:        "researcher",
            Description: "Research a topic. Supports background: true for async execution.",
            Agent:       researcher,
        },
    ),
)
```

## How It Works

1. The orchestrator calls a sub-agent tool with `background: true`
2. The sub-agent launches in a goroutine and returns a `task_id` immediately
3. Three task management tools are auto-registered for the orchestrator:

| Tool | Description |
|------|-------------|
| `get_task_result` | Check status or wait for a background task to complete |
| `stop_task` | Cancel a running background task |
| `list_tasks` | List all background tasks and their status |

## Task Lifecycle

Tasks move through these states:

| Status | Description |
|--------|-------------|
| `running` | Task is currently executing |
| `completed` | Task finished successfully |
| `failed` | Task encountered an error |
| `cancelled` | Task was explicitly cancelled |

## Tool Reference

### get_task_result

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `task_id` | string | yes | The task ID returned when the task was launched |
| `wait` | bool | no | If true, block until the task completes |
| `timeout` | int | no | Max wait time in milliseconds. 0 means no timeout |

### stop_task

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `task_id` | string | yes | The task ID to cancel |

### list_tasks

No parameters. Returns all tasks with their ID, agent name, and status.

## Streaming Example

```go
for event := range orchestrator.ChatStream(ctx, "Compare Go and Rust. Research each in the background.") {
    switch event.Type {
    case types.EventContentDelta:
        fmt.Print(event.Content)
    case types.EventError:
        log.Fatal(event.Error)
    }
    if event.ToolResult != nil {
        fmt.Printf("\n[Tool: %s → %s]\n", event.ToolResult.ToolName, event.ToolResult.Output)
    }
}
```

## Sub-Agent Input

When a sub-agent is called, it accepts these parameters:

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `task` | string | yes | The task or question to send to the sub-agent |
| `background` | bool | no | If true, run in background and return a task ID |
| `max_turns` | int | no | Maximum tool-execution turns. 0 uses the agent default |


---

## Agent Framework > Handoffs

> Source: agent/handoffs.md

# Handoffs

Handoffs transfer full control from one agent to another. Unlike sub-agents (which return results to the orchestrator), handoffs permanently switch the active agent.

## Setup

```go
billing := agent.New(llmClient,
    agent.WithSystemPrompt("You handle billing inquiries."),
)

support := agent.New(llmClient,
    agent.WithSystemPrompt("You handle technical support."),
)

triage := agent.New(llmClient,
    agent.WithSystemPrompt("Route the user to the right specialist."),
    agent.WithHandoffs(
        agent.HandoffConfig{Name: "billing", Description: "Billing questions", Agent: billing},
        agent.HandoffConfig{Name: "support", Description: "Technical issues", Agent: support},
    ),
)

response, _ := triage.Chat(ctx, "I was charged twice on my last invoice")
fmt.Println(response.AgentName) // "billing"
```

## How It Works

1. Each `HandoffConfig` auto-generates a `transfer_to_<name>` tool
2. When the triage agent calls `transfer_to_billing`, control transfers permanently
3. The billing agent's system prompt replaces the triage agent's
4. The conversation history carries over
5. `ChatResponse.AgentName` indicates which agent produced the final response

## HandoffConfig

```go
type HandoffConfig struct {
    Name        string  // Used to generate transfer_to_<name> tool
    Description string  // Tells the LLM when to transfer
    Agent       *Agent  // The target agent
}
```

## Handoffs vs Sub-Agents

| | Sub-Agents | Handoffs |
|---|---|---|
| Control flow | Returns to orchestrator | Permanent transfer |
| System prompt | Sub-agent uses its own | Replaces current |
| Use case | Task delegation | Routing/triage |


---

## Agent Framework > Fan-Out

> Source: agent/fan-out.md

# Fan-Out

Fan-out distributes multiple tasks to worker agents in parallel and collects results.

## Setup

```go
researcher := agent.New(llmClient,
    agent.WithSystemPrompt("Research the given topic thoroughly."),
)

coordinator := agent.New(llmClient,
    agent.WithSystemPrompt("You coordinate parallel research tasks."),
    agent.WithFanOut(agent.FanOutConfig{
        Name:           "research",
        Description:    "Research multiple topics in parallel",
        Agent:          researcher,
        MaxConcurrency: 3,
    }),
)

response, _ := coordinator.Chat(ctx, "Compare AI, blockchain, and quantum computing")
```

## How It Works

1. The `FanOutConfig` registers a tool that accepts multiple tasks
2. When the coordinator calls the fan-out tool, all tasks run concurrently
3. `MaxConcurrency` limits how many worker agents run at the same time
4. Results are collected and returned to the coordinator

## FanOutConfig

```go
type FanOutConfig struct {
    Name           string  // Tool name
    Description    string  // Describes when to use fan-out
    Agent          *Agent  // Worker agent (cloned per task)
    MaxConcurrency int     // Max parallel workers (0 = unlimited)
}
```


---

## Agent Framework > Team Coordination

> Source: agent/team-coordination.md

# Team Coordination

Team coordination enables peer-to-peer multi-agent collaboration where a lead agent dynamically spawns teammates that communicate via asynchronous messaging and coordinate work through a shared task board.

## Setup

```go
import (
    "github.com/joakimcarlsson/ai/agent"
    "github.com/joakimcarlsson/ai/agent/team"
)

lead := agent.New(llmClient,
    agent.WithSystemPrompt("You are a team lead. Spawn teammates and coordinate their work."),
    agent.WithTeam(team.Config{
        Name:    "research-team",
        MaxSize: 5,
    }),
)

response, _ := lead.Chat(ctx, "Research AI, blockchain, and quantum computing in parallel")
```

The lead agent automatically receives team management tools (`spawn_teammate`, `stop_teammate`) and communication tools (`send_message`, `read_messages`, `list_teammates`), plus task board tools (`create_board_task`, `claim_board_task`, `complete_board_task`, `list_board_tasks`).

Spawned teammates receive communication and task board tools but cannot spawn or stop other teammates.

## How It Works

1. The lead agent decides when to spawn teammates using `spawn_teammate`
2. Each teammate runs concurrently in its own goroutine with an isolated context
3. Teammates communicate with each other and the lead via the mailbox
4. The shared task board provides structured work coordination
5. When a teammate completes, its result is automatically sent to the lead's inbox
6. The lead can stop any teammate at any time via `stop_teammate`

## Configuration Options

| Option | Description |
|--------|-------------|
| `WithTeam(config)` | Configures the agent as a team lead with the given team config |
| `WithCoordinatorMode()` | Restricts the lead to only team tools, preventing direct tool execution |
| `WithMailbox(mb)` | Overrides the default in-memory mailbox (must be called after `WithTeam`) |
| `WithTeammateTemplates(map)` | Registers pre-configured agent templates by name |

### team.Config

| Field | Type | Description |
|-------|------|-------------|
| `Name` | `string` | Team name |
| `MaxSize` | `int` | Maximum concurrent active teammates (0 = unlimited) |

## Tools

### Lead-Only Tools

| Tool | Parameters | Description |
|------|-----------|-------------|
| `spawn_teammate` | `name`, `task`, `system_prompt?`, `max_turns?` | Spawn a new concurrent teammate |
| `stop_teammate` | `name` | Cancel a running teammate's context |

### Communication Tools (All Members)

| Tool | Parameters | Description |
|------|-----------|-------------|
| `send_message` | `to`, `content` | Send a message to a teammate or `*` for broadcast |
| `read_messages` | — | Read and consume all unread inbox messages |
| `list_teammates` | — | List all teammates and their current status |

### Task Board Tools (All Members)

| Tool | Parameters | Description |
|------|-----------|-------------|
| `create_board_task` | `title` | Create a new open task on the shared board |
| `claim_board_task` | `task_id` | Claim an open task |
| `complete_board_task` | `task_id`, `result` | Mark a claimed task as completed with a result |
| `list_board_tasks` | — | List all tasks on the board |

## Member Lifecycle

Each teammate moves through these states:

| Status | Description |
|--------|-------------|
| `active` | Running and processing its task |
| `completed` | Finished successfully |
| `failed` | Encountered an error or panic |
| `stopped` | Cancelled by the lead via `stop_teammate` |

## Messaging

The mailbox provides asynchronous, fire-and-forget message passing between team members.

```go
lead := agent.New(llmClient,
    agent.WithSystemPrompt(`You are a team lead.
Spawn a researcher and a writer.
The researcher should send findings to the writer via send_message.
The writer should read_messages to get the research before writing.`),
    agent.WithTeam(team.Config{Name: "content-team", MaxSize: 3}),
)
```

Key behaviors:

- **Point-to-point**: Set `to` to a teammate's name
- **Broadcast**: Set `to` to `*` to reach all teammates except the sender
- **Consumed on read**: Messages are removed from the inbox after `read_messages`
- **Lead inbox**: The lead's recipient name is `__lead__` — teammate completion results are automatically sent here

## Task Board

The task board provides structured coordination where teammates can create, claim, and complete shared tasks.

```go
lead := agent.New(llmClient,
    agent.WithSystemPrompt(`You are a team lead.
Create board tasks for each research topic.
Spawn teammates that claim and complete tasks from the board.`),
    agent.WithTeam(team.Config{Name: "task-team"}),
)
```

Task lifecycle: **Open** → **Claimed** → **Completed**

- Any member can create tasks
- Only open tasks can be claimed
- Only the assignee can complete their claimed task

## Teammate Templates

Pre-configure teammate agents with specific tools and settings:

```go
researcher := agent.New(llmClient,
    agent.WithSystemPrompt("You are a research specialist."),
    agent.WithTools(&webSearchTool{}),
)

writer := agent.New(llmClient,
    agent.WithSystemPrompt("You are a content writer."),
)

lead := agent.New(llmClient,
    agent.WithSystemPrompt("Coordinate research and writing."),
    agent.WithTeam(team.Config{Name: "content-team"}),
    agent.WithTeammateTemplates(map[string]*agent.Agent{
        "researcher": researcher,
        "writer":     writer,
    }),
)
```

When `spawn_teammate` is called with a name matching a template, the pre-configured agent is used instead of dynamically creating one.

## Coordinator Mode

Restrict the lead to only team management and communication tools:

```go
lead := agent.New(llmClient,
    agent.WithSystemPrompt("You only coordinate. Delegate all work to teammates."),
    agent.WithTeam(team.Config{Name: "my-team"}),
    agent.WithCoordinatorMode(),
)
```

In coordinator mode, any non-team tools registered on the lead are filtered out. The lead can only spawn teammates, communicate, and manage the task board.

## Streaming Events

Team coordination emits streaming events during `ChatStream`:

| Event | Description |
|-------|-------------|
| `EventTeammateSpawned` | A new teammate was launched |
| `EventTeamMessage` | A message was sent between members |
| `EventTeammateComplete` | A teammate finished successfully |
| `EventTeammateError` | A teammate encountered an error |

```go
for event := range lead.ChatStream(ctx, "Research these topics") {
    switch event.Type {
    case types.EventTeammateSpawned:
        fmt.Printf("Spawned: %s\n", event.AgentName)
    case types.EventTeammateComplete:
        fmt.Printf("Completed: %s\n", event.AgentName)
    case types.EventTeammateError:
        fmt.Printf("Error in %s: %v\n", event.AgentName, event.Error)
    case types.EventComplete:
        fmt.Println(event.Response.Content)
    }
}
```

## Hooks

Team hooks provide observation-only callbacks for teammate lifecycle and messaging events:

| Hook | Fires | Context |
|------|-------|---------|
| `OnTeammateJoin` | When a teammate is spawned | `TeammateEventContext` |
| `OnTeammateLeave` | When a teammate leaves (stopped) | `TeammateEventContext` |
| `OnTeammateComplete` | When a teammate finishes successfully | `TeammateEventContext` |
| `OnTeammateError` | When a teammate encounters an error | `TeammateEventContext` |
| `OnTeamMessage` | When a message is sent between members | `TeamMessageContext` |

```go
lead := agent.New(llmClient,
    agent.WithTeam(team.Config{Name: "my-team"}),
    agent.WithHooks(agent.Hooks{
        OnTeammateJoin: func(_ context.Context, tc agent.TeammateEventContext) {
            log.Printf("Teammate %s joined team %s", tc.MemberName, tc.TeamName)
        },
        OnTeammateComplete: func(_ context.Context, tc agent.TeammateEventContext) {
            log.Printf("Teammate %s completed in %s: %s", tc.MemberName, tc.Duration, tc.Result)
        },
        OnTeamMessage: func(_ context.Context, mc agent.TeamMessageContext) {
            log.Printf("Message from %s to %s: %s", mc.Message.From, mc.Message.To, mc.Message.Content)
        },
    }),
)
```

Hooks set on the lead automatically propagate to spawned teammates that don't have their own hooks.

### TeammateEventContext

```go
type TeammateEventContext struct {
    TeamName   string
    MemberID   string
    MemberName string
    Task       string
    Result     string
    Error      error
    Duration   time.Duration
}
```

### TeamMessageContext

```go
type TeamMessageContext struct {
    TeamName string
    Message  team.Message
}
```

### team.Message

```go
type Message struct {
    ID        string
    From      string
    To        string
    Content   string
    Timestamp time.Time
}
```

## Comparison with Other Multi-Agent Patterns

| Pattern | Use Case |
|---------|----------|
| [Sub-Agents](sub-agents.md) | Hierarchical delegation — orchestrator calls child agents as tools |
| [Handoffs](handoffs.md) | Sequential transfer — one agent passes control to a peer |
| [Fan-Out](fan-out.md) | Parallel execution — distribute independent tasks to cloned workers |
| **Team Coordination** | Peer-to-peer collaboration — dynamic spawning, messaging, shared task board |


---

## Agent Framework > Continue/Resume

> Source: agent/continue.md

# Continue/Resume

`Continue()` lets you manually execute tool calls and feed results back into the agent loop. This is useful when tools require human approval, external API calls, or custom execution logic.

## Setup

```go
myAgent := agent.New(llmClient,
    agent.WithAutoExecute(false), // Don't auto-execute tools
    agent.WithSession("conv-1", session.MemoryStore()),
)
```

## Usage

```go
// First call returns pending tool calls instead of executing them
response, _ := myAgent.Chat(ctx, "Search for flights to Tokyo")

// Inspect what tools the LLM wants to call
for _, tc := range response.ToolCalls {
    fmt.Printf("Tool: %s, Input: %s\n", tc.Name, tc.Input)
}

// Execute tools externally with your own logic
results := []message.ToolResult{
    {
        ToolCallID: response.ToolCalls[0].ID,
        Name:       "search_flights",
        Content:    `{"flights": [{"airline": "JAL", "price": 850}]}`,
    },
}

// Resume the agent loop with results
response, _ = myAgent.Continue(ctx, results)
fmt.Println(response.Content)
```

## Streaming Variant

```go
for event := range myAgent.ContinueStream(ctx, results) {
    switch event.Type {
    case types.EventContentDelta:
        fmt.Print(event.Content)
    case types.EventComplete:
        fmt.Println("\nDone!")
    }
}
```

> **Note:**
> `Continue()` requires a session to be configured, since it needs to restore conversation state from the previous `Chat()` call.
>

---

## Agent Framework > Context Strategies

> Source: agent/context-strategies.md

# Context Strategies

Context strategies automatically manage the context window when conversations grow beyond token limits.

## Available Strategies

### Sliding Window

Keep only the last N messages:

```go
import "github.com/joakimcarlsson/ai/tokens/sliding"

myAgent := agent.New(llmClient,
    agent.WithSystemPrompt("You are a helpful assistant."),
    agent.WithSession("conv-1", store),
    agent.WithContextStrategy(sliding.Strategy(sliding.KeepLast(10)), 0),
)
```

### Truncate

Remove oldest messages to fit the token budget:

```go
import "github.com/joakimcarlsson/ai/tokens/truncate"

myAgent := agent.New(llmClient,
    agent.WithContextStrategy(truncate.Strategy(), 0),
)
```

### Summarize

Use an LLM to compress older messages into a summary:

```go
import "github.com/joakimcarlsson/ai/tokens/summarize"

myAgent := agent.New(llmClient,
    agent.WithContextStrategy(summarize.Strategy(llmClient), 0),
)
```

## How It Works

Before each LLM call, the agent:

1. Counts tokens for all messages + system prompt + tools
2. If total exceeds the limit, applies the strategy
3. The strategy reduces messages while preserving recent context
4. The session is updated if the strategy produces a session update (e.g., summary message)

## Custom Max Tokens

The second argument to `WithContextStrategy` sets a custom max token limit. Pass `0` to auto-calculate from the model's context window minus a 4096-token reserve.

```go
// Custom limit: 50k tokens
agent.WithContextStrategy(sliding.Strategy(sliding.KeepLast(20)), 50000)
```

## Custom Strategy

Implement the `tokens.Strategy` interface:

```go
type Strategy interface {
    Fit(ctx context.Context, input StrategyInput) (*StrategyResult, error)
}
```


---

## Agent Framework > Toolsets

> Source: agent/toolsets.md

# Toolsets

Toolsets group multiple tools under a name with optional dynamic filtering. Unlike static tool lists, toolsets are resolved **per-call** — the predicate runs on every `Chat()` turn, so you can enable or disable tools based on runtime context.

## Creating a Toolset

A basic toolset is a named collection of tools:

```go
recon := tool.NewToolset("recon",
    &NmapTool{},
    &DnsLookupTool{},
    &WhoisTool{},
)

a := agent.New(llmClient,
    agent.WithToolsets(recon),
)
```

You can mix toolsets with individual tools:

```go
a := agent.New(llmClient,
    agent.WithTools(&AlwaysAvailableTool{}),
    agent.WithToolsets(recon, exploitation),
)
```

## Filtered Toolsets

`NewFilterToolset` wraps a toolset with a predicate that controls which tools are available. The predicate receives the `context.Context` and each tool, and returns whether that tool should be included.

```go
type phaseKey struct{}

allTools := tool.NewToolset("pentest",
    &NmapTool{},
    &SqlInjectionTool{},
    &BruteForcePasswordTool{},
)

filtered := tool.NewFilterToolset("phase-aware", allTools,
    func(ctx context.Context, t tool.BaseTool) bool {
        phase, _ := ctx.Value(phaseKey{}).(string)
        switch t.Info().Name {
        case "sql_injection", "brute_force_password":
            return phase == "exploitation"
        default:
            return true
        }
    },
)

a := agent.New(llmClient,
    agent.WithToolsets(filtered),
)

// During recon phase, only NmapTool is available
ctx := context.WithValue(ctx, phaseKey{}, "recon")
resp, _ := a.Chat(ctx, "Start scanning the target")

// During exploitation phase, all tools are available
ctx = context.WithValue(ctx, phaseKey{}, "exploitation")
resp, _ = a.Chat(ctx, "Try exploiting the SQL injection")
```

### Filtering by Configuration

Predicates can also read from engagement configuration or any other source:

```go
type EngagementConfig struct {
    AllowBruteForce bool
    AllowExploits   bool
}

configKey := struct{}{}

filtered := tool.NewFilterToolset("engagement", allTools,
    func(ctx context.Context, t tool.BaseTool) bool {
        cfg, _ := ctx.Value(configKey).(*EngagementConfig)
        if cfg == nil {
            return false
        }
        switch t.Info().Name {
        case "brute_force":
            return cfg.AllowBruteForce
        case "sql_injection", "xss_scanner":
            return cfg.AllowExploits
        default:
            return true
        }
    },
)
```

## Composing Toolsets

Toolsets compose — use `NewCompositeToolset` to merge multiple toolsets into one:

```go
recon := tool.NewToolset("recon", &NmapTool{}, &DnsLookupTool{})
exploit := tool.NewToolset("exploit", &SqlInjectionTool{})
reporting := tool.NewToolset("reporting", &ReportTool{})

all := tool.NewCompositeToolset("full-suite", recon, exploit, reporting)
```

Composite toolsets work with filtered toolsets too — you can filter individual groups and then compose them:

```go
filteredExploit := tool.NewFilterToolset("filtered-exploit", exploit, exploitPredicate)
combined := tool.NewCompositeToolset("suite", recon, filteredExploit, reporting)
```

## MCP Toolsets

Wrap MCP server tools as a toolset:

```go
mcpTools := tool.MCPToolset("external", map[string]tool.MCPServer{
    "filesystem": {
        Command: "npx",
        Args:    []string{"-y", "@modelcontextprotocol/server-filesystem", "/tmp"},
        Type:    tool.MCPStdio,
    },
})

a := agent.New(llmClient,
    agent.WithToolsets(mcpTools),
)
```

## Confirmation Wrapper

`tool.WithConfirmation` wraps a toolset so every tool in it requires human approval before execution. Pair it with `WithConfirmationProvider` on the agent:

```go
dangerous := tool.NewToolset("exploits",
    &SqlInjectionTool{},
    &BruteForcePasswordTool{},
)

a := agent.New(llmClient,
    agent.WithToolsets(tool.WithConfirmation(dangerous)),
    agent.WithConfirmationProvider(myApprovalHandler),
)
```

The original toolset is not modified. See [Tool Confirmation](confirmation.md) for the full protocol.

## Toolsets and Hooks

Since toolsets resolve to `[]tool.BaseTool`, [hooks](hooks.md) apply to individual tools regardless of how they were grouped:

```go
a := agent.New(llmClient,
    agent.WithToolsets(exploitToolset),
    agent.WithHooks(agent.Hooks{
        PreToolUse: func(ctx context.Context, tc agent.ToolUseContext) (agent.PreToolUseResult, error) {
            if tc.ToolName == "sql_injection" {
                return agent.PreToolUseResult{
                    Action:     agent.HookDeny,
                    DenyReason: "SQL injection blocked by policy",
                }, nil
            }
            return agent.PreToolUseResult{Action: agent.HookAllow}, nil
        },
    }),
)
```

## Custom Toolset Implementations

The `Toolset` interface is simple — implement it for custom resolution logic:

```go
type Toolset interface {
    Name() string
    Tools(ctx context.Context) []tool.BaseTool
}
```

For example, a toolset that loads tools from a database:

```go
type DBToolset struct {
    db *sql.DB
}

func (d *DBToolset) Name() string { return "db-tools" }

func (d *DBToolset) Tools(ctx context.Context) []tool.BaseTool {
    // Query available tools from database based on user permissions
    rows, _ := d.db.QueryContext(ctx, "SELECT name, config FROM tools WHERE enabled = true")
    // ... build and return tools
}
```


---

## Agent Framework > Instruction Templates

> Source: agent/instruction-templates.md

# Instruction Templates

Dynamic system prompts using template variables or runtime-generated instructions.

## Static Templates

Use Go template syntax (`{{.var}}`) with `WithState`:

```go
myAgent := agent.New(llmClient,
    agent.WithSystemPrompt("You are {{.role}}. Help {{.user_name}} with their tasks."),
    agent.WithState(map[string]any{
        "role":      "a coding assistant",
        "user_name": "Alice",
    }),
)
```

## Conditional Templates

```go
myAgent := agent.New(llmClient,
    agent.WithSystemPrompt(`You are a helpful assistant.
{{if .extra_context}}
Additional context: {{.extra_context}}
{{end}}`),
    agent.WithState(map[string]any{
        "extra_context": "The user prefers concise answers.",
    }),
)
```

## Dynamic Provider

For fully dynamic prompts generated at runtime:

```go
myAgent := agent.New(llmClient,
    agent.WithInstructionProvider(func(ctx context.Context, state map[string]any) (string, error) {
        return fmt.Sprintf(
            "Current time: %s\nYou are a helpful assistant.",
            time.Now().Format(time.RFC3339),
        ), nil
    }),
)
```

The instruction provider receives the state map and can use it alongside any other runtime data (database lookups, feature flags, etc.).


---

## Integrations > PostgreSQL

> Source: integrations/postgres.md

# PostgreSQL

PostgreSQL-backed session store for persistent conversation history. No
extensions required.

## Installation

```bash
go get github.com/joakimcarlsson/ai/memory/postgres
```

## Setup

```go
import pgsess "github.com/joakimcarlsson/ai/memory/postgres"

sessionStore, err := pgsess.SessionStore(ctx,
    "postgres://user:pass@localhost:5432/mydb?sslmode=disable",
)
if err != nil {
    log.Fatal(err)
}

myAgent := agent.New(llmClient,
    agent.WithSession("conv-1", sessionStore),
)
```

Tables and indexes are created automatically on first use.

## Schema

```sql
CREATE TABLE sessions (
    id TEXT PRIMARY KEY,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE messages (
    id TEXT PRIMARY KEY,
    session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
    role TEXT NOT NULL,
    parts JSONB NOT NULL,
    model TEXT,
    created_at BIGINT NOT NULL
);

CREATE INDEX messages_session_idx ON messages(session_id, created_at);
```

## Options

| Option | Description |
|---|---|
| `pgsess.WithIDGenerator(fn)` | Custom ID generator for message records. Default: UUID v4 |

```go
store, err := pgsess.SessionStore(ctx, connString,
    pgsess.WithIDGenerator(func() string {
        return myCustomID()
    }),
)
```

## Full example

```go
package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/joakimcarlsson/ai/agent"
    "github.com/joakimcarlsson/ai/memory"
    pgvectormem "github.com/joakimcarlsson/ai/memory/pgvector"
    pgsess "github.com/joakimcarlsson/ai/memory/postgres"
    embopenai "github.com/joakimcarlsson/ai/embeddings/openai"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
    embeddingsopenai "github.com/joakimcarlsson/ai/embeddings/openai"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
)

func main() {
    ctx := context.Background()
    connString := "postgres://postgres:password@localhost:5432/example?sslmode=disable"

    embedder := embopenai.NewEmbedding(
        embopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
        embopenai.WithModel(embeddingsopenai.Models[embeddingsopenai.TextEmbedding3Small]),
    )

    llmClient := llmopenai.NewLLM(
        llmopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
        llmopenai.WithModel(llmopenai.Models[llmopenai.GPT4o]),
    )

    sessionStore, err := pgsess.SessionStore(ctx, connString)
    if err != nil {
        log.Fatal(err)
    }

    memoryStore, err := pgvectormem.MemoryStore(ctx, connString, embedder)
    if err != nil {
        log.Fatal(err)
    }

    myAgent := agent.New(llmClient,
        agent.WithSystemPrompt("You are a personal assistant with memory."),
        agent.WithSession("conv-1", sessionStore),
        agent.WithMemory("alice", memoryStore,
            memory.AutoExtract(),
            memory.AutoDedup(),
        ),
    )

    response, err := myAgent.Chat(ctx, "Hi! My name is Alice and I love Italian food.")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(response.Content)
}
```


---

## Integrations > SQLite

> Source: integrations/sqlite.md

# SQLite

SQLite-backed session store for lightweight persistent conversation history.
Bring your own `*sql.DB` connection with any SQLite driver — the module
itself has zero external SDK deps.

## Installation

```bash
go get github.com/joakimcarlsson/ai/memory/sqlite
```

## Setup

```go
import (
    "database/sql"

    _ "modernc.org/sqlite" // or any SQLite driver
    sqlitemem "github.com/joakimcarlsson/ai/memory/sqlite"
)

db, err := sql.Open("sqlite", "./chat.db")
if err != nil {
    log.Fatal(err)
}

sessionStore, err := sqlitemem.SessionStore(ctx, db)
if err != nil {
    log.Fatal(err)
}

myAgent := agent.New(llmClient,
    agent.WithSession("conv-1", sessionStore),
)
```

Tables and indexes are created automatically on first use.

## Schema

```sql
CREATE TABLE sessions (
    id         TEXT PRIMARY KEY,
    created_at INTEGER NOT NULL
);

CREATE TABLE messages (
    id         INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
    role       TEXT NOT NULL,
    parts      TEXT NOT NULL,
    model      TEXT,
    created_at INTEGER NOT NULL
);

CREATE INDEX idx_messages_session ON messages(session_id, id);
```

## Options

| Option | Description |
|---|---|
| `sqlitemem.WithTablePrefix(prefix)` | Prefix for all table names. Useful for multi-tenant or multiple stores in one database |

```go
store, err := sqlitemem.SessionStore(ctx, db,
    sqlitemem.WithTablePrefix("chat_"),
)
// Creates "chat_sessions" and "chat_messages" instead of "sessions" and "messages"
```

## Full example

```go
package main

import (
    "context"
    "database/sql"
    "fmt"
    "log"
    "os"

    _ "modernc.org/sqlite"

    "github.com/joakimcarlsson/ai/agent"
    sqlitemem "github.com/joakimcarlsson/ai/memory/sqlite"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
)

func main() {
    ctx := context.Background()

    db, err := sql.Open("sqlite", "./chat.db")
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    llmClient := llmopenai.NewLLM(
        llmopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
        llmopenai.WithModel(llmopenai.Models[llmopenai.GPT4o]),
    )

    sessionStore, err := sqlitemem.SessionStore(ctx, db)
    if err != nil {
        log.Fatal(err)
    }

    myAgent := agent.New(llmClient,
        agent.WithSystemPrompt("You are a helpful assistant."),
        agent.WithSession("conv-1", sessionStore),
    )

    response, err := myAgent.Chat(ctx, "Hello!")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(response.Content)
}
```


---

## Integrations > pgvector

> Source: integrations/pgvector.md

# pgvector

PostgreSQL-backed memory store using [pgvector](https://github.com/pgvector/pgvector)
for semantic vector search. Stores facts as embeddings and retrieves them
using cosine similarity with HNSW indexing.

## Prerequisites

The pgvector extension must be available in your PostgreSQL instance. The
extension is enabled automatically on first use.

## Installation

```bash
go get github.com/joakimcarlsson/ai/memory/pgvector
```

## Setup

```go
import (
    "github.com/joakimcarlsson/ai/agent"
    "github.com/joakimcarlsson/ai/memory"
    pgvectormem "github.com/joakimcarlsson/ai/memory/pgvector"
)

memoryStore, err := pgvectormem.MemoryStore(ctx,
    "postgres://user:pass@localhost:5432/mydb?sslmode=disable",
    embedder,
)
if err != nil {
    log.Fatal(err)
}

myAgent := agent.New(llmClient,
    agent.WithMemory("user-123", memoryStore,
        memory.AutoExtract(),
        memory.AutoDedup(),
    ),
)
```

The table, pgvector extension, and HNSW index are created automatically on
first use. The vector dimension is auto-detected from the embedder's model
configuration.

## Schema

```sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE memories (
    id TEXT PRIMARY KEY,
    owner_id TEXT NOT NULL,
    content TEXT NOT NULL,
    vector vector(1536),  -- dimension from embedder
    metadata JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX memories_owner_idx ON memories(owner_id);
CREATE INDEX memories_vector_idx ON memories USING hnsw (vector vector_cosine_ops);
```

## Options

| Option | Description |
|---|---|
| `pgvectormem.WithIDGenerator(fn)` | Custom ID generator for memory records. Default: UUID v4 |

```go
store, err := pgvectormem.MemoryStore(ctx, connString, embedder,
    pgvectormem.WithIDGenerator(func() string {
        return myCustomID()
    }),
)
```

## Full example

```go
package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/joakimcarlsson/ai/agent"
    "github.com/joakimcarlsson/ai/memory"
    pgvectormem "github.com/joakimcarlsson/ai/memory/pgvector"
    pgsessmem "github.com/joakimcarlsson/ai/memory/postgres"
    embopenai "github.com/joakimcarlsson/ai/embeddings/openai"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
    embeddingsopenai "github.com/joakimcarlsson/ai/embeddings/openai"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
)

func main() {
    ctx := context.Background()
    connString := "postgres://postgres:password@localhost:5432/example?sslmode=disable"

    embedder := embopenai.NewEmbedding(
        embopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
        embopenai.WithModel(embeddingsopenai.Models[embeddingsopenai.TextEmbedding3Small]),
    )

    llmClient := llmopenai.NewLLM(
        llmopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
        llmopenai.WithModel(llmopenai.Models[llmopenai.GPT4o]),
    )

    // PostgreSQL sessions + pgvector memory
    sessionStore, err := pgsessmem.SessionStore(ctx, connString)
    if err != nil {
        log.Fatal(err)
    }

    memoryStore, err := pgvectormem.MemoryStore(ctx, connString, embedder)
    if err != nil {
        log.Fatal(err)
    }

    myAgent := agent.New(llmClient,
        agent.WithSystemPrompt("You are a personal assistant with memory."),
        agent.WithSession("conv-1", sessionStore),
        agent.WithMemory("alice", memoryStore,
            memory.AutoExtract(),
            memory.AutoDedup(),
        ),
    )

    // First conversation — agent learns facts.
    response, err := myAgent.Chat(ctx, "Hi! My name is Alice and I love Italian food.")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(response.Content)

    // New conversation — agent recalls memories via vector search.
    agent2 := agent.New(llmClient,
        agent.WithSystemPrompt("You are a personal assistant with memory."),
        agent.WithSession("conv-2", sessionStore),
        agent.WithMemory("alice", memoryStore,
            memory.AutoExtract(),
            memory.AutoDedup(),
        ),
    )

    response, err = agent2.Chat(ctx, "Can you recommend a restaurant for me?")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(response.Content)
}
```


---

## Advanced > Batch Processing

> Source: advanced/batch-processing.md

# Batch Processing

Process bulk LLM and embedding requests efficiently using provider-native
batch APIs or bounded concurrent execution. Each batch backend is its own Go
module under `batch/`.

## Native batch APIs

Native batch APIs submit all requests as a single async job. Providers may
offer reduced pricing for batch workloads. Results return within 24 hours,
often much faster.

### OpenAI

```go
import (
    "github.com/joakimcarlsson/ai/batch"
    batchopenai "github.com/joakimcarlsson/ai/batch/openai"
    "github.com/joakimcarlsson/ai/message"
    "github.com/joakimcarlsson/ai/llm/openai"
)

proc := batchopenai.NewProcessor(
    batchopenai.WithAPIKey("your-api-key"),
    batchopenai.WithModel(openai.Models[openai.GPT4o]),
    batchopenai.WithPollInterval(30 * time.Second),
)

requests := []batch.Request{
    {
        ID:   "q1",
        Type: batch.RequestTypeChat,
        Messages: []message.Message{
            message.NewUserMessage("What is the capital of France?"),
        },
    },
    {
        ID:   "q2",
        Type: batch.RequestTypeChat,
        Messages: []message.Message{
            message.NewUserMessage("What is the capital of Japan?"),
        },
    },
}

resp, err := proc.Process(ctx, requests)
for _, r := range resp.Results {
    if r.Err != nil {
        fmt.Printf("[%s] Error: %v\n", r.ID, r.Err)
        continue
    }
    fmt.Printf("[%s] %s\n", r.ID, r.ChatResponse.Content)
}
```

### Anthropic

```go
import batchanthropic "github.com/joakimcarlsson/ai/batch/anthropic"

proc := batchanthropic.NewProcessor(
    batchanthropic.WithAPIKey("your-api-key"),
    batchanthropic.WithModel(anthropic.Models[anthropic.Claude45Sonnet]),
    batchanthropic.WithMaxTokens(1024),
    batchanthropic.WithPollInterval(30 * time.Second),
)
```

Anthropic's batch API supports chat only — submitting an embedding request
returns an error.

### Gemini / Vertex AI

```go
import batchgemini "github.com/joakimcarlsson/ai/batch/gemini"

proc := batchgemini.NewProcessor(
    batchgemini.WithAPIKey("your-api-key"),
    batchgemini.WithModel(gemini.Models[gemini.Gemini25Flash]),
    batchgemini.WithPollInterval(30 * time.Second),
)
```

For Vertex AI:

```go
import "google.golang.org/genai"

proc := batchgemini.NewProcessor(
    batchgemini.WithModel(...),
    batchgemini.WithBackend(genai.BackendVertexAI),
)
```

## Concurrent fallback

For providers without a native batch API, pass an existing LLM and/or
embedding client to the concurrent runner:

```go
import (
    batchconcurrent "github.com/joakimcarlsson/ai/batch/concurrent"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
)

groq := llmopenai.NewLLM(
    llmopenai.WithAPIKey(os.Getenv("GROQ_API_KEY")),
    llmopenai.WithBaseURL("https://api.groq.com/openai/v1"),
    llmopenai.WithModel(groq.Models[groq.Llama4Scout]),
)

proc := batchconcurrent.NewProcessor(
    batchconcurrent.WithLLM(groq),
    batchconcurrent.WithMaxConcurrency(10),
)

resp, _ := proc.Process(ctx, requests)
```

The concurrent runner is no longer an automatic fallthrough — you pick it
explicitly. This means consumers know exactly which package they're getting
and which deps it pulls.

## Batch embeddings

```go
import (
    embvoyage "github.com/joakimcarlsson/ai/embeddings/voyage"
)

embedder := embvoyage.NewEmbedding(
    embvoyage.WithAPIKey(os.Getenv("VOYAGE_API_KEY")),
    embvoyage.WithModel(voyage.Models[voyage.Voyage35]),
)

proc := batchconcurrent.NewProcessor(
    batchconcurrent.WithEmbedding(embedder),
    batchconcurrent.WithMaxConcurrency(5),
)

requests := []batch.Request{
    {ID: "doc1", Type: batch.RequestTypeEmbedding, Texts: []string{"first document"}},
    {ID: "doc2", Type: batch.RequestTypeEmbedding, Texts: []string{"second document"}},
}

resp, _ := proc.Process(ctx, requests)
```

## Provider support

| Module | Native batch | Discount (as of writing) | Supported endpoints |
|---|---|---|---|
| `batch/openai` | ✅ | 50% | Chat, Embeddings |
| `batch/anthropic` | ✅ | 50% | Messages (chat only) |
| `batch/gemini` | ✅ | 50% | Content, Embeddings |
| `batch/concurrent` | (concurrent fallback) | — | Chat, Embeddings via existing client |

## Progress tracking

### Callback

```go
proc := batchopenai.NewProcessor(
    batchopenai.WithAPIKey("your-api-key"),
    batchopenai.WithModel(openai.Models[openai.GPT4o]),
    batchopenai.WithProgressCallback(func(p batch.Progress) {
        fmt.Printf("%d/%d completed, %d failed [%s]\n",
            p.Completed, p.Total, p.Failed, p.Status)
    }),
)
```

### Async event channel

```go
ch, err := proc.ProcessAsync(ctx, requests)

for event := range ch {
    switch event.Type {
    case batch.EventItem:
        fmt.Printf("[%s] done\n", event.Result.ID)
    case batch.EventProgress:
        fmt.Printf("%d/%d\n", event.Progress.Completed, event.Progress.Total)
    case batch.EventComplete:
        fmt.Println("all done")
    case batch.EventError:
        fmt.Printf("batch error: %v\n", event.Err)
    }
}
```

## Error handling

Individual request failures never fail the batch — each result carries its
own error:

```go
resp, err := proc.Process(ctx, requests)

for _, r := range resp.Results {
    if r.Err != nil {
        continue
    }
    // use r.ChatResponse or r.EmbedResponse
}

fmt.Printf("Completed: %d, Failed: %d\n", resp.Completed, resp.Failed)
```

## Common options

Across the native-batch processors:

```go
batchopenai.WithAPIKey("...")
batchopenai.WithModel(...)
batchopenai.WithEmbeddingModel(...)
batchopenai.WithMaxTokens(4096)
batchopenai.WithProgressCallback(fn)
batchopenai.WithPollInterval(30 * time.Second)
batchopenai.WithTimeout(60 * time.Second)
```

OpenAI-specific:

```go
batchopenai.WithBaseURL("https://custom-endpoint")
batchopenai.WithExtraHeaders(map[string]string{"X-Header": "value"})
```

Gemini-specific:

```go
batchgemini.WithBackend(genai.BackendVertexAI)
```

Concurrent fallback:

```go
batchconcurrent.WithLLM(client)
batchconcurrent.WithEmbedding(client)
batchconcurrent.WithMaxConcurrency(10)
batchconcurrent.WithProgressCallback(fn)
```


---

## Advanced > BYOM

> Source: advanced/byom.md

# BYOM (Bring Your Own Model)

Use Ollama, LocalAI, vLLM, LM Studio, or any OpenAI-compatible inference
server. After the modular refactor, BYOM is just `llm/openai` with a custom
base URL — no separate code path. The `llm` module also ships a tiny config
registry for organising multiple BYOM endpoints.

## Direct setup

```go
import (
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
    "github.com/joakimcarlsson/ai/llm"
)

llamaModel := llm.NewCustomModel(
    llm.WithModelID("llama3.2"),
    llm.WithAPIModel("llama3.2:latest"),
    llm.WithContextWindow(128_000),
)

client := llmopenai.NewLLM(
    llmopenai.WithBaseURL("http://localhost:11434/v1"),
    llmopenai.WithModel(llamaModel),
    llmopenai.WithMaxTokens(2000),
)

response, _ := client.SendMessages(ctx, messages, nil)
```

That's the whole story for a single endpoint.

## Registry helper (multiple endpoints)

When you've got several BYOM configurations and want to pass them around as
opaque IDs rather than re-typing URLs, use the `llm` module's registry:

```go
import "github.com/joakimcarlsson/ai/llm"

ollama := llm.RegisterCustomProvider("ollama", llm.CustomProviderConfig{
    BaseURL:      "http://localhost:11434/v1",
    DefaultModel: llamaModel,
})

// Later, in some other part of your code:
cfg, ok := llm.GetCustomProvider(ollama)
if !ok {
    log.Fatal("unknown provider")
}

client := llmopenai.NewLLM(
    llmopenai.WithBaseURL(cfg.BaseURL),
    llmopenai.WithExtraHeaders(cfg.ExtraHeaders),
    llmopenai.WithModel(cfg.DefaultModel),
)
```

The registry stores config; you still construct the client explicitly with
`llmopenai.NewLLM(...)`. There's no implicit factory that dispatches based
on provider ID — the modular refactor removed that, so callers know exactly
which vendor module they're invoking.

## Custom model options

```go
customModel := llm.NewCustomModel(
    llm.WithModelID("my-model"),
    llm.WithAPIModel("my-model-v1"),         // sent in API requests
    llm.WithName("My Custom Model"),         // human-readable
    llm.WithProvider("my-provider"),         // provider identifier
    llm.WithContextWindow(131_072),
    llm.WithDefaultMaxTokens(4096),
    llm.WithStructuredOutput(true),
    llm.WithAttachments(true),
    llm.WithReasoning(true),
    llm.WithImageGeneration(false),
    llm.WithCurrency("USD"),
    llm.WithCostPer1MIn(1.50),
    llm.WithCostPer1MOut(5.00),
    llm.WithCostPer1MInCached(0.15),
    llm.WithCostPer1MOutCached(2.50),
)
```

| Option | Description | Default |
|---|---|---|
| `WithModelID(id)` | Unique identifier | `""` |
| `WithAPIModel(name)` | Model name sent in API requests | `""` |
| `WithName(name)` | Human-readable display name | `""` |
| `WithProvider(provider)` | Provider identifier | `"custom"` |
| `WithContextWindow(tokens)` | Max input context size | `0` |
| `WithDefaultMaxTokens(tokens)` | Recommended max output tokens | `0` |
| `WithStructuredOutput(bool)` | Enable structured JSON output | `false` |
| `WithAttachments(bool)` | Enable image/file inputs | `false` |
| `WithReasoning(bool)` | Enable chain-of-thought | `false` |
| `WithImageGeneration(bool)` | Enable image generation | `false` |
| `WithCurrency(code)` | ISO 4217 code the costs are in | `""` (USD) |
| `WithCostPer1MIn(cost)` | Input token cost per million | `0` |
| `WithCostPer1MOut(cost)` | Output token cost per million | `0` |
| `WithCostPer1MInCached(cost)` | Cached input token cost | `0` |
| `WithCostPer1MOutCached(cost)` | Cached output token cost | `0` |

Setting these correctly enables features like structured output, context
strategies, and cost tracking against your custom model.

## Extra headers

For tenant headers, custom auth, etc.:

```go
client := llmopenai.NewLLM(
    llmopenai.WithBaseURL("https://my-service.com/v1"),
    llmopenai.WithExtraHeaders(map[string]string{
        "X-API-Tenant":  "my-tenant",
        "Authorization": "Bearer my-token",
    }),
    llmopenai.WithModel(customModel),
)
```

## Streaming

Streaming works the same as any other vendor:

```go
stream := client.StreamResponse(ctx, messages, nil)

for event := range stream {
    switch event.Type {
    case types.EventContentDelta:
        fmt.Print(event.Content)
    case types.EventComplete:
        fmt.Println()
    case types.EventError:
        log.Fatal(event.Error)
    }
}
```

## Supported servers

Any server that implements the OpenAI-compatible `/chat/completions` API:

- **Ollama** — `http://localhost:11434/v1`
- **LocalAI** — `http://localhost:8080/v1`
- **vLLM** — `http://localhost:8000/v1`
- **LM Studio** — `http://localhost:1234/v1`
- **Groq** — `https://api.groq.com/openai/v1` (cloud, OpenAI-compatible)
- **OpenRouter** — `https://openrouter.ai/api/v1` (cloud, multi-vendor proxy)
- **xAI** — `https://api.x.ai/v1`
- **Mistral** — `https://api.mistral.ai/v1`


---

## Advanced > MCP Integration

> Source: advanced/mcp.md

# MCP (Model Context Protocol) Integration

This library integrates with the official [Model Context Protocol Go SDK](https://github.com/modelcontextprotocol/go-sdk) to provide seamless access to MCP servers and their tools.

## Stdio Connection (subprocess)

```go
import "github.com/joakimcarlsson/ai/tool"

mcpServers := map[string]tool.MCPServer{
    "filesystem": {
        Type:    tool.MCPStdio,
        Command: "npx",
        Args:    []string{"-y", "@modelcontextprotocol/server-filesystem", "/path/to/directory"},
        Env:     []string{"NODE_ENV=production"},
    },
}

mcpTools, err := tool.GetMcpTools(ctx, mcpServers)
if err != nil {
    log.Fatal(err)
}

response, err := client.SendMessages(ctx, messages, mcpTools)

defer tool.CloseMCPPool()
```

## SSE Connection (HTTP)

```go
mcpServers := map[string]tool.MCPServer{
    "remote": {
        Type: tool.MCPSse,
        URL:  "https://your-mcp-server.com/mcp",
        Headers: map[string]string{
            "Authorization": "Bearer your-token",
        },
    },
}

mcpTools, err := tool.GetMcpTools(ctx, mcpServers)
if err != nil {
    log.Fatal(err)
}

defer tool.CloseMCPPool()
```

## Complete Example

```go
package main

import (
    "context"
    "fmt"
    "log"
    "os"

    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
    "github.com/joakimcarlsson/ai/message"
    "github.com/joakimcarlsson/ai/tool"
)

func main() {
    ctx := context.Background()

    mcpServers := map[string]tool.MCPServer{
        "context7": {
            Type:    tool.MCPStdio,
            Command: "npx",
            Args: []string{
                "-y",
                "@upstash/context7-mcp",
                "--api-key",
                os.Getenv("CONTEXT7_API_KEY"),
            },
        },
    }

    mcpTools, err := tool.GetMcpTools(ctx, mcpServers)
    if err != nil {
        log.Fatal(err)
    }
    defer tool.CloseMCPPool()

    client := llmopenai.NewLLM(
        llmopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
        llmopenai.WithModel(llmopenai.Models[llmopenai.GPT4oMini]),
    )

    messages := []message.Message{
        message.NewUserMessage("Explain React hooks using Context7 to fetch the latest documentation"),
    }

    response, err := client.SendMessages(ctx, messages, mcpTools)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(response.Content)
}
```

## StreamableHTTP Connection

The newer MCP transport for HTTP-based servers:

```go
mcpServers := map[string]tool.MCPServer{
    "remote": {
        Type: tool.MCPStreamableHTTP,
        URL:  "https://your-mcp-server.com/mcp",
        Headers: map[string]string{
            "Authorization": "Bearer your-token",
        },
    },
}

mcpTools, err := tool.GetMcpTools(ctx, mcpServers)
defer tool.CloseMCPPool()
```

## Transport Types

| Type | Constant | Use Case |
|------|----------|----------|
| Stdio | `tool.MCPStdio` | Local subprocess (e.g., `npx` commands) |
| SSE | `tool.MCPSse` | HTTP server with Server-Sent Events |
| StreamableHTTP | `tool.MCPStreamableHTTP` | HTTP server with streamable responses |

## MCPServer Config

```go
type MCPServer struct {
    Command string            // Stdio: command to run
    Args    []string          // Stdio: command arguments
    Env     []string          // Stdio: environment variables
    Type    MCPType           // Transport type
    URL     string            // SSE/StreamableHTTP: server URL
    Headers map[string]string // SSE/StreamableHTTP: custom HTTP headers
}
```

## Features

- Supports stdio, SSE, and StreamableHTTP transports
- Connection pooling for efficient reuse of MCP server connections
- Custom HTTP headers for authentication on remote servers
- Automatic tool discovery and registration
- Compatible with all official MCP servers
- Tools are namespaced with server name (e.g., `context7_search`)
- Graceful cleanup with `CloseMCPPool()`


---

## Advanced > Tool Calling

> Source: advanced/tools.md

# Tool Calling

## Defining a Tool

```go
import "github.com/joakimcarlsson/ai/tool"

type WeatherParams struct {
    Location string `json:"location" desc:"City name"`
    Units    string `json:"units" desc:"Temperature units" enum:"celsius,fahrenheit" required:"false"`
}

type WeatherTool struct{}

func (w *WeatherTool) Info() tool.Info {
    return tool.NewInfo("get_weather", "Get current weather for a location", WeatherParams{})
}

func (w *WeatherTool) Run(ctx context.Context, params tool.Call) (tool.Response, error) {
    var input WeatherParams
    json.Unmarshal([]byte(params.Input), &input)
    return tool.NewTextResponse("Sunny, 22°C"), nil
}
```

## Function Tools

For simple tools that are just a function, use `functiontool.New` to skip the struct boilerplate:

```go
import "github.com/joakimcarlsson/ai/tool/functiontool"

type WeatherParams struct {
    Location string `json:"location" desc:"City name"`
    Units    string `json:"units" desc:"Temperature units" enum:"celsius,fahrenheit" required:"false"`
}

weatherTool := functiontool.New("get_weather", "Get current weather for a location",
    func(ctx context.Context, p WeatherParams) (string, error) {
        return fmt.Sprintf("Sunny, 22°C in %s", p.Location), nil
    },
)
```

The JSON schema is inferred from the parameter struct using the same struct tags as `tool.NewInfo`. The result is a standard `BaseTool` that works with the registry, toolsets, hooks, and agent system.

### Supported Signatures

The function's first parameter can optionally be `context.Context`, and the second can be a struct for input parameters. Both are optional:

```go
// With context and params
functiontool.New("name", "desc", func(ctx context.Context, p Params) (string, error) { ... })

// Params only (no context)
functiontool.New("name", "desc", func(p Params) (string, error) { ... })

// Context only (no input schema)
functiontool.New("name", "desc", func(ctx context.Context) (string, error) { ... })

// No inputs at all
functiontool.New("name", "desc", func() (string, error) { ... })
```

### Return Types

The first return value determines the response type:

```go
// String → tool.NewTextResponse
func(p Params) (string, error)

// tool.Response → passed through directly
func(p Params) (tool.Response, error)

// Any other type → tool.NewJSONResponse (auto-marshaled)
func(p Params) (MyStruct, error)
```

### Options

```go
// Require human confirmation before execution
functiontool.New("delete", "Delete records", deleteFn, functiontool.WithConfirmation())
```

## Using Tools with LLM

```go
weatherTool := &WeatherTool{}
tools := []tool.BaseTool{weatherTool}

response, err := client.SendMessages(ctx, messages, tools)
```

## Provider Built-in Tools

Beyond client-side function tools (the `BaseTool` interface above), most
vendors expose **server-side built-in tools** that run inside the provider's
infrastructure: web search, code execution, file search, and so on. The agent
loop never sees these as `ToolCall` entries — the provider executes the tool
internally and inlines the result into the assistant message. Citations and
other structured output land in `Response.ProviderMetadata` under a
provider-namespaced key.

Built-ins are opt-in per-client via `With*` options. Off by default — some are
billed per call.

```go
import (
    llmanthropic "github.com/joakimcarlsson/ai/llm/anthropic"
    llmgemini "github.com/joakimcarlsson/ai/llm/gemini"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
    llmgroq "github.com/joakimcarlsson/ai/llm/groq"
    llmxai "github.com/joakimcarlsson/ai/llm/xai"
)

// Anthropic web_search
anthropic := llmanthropic.NewLLM(
    llmanthropic.WithAPIKey(os.Getenv("ANTHROPIC_API_KEY")),
    llmanthropic.WithModel(anthropic.Models[anthropic.Claude47Opus]),
    llmanthropic.WithWebSearch(llmanthropic.WebSearchConfig{
        MaxUses:        5,
        AllowedDomains: []string{"go.dev", "pkg.go.dev"},
    }),
)

// Gemini google_search + code_execution
gemini := llmgemini.NewLLM(
    llmgemini.WithAPIKey(os.Getenv("GEMINI_API_KEY")),
    llmgemini.WithModel(gemini.Models[gemini.Gemini25Flash]),
    llmgemini.WithGoogleSearch(),
    llmgemini.WithCodeExecution(),
)

// OpenAI Responses API: web_search, file_search, code_interpreter
openaiR := llmopenai.NewLLM(
    llmopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
    llmopenai.WithModel(openai.Models[openai.GPT5]),
    llmopenai.WithWebSearch(llmopenai.WebSearchOpts{
        SearchContextSize: llmopenai.SearchContextMedium,
    }),
)

// Groq compound models: browser_search, code_execution, visit_website
groq := llmgroq.NewCompoundLLM(
    llmgroq.WithCompoundAPIKey(os.Getenv("GROQ_API_KEY")),
    llmgroq.WithCompoundModel(llm.Model{APIModel: "groq/compound"}),
    llmgroq.WithBrowserSearch(),
)

// xAI Responses API: web_search, x_search, code_execution
xai := llmxai.NewLLM(
    llmxai.WithAPIKey(os.Getenv("XAI_API_KEY")),
    llmxai.WithModel(xai.Models[xai.Grok4]),
    llmxai.WithWebSearch(),
    llmxai.WithXSearch(),
)
```

Built-in tool results — citations, search chunks, executed-tool summaries —
arrive on `Response.ProviderMetadata` under namespaced keys:

```go
resp, _ := client.SendMessages(ctx, messages, nil)
fmt.Println(resp.Content)

if results, ok := resp.ProviderMetadata["anthropic.web_search_results"].([]map[string]any); ok {
    for _, r := range results {
        fmt.Printf("  ↳ %s (%s)\n", r["title"], r["url"])
    }
}
```

The keys vary per provider:

| Provider | Key | Shape |
|---|---|---|
| Anthropic | `anthropic.web_search_results` | `[]map[string]any` — URL, title, page age, encrypted content |
| Gemini | `gemini.grounding` | `map[string]any` — `web_search_queries`, `chunks` (URI/title/domain) |
| Gemini | `gemini.url_context` | `map[string]any` — retrieved URLs and status |
| OpenAI | `openai.url_citations` | `[]map[string]any` — URL, title, start/end indices |
| OpenAI | `openai.web_search_calls` | `[]map[string]any` — one entry per `web_search_call` item: `id`, `status`, `action` (`search`, `open_page`, `find_in_page`), `queries` |
| Groq | `groq.executed_tools` | `[]map[string]any` — Groq's raw executed-tool entries |
| xAI | `xai.citations` | `[]map[string]any` — URL, title, start/end indices |

Built-in tools are **not dispatched through the agent loop** — they don't
appear as `message.ToolCall` entries and don't go through `registry.Execute`.
This means you can use them alongside ordinary function tools without
collision. See the [`builtin-tools`](https://github.com/joakimcarlsson/ai/tree/main/examples/llm/builtin-tools)
example for a runnable provider-switch demo.

## Struct Tag Schema Generation

Generate JSON schemas automatically from Go structs:

```go
type SearchParams struct {
    Query   string   `json:"query" desc:"Search query"`
    Limit   int      `json:"limit" desc:"Max results" required:"false"`
    Filters []string `json:"filters" desc:"Filter tags" required:"false"`
}

info := tool.NewInfo("search", "Search documents", SearchParams{})
```

Supported tags:

| Tag | Description |
|-----|-------------|
| `json` | Parameter name |
| `desc` | Parameter description |
| `required` | `"true"` or `"false"` (non-pointer fields default to required) |
| `enum` | Comma-separated allowed values |

## Rich Tool Responses

```go
// Text response
tool.NewTextResponse("Result text")

// JSON response (auto-marshals any value)
tool.NewJSONResponse(map[string]any{"status": "ok", "count": 42})

// File/binary response
tool.NewFileResponse(pdfBytes, "application/pdf")

// Image response (base64)
tool.NewImageResponse(base64ImageData)

// Error response
tool.NewTextErrorResponse("Something went wrong")
```

## Parsing Tool Input

The agent package provides a generic helper:

```go
input, err := agent.ParseToolInput[WeatherParams](params.Input)
```

## Requiring Confirmation

Set `RequireConfirmation` on a tool's `Info` to require human approval before execution:

```go
func (t *DeleteTool) Info() tool.Info {
    info := tool.NewInfo("delete_records", "Delete database records", DeleteParams{})
    info.RequireConfirmation = true
    return info
}
```

Tools can also request confirmation dynamically from within `Run()`:

```go
func (t *TransferTool) Run(ctx context.Context, params tool.Call) (tool.Response, error) {
    if amount > 10000 {
        if err := tool.RequestConfirmation(ctx, "Large transfer", params); err != nil {
            return tool.Response{}, err
        }
    }
    // ...
}
```

Both require a `ConfirmationProvider` on the agent. See [Tool Confirmation](../agent/confirmation.md) for the full protocol.

## Toolsets

For grouping, filtering, and dynamically controlling which tools are available at runtime, see [Toolsets](../agent/toolsets.md).


---

## Advanced > Structured Output

> Source: advanced/structured-output.md

# Structured Output

Constrained generation that forces the LLM to return valid JSON matching a schema.

## Usage

```go
type CodeAnalysis struct {
    Language   string   `json:"language"`
    Functions  []string `json:"functions"`
    Complexity string   `json:"complexity"`
}

schema := &schema.StructuredOutputInfo{
    Name:        "code_analysis",
    Description: "Analyze code structure",
    Parameters: map[string]any{
        "language": map[string]any{
            "type":        "string",
            "description": "Programming language",
        },
        "functions": map[string]any{
            "type": "array",
            "items": map[string]any{"type": "string"},
            "description": "List of function names",
        },
        "complexity": map[string]any{
            "type": "string",
            "enum": []string{"low", "medium", "high"},
        },
    },
    Required: []string{"language", "functions", "complexity"},
}

response, err := client.SendMessagesWithStructuredOutput(ctx, messages, nil, schema)
if err != nil {
    log.Fatal(err)
}

var analysis CodeAnalysis
json.Unmarshal([]byte(*response.StructuredOutput), &analysis)
```

> **Note:**
> Structured output is supported by OpenAI, Gemini, Azure OpenAI, Vertex AI, Groq, OpenRouter, and xAI. Anthropic and AWS Bedrock do not currently support it.
>

---

## Advanced > Reasoning

> Source: advanced/reasoning.md

# Reasoning / Extended Thinking

Models that support chain-of-thought reasoning can be configured to control
reasoning depth. Some providers also expose the model's thinking process via
`EventThinkingDelta` streaming events.

## Reasoning effort

Each LLM vendor module exports its own `WithReasoningEffort` (or
`WithThinkingLevel`) helper.

**OpenAI:**

    ```go
    import llmopenai "github.com/joakimcarlsson/ai/llm/openai"

    client := llmopenai.NewLLM(
        llmopenai.WithAPIKey("your-key"),
        llmopenai.WithModel(openai.Models[openai.O4Mini]),
        llmopenai.WithMaxTokens(16000),
        llmopenai.WithReasoningEffort(llmopenai.ReasoningEffortHigh),
    )
    ```

    | Level | Constant |
    |---|---|
    | None | `llmopenai.ReasoningEffortNone` |
    | Minimal | `llmopenai.ReasoningEffortMinimal` |
    | Low | `llmopenai.ReasoningEffortLow` |
    | Medium | `llmopenai.ReasoningEffortMedium` |
    | High | `llmopenai.ReasoningEffortHigh` |
    | Xhigh | `llmopenai.ReasoningEffortXhigh` |

    Which levels a model accepts depends on the model; the level is sent
    only when the model can reason. `llmopenai.NewLLM` uses the Responses API,
    which accepts function tools on every OpenAI model. A model can still
    refuse a specific level, for example `gpt-6-astra` and `gpt-6.1-sol` do not
    accept `none`; the API's message is returned in the error.

    OpenAI's Chat Completions API does not expose thinking content. The
    model reasons internally but `EventThinkingDelta` events are not emitted.

**Anthropic:**

    ```go
    import llmanthropic "github.com/joakimcarlsson/ai/llm/anthropic"

    client := llmanthropic.NewLLM(
        llmanthropic.WithAPIKey("your-key"),
        llmanthropic.WithModel(anthropic.Models[anthropic.Claude45Sonnet]),
        llmanthropic.WithMaxTokens(16000),
        llmanthropic.WithReasoningEffort(llmanthropic.ReasoningEffortHigh),
    )
    ```

    | Level | Constant |
    |---|---|
    | Low | `llmanthropic.ReasoningEffortLow` |
    | Medium | `llmanthropic.ReasoningEffortMedium` |
    | High | `llmanthropic.ReasoningEffortHigh` |
    | Max | `llmanthropic.ReasoningEffortMax` |

**Gemini:**

    ```go
    import llmgemini "github.com/joakimcarlsson/ai/llm/gemini"

    client := llmgemini.NewLLM(
        llmgemini.WithAPIKey("your-key"),
        llmgemini.WithModel(gemini.Models[gemini.Gemini3Pro]),
        llmgemini.WithMaxTokens(16000),
        llmgemini.WithThinkingLevel(llmgemini.ThinkingLevelHigh),
    )
    ```

    | Level | Constant |
    |---|---|
    | Minimal | `llmgemini.ThinkingLevelMinimal` |
    | Low | `llmgemini.ThinkingLevelLow` |
    | Medium | `llmgemini.ThinkingLevelMedium` |
    | High | `llmgemini.ThinkingLevelHigh` |

## Streaming thinking events

Anthropic, Gemini, and OpenAI-compatible providers (Ollama, vLLM, etc.) that
expose `reasoning` deltas stream thinking content via `EventThinkingDelta`:

```go
import "github.com/joakimcarlsson/ai/types"

for event := range client.StreamResponse(ctx, messages, nil) {
    switch event.Type {
    case types.EventThinkingDelta:
        fmt.Print(event.Thinking)
    case types.EventContentDelta:
        fmt.Print(event.Content)
    case types.EventComplete:
        fmt.Printf("\nTokens: %d in, %d out\n",
            event.Response.Usage.InputTokens,
            event.Response.Usage.OutputTokens,
        )
    case types.EventError:
        log.Fatal(event.Error)
    }
}
```

The same pattern works with agents via `ChatStream`:

```go
for event := range myAgent.ChatStream(ctx, "Think about this carefully...") {
    switch event.Type {
    case types.EventThinkingDelta:
        fmt.Print(event.Thinking)
    case types.EventContentDelta:
        fmt.Print(event.Content)
    }
}
```

## OpenAI-compatible providers (Ollama, vLLM)

Reasoning models served via OpenAI-compatible APIs (Qwen, DeepSeek, etc.)
stream thinking content over the same `reasoning` delta channel. Use
`llm/openai` with a custom base URL and a custom model:

```go
import (
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
    "github.com/joakimcarlsson/ai/llm"
)

ollama := llmopenai.NewLLM(
    llmopenai.WithAPIKey("ollama"),
    llmopenai.WithBaseURL("http://localhost:11434/v1"),
    llmopenai.WithModel(llm.Model{
        ID:               "qwen3:14b",
        Name:             "Qwen3 14B",
        APIModel:         "qwen3:14b",
        Provider:         "openai",
        ContextWindow:    32768,
        DefaultMaxTokens: 4096,
        CanReason:        true,
    }),
    llmopenai.WithMaxTokens(4096),
)
```


---

## Advanced > Cost Tracking

> Source: advanced/cost-tracking.md

# Cost Tracking

All models include built-in pricing information for cost calculation.

## Currency

Every model states the currency its cost fields are denominated in. Most
providers bill in USD, but not all: Berget bills in EUR, so its catalogs carry
EUR rates.

```go
model := berget.Models[berget.GLM52]
fmt.Printf("%.2f %s per 1M input tokens\n", model.CostPer1MIn, model.Currency)
// 1.40 EUR per 1M input tokens
```

Check `Currency` before summing costs across providers, and before formatting a
figure with a currency symbol. An empty value means USD.

## LLM Models

```go
model := openai.Models[openai.GPT4o]
fmt.Printf("Input cost: $%.2f per 1M tokens\n", model.CostPer1MIn)
fmt.Printf("Output cost: $%.2f per 1M tokens\n", model.CostPer1MOut)

response, err := client.SendMessages(ctx, messages, nil)
inputCost := float64(response.Usage.InputTokens) * model.CostPer1MIn / 1_000_000
outputCost := float64(response.Usage.OutputTokens) * model.CostPer1MOut / 1_000_000
```

## Image Generation Models

```go
gptImageModel := openai.Models[openai.GPTImage15]
lowCost := gptImageModel.Pricing["1024x1024"]["low"]       // $0.009
mediumCost := gptImageModel.Pricing["1024x1024"]["medium"] // $0.034
highCost := gptImageModel.Pricing["1024x1024"]["high"]     // $0.133
```

## Audio Generation Models

```go
model := elevenlabs.Models[elevenlabs.TurboV2_5]

fmt.Printf("Cost per 1M chars: $%.2f\n", model.CostPer1MChars)
fmt.Printf("Max characters per request: %d\n", model.MaxCharacters)
fmt.Printf("Supports streaming: %v\n", model.SupportsStreaming)

response, err := client.GenerateAudio(ctx, text)
cost := float64(response.Usage.Characters) * model.CostPer1MChars / 1_000_000
fmt.Printf("Cost: $%.4f\n", cost)
```


---

## Advanced > Prompt Templates

> Source: advanced/prompt-templates.md

# Prompt Templates

A template engine for building dynamic prompts with variable substitution, built-in functions, caching, and validation.

## Basic Usage

```go
import "github.com/joakimcarlsson/ai/prompt"

result, err := prompt.Process("Hello, {{.name}}!", map[string]any{
    "name": "World",
})
// "Hello, World!"
```

## Reusable Templates

```go
tmpl, err := prompt.New("You are {{.role}}. Help with {{.task}}.")
if err != nil {
    log.Fatal(err)
}

result, err := tmpl.Process(map[string]any{
    "role": "a coding assistant",
    "task": "debugging",
})
// "You are a coding assistant. Help with debugging."
```

## Caching

Thread-safe template caching avoids re-parsing the same template repeatedly.

```go
cache := prompt.NewCache()

tmpl, err := prompt.New("You are {{.role}}.",
    prompt.WithCache(cache),
    prompt.WithName("system"),  // cache key
)
```

When using a cache without `WithName`, the template source is hashed automatically as the cache key.

## Validation

Require specific variables to be present in the data map:

```go
_, err := prompt.Process("Hello, {{.name}}!", map[string]any{},
    prompt.WithRequired("name"),
)
// error: missing required variables: name
```

## Strict Mode

Error on any missing variable instead of using zero values:

```go
tmpl, err := prompt.New("{{.name}} is {{.age}} years old.",
    prompt.WithStrictMode(),
)

_, err = tmpl.Process(map[string]any{"name": "Alice"})
// error: template execution fails because .age is missing
```

## Built-in Functions

### String

| Function | Description | Example |
|----------|-------------|---------|
| `upper` | Uppercase | `{{upper .name}}` |
| `lower` | Lowercase | `{{lower .name}}` |
| `title` | Title case | `{{title .name}}` |
| `trim` | Trim whitespace | `{{trim .text}}` |
| `trimPrefix` | Remove prefix | `{{trimPrefix "Mr. " .name}}` |
| `trimSuffix` | Remove suffix | `{{trimSuffix "." .text}}` |
| `replace` | Replace all | `{{replace "old" "new" .text}}` |
| `contains` | Check substring | `{{if contains .text "error"}}...{{end}}` |
| `hasPrefix` | Check prefix | `{{if hasPrefix .name "Dr."}}...{{end}}` |
| `hasSuffix` | Check suffix | `{{if hasSuffix .file ".go"}}...{{end}}` |

### Collections

| Function | Description | Example |
|----------|-------------|---------|
| `join` | Join slice | `{{join ", " .items}}` |
| `split` | Split string | `{{split "," .csv}}` |
| `first` | First element | `{{first .items}}` |
| `last` | Last element | `{{last .items}}` |
| `list` | Create slice | `{{list "a" "b" "c"}}` |

### Comparison

| Function | Description | Example |
|----------|-------------|---------|
| `eq` | Equal | `{{if eq .role "admin"}}...{{end}}` |
| `ne` / `neq` | Not equal | `{{if ne .status "done"}}...{{end}}` |
| `lt` | Less than | `{{if lt .count 10}}...{{end}}` |
| `le` | Less or equal | `{{if le .count 10}}...{{end}}` |
| `gt` | Greater than | `{{if gt .count 0}}...{{end}}` |
| `ge` | Greater or equal | `{{if ge .count 1}}...{{end}}` |

### Defaults

| Function | Description | Example |
|----------|-------------|---------|
| `default` | Default value | `{{default "anonymous" .name}}` |
| `coalesce` | First non-empty | `{{coalesce .nickname .name "unknown"}}` |
| `empty` | Check if empty | `{{if empty .list}}...{{end}}` |
| `ternary` | Conditional | `{{ternary .admin "admin" "user"}}` |

### Formatting

| Function | Description | Example |
|----------|-------------|---------|
| `indent` | Indent text | `{{indent 4 .code}}` |
| `nindent` | Newline + indent | `{{nindent 4 .code}}` |
| `quote` | Double quote | `{{quote .name}}` |
| `squote` | Single quote | `{{squote .name}}` |

## Custom Functions

Add your own template functions:

```go
import "text/template"

result, err := prompt.Process("{{shout .name}}", data,
    prompt.WithFuncs(template.FuncMap{
        "shout": func(s string) string {
            return strings.ToUpper(s) + "!!!"
        },
    }),
)
```

## Options

| Option | Description |
|--------|-------------|
| `prompt.WithCache(c)` | Enable template caching |
| `prompt.WithName(name)` | Set template name (used as cache key) |
| `prompt.WithRequired(vars...)` | Require specific variables |
| `prompt.WithStrictMode()` | Error on missing variables |
| `prompt.WithFuncs(funcs)` | Add custom template functions |

## With Agent Instruction Templates

The prompt package powers the agent's [instruction templates](../agent/instruction-templates.md) feature:

```go
myAgent := agent.New(llmClient,
    agent.WithSystemPrompt("You are {{.role}}. The user's name is {{.userName}}."),
    agent.WithState(map[string]any{
        "role":     "a helpful assistant",
        "userName": "Alice",
    }),
)
```


---

## Advanced > OpenTelemetry Tracing

> Source: advanced/tracing.md

# OpenTelemetry Tracing

Built-in OpenTelemetry instrumentation for all provider calls and agent execution. Includes traces, metrics, and structured log records following [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). When no providers are configured, everything is a zero-cost no-op.

## Setup

Use the built-in setup helper to initialize traces, metrics, and logs in one call:

```go
import (
    "github.com/joakimcarlsson/ai/tracing"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.36.0"
)

res, _ := resource.New(ctx, resource.WithAttributes(
    semconv.ServiceNameKey.String("my-ai-service"),
    semconv.ServiceVersionKey.String("1.0.0"),
))

providers, _ := tracing.New(ctx,
    tracing.WithResource(res),
    tracing.WithOTLPEndpoint("localhost:4318"),
)
defer providers.Shutdown(ctx)
```

This creates and globally registers a `TracerProvider`, `MeterProvider`, and `LoggerProvider` — all configured with OTLP HTTP exporters pointing at the given endpoint.

### Setup Options

| Option | Description |
|--------|-------------|
| `WithResource(r)` | Set service resource (name, version, environment) |
| `WithOTLPEndpoint(url)` | Configure OTLP HTTP exporters for all signals |
| `WithSpanProcessors(p...)` | Register custom span processors |
| `WithMetricReaders(r...)` | Register custom metric readers |
| `WithLogProcessors(p...)` | Register custom log processors |

If no `WithOTLPEndpoint` is provided, the helper checks the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable.

### Manual Setup

You can also configure providers manually using the standard OpenTelemetry SDK:

```go
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

exporter, _ := stdouttrace.New(stdouttrace.WithPrettyPrint())
tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter))
defer tp.Shutdown(ctx)

otel.SetTracerProvider(tp)
```

All subsequent LLM calls, tool executions, and agent runs will produce spans and metrics.

## Span Hierarchy

When using the agent framework, spans form a parent-child tree:

```
invoke_agent {agent_name}
├── generate_content {model}       (LLM turn 1)
├── execute_tool {tool_name}       (single tool call)
├── generate_content {model}       (LLM turn 2)
└── ...
```

When the LLM requests multiple tool calls at once, they are grouped under a parent span:

```
invoke_agent {agent_name}
├── generate_content {model}
├── execute_tools                  (merged parent for 2+ tools)
│   ├── execute_tool {tool_a}
│   └── execute_tool {tool_b}
└── generate_content {model}
```

When using providers standalone (no agent), each call produces a root span:

```
generate_content {model}
generate_embeddings {model}
rerank {model}
generate_audio {model}
generate_image {model}
transcribe {model}
fim_complete {model}
```

## Instrumented Operations

Every provider package is instrumented at the public API level — one span per call, covering all underlying providers.

| Package | Span Name | Methods |
|---------|-----------|---------|
| `providers` (LLM) | `generate_content` | `SendMessages`, `StreamResponse`, and structured output variants |
| `embeddings` | `generate_embeddings` | `GenerateEmbeddings`, `GenerateMultimodalEmbeddings`, `GenerateContextualizedEmbeddings` |
| `rerankers` | `rerank` | `Rerank` |
| `audio` | `generate_audio` | `GenerateAudio`, `StreamAudio` |
| `image_generation` | `generate_image` | `GenerateImage`, `GenerateImageStreaming` |
| `transcription` | `transcribe` / `translate` | `Transcribe`, `Translate` |
| `fim` | `fim_complete` | `Complete`, `CompleteStream` |
| `agent` | `invoke_agent` | `Chat`, `ChatStream`, `Continue`, `ContinueStream` |
| `agent` (tools) | `execute_tool` | Each tool call during agent execution |

## Span Attributes

Spans carry GenAI semantic convention attributes.

### LLM (`generate_content`)

| Attribute | When |
|-----------|------|
| `gen_ai.system` | Always |
| `gen_ai.request.model` | Always |
| `gen_ai.request.max_tokens` | Always |
| `gen_ai.request.temperature` | If set |
| `gen_ai.request.top_p` | If set |
| `gen_ai.usage.input_tokens` | On completion |
| `gen_ai.usage.output_tokens` | On completion |
| `gen_ai.usage.cache_creation_tokens` | If non-zero |
| `gen_ai.usage.cache_read_tokens` | If non-zero |
| `gen_ai.response.finish_reason` | On completion |
| `gen_ai.response.tool_call_count` | If tool calls present |

### Agent (`invoke_agent`)

| Attribute | When |
|-----------|------|
| `gen_ai.agent.name` | Always |
| `gen_ai.usage.input_tokens` | On completion (aggregated) |
| `gen_ai.usage.output_tokens` | On completion (aggregated) |
| `gen_ai.agent.total_turns` | On completion |
| `gen_ai.agent.total_tool_calls` | On completion |

### Tool (`execute_tool`)

| Attribute | When |
|-----------|------|
| `gen_ai.tool.name` | Always |
| `gen_ai.tool.call_id` | Always |

## Streaming

Streaming calls (`StreamResponse`, `StreamAudio`, `CompleteStream`) are fully traced. The span covers the entire stream lifetime — from the initial call until the channel closes. Response attributes (token usage, finish reason) are recorded when the final event arrives.

## Metrics

Every provider call records two metrics via the global `MeterProvider`:

| Metric | Type | Unit | Description |
|--------|------|------|-------------|
| `gen_ai.client.operation.duration` | Float64Histogram | `s` | Duration of each provider call |
| `gen_ai.client.token.usage` | Int64Counter | `{token}` | Token consumption per call |

Both metrics carry these attributes:

| Attribute | Description |
|-----------|-------------|
| `gen_ai.operation.name` | Operation type (`generate_content`, `generate_embeddings`, `rerank`, etc.) |
| `gen_ai.system` | Provider name (`openai`, `anthropic`, `voyage`, etc.) |
| `gen_ai.request.model` | Model identifier |
| `error.type` | Error message (only on failed calls) |

The token usage counter additionally carries `gen_ai.token.type` (`input` or `output`) to distinguish token direction. Token metrics are only recorded when the count is non-zero.

### Metrics Setup

Metrics work the same as traces — configure a global `MeterProvider`:

```go
import (
    "go.opentelemetry.io/otel"
    sdkmetric "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
)

exporter, _ := otlpmetrichttp.New(ctx)
mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(
    sdkmetric.NewPeriodicReader(exporter),
))
defer mp.Shutdown(ctx)

otel.SetMeterProvider(mp)
```

## Log Records

LLM calls emit OpenTelemetry log records tied to the active span. Log bodies are structured JSON following GenAI semantic conventions:

| Event Name | Body Structure |
|------------|----------------|
| `gen_ai.system.message` | `{"content": "..."}` |
| `gen_ai.user.message` | `{"content": "..."}` |
| `gen_ai.choice` | `{"index": 0, "content": "...", "finish_reason": "..."}` |

Log records require a global `LoggerProvider` to be configured. Without one, they are silently dropped.

### Content Capture

Message content is **elided by default** for privacy. To include actual message content in log records, set:

```bash
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
```

When disabled, log bodies contain `<elided>` instead of the actual content.

## Retry Visibility

When a provider call is retried (rate limits, transient errors), each retry attempt is recorded as a span event on the `generate_content` span:

```
Event: "retry"
  attempt = 1
  retry_after_ms = 2000
  error = "429 Too Many Requests"
```

This gives visibility into retries without creating additional spans, making it easy to diagnose latency spikes caused by rate limiting.

## OTLP Export

To export traces to Jaeger, Grafana Tempo, Datadog, or any OTLP-compatible backend:

```go
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.36.0"
)

exporter, _ := otlptracehttp.New(ctx)

res, _ := resource.New(ctx,
    resource.WithAttributes(
        semconv.ServiceNameKey.String("my-ai-service"),
        semconv.ServiceVersionKey.String("1.0.0"),
    ),
)

tp := sdktrace.NewTracerProvider(
    sdktrace.WithBatcher(exporter),
    sdktrace.WithResource(res),
)
defer tp.Shutdown(ctx)

otel.SetTracerProvider(tp)
```

Configure the OTLP endpoint via environment variable:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
```

## Standalone Provider Tracing

Tracing works without the agent framework. Any provider call creates spans and records metrics automatically:

```go
import llmanthropic "github.com/joakimcarlsson/ai/llm/anthropic"

otel.SetTracerProvider(tp)

client := llmanthropic.NewLLM(
    llmanthropic.WithAPIKey(os.Getenv("ANTHROPIC_API_KEY")),
    llmanthropic.WithModel(anthropic.Models[anthropic.Claude45Sonnet]),
)

// This call produces a "generate_content claude-sonnet-4-6-20250514" span
// and records duration + token usage metrics
response, _ := client.SendMessages(ctx, messages, nil)
```

The same applies to embeddings, audio, image generation, transcription, rerankers, and FIM.

## Complete Example

```go
package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.36.0"

    "github.com/joakimcarlsson/ai/agent"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
    "github.com/joakimcarlsson/ai/tool/functiontool"
    "github.com/joakimcarlsson/ai/tracing"
)

func main() {
    ctx := context.Background()

    res, _ := resource.New(ctx, resource.WithAttributes(
        semconv.ServiceNameKey.String("my-ai-service"),
    ))
    providers, _ := tracing.New(ctx,
        tracing.WithResource(res),
        tracing.WithOTLPEndpoint("localhost:4318"),
    )
    defer func() { _ = providers.Shutdown(ctx) }()

    client := llmopenai.NewLLM(
        llmopenai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
        llmopenai.WithModel(llmopenai.Models[llmopenai.GPT5Nano]),
    )

    timeTool := functiontool.New(
        "get_time",
        "Get the current time",
        func(_ context.Context, p struct{}) (string, error) {
            return "14:30 UTC", nil
        },
    )

    myAgent := agent.New(client,
        agent.WithTools(timeTool),
    )

    resp, err := myAgent.Chat(ctx, "What time is it?")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(resp.Content)
}
```

This produces spans:

```
invoke_agent
├── generate_content gpt-5-nano
├── execute_tool get_time
└── generate_content gpt-5-nano
```


---

## Advanced > Configuration

> Source: advanced/configuration.md

# Configuration

After the modular split, every option lives on the vendor module. Each
`llm/<vendor>`, `embeddings/<vendor>`, etc. exports its own `Options` type
and `WithXxx` helpers — there's no generic factory or provider switch.

## LLM client options

Each LLM vendor module exports a `NewLLM` plus shared and vendor-specific
options.

### OpenAI

```go
import (
    "github.com/joakimcarlsson/ai/llm"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
)

client := llmopenai.NewLLM(
    llmopenai.WithAPIKey("your-key"),
    llmopenai.WithModel(llmopenai.Models[llmopenai.GPT4o]),
    llmopenai.WithMaxTokens(2000),
    llmopenai.WithTemperature(0.7),
    llmopenai.WithTopP(0.9),
    llmopenai.WithTimeout(30*time.Second),
    llmopenai.WithStopSequences("STOP", "END"),
    llmopenai.WithBaseURL("https://custom-endpoint"),
    llmopenai.WithExtraHeaders(map[string]string{"X-Header": "value"}),
    llmopenai.WithDisableCache(),
    llmopenai.WithReasoningEffort(llmopenai.ReasoningEffortHigh),
    llmopenai.WithFrequencyPenalty(0.5),
    llmopenai.WithPresencePenalty(0.3),
    llmopenai.WithSeed(42),
    llmopenai.WithParallelToolCalls(false),
    llmopenai.WithToolChoice(llm.ToolChoice{Mode: llm.ToolChoiceRequired}),
    llmopenai.WithLogitBias(map[string]int{"50256": -100}),
    llmopenai.WithLogprobs(3),
    llmopenai.WithN(3),
)
```

`WithLogitBias`, `WithLogprobs`, and `WithN` are OpenAI-only sampling knobs
(inherited by every OpenAI-compatible provider) and are emitted only when set.
`WithLogitBias` biases or bans tokens by tokenizer id (-100 to 100).
`WithLogprobs(n)` surfaces per-token log probabilities with up to `n`
alternatives on `Response.LogProbs`. `WithN(n)` requests `n` completions on
`Response.Choices`, with the top-level fields mirroring choice 0 (when
streaming, deltas cover choice 0 only and the rest arrive on the final
response). `logit_bias` is rejected by reasoning-tier models (the
gpt-5 family); use a classic chat model such as `gpt-4o-mini` for it.

`WithToolChoice` controls whether and which tool the model may call. It takes
the shared `llm.ToolChoice` type — `Mode` is one of `ToolChoiceAuto` (default),
`ToolChoiceNone`, `ToolChoiceRequired`, or `ToolChoiceSpecific` (which also sets
`Name`). The field is only emitted when tools are supplied, and is available on
the OpenAI, Anthropic, and Gemini modules (OpenAI-compatible providers inherit
it through `llm/openai`).

### Anthropic

```go
import llmanthropic "github.com/joakimcarlsson/ai/llm/anthropic"

client := llmanthropic.NewLLM(
    llmanthropic.WithAPIKey("your-key"),
    llmanthropic.WithModel(anthropic.Models[anthropic.Claude45Sonnet]),
    llmanthropic.WithMaxTokens(4000),
    llmanthropic.WithTemperature(0.7),
    llmanthropic.WithBeta("beta-feature"),
    llmanthropic.WithBedrock(true),
    llmanthropic.WithDisableCache(),
    llmanthropic.WithReasoningEffort(llmanthropic.ReasoningEffortMedium),
    llmanthropic.WithToolChoice(llm.ToolChoice{Mode: llm.ToolChoiceRequired}),
)
```

### Gemini

```go
import llmgemini "github.com/joakimcarlsson/ai/llm/gemini"

client := llmgemini.NewLLM(
    llmgemini.WithAPIKey("your-key"),
    llmgemini.WithModel(gemini.Models[gemini.Gemini25Flash]),
    llmgemini.WithMaxTokens(2000),
    llmgemini.WithDisableCache(),
    llmgemini.WithFrequencyPenalty(0.5),
    llmgemini.WithPresencePenalty(0.3),
    llmgemini.WithSeed(42),
    llmgemini.WithThinkingLevel(llmgemini.ThinkingLevelHigh),
    llmgemini.WithToolChoice(llm.ToolChoice{Mode: llm.ToolChoiceRequired}),
)
```

### Azure OpenAI

```go
import llmazure "github.com/joakimcarlsson/ai/llm/azure"

client := llmazure.NewLLM(
    llmazure.WithAPIKey("your-key"),
    llmazure.WithEndpoint("https://your-resource.openai.azure.com"),
    llmazure.WithDeployment("my-chat-deployment"),
)
```

### Bedrock

```go
import (
    "github.com/joakimcarlsson/ai/llm"
    llmbedrock "github.com/joakimcarlsson/ai/llm/bedrock"
)

// Bedrock ships no bundled catalog: pass the AWS model id directly.
client := llmbedrock.NewLLM(
    llmbedrock.WithRegion("us-east-1"),
    llmbedrock.WithModel(llm.NewCustomModel(
        llm.WithAPIModel("anthropic.claude-sonnet-4-5-20250929-v1:0"),
        llm.WithContextWindow(200_000),
    )),
)
```

## Embedding client options

```go
import embvoyage "github.com/joakimcarlsson/ai/embeddings/voyage"

embedder := embvoyage.NewEmbedding(
    embvoyage.WithAPIKey(""),
    embvoyage.WithModel(voyage.Models[voyage.Voyage35]),
    embvoyage.WithBatchSize(100),
    embvoyage.WithTimeout(30*time.Second),
    embvoyage.WithInputType("document"),
    embvoyage.WithOutputDimension(1024),
    embvoyage.WithOutputDtype("float"),
)
```

The same shape applies to `embeddings/openai`, `embeddings/gemini`,
`embeddings/cohere`, `embeddings/mistral`, `embeddings/jina`. Vendor-specific
options live on the vendor's module.

## Reranker client options

```go
import rerankvoyage "github.com/joakimcarlsson/ai/rerankers/voyage"

reranker := rerankvoyage.NewReranker(
    rerankvoyage.WithAPIKey(""),
    rerankvoyage.WithModel(voyage.Models[voyage.Rerank25Lite]),
    rerankvoyage.WithTopK(10),
    rerankvoyage.WithReturnDocuments(true),
    rerankvoyage.WithTruncation(true),
    rerankvoyage.WithTimeout(30*time.Second),
)
```

## Image generation client options

```go
import (
    imageopenai "github.com/joakimcarlsson/ai/image/openai"
    imagegemini "github.com/joakimcarlsson/ai/image/gemini"
    "google.golang.org/genai"
)

// OpenAI
client := imageopenai.NewGeneration(
    imageopenai.WithAPIKey("your-key"),
    imageopenai.WithModel(openai.Models[openai.GPTImage15]),
    imageopenai.WithTimeout(60*time.Second),
    imageopenai.WithBaseURL("custom-endpoint"),
)

// Gemini / Vertex AI
client := imagegemini.NewImageGeneration(
    imagegemini.WithAPIKey("your-key"),
    imagegemini.WithModel(gemini.Models[gemini.Imagen4]),
    imagegemini.WithTimeout(60*time.Second),
    imagegemini.WithBackend(genai.BackendVertexAI),
)
```

## TTS client options

```go
import ttseleven "github.com/joakimcarlsson/ai/tts/elevenlabs"

client := ttseleven.NewAudioGeneration(
    ttseleven.WithAPIKey("your-key"),
    ttseleven.WithModel(elevenlabs.Models[elevenlabs.TurboV2_5]),
    ttseleven.WithTimeout(30*time.Second),
    ttseleven.WithBaseURL("custom-endpoint"),
)
```

## STT client options

```go
import sttopenai "github.com/joakimcarlsson/ai/stt/openai"

client := sttopenai.NewSpeechToText(
    sttopenai.WithAPIKey("your-key"),
    sttopenai.WithModel(openai.Models[openai.GPT4oTranscribe]),
    sttopenai.WithTimeout(30*time.Second),
)
```

## Retry configuration

Each LLM vendor module exposes its own retry config with provider-tuned
defaults. They all share the same field shape via `llm.RetryConfig`:

```go
import (
    "github.com/joakimcarlsson/ai/llm"
    llmopenai "github.com/joakimcarlsson/ai/llm/openai"
    llmanthropic "github.com/joakimcarlsson/ai/llm/anthropic"
)

// Defaults baked into each vendor
llmopenai.DefaultRetryConfig()      // retries: 429, 500
llmanthropic.DefaultRetryConfig()   // retries: 429, 529

// Override per-client
client := llmopenai.NewLLM(
    llmopenai.WithRetryConfig(llm.RetryConfig{
        MaxRetries:       5,
        BaseBackoffMs:    1000,
        JitterPercent:    0.2,
        RetryStatusCodes: []int{429, 500, 502, 503, 504},
        CheckRetryAfter:  true,
    }),
)
```

| Setting | Default | Description |
|---------|---------|-------------|
| `MaxRetries` | 3 | Maximum retry attempts |
| `BaseBackoffMs` | 2000 | Initial backoff in milliseconds |
| `JitterPercent` | 0.2 | Jitter added to backoff (20%) |
| `RetryStatusCodes` | varies | HTTP status codes that trigger retries |
| `CheckRetryAfter` | true | Respect the `Retry-After` header |

Retries use exponential backoff: `base * 2^(attempt-1) + jitter`. When
`CheckRetryAfter` is enabled and the server sends a `Retry-After` header,
that value takes precedence.

## Tracing wrappers

Every modality interface module exports a `WithTracing` helper that wraps
any vendor client without dragging an SDK into the import:

```go
import "github.com/joakimcarlsson/ai/llm"

traced := llm.WithTracing(client, llm.TracingAttrs{
    Provider: "openai",
    Model:    "gpt-4o",
})
```

See [Tracing](tracing.md) for the full picture.

## Agent options

See the [Agent Framework Overview](../agent/overview.md) for a full table of
agent configuration options.