Switching Judge Providers

Regtrace supports multiple LLM providers for judging factuality and tone metrics. Swap between them by changing a single config line.

Supported providers

| Provider | Env variable | Models | |---|---|---|---| | Anthropic | ANTHROPIC_API_KEY | claude-haiku-4-5-20251001, claude-sonnet-4-20250514 | | OpenAI | OPENAI_API_KEY | gpt-4o, gpt-4o-mini, gpt-4.1, gpt-4.1-mini | | Gemini | GEMINI_API_KEY | gemini-2.5-pro, gemini-2.5-flash | | Groq | GROQ_API_KEY | llama-3.1-8b-instant, llama-3.3-70b-versatile | | Ollama | (none) | Any local model |

API key location

Regtrace loads .env from the current working directory using dotenv.config(). Place your .env file in the project root where you run regtrace run.

echo "GROQ_API_KEY=gsk_..." > .env

Change provider

Set the provider in regtrace.config.yaml:

judge:
  primary:
    provider: anthropic
    model: claude-haiku-4-5-20251001
    temperature: 0.1
    max_tokens: 4096
    timeout_ms: 30000
    retry_attempts: 3

judge:
  primary:
    provider: openai
model: gpt-4.1-mini
temperature: 0.1
max_tokens: 4096
timeout_ms: 30000
retry_attempts: 3

judge:
  primary:
    provider: gemini
    model: gemini-2.5-flash
    temperature: 0.1
    max_tokens: 4096
    timeout_ms: 30000
    retry_attempts: 3

judge:
  primary:
    provider: groq
    model: llama-3.1-8b-instant
    temperature: 0.1
    max_tokens: 4096
    timeout_ms: 30000
    retry_attempts: 3

No API key needed. Regtrace sends prompts to your local Ollama instance.

judge:
  primary:
    provider: ollama
    model: llama3
    local_endpoint: http://localhost:11434

An Anthropic key is the default when no judge block is specified. Set ANTHROPIC_API_KEY in your .env file to use the default provider.

Fallback provider

Configure a fallback if the primary provider fails after retries:

judge:
  primary:
    provider: anthropic
    model: claude-haiku-4-5-20251001
    temperature: 0.1
    max_tokens: 4096
    timeout_ms: 30000
    retry_attempts: 3

  fallback:
    provider: openai
    model: gpt-4o-mini
    temperature: 0.1
    max_tokens: 4096
    timeout_ms: 30000
    retry_attempts: 2

API key validation

Regtrace validates required API keys before making any HTTP calls. If a key is missing, the CLI reports the error immediately rather than timing out.

Deterministic fallback

If no provider is configured or all providers fail, deterministic metrics (format) still run normally. LLM-judged metrics (factuality, tone) fall back to heuristic scoring.