mrsentry
Docs › LLM providers

LLM providers

mrsentry talks to models through LiteLLM, so any provider LiteLLM supports works. You set up named profiles, assign them to tasks, and add fallbacks and spending limits.

mrsentry routes each task to a named LLM profile through LiteLLM: Amazon Bedrock, Azure OpenAI, Anthropic, OpenAI, a LiteLLM proxy, or local OpenAI-compatible servers. mrsentry via LiteLLM Amazon Bedrock bedrock/… Azure OpenAI azure/<deployment> Anthropic anthropic/… OpenAI openai/… LiteLLM proxy litellm_proxy/… vLLM · Ollama openai/ + api_base

Concepts

ConceptConfigMeaning
Profilellm.profiles.<name>A model and how to reach it: model, api_base, api_key_env, api_version, aws_region, temperature, max_tokens, timeout_s, extra.
Rolellm.rolesWhich profile each task uses: review (main pass), summarize (critic pass), fix_repair (repairing a build broken by an upgrade).
Fallbacksllm.fallbacksProfiles tried in order when a call fails after its retries.
Limitsllm.max_cost_usd_per_run, llm.input_token_budget, llm.max_retriesStops further calls once estimated cost or input tokens exceed the limit. Transient errors such as 429 or 5xx are retried with backoff.

Keys never go in files. api_key_env names the environment variable that holds the key. Provider-standard variables (ANTHROPIC_API_KEY, OPENAI_API_KEY, AZURE_API_KEY, the AWS credential chain) are also picked up automatically by LiteLLM.

Provider recipes

Amazon Bedrock

Uses the normal AWS credential chain, so in GitLab CI the same OIDC role that reads ECR can call Bedrock. Use a cross-region inference profile id where your account requires one.

llm:
  profiles:
    default:
      model: "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0"
      aws_region: us-east-1

IAM: bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream on the model or inference profile.

Azure OpenAI

The model string is azure/<deployment name>, not the base model name.

llm:
  profiles:
    default:
      model: "azure/gpt-4o-review"            # your deployment name
      api_base: "https://acme-ai.openai.azure.com"
      api_version: "2024-10-21"
      api_key_env: AZURE_OPENAI_API_KEY

No config at all: if AZURE_OPENAI_API_ENDPOINT (or AZURE_OPENAI_ENDPOINT), AZURE_OPENAI_API_KEY and AZURE_OPENAI_API_DEPLOYMENT_NAME (or AZURE_OPENAI_DEPLOYMENT) are set and nothing else names a model, mrsentry builds this profile for you. AZURE_OPENAI_API_VERSION is optional.

Anthropic

This is the default profile. Set the key and go.

export ANTHROPIC_API_KEY=sk-ant-…

llm:
  profiles:
    default: { model: "anthropic/claude-sonnet-4-5", api_key_env: ANTHROPIC_API_KEY }

OpenAI

llm:
  profiles:
    default: { model: "openai/gpt-4.1", api_key_env: OPENAI_API_KEY }

LiteLLM proxy

Route through your company's LiteLLM gateway to get central keys, budgets, logging and model aliases.

llm:
  profiles:
    default:
      model: "litellm_proxy/claude-sonnet"    # alias defined on the proxy
      api_base: "https://litellm.internal.example.com"
      api_key_env: LITELLM_KEY

Self-hosted (vLLM, Ollama, LM Studio, TGI)

Anything that speaks the OpenAI API works with the openai/ prefix and an api_base. Choose a model with solid tool calling, such as a recent Qwen-Coder or Llama. Models without it still work through the JSON fallback.

llm:
  profiles:
    default:
      model: "openai/qwen2.5-coder-32b-instruct"
      api_base: "http://vllm.internal:8000/v1"
      api_key_env: VLLM_KEY                   # any value if the server has no auth
      max_tokens: 4096
    ollama:
      model: "ollama_chat/qwen2.5-coder:14b"
      api_base: "http://ollama:11434"

Mixing models per task

Run the main review on your strongest model and the critic pass on a cheaper one, with a gateway as the safety net:

llm:
  profiles:
    strong: { model: "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0" }
    cheap:  { model: "azure/gpt-4o-mini", api_base: "https://acme-ai.openai.azure.com",
              api_version: "2024-10-21", api_key_env: AZURE_OPENAI_API_KEY }
    proxy:  { model: "litellm_proxy/claude-sonnet", api_base: "https://litellm.internal" }
  roles:
    review: strong
    summarize: cheap
    fix_repair: strong
  fallbacks: [proxy]
  max_cost_usd_per_run: 3

Quick overrides

# one run, different model
mrsentry review --mr-url … --model openai/gpt-4.1

# environment (CI variables)
MRSENTRY_LLM__PROFILES__DEFAULT__MODEL=azure/gpt-4o-review
MRSENTRY_LLM__MAX_COST_USD_PER_RUN=2

Provider-specific options

extra passes keyword arguments through to litellm.completion unchanged, for things like custom headers or a provider's reasoning settings:

llm:
  profiles:
    default:
      model: "openai/gpt-4.1"
      extra:
        extra_headers: { "X-Team": "payments" }

Cost and transparency