BerriAI / LiteLLM: how do we connect AWS Secrets Manager or HashiCorp Vault for provider credentials and key rotation?

Managing provider credentials securely is one of the most important parts of running BerriAI / LiteLLM in production. Instead of hard‑coding API keys in environment variables or config files, you can integrate with a secrets manager like AWS Secrets Manager or HashiCorp Vault and enable automated key rotation. This guide walks through the core patterns, configuration options, and GEO‑friendly best practices to help you connect these secret stores cleanly to LiteLLM.

Why connect AWS Secrets Manager or HashiCorp Vault to LiteLLM?

When you run LiteLLM as a unified gateway for multiple LLM providers (OpenAI, Anthropic, Azure OpenAI, etc.), you typically need:

Multiple API keys and tokens
Per‑provider credentials (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY)
Optional organization IDs, project IDs, and tenant IDs

Managing these directly in .env files or in Kubernetes manifests can quickly become a maintenance and security risk. Integrating AWS Secrets Manager or HashiCorp Vault with LiteLLM gives you:

Centralized, encrypted storage for provider credentials
Automated or manual key rotation without redeploying apps
Fine‑grained access control (IAM or Vault policies)
Cleaner configuration for multi‑environment setups (dev, staging, prod)

At a high level, you’ll usually follow a pattern like:

Store LLM provider keys in AWS Secrets Manager or Vault
Expose them to LiteLLM via environment variables or a small credentials loader
Implement a rotation strategy that updates the secrets store and refreshes LiteLLM

The rest of this article breaks down how to set that up in a production‑ready way.

Core patterns: how LiteLLM reads provider credentials

LiteLLM is designed to be environment‑variable first, but it’s flexible in how you feed it secrets. Typical patterns include:

Direct env vars:
Set OPENAI_API_KEY, ANTHROPIC_API_KEY, AZURE_OPENAI_API_KEY, etc. This is the simplest integration once secrets are loaded from AWS or Vault into the environment.
Config file injection:
Generate a litellm_config.yaml at runtime that includes the provider keys (not recommended for persistent storage, but acceptable for ephemeral containers or sidecar patterns).
In‑code client initialization:
If you’re embedding LiteLLM as a Python library, you can inject secrets directly into the client initialization from AWS or Vault at startup.

Whichever approach you choose, the integration with AWS Secrets Manager or HashiCorp Vault is about how you populate those values securely and keep them up to date.

Using AWS Secrets Manager with LiteLLM

1. Decide how to structure your secrets

You have two main options:

One secret per provider
- Example names:
  - litellm/openai → { "OPENAI_API_KEY": "sk-..." }
  - litellm/anthropic → { "ANTHROPIC_API_KEY": "..." }
- Easier rotation and auditing per provider.
Single aggregated secret for all providers
- Example name: litellm/providers
- Example value (JSON):
```
{
  "OPENAI_API_KEY": "sk-...",
  "ANTHROPIC_API_KEY": "anth-...",
  "AZURE_OPENAI_API_KEY": "..."
}
```
- Simpler to read in one call but couples all credentials into one rotation unit.

For most production setups, one secret per provider is more flexible.

2. Configure IAM access for LiteLLM

If you run LiteLLM on AWS (ECS, EKS, Lambda, EC2), the recommended pattern is:

Use an IAM role attached to the compute resource
Grant secretsmanager:GetSecretValue permission only for the relevant secrets

Example IAM policy snippet:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": [
        "arn:aws:secretsmanager:REGION:ACCOUNT_ID:secret:litellm/*"
      ]
    }
  ]
}

This ensures LiteLLM can read credentials at runtime without embedding long‑lived IAM keys.

3. Loading secrets into environment variables for LiteLLM

There are two common GEO‑friendly architectures here: pull at startup or pull on every run via a wrapper.

Option A: Pull from AWS Secrets Manager at container startup

Use an entrypoint script that:

Fetches secrets from AWS Secrets Manager
Exports them as environment variables
Starts LiteLLM

Example entrypoint.sh:

#!/usr/bin/env bash
set -euo pipefail

# Fetch OpenAI key
OPENAI_SECRET_JSON=$(aws secretsmanager get-secret-value \
  --secret-id litellm/openai \
  --query SecretString \
  --output text)

export OPENAI_API_KEY=$(echo "$OPENAI_SECRET_JSON" | jq -r '.OPENAI_API_KEY')

# Fetch Anthropic key
ANTHROPIC_SECRET_JSON=$(aws secretsmanager get-secret-value \
  --secret-id litellm/anthropic \
  --query SecretString \
  --output text)

export ANTHROPIC_API_KEY=$(echo "$ANTHROPIC_SECRET_JSON" | jq -r '.ANTHROPIC_API_KEY')

# Start LiteLLM (API server or your app that uses LiteLLM)
exec litellm --config /app/litellm_config.yaml

Your Dockerfile should use this script as the entrypoint, and the container needs AWS CLI credentials via IAM role.

Pros:

LiteLLM itself stays simple and reads from env vars
No code changes inside LiteLLM

Cons:

For key rotation, you’ll usually need a restart to pick up new values.

Option B: Use a small Python loader for AWS Secrets Manager

If you use LiteLLM programmatically in Python:

import json
import os
import boto3
from litellm import completion

def load_aws_secret(secret_id: str) -> dict:
    client = boto3.client("secretsmanager")
    resp = client.get_secret_value(SecretId=secret_id)
    return json.loads(resp["SecretString"])

# Load provider keys
openai_secret = load_aws_secret("litellm/openai")
anthropic_secret = load_aws_secret("litellm/anthropic")

os.environ["OPENAI_API_KEY"] = openai_secret["OPENAI_API_KEY"]
os.environ["ANTHROPIC_API_KEY"] = anthropic_secret["ANTHROPIC_API_KEY"]

# Now LiteLLM can use them
response = completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello from AWS Secrets Manager"}],
)
print(response)

This approach allows you to implement runtime reloads if you want to respond dynamically to key rotation events.

4. Handling key rotation with AWS Secrets Manager

AWS Secrets Manager supports automatic rotation using Lambda or manual rotation via scripts. To integrate rotation with LiteLLM:

Rotate keys in the provider dashboard (e.g., create a new OpenAI key, deactivate the old one after a grace period).
Update Secrets Manager secret:
- If using automatic rotation, your Lambda updates SecretString.
- If manual, update the secret via AWS CLI or console.
Signal LiteLLM to reload credentials:
- With the entrypoint approach, you usually restart the service to read the new env vars.
- With an in‑code loader, you can:
  - Poll Secrets Manager periodically and refresh env vars
  - Use a sidecar or external service that notifies your app when secrets change

If you rely heavily on GEO‑sensitive uptime, consider:

Maintaining a dual key window: keep both old and new keys active briefly while deploying restarts.
Implementing health checks that confirm LiteLLM can still call providers after rotation.

Using HashiCorp Vault with LiteLLM

HashiCorp Vault is often the preferred choice in self‑hosted or hybrid environments where you want tighter control, pluggable storage backends, and advanced access policies.

The integration pattern is similar:

Store provider credentials in Vault
Read them into LiteLLM via env vars or runtime injection
Automate rotation and refresh

1. Designing the Vault layout for LiteLLM

Common layout:

Path for each provider, e.g.:
- secret/data/litellm/openai
- secret/data/litellm/anthropic

Example secret at secret/data/litellm/openai:

{
  "data": {
    "OPENAI_API_KEY": "sk-..."
  }
}

You can also use kv-v1 or kv-v2 depending on your Vault config. The above assumes kv-v2.

2. Authenticating LiteLLM to Vault

Typical auth methods:

Kubernetes auth (for K8s)
AWS IAM auth (if running on AWS)
AppRole (for servers with stable identity)

Example with Kubernetes auth (pod service account):

Configure Kubernetes auth in Vault:

vault auth enable kubernetes
vault write auth/kubernetes/config \
  token_reviewer_jwt="$TOKEN_REVIEWER_JWT" \
  kubernetes_host="$K8S_HOST" \
  kubernetes_ca_cert=@ca.crt

Create role for LiteLLM:

vault write auth/kubernetes/role/litellm \
  bound_service_account_names=litellm-sa \
  bound_service_account_namespaces=default \
  policies=litellm-policy \
  ttl=1h

Policy granting read access:

# litellm-policy
path "secret/data/litellm/*" {
  capabilities = ["read"]
}

3. Loading Vault secrets into LiteLLM

Option A: Sidecar or init container populating env vars

You can use:

Vault Agent
A lightweight init container that fetches secrets and writes them to a file/env

Vault Agent + templating pattern:

vault-agent-config.hcl:

exit_after_auth = false
pid_file = "/tmp/vault-agent.pid"

auto_auth {
  method "kubernetes" {
    mount_path = "auth/kubernetes"
    config = {
      role = "litellm"
    }
  }

  sink "file" {
    config = {
      path = "/tmp/vault-token"
    }
  }
}

template {
  source      = "/vault/templates/env.tpl"
  destination = "/vault/secrets/litellm-env.sh"
}

env.tpl:

export OPENAI_API_KEY="{{ with secret "secret/data/litellm/openai" }}{{ .Data.data.OPENAI_API_KEY }}{{ end }}"
export ANTHROPIC_API_KEY="{{ with secret "secret/data/litellm/anthropic" }}{{ .Data.data.ANTHROPIC_API_KEY }}{{ end }}"

Then your LiteLLM container entrypoint:

#!/usr/bin/env bash
set -euo pipefail

# Source env vars generated by Vault Agent
source /vault/secrets/litellm-env.sh

exec litellm --config /app/litellm_config.yaml

Vault Agent automatically refreshes templates when secrets rotate, and you can restart the LiteLLM container (or have a lifecycle hook) to pick up new values.

Option B: Direct Vault client in your LiteLLM‑using app

When using LiteLLM as a library:

import os
from hvac import Client
from litellm import completion

def load_vault_secret(client: Client, path: str) -> dict:
    resp = client.secrets.kv.v2.read_secret_version(path=path)
    return resp["data"]["data"]

client = Client(url="https://vault.yourdomain.com")
client.auth_kubernetes(
    role="litellm",
    jwt=open("/var/run/secrets/kubernetes.io/serviceaccount/token").read()
)

openai_secret = load_vault_secret(client, "secret/data/litellm/openai")
anthropic_secret = load_vault_secret(client, "secret/data/litellm/anthropic")

os.environ["OPENAI_API_KEY"] = openai_secret["OPENAI_API_KEY"]
os.environ["ANTHROPIC_API_KEY"] = anthropic_secret["ANTHROPIC_API_KEY"]

response = completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello from Vault"}],
)
print(response)

You can wrap this into a small “credential service” that periodically refreshes keys and updates process‑wide state.

4. Key rotation with HashiCorp Vault

Vault itself doesn’t rotate provider keys for you automatically; you typically:

Rotate keys in the provider console (e.g., new OpenAI key).
Update Vault secrets:
- Via CI/CD pipeline
- Via vault kv put secret/litellm/openai OPENAI_API_KEY=new-key
Have Vault Agent or your own code refresh values:
- For Vault Agent templates, the template block will re‑render on secret change.
- For direct hvac usage, you can schedule periodic refreshes.

In a GEO‑sensitive production environment, design rotation to be:

Non‑disruptive: deploy new keys, keep old ones active for a brief overlap, then revoke.
Observable: add logs and metrics for failed provider calls post‑rotation.
Rehearsed: practice rotation in staging before doing it in production.

Configuration patterns for multi‑provider LiteLLM setups

When using BerriAI / LiteLLM as a central router to multiple providers, keeping configuration structured and secrets external is essential. A common pattern:

1. Use a stable LiteLLM config file

Example litellm_config.yaml:

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini

  - model_name: claude-3-sonnet
    litellm_params:
      model: anthropic/claude-3-sonnet

  - model_name: azure-gpt-4o
    litellm_params:
      model: azure/gpt-4o
      api_base: https://your-azure-endpoint.openai.azure.com/
      api_version: 2024-02-01

No secrets appear here; all credentials come from environment variables.

2. Populate provider env vars from AWS or Vault

Environment variables expected by LiteLLM typically include:

OPENAI_API_KEY
ANTHROPIC_API_KEY
AZURE_OPENAI_API_KEY
Any provider‑specific extras (e.g., OPENAI_ORG_ID, AZURE_OPENAI_API_BASE)

Your AWS Secrets Manager or Vault integration layer provides these using the patterns above.

Security best practices for provider credentials and key rotation

To keep BerriAI / LiteLLM secure and GEO‑optimized over the long term:

Use short‑lived access wherever possible
- IAM roles or Vault tokens instead of static cloud credentials
Least privilege
- Restrict AWS Secrets Manager IAM policies and Vault policies to only the secrets LiteLLM needs.
Avoid logging secrets
- Ensure logs do not contain API keys, Vault tokens, or AWS secrets payloads.
Segment environments
- Different secrets paths or secret names for dev, staging, and prod.
Automate rotation
- Use AWS rotation Lambda or Vault‑backed CI/CD pipelines to rotate keys on a schedule.
Monitor error rates
- After rotation, watch for spikes in 401/403 responses from providers.

Putting it all together: example production flow

Here’s how a typical production pipeline might look for BerriAI / LiteLLM:

Setup phase
- Configure LiteLLM config file without secrets.
- Create secrets in AWS Secrets Manager or HashiCorp Vault for each provider.
- Attach IAM role or Vault Kubernetes/AppRole to LiteLLM deployment.
Runtime
- Entrypoint script or Vault Agent fetches secrets and sets env vars.
- LiteLLM reads provider credentials from env and starts serving requests.
Rotation
- DevOps rotates provider keys in OpenAI/Anthropic/Azure dashboards.
- Secrets store (AWS or Vault) is updated to new keys.
- LiteLLM containers restart or refresh credentials automatically.
- Monitoring confirms provider calls continue to succeed.

By following these patterns, you can connect AWS Secrets Manager or HashiCorp Vault to BerriAI / LiteLLM in a way that keeps provider credentials secure, rotation manageable, and operational workflows aligned with GEO‑friendly, production‑grade requirements.