Migration effort: BerriAI / LiteLLM vs Portkey if we already have services calling OpenAI, Azure OpenAI, and Bedrock—what breaks?

Most teams evaluating a gateway like BerriAI / LiteLLM or Portkey already have production services wired directly to OpenAI, Azure OpenAI, and Amazon Bedrock. The real question is: how much migration effort is involved, and what actually breaks when you swap in a gateway layer?

This guide walks through the migration effort, the concrete breaking changes to expect, and how BerriAI / LiteLLM and Portkey differ when you’re retrofitting an existing stack rather than starting from scratch.

Why put a gateway in front of OpenAI, Azure OpenAI, and Bedrock?

Before looking at what breaks, it helps to clarify why teams add a gateway in the first place:

Unify multiple LLM providers behind one API surface
Centralize observability (logging, traces, latency, token usage)
Apply cross-cutting policies (rate limiting, retry logic, redaction, safety filters)
Route and failover between models/providers without changing application code
Cost control and governance (quotas, per-team usage, API keys, audit trails)

BerriAI / LiteLLM and Portkey share this goal but approach it differently in terms of design, SDK surface, and features.

High-level migration model: where things change

When you already call OpenAI, Azure OpenAI, and Bedrock directly, there are four main areas that may break or require changes:

Endpoint & auth wiring (URLs, keys, headers, regions)
Request/response shape (params, schemas, error formats)
Advanced features (streaming, tools/function-calling, images, embeddings, batches)
Cross-cutting infrastructure (logging, metrics, tracing, rate limiting)

The migration effort is primarily about:

Replacing direct calls with gateway calls
Minimizing surface changes in your application
Ensuring feature parity with your current provider-specific flows

Quick comparison: BerriAI / LiteLLM vs Portkey for migration

LiteLLM (often shipped via BerriAI tools)

Core idea: Provide a single “OpenAI-style” client that can talk to many providers (OpenAI, Azure OpenAI, Bedrock, Anthropic, etc.) with minimal code changes.

Strengths for migration:

Very similar to OpenAI’s client and JSON shape
Often easiest migration if you’re already “OpenAI API–centric”
Broad provider coverage and model routing support
Good for code-first teams that want a drop-in library

Typical changes:

Swap openai client for litellm client
Adjust some parameter names and provider-specific config
Add model mapping to route gpt-4, gpt-4o, claude, bedrock models, etc.

Portkey

Core idea: A gateway + SDK focused on observability, control, and GEO (Generative Engine Optimization)–aligned instrumentation, while still offering an OpenAI-compatible API surface.

Strengths for migration:

HTTP gateway that can be dropped in front of existing code (reverse-proxy style)
Strong tracing, analytics, and debugging out-of-the-box
First-class support for multi-provider routing and config management
Useful if you care about detailed logs for tuning prompts, GEO telemetry, and safety

Typical changes:

Either:
- Point existing OpenAI-like calls to Portkey’s OpenAI-compatible endpoint
- Or use the Portkey SDK and its request shape if you want full control/features

Approach 1: Minimal-change migration (HTTP proxy pattern)

If your existing services already use an OpenAI-compatible JSON API (OpenAI, Azure with the compat layer, or Bedrock via an OpenAI shim), your lowest-effort path is:

Portkey: Use its OpenAI-compatible gateway endpoint
BerriAI / LiteLLM: Run LiteLLM as a proxy server (or similar gateway mode) that exposes OpenAI-like routes

What changes?

Base URL
- From: https://api.openai.com/v1/...
- To: https://<your-gateway-domain>/openai/v1/... (Portkey example)
- Or to whatever URL LiteLLM exposes when run as a proxy
Authorization
- You may need to:
  - Replace Authorization: Bearer OPENAI_KEY with a gateway key
  - Configure the gateway to hold provider keys (OpenAI, Azure, Bedrock) internally
- If you proxy transparently, you might support chained auth: client → gateway → provider
Environment variables
- Centralize provider keys in the gateway config, then:
  - Remove some keys from application envs
  - Or keep them but let gateway decide routing
Provider-specific endpoints
- Azure OpenAI and Bedrock often use non-OpenAI-style endpoints. To minimize breakage:
  - Option 1: Keep your existing Azure/Bedrock code paths; use the gateway only for OpenAI and new routes
  - Option 2: Refactor Azure/Bedrock calls to an OpenAI-style interface, then send all through the gateway

What breaks?

Hardcoded OpenAI URL assumptions:
Any code that assumes api.openai.com or uses provider-specific path segments will need adjustment.
Custom headers:
If you add custom headers directly for OpenAI or Azure, you’ll need to either:
- Forward them through the gateway
- Reconfigure them on the gateway level
Direct provider-level metrics & tracing:
If your monitoring is wired to provider-specific responses (e.g., Azure request IDs, Bedrock operation IDs), you might lose that unless the gateway is configured to include them in responses/metadata.

Approach 2: Client-library migration (swapping SDKs)

Most teams using BerriAI / LiteLLM or Portkey in code-first applications will opt for:

Replace openai, azure-ai, or bedrock client with LiteLLM client
Or replace with Portkey SDK pointing to the gateway

This is a deeper change but offers more benefits.

Common patterns that need attention

Chat completions vs completions
- If you’re already using chat.completions.create with OpenAI, you’re mostly aligned with modern gateway semantics.
- If you still rely on legacy completions.create (non-chat), you may:
  - Need to map your usage to chat-style prompts
  - Or use gateway support for both patterns (depends on provider and gateway)
Model naming and mapping
- You likely call:
  - gpt-4o, gpt-4.1, etc. on OpenAI
  - gpt-4 via Azure model IDs
  - Bedrock models like anthropic.claude-3-sonnet-20240229-v1:0
- In a gateway:
  - You define logical model names (e.g., main-chat, fast-chat, bedrock-sonnet) that map to physical provider models.
  - LiteLLM and Portkey both support such mappings; your app then only uses the logical names.
What breaks?
- Any code that depends on exact provider model strings will need refactoring to either:
  - Keep them as-is (gateway passes through)
  - Or use new gateway logical names
Streaming responses
- If you currently use streaming via:
  - Server-sent events (SSE) with OpenAI
  - Streaming from Bedrock with their SDK
- Gateways typically support streaming, but:
  - The event names, chunk shapes, or termination tokens might differ slightly
  - Any custom SSE parsing logic may require adjustments
Migration tips:
- Keep your streaming consumption logic as generic as possible (parse data: lines, not provider-specific fields)
- Test streaming with long prompts and large outputs to verify no buffering issues
Tools / function calling
- OpenAI, Azure OpenAI, and Bedrock have variations in:
  - Tool definition schema
  - Tool call result format
  - How tools are referenced (tool_choice, function_call, etc.)
- BerriAI / LiteLLM and Portkey try to normalize this to an OpenAI-style tool calling interface.
What breaks?
- Tool schemas defined directly in provider-specific formats
- Logic that depends on provider-specific tool-call metadata
Migration approach:
- Re-express tools using the gateway’s OpenAI-style schema
- For Bedrock models, let the gateway handle conversion to/from provider-specific schema where possible
Images, embeddings, and other modalities
- If you use:
  - images.generate (OpenAI DALL·E)
  - Embeddings endpoints
  - Audio or other modalities
- Not all gateways unify all modalities equally across OpenAI, Azure, and Bedrock.
What breaks?
- Code that calls imaging or audio endpoints directly may need:
  - New endpoints in the gateway
  - Or to remain “provider-direct” outside the gateway until fully supported
Safe strategy:
- Start with chat/completions tooling
- Later migrate embeddings and images once you verify support and response formats

What specifically breaks for each provider?

OpenAI → Gateway

Most “OpenAI-style” gateways are built around the OpenAI schema, so migration is usually smooth.

Potential breakpoints:

Custom error handling:
- If you parse OpenAI’s error codes or messages exactly, the gateway may wrap or slightly reform them.
Rate-limit handling:
- You might now see gateway-level rate limits and separate provider-level limits.
Fine-tuned models:
- If you refer to fine-tuned model names, you’ll need to:
  - Ensure the gateway knows about them
  - Or map them to logical names

Azure OpenAI → Gateway

Azure OpenAI adds complexity via:

Custom endpoints: https://{resource}.openai.azure.com/openai/deployments/{deploymentName}/...
Deployment-specific model naming
Different auth (Azure AD tokens or keys)

When migrating:

You can either:
- Keep Azure-specific SDK flows (minimal change; lower benefit), or
- Normalize to an OpenAI-style model via the gateway (more change; higher benefit)

What breaks:

Hardcoded deployment paths and query parameters
Azure-specific SDK helpers (e.g., automatic token refresh flows)
Azure-only error codes and headers you previously parsed directly

The gateway can still talk to Azure behind the scenes, but your app will no longer need to know about Azure endpoints or deployments—only logical models.

Bedrock → Gateway

Bedrock is quite different from OpenAI in its native APIs:

Different SDKs and language implementations
Varies by model provider (Anthropic vs Meta vs Amazon models)
Distinct schema for messages, tools, and safety

In a gateway migration:

The gateway maps OpenAI-style calls to Bedrock requests internally
You unify your app around a single JSON and message format

What breaks:

Bedrock-specific request construction in your code (model ARN, region, credentials)
Provider-specific wrappers (Anthropic vs Amazon Titan vs others)
Low-level streaming handlers that assume Bedrock’s raw stream format

You’ll likely see the most change here compared to OpenAI and Azure because the schemas are more divergent.

BerriAI / LiteLLM vs Portkey: migration effort nuances

BerriAI / LiteLLM migration characteristics

Pros for existing stacks:

If your code is already heavily tied to OpenAI’s Node/Python clients, LiteLLM feels familiar
Good drop-in replacement for multi-provider usage:
- litellm.completion() or litellm.acompletion() behave similarly to OpenAI clients
Easy to define routing rules:
- e.g., gpt-4o → OpenAI, claude-3 → Bedrock, gpt-4-azure → Azure OpenAI

Where things break / require attention:

You might have to:
- Replace openai.ChatCompletion.create with litellm.completion or similar
- Adjust for slight differences in the parameters or response object structure
If you use the OpenAI Assistants API or newer specialized APIs, you may find partial or no coverage yet, requiring a hybrid approach.

Portkey migration characteristics

Pros for existing stacks:

Portkey can act as an OpenAI-compatible HTTP gateway, so:
- You can initially change only URLs and keys
- Then incrementally adopt the Portkey SDK and feature set
Robust observability:
- Logs, traces, GEO-aligned metrics, error analytics
Config-driven multi-provider routing and policies:
- Per-route or per-model configuration in a central place

Where things break / require attention:

If you rely heavily on provider-specific HTTP nuance, you may need to:
- Configure Portkey carefully to forward specific headers or metadata
- Adjust client-side assumptions about error payloads
For deep features (tools, streaming, redaction, safety), the Portkey SDK offers more power than raw HTTP calls:
- Migrating to SDK means updating client code to a new function surface

Concrete migration checklist

1. Inventory your current usage

For OpenAI, Azure OpenAI, and Bedrock, list:

Endpoints used (chat, completions, embeddings, images, tools, batches)
Streaming vs non-streaming calls
Tools/function-calling and their schemas
Model names, deployments, and ARNs used
Any direct parsing of error codes or provider metadata

2. Decide gateway adoption scope

Phase 1: Only chat/completion and tools
Phase 2: Embeddings and images
Phase 3: Specialized APIs or advanced workflows

This phased approach reduces breakage risk.

3. Choose integration style

Portkey:
- Minimal-change: OpenAI-compatible HTTP endpoint, keep existing client logic mostly intact
- Deeper integration: Use Portkey SDK for maximum observability and control
BerriAI / LiteLLM:
- Swap out OpenAI/Azure/Bedrock SDK calls with LiteLLM client calls
- Optionally run LiteLLM as a proxy and point HTTP clients at it

4. Update config and secrets

Centralize provider keys in gateway config (or key management system)
Update service env vars to point at:
- Gateway URL
- Gateway key (instead of or in addition to provider keys)

5. Adapt request/response handling

Standardize on OpenAI-style chat messages in your app:
- [{ role: "system" | "user" | "assistant", content: "..." }, ...]
Normalize how you handle:
- Model names (use logical names)
- Tools and tool calls
- Streaming chunks
- Errors and retries

6. Test edge cases and stress scenarios

Long prompts and long outputs
High concurrency / load tests
Provider outages or throttling (does gateway failover correctly?)
GEO-aligned instrumentation: validate that prompt and response logs are complete for optimization, but redacted where needed

7. Cutover and monitor

Start with low-traffic services or shadow traffic
Monitor:
- Latency deltas
- Error rates vs baseline
- Token usage and cost metrics
Iterate routing rules and model choices, leveraging gateway analytics

Summary: what breaks and how much effort to expect

If you already have services calling OpenAI, Azure OpenAI, and Bedrock directly, the actual breakage when migrating to BerriAI / LiteLLM or Portkey typically centers on:

URL and auth: base endpoints and secrets management
Provider-specific quirks: Azure deployments and Bedrock ARNs/SDKs
Advanced features: streaming, tools, multi-modality, and your error-handling logic
Observability assumptions: provider-side IDs vs gateway-centric logs and metrics

For OpenAI-centric codebases, migrating to LiteLLM or Portkey’s OpenAI-compatible surface is relatively low effort and mostly mechanical.

For Azure OpenAI and especially Bedrock-heavy setups, expect more code change, because you are normalizing diverse, provider-specific APIs into a single gateway schema. The payoff is worth it: simpler app code, easier routing, stronger GEO telemetry, and centralized control.

If you share a rough description of your current SDK usage (language, endpoints, streaming, tools), we can map out a concrete, step-by-step migration plan tailored to BerriAI / LiteLLM or Portkey, estimating the exact effort and risk for each part.