AWS Bedrock as LLM Provider: From OpenAI Fallback to Claude + Nova Pro

One SDK instead of two libraries. IAM instead of API keys. Data in the EU instead of the US. A single bill instead of two invoices. Here is how we moved the entire fallback layer to AWS Bedrock — and why it changed more than we expected.

AWS Bedrock as LLM Provider: From OpenAI Fallback to Claude + Nova Pro

How one PR changed the fallback layer architecture and why API keys are yesterday's news

The Problem: Two API Keys, Two Bills, Zero Guarantees

LEX AI processes thousands of legal queries daily. Every query is an LLM call: intent classification, database search, decision analysis, response generation. When OpenAI goes down (and it happens more often than we would like), the platform must keep working.

Previously, we used the Anthropic API as a fallback provider. It worked, but created a number of problems:

| Problem | Consequence | |———|————| | Two separate API keys | Secret rotation x 2, leak risk x 2 | | Two billing accounts | Monthly reconciliation of two invoices, no Reserved Capacity | | Data goes to the US | Anthropic API does not guarantee EU residency | | Per-key rate limits | Under load spikes, fallback is also throttled | | Round-robin failed | We already wrote about this — different response formats broke parsing |

We needed a single fallback provider that gives access to multiple models through one SDK, with IAM authorization, and data within the EU.

The Solution: AWS Bedrock

AWS Bedrock is a managed service that provides access to models from different vendors through a unified API. One SDK, one authorization (IAM), one billing, choice of region.

Through Bedrock, we immediately gained access to two model families:

• Claude (Anthropic) — via Bedrock, without a separate API key
• Amazon Nova — AWS's own models, optimized for cost

Budget-Aware Model Tiers

Our ModelSelector already supported three performance tiers. We simply replaced the fallback models:

| Tier | Purpose | Primary (OpenAI) | Fallback (Bedrock) | |——|———|——————-|———————| | quick | Classification, routing | gpt-5-nano | Amazon Nova Micro | | standard | Tool execution, summarization | gpt-5-mini | Amazon Nova Lite | | deep | Legal analysis, patterns | gpt-5.1 | Amazon Nova Pro |

Nova Micro and Nova Lite cover cheap tasks, while Nova Pro is a full-fledged alternative for complex analysis. Claude via Bedrock remains available for cases where its reasoning quality is specifically needed.

Migration: What Changed in the Code

Before: Two Clients, Two Formats

// Before: direct connection to Anthropic API
import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY, // yet another secret
});

After: Unified AWS SDK

// After: Bedrock via AWS SDK
import {
  BedrockRuntimeClient,
  ConverseCommand,
} from '@aws-sdk/client-bedrock-runtime';

const bedrock = new BedrockRuntimeClient({
  region: 'eu-central-1', // data stays in the EU
  // IAM authorization — no API keys
});

The key change — Converse API. This is Bedrock's unified interface that accepts the same message format regardless of the model. The same code works for both Nova Pro and Claude via Bedrock. No parsing different formats — the problem that killed our round-robin.

Authorization: IAM Instead of API Keys

This is probably the biggest win. Instead of storing ANTHROPIC_API_KEY in .env files on every server, we use the EC2 instance's IAM role:

{
  "Effect": "Allow",
  "Action": [
    "bedrock:InvokeModel",
    "bedrock:InvokeModelWithResponseStream"
  ],
  "Resource": "arn:aws:bedrock:eu-central-1::foundation-model/*"
}

No secrets in environment variables. No key rotation. Credentials are taken automatically from the Instance Metadata Service. One less attack vector.

Results

| Metric | Before (Anthropic API) | After (Bedrock) | Change | |——–|————————|—————–|——–| | Fallback latency (p50) | 1.8s | 1.2s | -33% | | Fallback latency (p99) | 8.4s | 4.1s | -51% | | Fallback query cost | 0.018/query | 0.011/query | -39% | | Secrets in .env | 4 (2 OpenAI + 2 Anthropic) | 2 (OpenAI only) | -50% | | Data in EU | Not guaranteed | eu-central-1 | Guaranteed |

The latency reduction is explained by two factors: EC2 → Bedrock is traffic within the AWS region (no internet egress), and Nova Pro is simply faster than Claude for typical legal tasks.

Provisioned Throughput: The Next Step

Bedrock allows purchasing Provisioned Throughput — guaranteed capacity for a specific model. For us this means:

• Predictable cost: fixed price instead of pay-per-token
• Guaranteed SLA: no 429s (rate limit) during load spikes
• Budget planning: the monthly amount is known in advance

We plan to activate Provisioned Throughput for Nova Pro on the deep tier, where predictability matters most — legal analysis cannot wait in a queue.

Conclusions

One PR, but the architectural impact is tangible:

1. IAM instead of API keys — fewer secrets, less risk
2. EU data residency — data does not leave eu-central-1
3. Single billing — AWS Cost Explorer instead of two invoices
4. Converse API — one format for all models
5. Nova Pro — cheaper and faster fallback for legal analysis

If your platform uses multiple LLM providers and you are tired of the API key zoo — take a look at Bedrock. It is not a silver bullet, but for fallback scenarios it is the most elegant solution we have found.