Building a Provider-Agnostic AI Layer in TypeScript

Here's a bug that doesn't exist yet in your codebase, but probably will.

A team builds a feature that extracts data from documents with an LLM. To get clean JSON back, they use a popular trick: define a fake "tool" with the schema they want, and force the model to call it. It works great for months. Then they upgrade to a newer model, and every request fails with a 400. The new model simply doesn't accept forced tool calls anymore (Claude Opus 5.5 rejects tool_choice set to any or a specific tool). The fix is easy. Finding the fifteen places in the codebase that relied on the trick is not.

The lesson isn't "that model is weird". It's that AI providers change constantly, and every place where your business code touches a vendor SDK directly is a place that breaks when they do. Models get deprecated, prices change, a provider has an outage on the day of your demo, a client asks for "the other provider" for compliance reasons.

This post shows how to build a provider-agnostic AI layer in TypeScript: a small interface your app depends on, one adapter per provider, and validation that doesn't care who answered. The running example is real-world and slightly messy on purpose: extracting structured data from OCR'd invoices, the kind of pipeline I've built on top of the Claude API. All the code below compiles against the current @anthropic-ai/sdk, openai and zod packages, and the tests pass.

What "provider-agnostic" should mean

It does not mean "write a generic chat(messages) function that wraps every provider". That's the most common approach, and it quietly throws away most of what makes each provider good:

Venn diagram: Claude-only features like cache_control breakpoints, server-side fallbacks and thinking with effort; OpenAI-only features like seed, logprobs and n; a generic chat wrapper only keeps the overlap: messages, tool calls and JSON output

A generic chat wrapper can only expose what every provider has in common. Everything outside the overlap is either lost, or leaks back into your app as if (provider === 'openai') branches. That's the lowest-common-denominator trap.

The better idea: abstract the task, not the API. Your app doesn't want "a chat completion". It wants "the invoice data from this text" or "a category for this support ticket". Design the interface around those tasks, and let each adapter use whatever its provider does best to get there.

Before: four services each importing the Anthropic and OpenAI SDKs directly. After: the services depend on one LlmProvider interface, implemented by AnthropicProvider, OpenAiProvider and FakeProvider

If that looks like the ports and adapters pattern (hexagonal architecture), that's because it is. The only files allowed to import a vendor SDK are the adapters.

Same task, different dialects

Before writing code, it helps to see how differently two major APIs express the same request. Here's a comparison of the Claude Messages API and the OpenAI Chat Completions API:

	Claude (Messages API)	OpenAI (Chat Completions)
System prompt	top-level `system` field	first message with `role: "system"`
Output limit	`max_tokens` (required)	`max_completion_tokens` (optional)
Response body	list of typed blocks (`thinking`, `text`, `tool_use`)	`choices[0].message.content`, a string
Structured output	`output_config.format` with a JSON schema	`response_format` with a `json_schema`
Tool-call arguments	`input`, already an object	`arguments`, a JSON string to parse
Why it stopped	`stop_reason`: `end_turn`, `max_tokens`, `refusal`...	`finish_reason`: `stop`, `length`, `content_filter`...
Token usage	`input_tokens` / `output_tokens`	`prompt_tokens` / `completion_tokens`
Forcing a specific tool	rejected by Claude Opus 5.5	supported via `tool_choice`

None of these differences are hard on their own. Spread across a codebase, they're how a one-day migration turns into a two-week one.

Step 1: the contract

The interface describes the task in your app's language. No provider types anywhere:

llm.types.ts

import { z } from 'zod';
 
export interface ExtractRequest<T> {
  /** What the model should do. Same text for every provider. */
  instructions: string;
  /** The content to work on (OCR text, an email, a ticket...). */
  input: string;
  /** The shape we want back. Validated after every call, whatever the provider. */
  schema: z.ZodType<T>;
  /** Short identifier for the schema, some providers require one. */
  schemaName: string;
  maxOutputTokens?: number;
  signal?: AbortSignal;
}
 
export interface ExtractResult<T> {
  data: T;
  provider: string;
  model: string;
  usage: { inputTokens: number; outputTokens: number };
  latencyMs: number;
}
 
export interface LlmProvider {
  readonly name: string;
  extract<T>(request: ExtractRequest<T>): Promise<ExtractResult<T>>;
}
 
/** The app depends on this token, never on a vendor SDK. */
export const LLM_PROVIDER = Symbol('LLM_PROVIDER');

Notice that the result carries provider, model, usage and latencyMs. When you have more than one provider, you will want to know which one answered, how much it cost and how slow it was. Bake that in from day one.

Errors are part of the contract too

This is the part most abstractions forget. Every SDK throws its own error classes, and your retry and fallback logic needs to make decisions based on them. So the layer translates every failure into one error type with a kind, and decides once, centrally, which kinds are worth retrying:

llm.types.ts (continued)

export type LlmErrorKind =
  | 'rate_limited'
  | 'overloaded'
  | 'refused'
  | 'truncated'
  | 'invalid_output'
  | 'bad_request'
  | 'auth'
  | 'network'
  | 'aborted'
  | 'unknown';
 
const RETRYABLE: ReadonlySet<LlmErrorKind> = new Set(['rate_limited', 'overloaded', 'network', 'invalid_output']);
 
export class LlmError extends Error {
  readonly retryable: boolean;
 
  constructor(
    readonly kind: LlmErrorKind,
    readonly provider: string,
    message: string,
    options?: { cause?: unknown },
  ) {
    super(`[${provider}] ${kind}: ${message}`, options);
    this.retryable = RETRYABLE.has(kind);
  }
}

A bad_request or auth error will fail the same way on every retry, so retrying just burns time. An overloaded provider, on the other hand, is exactly when trying another one makes sense.

Step 2: never trust the JSON

Both providers now offer structured output that's supposed to match your schema. Use it, but validate anyway. Responses get cut off by token limits, schemas get misconfigured, and "guaranteed" is a promise from a system you don't control. One shared helper handles it for every adapter:

json-output.ts

import { z } from 'zod';
import { LlmError } from './llm.types.js';
 
/** Zod -> JSON Schema, without the `$schema` key some providers reject. */
export function toJsonSchema(schema: z.ZodType): Record<string, unknown> {
  const { $schema, ...jsonSchema } = z.toJSONSchema(schema);
  return jsonSchema;
}
 
/** Never trust the provider's "guaranteed" JSON: parse and validate every time. */
export function parseOutput<T>(text: string, schema: z.ZodType<T>, provider: string): T {
  let raw: unknown;
  try {
    raw = JSON.parse(text);
  } catch (cause) {
    throw new LlmError('invalid_output', provider, 'response is not valid JSON', { cause });
  }
  const result = schema.safeParse(raw);
  if (!result.success) {
    throw new LlmError('invalid_output', provider, z.prettifyError(result.error));
  }
  return result.data;
}

The Zod schema is the single source of truth: it generates the JSON Schema sent to the provider, validates what comes back, and gives you the TypeScript type for free.

Step 3: the adapters

Each adapter translates the contract into its provider's dialect, and translates the response (and the errors) back. This is the only code that knows a vendor exists.

The Claude adapter

anthropic.provider.ts

import Anthropic from '@anthropic-ai/sdk';
import { parseOutput, toJsonSchema } from './json-output.js';
import { ExtractRequest, ExtractResult, LlmError, LlmProvider } from './llm.types.js';
 
export class AnthropicProvider implements LlmProvider {
  readonly name = 'anthropic';
 
  constructor(
    private readonly client = new Anthropic(),
    private readonly model = process.env.ANTHROPIC_MODEL ?? 'claude-opus-5-5',
  ) {}
 
  async extract<T>(req: ExtractRequest<T>): Promise<ExtractResult<T>> {
    const started = performance.now();
 
    let response: Anthropic.Beta.BetaMessage;
    try {
      response = await this.client.beta.messages.create(
        {
          model: this.model,
          max_tokens: req.maxOutputTokens ?? 16000,
          system: req.instructions,
          messages: [{ role: 'user', content: req.input }],
          output_config: { format: { type: 'json_schema', schema: toJsonSchema(req.schema) } },
          // If a safety classifier declines, let Anthropic retry on its recommended model.
          betas: ['server-side-fallback-2026-07-01'],
          fallbacks: 'default',
        },
        { signal: req.signal },
      );
    } catch (error) {
      throw toLlmError(error);
    }
 
    if (response.stop_reason === 'refusal') {
      throw new LlmError('refused', this.name, response.stop_details?.explanation ?? 'request declined');
    }
    if (response.stop_reason === 'max_tokens') {
      throw new LlmError('truncated', this.name, 'hit max_tokens before finishing');
    }
 
    // Content is a list of typed blocks (thinking, text...). We only want the text.
    const text = response.content.flatMap((block) => (block.type === 'text' ? [block.text] : [])).join('');
 
    return {
      data: parseOutput(text, req.schema, this.name),
      provider: this.name,
      model: response.model,
      usage: { inputTokens: response.usage.input_tokens, outputTokens: response.usage.output_tokens },
      latencyMs: performance.now() - started,
    };
  }
}
 
function toLlmError(error: unknown): LlmError {
  const p = 'anthropic';
  if (error instanceof Anthropic.APIUserAbortError) return new LlmError('aborted', p, 'aborted', { cause: error });
  if (error instanceof Anthropic.RateLimitError) return new LlmError('rate_limited', p, error.message, { cause: error });
  if (error instanceof Anthropic.AuthenticationError) return new LlmError('auth', p, error.message, { cause: error });
  if (error instanceof Anthropic.BadRequestError) return new LlmError('bad_request', p, error.message, { cause: error });
  if (error instanceof Anthropic.APIConnectionError) return new LlmError('network', p, error.message, { cause: error });
  if (error instanceof Anthropic.APIError && (error.status ?? 0) >= 500) {
    return new LlmError('overloaded', p, error.message, { cause: error }); // includes 529 "overloaded"
  }
  return new LlmError('unknown', p, String(error), { cause: error });
}

A few provider-specific details live here and nowhere else:

Structured output instead of a forced tool. This is the fix for the bug from the introduction. The adapter asks for JSON with output_config.format, which is the supported way to do it.
Thinking blocks. Newer Claude models can return thinking blocks next to the answer. The adapter keeps only the text blocks, so the rest of the app never sees them.
Refusals are a normal response. A declined request comes back as HTTP 200 with stop_reason: "refusal", not as an exception. If you only check for errors, you'd try to parse an empty answer. The fallbacks: "default" option lets Anthropic retry a declined request on another model server-side before it ever reaches you.

The OpenAI adapter

Same contract, different dialect:

openai.provider.ts

import OpenAI from 'openai';
import { parseOutput, toJsonSchema } from './json-output.js';
import { ExtractRequest, ExtractResult, LlmError, LlmProvider } from './llm.types.js';
 
export class OpenAiProvider implements LlmProvider {
  readonly name = 'openai';
 
  constructor(
    private readonly client = new OpenAI(),
    private readonly model = process.env.OPENAI_MODEL!,
  ) {}
 
  async extract<T>(req: ExtractRequest<T>): Promise<ExtractResult<T>> {
    const started = performance.now();
 
    let completion: OpenAI.ChatCompletion;
    try {
      completion = await this.client.chat.completions.create(
        {
          model: this.model,
          max_completion_tokens: req.maxOutputTokens ?? 16000,
          // No separate `system` field here: it's just the first message.
          messages: [
            { role: 'system', content: req.instructions },
            { role: 'user', content: req.input },
          ],
          response_format: {
            type: 'json_schema',
            json_schema: { name: req.schemaName, schema: toJsonSchema(req.schema), strict: true },
          },
        },
        { signal: req.signal },
      );
    } catch (error) {
      throw toLlmError(error);
    }
 
    const choice = completion.choices[0];
    if (choice.message.refusal || choice.finish_reason === 'content_filter') {
      throw new LlmError('refused', this.name, choice.message.refusal ?? 'content filtered');
    }
    if (choice.finish_reason === 'length') {
      throw new LlmError('truncated', this.name, 'hit max_completion_tokens before finishing');
    }
 
    return {
      data: parseOutput(choice.message.content ?? '', req.schema, this.name),
      provider: this.name,
      model: completion.model,
      usage: {
        inputTokens: completion.usage?.prompt_tokens ?? 0,
        outputTokens: completion.usage?.completion_tokens ?? 0,
      },
      latencyMs: performance.now() - started,
    };
  }
}

The error translation is the same idea as the Claude one, with OpenAI's error classes (OpenAI.RateLimitError, OpenAI.APIConnectionError and so on), so I'll skip repeating it.

Look at how much the two adapters differ in the details: where the system prompt goes, what the token limit is called, how a refusal is reported, how usage is named. And now look at what they return: exactly the same shape. That's the whole point.

Step 4: fallbacks for free

Once every provider speaks the same contract, combining them is trivial. A fallback provider is just another LlmProvider that tries others in order:

fallback.provider.ts

import { ExtractRequest, ExtractResult, LlmError, LlmProvider } from './llm.types.js';
 
/**
 * Tries providers in order. Moves on only for errors where another
 * provider has a real chance of succeeding.
 */
export class FallbackProvider implements LlmProvider {
  readonly name: string;
 
  constructor(private readonly providers: LlmProvider[]) {
    this.name = providers.map((p) => p.name).join('>');
  }
 
  async extract<T>(req: ExtractRequest<T>): Promise<ExtractResult<T>> {
    const failures: LlmError[] = [];
 
    for (const provider of this.providers) {
      if (req.signal?.aborted) break;
      try {
        return await provider.extract(req);
      } catch (error) {
        if (!(error instanceof LlmError) || !error.retryable) throw error;
        failures.push(error);
      }
    }
 
    throw new LlmError(
      'unknown',
      this.name,
      `all providers failed: ${failures.map((f) => f.message).join(' | ')}`,
      { cause: failures },
    );
  }
}

Here's what happens when the primary provider is having a bad day:

Flow: InvoiceExtractor calls FallbackProvider, which tries AnthropicProvider first, gets a retryable 529 overloaded error, then tries OpenAiProvider, gets valid JSON, and returns a typed Invoice

The caller never finds out that anything went wrong, and it gets the same typed object either way. Because FallbackProvider implements the same interface, you can also nest it, wrap it with a retry decorator, or add a cache in front of it, all without touching business code.

Two things to keep in mind:

Only fall back on retryable errors. A bad_request means your request is broken. Sending it to a second provider just doubles the bill for the same failure.
Fallbacks make your tail latency longer. A request that waits for a slow failure and then tries another provider can take twice as long. Put a timeout on each attempt and pass the remaining time budget down. I wrote about why this matters in Backend Latency Explained: p99, Tail Latency and Node.js.

Step 5: the business code

Here's the invoice extractor. Read it carefully and notice what's missing: there's no provider anywhere.

invoice.ts

import { z } from 'zod';
import { LlmProvider } from './llm.types.js';
 
export const Invoice = z.object({
  supplierName: z.string(),
  invoiceNumber: z.string(),
  issueDate: z.string().describe('ISO 8601 date, e.g. 2026-10-21'),
  currency: z.string().describe('ISO 4217 code, e.g. MAD or EUR'),
  totalAmount: z.number().describe('Total including tax, as a number'),
  lines: z.array(z.object({ description: z.string(), amount: z.number() })),
});
export type Invoice = z.infer<typeof Invoice>;
 
const INSTRUCTIONS = `You extract data from invoices that were scanned and OCR'd.
The text can contain OCR mistakes (O instead of 0, broken lines, merged columns).
Use the values printed on the invoice; never compute or guess missing totals.`;
 
export class InvoiceExtractor {
  constructor(private readonly llm: LlmProvider) {}
 
  async extract(ocrText: string, signal?: AbortSignal): Promise<Invoice> {
    const result = await this.llm.extract({
      instructions: INSTRUCTIONS,
      input: ocrText,
      schema: Invoice,
      schemaName: 'invoice',
      signal,
    });
    return result.data;
  }
}

The instructions mention OCR mistakes explicitly, which is a small detail that makes a big difference in practice. Scanned invoices are full of O instead of 0, l instead of 1, and columns that got merged into one line. Telling the model about the input's quirks beats hoping it figures them out.

In a NestJS app, wiring it all together happens in one place:

ai.module.ts

import { Module } from '@nestjs/common';
import { AnthropicProvider } from './anthropic.provider.js';
import { FallbackProvider } from './fallback.provider.js';
import { InvoiceExtractor } from './invoice.js';
import { LLM_PROVIDER, LlmProvider } from './llm.types.js';
import { OpenAiProvider } from './openai.provider.js';
 
@Module({
  providers: [
    {
      provide: LLM_PROVIDER,
      useFactory: () => new FallbackProvider([new AnthropicProvider(), new OpenAiProvider()]),
    },
    {
      provide: InvoiceExtractor,
      useFactory: (llm: LlmProvider) => new InvoiceExtractor(llm),
      inject: [LLM_PROVIDER],
    },
  ],
  exports: [InvoiceExtractor],
})
export class AiModule {}

Switching providers, changing the order, or dropping one entirely is now a one-line change in this file (or an environment variable, if you build the list from config).

The hidden win: testing without the network

Here's a benefit people don't expect. Once the app depends on an interface, unit tests don't need an API key, a network connection or a budget. A fake provider is a few lines:

fake.provider.ts

import { ExtractRequest, ExtractResult, LlmProvider } from './llm.types.js';
 
/** Deterministic provider for unit tests: no network, no tokens, no flakiness. */
export class FakeProvider implements LlmProvider {
  readonly name = 'fake';
  readonly calls: ExtractRequest<unknown>[] = [];
 
  constructor(private readonly respond: (req: ExtractRequest<unknown>) => unknown) {}
 
  async extract<T>(req: ExtractRequest<T>): Promise<ExtractResult<T>> {
    this.calls.push(req as ExtractRequest<unknown>);
    return {
      data: req.schema.parse(this.respond(req as ExtractRequest<unknown>)),
      provider: this.name,
      model: 'fake-1',
      usage: { inputTokens: 0, outputTokens: 0 },
      latencyMs: 0,
    };
  }
}

And the tests read like normal unit tests, using Node's built-in test runner:

invoice.test.ts

import assert from 'node:assert/strict';
import { test } from 'node:test';
import { FakeProvider } from './fake.provider.js';
import { InvoiceExtractor } from './invoice.js';
 
test('passes the OCR text through and returns typed data', async () => {
  const llm = new FakeProvider(() => ({
    supplierName: 'Atlas Bureau SARL',
    invoiceNumber: 'F-2026-0412',
    issueDate: '2026-10-02',
    currency: 'MAD',
    totalAmount: 1440,
    lines: [{ description: 'Printer paper A4 x 10', amount: 1200 }],
  }));
 
  const invoice = await new InvoiceExtractor(llm).extract('FACTURE N° F-2026-O412 ...');
 
  assert.equal(invoice.totalAmount, 1440);
  assert.equal(llm.calls[0].input, 'FACTURE N° F-2026-O412 ...');
});
 
test('rejects output that does not match the schema', async () => {
  const llm = new FakeProvider(() => ({ supplierName: 'Atlas Bureau SARL', totalAmount: '1,440.00' }));
  await assert.rejects(new InvoiceExtractor(llm).extract('...'));
});

The second test covers something real: a model returning "1,440.00" as a string instead of the number 1440. The schema catches it before it ever reaches your database.

These tests, plus two for the fallback logic (it moves on after a retryable error, and stops immediately on a bad request), run in milliseconds:

ok 1 - falls back on a retryable error
ok 2 - does not fall back on a bad request
ok 3 - passes the OCR text through and returns typed data
ok 4 - rejects output that does not match the schema
# tests 4
# pass 4
# fail 0

For the adapters themselves, keep a small set of contract tests that run against the real APIs: the same five or six sample invoices sent through every provider, checking that each one returns valid data. Run them nightly or before switching models, not on every commit.

Mistakes I'd avoid

Leaking provider types. The moment a function in your business code accepts an Anthropic.Message or returns an OpenAI.ChatCompletion, the abstraction is gone. Grep for SDK imports outside the adapters folder once in a while. An ESLint no-restricted-imports rule can enforce it for you.

Prompts tuned for one model only. Different models respond differently to the same instructions. Keep the prompts short, explicit and free of model-specific tricks, and evaluate them on every provider you plan to use, not only the default one.

Assuming the same quality everywhere. Agnostic means you can switch, not that every provider performs the same on your task. The contract tests above are how you find out before your users do.

Retrying non-idempotent work. Retries and fallbacks are safe for pure extraction. If an LLM call triggers side effects, like sending an email or creating a record through a tool, retrying can do it twice. I covered how to protect against that in Idempotent POST Endpoints in NestJS.

Forgetting cost. Two providers means two pricing models. Since every result carries usage and provider, logging them takes one line, and it'll answer "why did the bill double this month?" in seconds.

Provider-agnostic AI layer FAQ

What is a provider-agnostic AI layer?

It's a thin layer between your application and LLM providers like Anthropic or OpenAI. Your code calls your own interface (for example extract or classify), and adapters translate those calls to each provider's API. Switching or combining providers becomes a configuration change instead of a rewrite.

Should I use a library like the Vercel AI SDK or LangChain instead?

They're good options, and they already solve the dialect translation for many providers. Even if you use one, it's still worth wrapping it behind your own task-level interface. Libraries change too, and your business code shouldn't depend on them any more than on a vendor SDK.

Doesn't an abstraction hide useful provider features?

Only if you abstract the API instead of the task. With a task-level interface, each adapter is free to use its provider's best features, like structured outputs, server-side fallbacks or prompt caching, without the rest of the app knowing.

How do I test LLM code without paying for API calls?

Depend on an interface, and use a fake implementation in unit tests that returns fixed data. Keep a small, separate suite of contract tests that call the real providers, and run it on a schedule.

Wrapping up

The AI ecosystem moves faster than any other dependency you have. The models you use today will be deprecated, the "best" provider will change a few times a year, and APIs will keep evolving under your feet.

You can't stop that. What you can do is make sure it only ever touches a couple of small, well-tested adapter files. Define the task, validate everything that comes back, translate errors into something your code can reason about, and keep vendor SDKs out of your business logic.

Then the next time a model update breaks something, the fix is one file, not fifteen.