How to integrate an LLM into your product: architecture and pitfalls

Integrating an LLM into a product means more than calling a model's API: you build a layer around it that prepares context, validates the output, tracks cost and survives provider failures. The call itself is a couple of lines of code; the real work lies in the data you give the model and in checking what it returns. That is where design should start.

Below is the architecture we use for AI features and the pitfalls that usually surface only after launch.

Where do you start integrating an LLM into a product?

Start with one specific user task and a way to measure how well the model solves it. "Add AI" is not a task; "cut support response time on common questions" or "extract payment details from uploaded invoices" is a task with a metric and a clear cost of error.

Before development, answer four questions:

1. What does the user get back, and in what form — text, structured data, an action in the system?
2. What data does the model need to answer correctly, and where does it live?
3. What happens if the model is wrong — and who will notice?
4. How many requests a day do you expect, and what budget is acceptable?

The answer to the third question shapes the architecture most. A draft email that a person will reread forgives mistakes. An automatic price change in your catalogue does not.

What does the architecture of an AI feature look like?

An AI feature consists of several layers around the model: context retrieval, a prompt template, a call through a gateway with limits, output validation and logging. The model is a replaceable component in this design; the product's value sits in the other layers.

Layer	What it does	Why
Context retrieval	Fetches the data needed: documents, history, profile	The model only knows what you pass it
Prompt template	Instructions, examples, output format — versioned	Prompt changes are tracked like code
LLM gateway	Model choice, retries, timeouts, spend limits	Switching provider does not touch business logic
Output validation	Checks structure, allowed values, restrictions	A model error never reaches the user
Logging and evaluation	Stores requests, responses, cost and feedback	Quality and cost are visible on real data

Photo: Unsplash

Structured output instead of free text

If code rather than a person consumes the result, ask the model to return data against a schema and validate it as strictly as user input. Most providers support structured output and tool calling.

import { z } from 'zod'

const Invoice = z.object({
  supplier: z.string().min(1),
  total: z.number().positive(),
  currency: z.enum(['USD', 'EUR', 'GBP']),
  dueDate: z.string().date(),
})

export async function extractInvoice(text: string) {
  const raw = await llm.complete({
    prompt: prompts.get('invoice-extract', 'v3'),
    input: text,
    responseFormat: 'json',
    maxCostUsd: 0.02,
  })
  const parsed = Invoice.safeParse(JSON.parse(raw))
  if (!parsed.success) {
    // Do not guess: send the document for manual review
    return { status: 'needs_review', errors: parsed.error.issues }
  }
  return { status: 'ok', invoice: parsed.data }
}

Here llm is your gateway rather than a specific provider's SDK, and the prompt is stored with a version. That lets you swap the model or improve instructions without rewriting the function.

How do you choose a model and provider?

Choose a model by its results on your data, not by leaderboards: collect 50–100 real examples with correct answers and run them through several models. Often a fast, cheap model is enough for the specific task, and the powerful one can be reserved for hard cases.

Other factors to weigh:

• where data is processed — for personal and commercial data, the region and the provider's terms matter;
• latency — a chat interface needs streaming, background processing does not;
• cost at volume — multiply the average request size by the expected number of calls;
• open-weight models — self-hosting gives control over data but requires GPU infrastructure and a team.

The gateway means the choice is not permanent: route simple requests to a cheap model, hard ones to a strong one, and fail over when a provider is down.

What pitfalls appear after launch?

After launch, problems show up where demos do not reach: rare inputs, volume and cost. A model that worked perfectly on ten examples starts making mistakes on a thousand real ones.

What we build in from the start:

• Evaluation on an example set for every prompt or model change — like regression tests.
• Spend limits per request, per user and per day, with alerts when exceeded.
• Caching of identical requests and reuse of context where the provider supports it.
• Prompt injection defences: user data and documents must not be able to change the model's instructions or grant it dangerous actions.
• A fallback path: what the user sees if the provider is down or the output fails validation.

Personal data deserves special care. Before sending it to an external provider, confirm that your policy and the law allow it, and mask anything the answer does not need.

How do you fit an AI feature into an existing architecture?

The simplest way is to add the AI feature as a separate module with a clear interface inside the existing application and run heavy calls as background jobs. A dedicated service from day one is usually unnecessary — the same reasoning as in our piece on monoliths and microservices.

Practical rules:

1. Long calls go through a job queue; the user sees a status or gets a streamed response.
2. Prompts and output schemas live in the repository with versions and review, like any code.
3. Every call is logged with the prompt version, model, cost and response time.
4. Users can rate answers — the best source of examples for improvement.

In short

• Start with a specific task, a metric and an understanding of the cost of error, not with a model.
• The model is replaceable; the value lies in the context, validation and logging around it.
• Ask for structured output and validate it as strictly as user input.
• Choose the model by testing on your own data and set spend limits from day one.
• Add AI as a module of the existing application and push heavy calls into the background.