RAG agents in production

AI agents with retrieval-augmented generation running for real customers: a retrieval chatbot for a photography studio, and Juanito, the conversational assistant inside El Empleado Juan — it classifies conversations, queries attendance and payroll data, and acts against the platform.

What the tutorials leave out

A RAG demo takes an afternoon. The hard part starts afterwards, when someone on the other side is waiting for a correct figure:

• Retrieval fails silently. When the search returns nothing useful the model does not go quiet: it answers anyway, in a perfectly confident tone and without the information. It is the worst possible failure mode, because it does not look like one.
• Chunking is a product decision, not an infrastructure one. Fragment size determines which questions can be answered at all.
• An agent that queries real data needs a contract, not a prompt. Typed tools, validated results, and an explicit path to “I don’t know” that is cheaper than making something up.
• Without evaluation there is no improvement, only a feeling. A set of questions with known answers is worth more than any temperature tweak.

The rule I work by: the agent may get the tone wrong, never the fact.

The contract for “I don’t know”

The expensive failure is not being wrong: it is answering confidently when retrieval brought back nothing. Here that cannot happen silently, because “I don’t know” is a return value rather than a virtue asked of the model.

// Two ways to finish, and both are valid. If the only successful shape
// is "fact", the model learns that staying quiet counts as failure — and
// it stops staying quiet.
const Answer = z.discriminatedUnion('kind', [
  z.object({ kind: z.literal('fact'), value: z.string(), source: z.string().min(1) }),
  z.object({ kind: z.literal('ungrounded'), missing: z.string() }),
]);

async function answer(question: string): Promise<Answer> {
  const chunks = await retrieve(question, { k: 8, minScore: 0.35 });

  // With no sufficient context the model is not called at all. Calling it
  // here is explicitly asking it to make something up, and it will — in a
  // perfectly confident tone.
  if (chunks.length === 0) {
    return { kind: 'ungrounded', missing: question };
  }

  const out = await model.complete({ question, chunks, schema: Answer });

  // The citation is checked against what was retrieved, not against what
  // the model claims it cited. That is the difference between having
  // sources and appearing to have them.
  return chunks.some((c) => c.id === out.source)
    ? out
    : { kind: 'ungrounded', missing: 'the citation does not match what was retrieved' };
}

minScore is the parameter that saves the most argument: below it no answer counts, and that gets decided once against a set of questions with known answers instead of being relitigated case by case.