6 min read

Green Tests, Broken LLM: Five Things the First Real Model Call Exposed

Mocks repeat your assumptions back to you. Five integration failures that only appeared on the first real LLM call, and the smoke test that catches them.
a group of mannequins sitting next to each other
Photo by David Underland on Unsplash

I have shipped more than one LLM feature with a fully green test suite that had never once talked to a model. The client was mocked, the mock returned a tidy JSON object, the parser parsed it, the database stored it, and every assertion passed. Then credentials arrived, I made the first real call, and things broke in ways no test of mine could have caught.

This isn't a post about evaluating model quality. Whether the answer is good is a separate problem. This is about the plumbing: the request the provider actually accepts, the shape of what comes back, and the types that come out the other end. I made five of these mistakes across a few projects. I now treat the first real call as a separate test with its own checklist.

Why mocks can't catch this

A mock returns what you think the provider returns. If you read the docs for one API and then deploy against another, or you read the SDK types instead of the endpoint's behaviour, the mock just repeats your misunderstanding back to you. The test passes because the test and the code share the same wrong belief.

Every failure below had the same structure: my assumption was baked into both the code and the fixture.

1. The provider rejected the structured-output parameter

I wanted JSON that matched a schema, so I used the native structured-output option that the first-party Anthropic API supports and that the SDK's types describe. On AWS Bedrock, the same request came back as a 400 saying that field was not permitted. Same model family, different endpoint, different set of accepted parameters.

The fix was forced tool use. You define a single tool whose input schema is the output you want, and set tool_choice to that tool. The model's answer arrives as the tool's input, already parsed.

The trap is in what happens when it goes wrong. If the model writes its answer as prose instead of calling the tool, the turn still succeeds, so a naive client finds no tool block and returns an empty result. Downstream, "nothing extracted" looks exactly like "there was nothing to extract". So the client now fails loudly and quotes what the model actually wrote:

const block = response.content.find(
  (b) => b.type === "tool_use" && b.name === OUTPUT_TOOL
);
if (!block) {
  const text = response.content
    .filter((b) => b.type === "text")
    .map((b) => b.text).join("").trim();
  throw new Error(
    `Model did not call ${OUTPUT_TOOL} (stop: ${response.stop_reason})` +
    (text ? `; it wrote: ${text.slice(0, 200)}` : "")
  );
}
return block.input;

One more lesson here. My original design avoided forced tools because I'd read that forcing a tool required turning extended thinking off. I had never tested that. When I finally probed the endpoint, thinking and a forced tool worked together fine. Check what the endpoint does, not what you remember reading.

2. The database driver returned numbers as strings

The model returned a confidence score, I stored it in a Postgres NUMERIC column, and when I read it back to accept the proposal, the validator rejected it as not a number. node-postgres returns NUMERIC as a string by default because it can't always fit in a JavaScript number without losing precision. My fixture had a float in it, since that's what I expected.

Making it worse, the obvious fix (registering a type parser on the driver) did nothing, because the ORM on top installed its own parsers over the driver's. I only found that by trying both fixes against a real database. Nothing in the error pointed at the driver. It just said a number wasn't a number.

3. A field name that lied made the model contradict itself

I passed the model a list called something like pending. It actually held pending items and in-progress items. Separately, I passed a total for the pending items alone. The model wrote one paragraph saying "two remain" and another saying "three remain". Both numbers were correct for what they measured, but they shared a name, so the text contradicted itself.

My first instinct was a prompt rule: "take quantities only from the totals". It didn't help, and it couldn't have, because the model wasn't counting. It was naming the list. Renaming the field to say what it actually contained fixed it at once. The model trusts your field names more than your instructions, so a misleading name will produce misleading text however you word the prompt.

4. Prompt caching silently never engaged

I'd put a cache breakpoint on the stable part of the prompt and assumed it was working. It wasn't. Providers have a minimum cacheable prefix length that depends on the model and the platform, and my prompt was under it. Below the minimum the breakpoint does nothing at all. There's no warning.

What made it easy to miss was that the usage block reported a cache-read count of 1 on every call. At a glance, a non-zero number in the cache field looks like a hit. It's a sentinel. A real hit reads hundreds or thousands of tokens. Another prompt in the same service was over the limit and caching properly, which made it even easier to assume they all were.

I didn't fix it by padding. The prompt was missing things it should have said anyway, like what each output format actually meant. Once those were in, it cleared the minimum.

5. An LLM CLI answered "Done." instead of JSON

In a side project I wrapped a coding-agent CLI in a shell script so an automation tool could call it and get JSON back. I passed a schema flag that, as it turned out, the CLI didn't have. It didn't error. The model took the schema as a task, ran an agentic loop, and replied Done.

Getting real JSON out took four separate fixes, and each one only showed up after the previous one was in:

  • Put the schema in the system prompt, with an explicit "respond with only one JSON object, no fences, no commentary".
  • Strip markdown fences anyway, and clip to the outermost braces, because sometimes you get ```json regardless.
  • Unwrap the envelope. The CLI's JSON output format wraps the reply in an object with the text in a result field, and the automation layer wrapped that again. The JSON you want is a string inside two layers of JSON.
  • Check the model ID. In a different project the "parsing bug" was a model ID that didn't exist. Read the error body before you debug the parser.

The first-real-call smoke test

None of these needed a clever test. They needed one test that runs against the real provider, with real credentials, once per environment and after any change to the client, and that asserts on the plumbing rather than the prose. Mine checks:

  1. Request accepted. The exact request shape you send in production gets a 2xx from the exact endpoint you deploy to, not the one in the docs.
  2. Shape. The structured result is where you expect it (a tool block, a result field) and the test fails loudly, quoting the text, if it isn't.
  3. Types after a round trip. Write the model's output to the real database, read it back, and assert typeof on every numeric and date field.
  4. Cache counters. Make the same call twice. Assert the second call's cache-read count is close to the prompt's stable prefix size, not just greater than zero.
  5. Consistency. For anything that produces prose from data, check that every number in the text traces back to a field and that no two fields with similar names disagree.
  6. Identity. Log the model ID the response says it used, and compare it to the one you configured.

It costs a few cents and a few seconds. Keep it out of the unit suite: it needs secrets, it's slow, and the model isn't deterministic. Run it as a separate gate before you trust the green tick.

Takeaway

Your mocks are only as accurate as your assumptions about the provider, and the first real call is where you find out which assumptions were wrong. Make that call on purpose and early, and write down what it should prove: the request is accepted, the shape is right, the types survive storage, and the cache counters mean what you think they mean.