Treat the model like an unreliable dependency
Language models and vision services fail in ordinary ways: timeouts, rate limits, malformed output, and partial responses. They also fail in less ordinary ways: confident fabrications and outputs that are valid JSON but semantically wrong.
An API boundary should assume both. Validate the shape of the response, check the parts that must be true, and define what the application does when the check fails.
Design the contract before the prompt
The API contract should describe inputs, outputs, limits, and error states independently of any particular model. That separation lets the model change without forcing every client to change.
Version the contract. When a prompt or model change alters behavior, clients need a predictable way to opt in rather than discovering the change in production.
Make failure explicit
A silent fallback is often worse than a visible error. If the system cannot answer, it should say so in a way the interface can render. If it returns a degraded result, that should be labelled.
Timeouts and retries need budgets. Retrying an expensive generation twice may be reasonable; retrying a tool call that already produced a side effect is not. Idempotency keys make that distinction possible.
Log decisions, not just requests
A request log tells you that something happened. A decision log tells you why. Capture the model version, retrieved context identifiers, tool calls, validation outcomes, and the final response status.
That record supports debugging, evaluation, and user trust. It also makes it possible to answer the question that eventually arrives: why did the system do that?
TechieMike
AI Systems Engineer