From Prompt Engineering to Control Flow: Designing Deterministic State Machines for LLM Apps

The usual diagnosis is that people treat the LLM like a chat buddy, and the usual cure is a state machine. I agree with the cure and I think the diagnosis is slightly off. Being friendly to the model is not the problem. The problem is that in most LLM apps the transcript is the program. Where you are in the flow, what has been confirmed, what is still missing: all of it lives in a list of messages, and the model re-reads that list on every turn to work out what to do next.

That is what a state machine replaces. Not the model, and not the prompts. It replaces the habit of storing control state in prose and asking a probabilistic reader to recover it.

§The transcript is not a database

Picture a refund assistant built the chat way. The system prompt says: ask for the order number, check eligibility, confirm the amount, then issue the refund. It works in the demo. Then a real customer gives the order number in their first message, before being asked. Another changes their mind halfway through confirming. A third pastes two order numbers. A fourth says “yes” to a question the model asked three turns ago, not the one it just asked. Every one of these is a question about state, and in the chat design every one of them is answered by the model reinterpreting the history, slightly differently each time.

You fix it the way people always fix it: more prompt. A paragraph about order numbers given early. A rule about changes of mind. A capitalised NEVER about issuing a refund without confirmation. The prompt grows into a specification written in a language with no compiler, enforced by something that follows it most of the time. You cannot test it except by running conversations. You cannot say, with certainty, that there is no path to a refund without a yes.

12-Factor Agents, from HumanLayer, makes the observation that a lot of products sold as AI agents are mostly deterministic code, with LLM steps sprinkled in at just the right points, and one of its factors is simply: own your control flow.^[1]1. Dex at HumanLayer wrote 12-Factor Agents. Factor 8 is “Own your control flow”. I would go one step further. If the control flow matters, it should exist somewhere other than the model’s head, in a form you can print out.

§Temperature zero will not save you

The tempting shortcut is to keep the chat design and make the model deterministic. Set temperature to zero, pin the model version, and the same history should give the same answer. It does not, and the reason is interesting.

Thinking Machines ran 1,000 completions of the same prompt at temperature zero on Qwen3-235B and got 80 distinct outputs. The first divergence appeared at token 103.^[2]2. Horace He and Thinking Machines Lab, September 2025. Defeating Nondeterminism in LLM Inference. Their explanation is not the usual hand-wave about floating point and concurrency. It is that common inference kernels are not batch-invariant: the numbers your request gets depend on how many other requests the server happened to batch it with, and that depends on load you cannot see or control. They built batch-invariant kernels and got 1,000 identical completions, at a real cost in speed.

Two lessons there. One: unless you run your own inference stack with that kind of care, the same input to a hosted model is not guaranteed to produce the same output. Two, and more important: even if it did, deterministic is not the same as correct. A model that misreads a stale “yes” the same way every time is reproducibly wrong. The determinism you want is not in the model’s tokens. It is in the consequences, and those you can own completely.

§Models translate, code decides

The design I keep coming back to splits the app into two kinds of step. Probabilistic steps take messy input and translate it into a small, closed vocabulary of events: this message means want_refund, or give_order_id with an ID attached, or other. Deterministic steps take the current state and that event and decide what happens. The model is the parser. The state machine is the program.

Here is the translator. It is a classifier, so it uses the small model, which is also the tier Anthropic recommends for classification and routing.^[3]3. Anthropic’s models overview describes Haiku as the tier for classification, extraction and routing. Models overview. The output is constrained to a Pydantic model so the result is a value, not a paragraph.

from enum import StrEnum
from typing import Literal
from pydantic import BaseModel
import anthropic
 
class S(StrEnum):
    START = "start"
    NEED_ORDER = "need_order"
    CHECKING = "checking"
    CONFIRMING = "confirming"
    REFUNDED = "refunded"
    CLOSED = "closed"
    HUMAN = "human"
 
Kind = Literal["want_refund", "give_order_id", "yes", "no", "other"]
 
class Intent(BaseModel):
    kind: Kind
    order_id: str | None
 
client = anthropic.Anthropic()
 
def interpret(state: S, message: str) -> Intent:
    resp = client.messages.parse(
        model="claude-haiku-5-5",
        max_tokens=1024,
        system=(
            "Classify the customer's latest message as exactly one intent. "
            f"The conversation is in state '{state}'. "
            "Set order_id only if the customer wrote one in this message."
        ),
        messages=[{"role": "user", "content": message}],
        output_format=Intent,
    )
    if resp.stop_reason != "end_turn" or resp.parsed_output is None:
        return Intent(kind="other", order_id=None)
    return resp.parsed_output

The stop_reason guard is not paranoia. Anthropic’s docs say plainly that a refusal or a response cut off at max_tokens can come back not matching the schema.^[4]4. A refusal or a max_tokens cut-off can return output that does not match the schema. Anthropic, Structured outputs. When the translator fails, it fails to other, which the state machine already knows how to handle. That is the general trick: every probabilistic step gets a boring, safe default, and the default is a member of the same event vocabulary as the success cases.

Now the program. The transition table is a dictionary, and handle is the only function that changes state:

TRANSITIONS: dict[tuple[S, str], S] = {
    (S.START, "want_refund"): S.NEED_ORDER,
    (S.START, "give_order_id"): S.CHECKING,
    (S.NEED_ORDER, "give_order_id"): S.CHECKING,
    (S.CONFIRMING, "yes"): S.REFUNDED,
    (S.CONFIRMING, "no"): S.CLOSED,
}
TERMINAL = {S.REFUNDED, S.CLOSED, S.HUMAN}
MAX_STRIKES = 3
 
def step(state: S, kind: str) -> S | None:
    return TRANSITIONS.get((state, kind))  # None: not a valid event here
 
def handle(convo, message: str) -> str:
    if convo.state in TERMINAL:
        return render_reply(convo)
 
    intent = interpret(convo.state, message)
    kind = intent.kind
    if kind == "give_order_id" and not intent.order_id:
        kind = "other"
 
    nxt = step(convo.state, kind)
    if nxt is None:
        convo.strikes += 1
        if convo.strikes >= MAX_STRIKES:
            convo.state = S.HUMAN
        return render_reply(convo)
 
    convo.strikes = 0
    if intent.order_id:
        convo.order_id = intent.order_id
    convo.state = nxt
 
    if convo.state is S.CHECKING:  # no model in this state
        ok = eligible(convo.order_id, convo.customer_id)
        convo.state = S.CONFIRMING if ok else S.HUMAN
    if convo.state is S.REFUNDED:
        issue_refund(convo.order_id, idempotency_key=f"refund:{convo.id}")
 
    return render_reply(convo)  # the model writes words, not decisions

Walk the awkward customers through it. The order number in the first message is (START, give_order_id), an explicit row, so it skips straight to checking. The change of mind is (CONFIRMING, no). The stale “yes” outside the confirming state has no row, so it is a strike, not a refund. Three strikes and a human takes over, which is the failure exit every loop in an LLM app needs. The case I have not handled, two order numbers in one message, is now a visible gap: the translator only has room for one ID, and you can decide deliberately whether to widen the schema or treat it as other.

Notice what the model is allowed to do. It classifies one message. Then, in render_reply, it writes the words the customer sees, given the state it is handed. It never decides whether a refund happens. CHECKING has no model in it at all, because eligibility is a database question. And the order ID the translator extracted still goes through eligible, which should check the order exists and belongs to this customer; the reasons that check is not optional are the subject of Tool Use vs. Hallucination.

§Test the table, evaluate the translator

The payoff is that the two halves can be checked in completely different ways, each suited to what it is. The state machine is finite, so you can test it exhaustively. Seven states, five event kinds, thirty-five pairs. You can assert properties over all of them, and the properties are the sentences you used to write in capitals in the prompt:

from itertools import product
from typing import get_args
 
def test_refund_only_follows_confirmation():
    for state, kind in product(S, get_args(Kind)):
        if step(state, kind) is S.REFUNDED:
            assert (state, kind) == (S.CONFIRMING, "yes")
 
def test_terminal_states_have_no_exits():
    for state, kind in product(TERMINAL, get_args(Kind)):
        assert step(state, kind) is None

That first test is the NEVER from the prompt, turned into something that runs in CI in milliseconds and fails loudly if a teammate adds a careless row. No conversation simulation, no model calls, no flakiness. You will never get that guarantee about a prompt.

The translator is the opposite kind of thing. It is a classifier, so you evaluate it like one: a few hundred real customer messages, each labelled with the state it arrived in and the intent it should map to, scored as a confusion matrix. Which mistakes it makes matters more than how many. Confusing yes with other costs the customer an extra turn. Confusing other with yes in the confirming state costs you money, and that asymmetry is exactly what tells you where to spend prompt effort, examples and model size.

None of this is new. Temporal, the workflow engine, requires workflow code to be deterministic so it can be replayed from its event history, and pushes everything unpredictable, like calls to external services, into Activities that run outside the replay path.^[5]5. Workflow code must make the same calls in the same order given the same input; external calls go in Activities, outside the replay path. Temporal, Workflow Definition. Swap “Activity” for “LLM node” and that is the architecture above. When a flat state machine starts sprouting dozens of near-duplicate states, the answer is also old: David Harel’s statecharts, published in 1987, extended plain state diagrams with hierarchy, concurrency and communication, and were demonstrated on, of all things, his digital wristwatch.^[6]6. David Harel, Science of Computer Programming, 1987. A good short introduction, including the wristwatch, is the Recurse Center’s paper of the week. Nested states are how you add “the customer can ask a shipping question at any point” without doubling the table.

§Where the state machine is the wrong tool

It would be dishonest to stop there, because “industrial-grade reliability” oversells what this buys. A state machine does not make the model reliable. It makes the model’s unreliability bounded and legible: a misclassification turns into a strike, a wrong transition is impossible rather than unlikely, and every failure lands in a state with a name. That is a large improvement, and the kind that lets you sleep, but the translator is still wrong some of the time, and your eval tells you how often.

It also only works when you can enumerate the states. A refund flow, an onboarding form, a support triage, a document pipeline: these have a shape you can draw before the first user arrives. Open-ended work does not. You cannot write the transition table for “find out why our churn went up” or “fix this failing test”, because the steps depend on what each step discovers. That is the line Anthropic draws between workflows, which follow predefined code paths, and agents, which direct their own process.^[7]7. Anthropic’s distinction between workflows, on predefined code paths, and agents, which direct their own process. Building effective agents. Forcing an agentic task into a rigid machine gives you a brittle flowchart that is wrong in new and interesting ways.

The answer, in practice, is to nest them. The outer program is a state machine with a small number of coarse states. One of those states can contain a real agent loop, with its own tools and a hard budget, and it exits through an edge the outer machine defines: done, gave up, needs a human. The agent gets freedom where freedom helps and walls where the consequences live. When several agents need to share those walls, the problems change shape again, into bounded cycles, checkpoints and recovery, which is the subject of Beyond the Wrapper.

What I like about this way of building is that it gives prompts back their proper job. A prompt is very good at telling a model what a message means. It is a terrible place to keep the rules of a business. Once the rules live in a table, the prompt can shrink to the size of the actual judgement it asks for, and the sentence you used to shout in capitals becomes a test that passes.