Decide when your AI agent should stop

An agent can have a clear goal, the right tools, and still behave badly. It keeps searching after it has enough evidence, retries a failed action with a different guess, or quietly stops halfway through a task. The missing part of the spec is often not another capability. It is a rule for when the system is done.

A stopping rule sounds like an implementation detail. It is a product decision: it defines how much work the system may do, what it is allowed to infer, and when a person takes over.

A goal is not a completion contract#

“Find the renewal date and prepare a reminder” looks specific until an agent has to execute it. Which folder should it search? What if two documents contain different dates? Is drafting the reminder enough, or should it send one? If it cannot find the date, should it search every connected drive or ask the user?

A goal describes a destination. A completion contract describes the evidence that the destination was reached and the boundary beyond which the agent must not continue.

For that example, a first contract could say:

• Search only the user’s selected contract folder.
• Return the date with a link to the source passage.
• Draft a reminder, but do not send it.
• If the sources disagree or no date is found, stop and show what was checked.

That is less magical than “handle renewals for me.” It is also testable. The user can tell what happened, and the team can tell whether the agent followed the intended path.

Define “done” in observable terms#

An agent should not decide it is done because its last message sounds complete. Done should correspond to a state a person or system can verify.

For a read-only research task, the finish line might require a result and a source. For a record update, it might require a successful response from the system and a read-back showing the new value. For a draft, it might mean the draft exists in the expected place and remains unsent.

Write the completion condition as a small checklist:

The distinction between “attempted” and “completed” matters. A tool returning without an error may only mean the request was accepted. If the outcome matters, read it back. If it cannot be verified, say so instead of converting an attempt into a success claim.

Put a budget on actions#

Token limits and timeouts protect infrastructure. They do not tell the product how much autonomy is appropriate. A user-facing agent also needs an action budget: a limit on searches, retries, mutations, or elapsed time for one request.

For a first version of the read-only renewal lookup, I might start with five document searches, one follow-up search if a source is incomplete, and a two-minute deadline. Those are hypotheses to test against real tasks, not universal numbers. The important part is that the agent has a defined point at which it must stop and report what it found.

Separate the budget by action type. Reading one more document is not equivalent to sending a message or changing a record. A useful policy might allow several read-only lookups, permit one retry after a transient error, and require explicit approval before any irreversible action. Never let a retry silently broaden permissions or turn a draft into a sent message.

Then measure where the budget is spent. If successful tasks regularly need one more search, adjust the search cap. If retries repeat the same failure, change the recovery path rather than raising the cap. More autonomy should follow evidence that the extra action improves the result without creating a worse failure mode.

Make stopping conditions explicit#

“Stop when confused” is too vague to implement or evaluate. Name the conditions that end the run:

• Missing information: a required field is absent from the permitted sources.
• Conflicting evidence: two sources disagree and no precedence rule resolves them.
• Permission boundary: the next step needs access or approval the agent does not have.
• Repeated failure: the same tool action fails again after its allowed retry.
• Budget exhausted: the remaining task cannot be completed within the action or time limit.

Each condition needs a user-facing result. Missing information should name the missing field. Conflicting evidence should show both sources. A permission boundary should ask for the specific approval needed. Budget exhaustion should summarize completed and unfinished steps.

This turns stopping into a designed path instead of an error state. It also gives the team a useful set of evaluation cases: one for each reason the agent should stop, plus one where it should continue.

Make partial progress legible#

A task is not always all-or-nothing. If an agent finds the date and prepares the reminder but cannot save the draft, it should not report “done” or return a generic failure. It should say which parts are complete, which are not, and why it stopped.

I would make the final status one of three things:

1. Complete: all required outcomes were verified.
2. Partial: some outcomes were verified; unfinished steps are named.
3. Blocked: no safe next action is available without new information or approval.

Keep that status separate from the natural-language summary. The interface can show a concise sentence, but the underlying result should preserve the status, completed actions, evidence, and stop reason. That makes the experience clearer for users and the behavior easier to audit later.

The stopping-rule review#

Before shipping an agent, take one normal request and walk it through these questions:

1. What exact evidence proves the task is complete?
2. How many searches, retries, and state-changing actions are allowed?
3. Which conditions require stopping or asking the user?
4. Can the agent distinguish a successful attempt from a verified outcome?
5. If it stops halfway through, can the user see what remains?

If those answers are missing, the agent does not yet have a product contract. Add the stopping rule before adding another tool.