Blogs / LLMs & agents

What Is Loop Engineering? Inner vs Outer Loops, Stop Rules and Budgets

By Kadam Parikh · · 14 min read · Worked example: an invented upgrade bot

Run an AI agent once and it either finishes or it does not. Run it again and again and you have a new problem: when does it stop, who says the work is good, what does it cost, and what does the next run remember? That is loop engineering. This post explains it with one small example.

What is loop engineering?

Loop engineering is the practice of designing the repeating cycle around an AI agent: what starts each run, how its result is checked, what ends the cycle and what each run remembers, so that a system, not a person, prompts the agent next. Some writers call the result an agentic loop.

Suppose you have a job an agent can do, but not in one go. You run it, read what it did, run it again with a nudge, and repeat. In that setup you are the loop, and you are also the bottleneck: nothing happens while you are away. Loop engineering is building that repetition into a system, so code decides when the agent runs again instead of you.

Addy Osmani, an engineer on Anthropic’s Claude Code team, put it this way in June 2026: “Loop engineering is replacing yourself as the person who prompts the agent. You design the system that does it instead.” A research paper from August 2026, by Jai Lal Lulla and colleagues at Singapore Management University and Heidelberg University, gave the formal version: automated control structures that repeatedly invoke coding agents, each loop bounded by a machine-checkable stop condition, with state kept between runs, independent checking, capped cost and defined points where a human steps in. Most of the writing so far is about coding agents, but the ideas apply to any agent you run more than once, which is why the example below is deliberately plain.

One example for the whole post: an upgrade bot

Imagine a team upgrades a payments library from version 4 to version 5. The old calls no longer match, and 140 of the project’s 600 tests now fail. Fixing them is dull work: change a call, run the tests, change the next one. It is a good job for an agent. It is also too long for one run. A model can only keep a limited amount of text in view at once (its context window), and a long job fills it before the work is done.

So you want a bot that runs the agent, looks at the result and, if tests still fail, runs it again. The rest of the post is about how to build that repetition so it ends well.

The simplest possible version

The best-known loop of this kind is the “Ralph” loop, introduced by the developer Geoffrey Huntley in July 2025. In his words, in its purest form it is a Bash loop:

while :; do cat PROMPT.md | claude-code ; done

The same prompt goes to the agent, over and over. Each pass starts with a fresh context window, and the agent works out what is left to do from files in the project. Huntley asks for one task per loop. His post shows no exit condition. He writes that the loop ends when the agent runs out of things to do “or, it goes completely off track”, and that when it goes off track he sometimes throws the plan away or resets the code with git reset --hard.

Huntley also says this works best for starting a new project, and that he would not use it on an existing codebase. The upgrade bot works on an existing codebase, so it needs more than the one-liner. Four things could go wrong with the bare loop:

Inner loop vs outer loop: two loops, not one

It helps to separate the two cycles going on.

The inner loop sits inside the outer loopA large ring labelled outer loop contains a smaller ring labelled inner loop with the agent on it. On the outer ring sit a notes file and a check. The check decides whether the outer ring runs again.AgentInner loop: edit, test, editCheckcode counts the failuresNotes filewhat earlier passes learnedOuter loop: run again until the check says stopStop rule, budget and a human hand-over live on the outer loop
Two loops. The inner loop is what an agent tool already does for you. The outer loop, with its check, notes and limits, is what you build.
  1. 1 · Start

    • Triggerthe library is upgraded and 140 tests fail
    • Notes filewhat earlier passes tried and learned
  2. 2 · One pass (inner loop)

    • Agentedits code, runs tests, edits again, until it stops
  3. 3 · Check, outside the model

    • Test runnercounts failing tests itself
    • Cheat checkwere any tests deleted or skipped?
  4. 4 · Decide (outer loop)

    • 0 failingopen a pull request for a human
    • Fewer failinganother pass, with updated notes
    • No progress or over budgetstop and hand over to a human
How the bot fits together. Shaded box: the model. Dashed boxes: ordinary code the model cannot change. The inner loop lives inside box 2; the outer loop is boxes 1 to 4 repeating.

Notice where the judgment sits. The agent does the creative work of editing code. Whether the work is finished is decided by ordinary code that counts failing tests. That split is the most important design choice in the post.

Two runs, pass by pass

Here is the finished bot (the code is at the end of the next section) on two imagined nights. Read the “what the bot saw” column: each rule in the next section exists because of a row here.

RunPassFailing / total testsWhat the bot sawWhat it did
Astart140 / 600The upgrade broke 140 testsStarted pass 1
185 / 600Fewer failing, same number of testsWrote notes, ran pass 2
231 / 600Fewer failing againWrote notes, ran pass 3
30 / 571Zero failing, but 29 tests are goneUndid pass 3, stopped, asked a human
Bstart140 / 600Same starting point, a different nightStarted pass 1
185 / 600Fewer failingWrote notes, ran pass 2
285 / 600No change from pass 1Stopped and asked a human, with the notes
  • Run A: start140
  • Run A: after pass 185
  • Run A: after pass 231
  • Run A: after pass 30 (29 tests gone)
  • Run B: start140
  • Run B: after pass 185
  • Run B: after pass 285 (no progress)
The same two runs as bars. Run A (top) looks like a win until you see the test count. Run B (bottom) stalls. Invented numbers.

Run A is the dangerous one. A loop that only asked “are there zero failing tests?” would have called pass 3 a success and opened a pull request. Run B is the expensive one: without a rule for “no progress”, passes 3, 4 and 5 would probably have been more of pass 2.

The four decisions of loop engineering

Every outer loop has to settle the same four questions. Recent guides name them in different words, and the research paper lists more building blocks, such as triggers and parallel work. These four are the ones the loop cannot work without.

1 · The stop rule: when does it end?

Run B shows why “until the goal is met” is not enough. The goal was never going to be met, and nothing in the loop knew. A workable stop rule has three parts:

The first two are common practice. Osmani quotes the Claude Code team describing a version where, each time the agent tries to stop, a separate evaluator model checks your condition, and the run continues until the goal is met or a number of turns you set is reached. His example of a turn limit is “stop after 10 turns”. The no-progress rule is my addition, and it is the cheapest of the three to build.

Together they mean the bot always ends in one of a few known states: done, cheated, no progress, over budget or out of passes. Each tells you what to do next, which a loop that simply runs until you kill it cannot.

2 · A check the model cannot influence

Look at pass 3 of run A again. The agent got to zero failing tests, and it did so by removing 29 of them. If the loop had asked the agent “are you done?”, it would have said yes, and meant it.

Two separate problems are hiding here.

So the bot does not ask. It runs the test suite itself, counts, and compares with what it saw at the start. Counting is mechanical. Not every goal can be checked this way: “does the design look good?” needs a judge, and Rajasekaran warns that an evaluator model is still a model, inclined to be generous towards the work it reviews. The more mechanical you can make the check, the less it matters. The same idea applies inside a single run; the tool design post covers when to put a rule in code instead of a prompt.

3 · A budget

Every pass costs time and tokens (the units models are billed in). You want a ceiling that does not depend on the agent’s good sense, and you can work one out in three steps:

  1. Run one pass by hand and note what it costs. Say it comes to about $5 (an invented figure).
  2. Multiply by the pass cap. Six passes at $5 is $30 in the normal worst case.
  3. Add margin for passes that run long, and set a spend limit above that. The bot uses $40.

The pass cap and the spend limit protect against different things. The cap catches a loop that crawls. The spend limit catches a single pass that goes wildly over, say because the agent got stuck re-reading a huge file. You want both.

Real loops can cost more than expected. The research paper reports a “CI sweeper” loop that used roughly eight million tokens in 48 hours. And in Rajasekaran’s comparison of ways to build the same small game, a single agent run took about 20 minutes and cost $9, while a setup with a planner, a builder and a separate evaluator took about 6 hours and cost $200. He judged the second clearly better. Neither is an upgrade bot, but the shape of the trade is the same: more passes and more checking buy quality at a real price, so decide the ceiling before you start.

4 · State: what does the next pass remember?

A new pass starts with an empty context window, so the agent has no memory of pass 1. Whatever it should know has to be written down where it can read it. Osmani calls the state file “the spine of the whole thing”. In Young’s long-running agents, each session leaves a progress file and commits its work to version control, so a fresh session can read the history. Huntley’s loop uses a plan file and a specs folder that every pass re-reads.

For the bot, the state is a short notes file the agent updates at the end of each pass. After pass 1 of run A it might read:

# Upgrade notes (the agent updates this after each pass)

Done: charge() and refund() moved to the v5 client.
      85 tests still failing.
Tried and reverted: wrapping the old client in an adapter.
      It broke 12 tests.
Left: webhook handlers (about 50 tests),
      3 date-format tests in billing/.
Rule: do not edit tests/ except to update expected values
      that the upgrade changed.

Without this, pass 2 might try the adapter again, find out the hard way that it breaks 12 tests, and lose a pass doing it. This is also why the bot’s notes were useful to the human in run B: they showed exactly where it was stuck.

Notes have a limit. Suppose a pass fills its context window before it can write them. Young describes an agent that ran out of context in the middle of a feature and left the next session with work that was half done and undocumented. He adds that summarising the conversation in place (compaction) is not enough on its own. Rajasekaran’s alternative is to clear the context completely and start a fresh agent from a structured hand-off, which he says solves the problem but adds complexity, cost and delay to each run. A simple defence for the bot is to ask for one small piece of work per pass and update the notes after each piece.

Putting it together

Here is the whole bot as a sketch. It is about 25 lines, and almost all of them are decisions rather than AI. Match each comment to a row in the two-run table above.

# Illustrative sketch of the upgrade bot's outer loop. Not runnable.
MAX_PASSES = 6          # stop rule, part 2: a hard cap on passes
MAX_SPEND_USD = 40      # budget: a hard cap on spend

tests_at_start = count_tests()
failing_before = run_tests().failing       # 140

for n in range(1, MAX_PASSES + 1):
    notes = read("UPGRADE_NOTES.md")       # state: what earlier passes learned
    agent.run(PROMPT, context=notes)       # inner loop: edit, test, edit...

    result = run_tests()                   # feedback the agent cannot influence
    if result.total < tests_at_start or result.skipped_new:
        revert_changes()
        stop("Tests were deleted or skipped")

    if result.failing == 0:                # stop rule, part 1: the goal is met
        open_pull_request()
        break
    if result.failing >= failing_before:   # stop rule, part 3: no progress
        stop("A pass made no progress")
        break
    if spent_usd() > MAX_SPEND_USD:
        stop("Over budget")
        break
    failing_before = result.failing
else:
    stop("Out of passes")

Loop engineering vs harness engineering

This is the comparison most readers search for, and the honest answer is that the boundary is a convention, not a standard. A harness is everything around one agent: its loop, tools, context, rules and checks. The outer loop is what runs that agent again.

Harness engineeringLoop engineering
Unit of designOne agent runMany runs over time
Main questionWhat can the agent touch, and what happens when it errs?When does it run, how is each run judged, and when does it stop?
Upgrade bot exampleThe tools for editing files and running tests; what the agent seesThe pass cap, the test count, the spend limit, the notes file
Fails whenThe agent can break things or cannot see what it needsThe loop never ends, ends early, costs too much or forgets

Osmani says loop engineering “sits one floor above the harness”, and I use that framing here because it makes the question easy to ask: is the problem inside one run, or between runs? If a single run goes wrong, look at the harness. If the repetition goes wrong, look at the loop. The term is only a few months old, so other writers may draw the boxes differently. Check which meaning an article uses before comparing it with this one.

They also depend on each other. A loop that repeats a weak agent just repeats its mistakes faster, so the harness comes first.

Do you need an outer loop at all?

Often not. If one run, with good checks inside it, finishes the job, adding a loop adds cost and one more thing that can fail. The upgrade bot needs one only because 140 failing tests do not fit in one context window. Reach for an outer loop when:

It also helps to know how new and unproven this is. The research paper calls itself exploratory. Of the sources it collected it writes that they “provide no systematic evidence” to evaluate the claimed benefits, and it describes the biggest claims as self-reports and anecdotes. Its scan of public repositories found 217 with autonomous agent loops among roughly 36,700 it examined, which says loops exist, not that they pay off. Treat the advice here as sensible practice from early writing, not as settled results.

Frequently asked questions

What is loop engineering in simple terms?

It is designing the repeating cycle around an AI agent: what starts each run, how the result is checked, what ends the cycle and what each run remembers. The aim is that the system, not a person, prompts the agent next.

What is the difference between the inner loop and the outer loop?

The inner loop is one agent run: act, see the result, act again. The outer loop repeats whole runs until a check outside the model says the work is done or a limit is hit.

Is loop engineering the same as harness engineering?

No. Harness engineering designs the environment around one agent. Loop engineering designs how that agent is run repeatedly. The boundary is a convention, so check which meaning an article uses. See the comparison above.

What is the Ralph loop?

A technique by Geoffrey Huntley, from July 2025: run a coding agent in a Bash loop that feeds it the same prompt on every pass, with plan and spec files carrying what it needs to know. In its original form it has no built-in stop, which is why people add stop rules and caps around it.

How do I stop an agent loop from running forever?

Give it a goal that code can check, a hard cap on passes or spend, and a rule that stops it when a pass makes no measurable progress. The agent saying it is finished should not be what ends the loop; the upgrade-bot trace above shows why.

How do I know when the agent is really done?

Check with something the agent cannot influence, such as a test runner or a separate evaluator, and close the obvious loopholes, like deleting the tests that fail. Rajasekaran at Anthropic found that agents tend to rate their own work too kindly.

Does a bigger model remove the need for any of this?

Not for the stop rule, the budget or the state. A stronger model may need fewer passes, but it can still run past its goal or lose track across a long task. See the harness post for how to measure what a model gets wrong before deciding what to build.

Terms used in this post

Agent
A language model that decides its own next step, such as which tool to call, in a loop until it judges the task done.
Inner loop
The cycle inside one agent run: the model picks an action, the program runs it, the model sees the result, and this repeats until the model stops.
Outer loop
A cycle around whole runs: start the agent, check what it did, then decide whether to run it again, stop, or call a human.
Stop rule
The condition that ends a loop, ideally a check that code can make, such as “zero failing tests”, plus a hard cap in case it is never met.
Maker and checker
Splitting the agent that does the work from whatever judges it, so the work is not graded by the one who did it.
Harness
Everything around the model that makes it usable: the loop, tools, context, rules, checks and logs.
Compaction
Summarising a long conversation so a new context window can start with the important parts.
Context window
The text a model can see at one time. When it fills up, older material is dropped or summarised.
Token
The unit models read and are billed in, roughly a short word or part of one.

Sources

Claims about other people’s work come from the pages below, read in November 2026. The upgrade bot is my own invention for illustration, and its code is a sketch, not a tested program.