Blogs / LLMs & agents

AI Agent Tool Design: Tool or Prompt, and How Many Tools Is Too Many

By Kadam Parikh · · 17 min read · Worked example: an invented refund assistant

When your model is strong, it is fair to ask why you would build a tool at all. This post answers that, then covers how to design the tools you do build and how to tell when you have too many, using one small made-up assistant to show each step.

First: do you need a tool at all?

Before choosing between a tool and a prompt, ask whether you need either. Anthropic’s guide Building effective agents, written by Erik S. and Barry Zhang of Anthropic’s engineering team, recommends finding “the simplest solution possible” and adding complexity only when it demonstrably improves outcomes. For many applications, they write, a single model call with retrieval and examples is usually enough.

Tools earn their place when the model needs something a prompt cannot give it: data it cannot see, an exact calculation, an action with side effects, or state that outlasts one call. A fixed sequence of steps in your own code is also still an option. You only need the model to choose the steps when you cannot know them in advance.

A harness is the code around the model, and the tools are one layer of it. This post is about that layer.

One example for the whole post: a refund assistant

Imagine an online shop that sells mugs. A chat assistant handles refund requests. Its instructions contain five things:

The assistant could get all of this from one long prompt. The question for the rest of the post is which of those five things should stay in the prompt, which should become a tool, and which should become a check in code.

  1. 1 · Customer

    • Message“My mug arrived cracked. Can I get a refund?”
  2. 2 · Model decides

    • Agentreads the message, chooses a tool, writes the reply
  3. 3 · Tools

    • find_orderlooks up the order; read only
    • issue_refundmoves money; side effect
  4. 4 · Checks inside issue_refund

    • 30-day windowdate arithmetic, done by code
    • $200 limitabove it, a human approves
    • Rejected or donea specific reason, or a confirmation
How the assistant fits together. Shaded box: the model. Dashed boxes: ordinary code the model cannot change. The model decides what to do and what to say; code decides whether a refund is allowed.

Tool or prompt? Sorting the rules

A rule belongs in code when a failure would be costly, when the result must be the same every time, or when the model cannot see what it needs. It belongs in the prompt when it is a matter of judgment. A stronger model lowers the failure rate but does not set it to zero. A failure in 1 chat out of 20 is easy to miss in a demo and shows up every day in production.

Here is how they sort, with the refund action itself added as a sixth row:

Rule or needWhere it goesWhy
Order details: items, delivery date, priceA tool (find_order)The model cannot see the shop’s database; no amount of reasoning replaces access
“Delivered within 30 days?”A check in codeDate arithmetic has one right answer, and the refund depends on it every time
“Over $200 needs a human”A guardrail in codeA mistake costs real money, so the result should not depend on the model obeying
Issuing the refundA tool with a permission check (issue_refund)It has a side effect, so code should be able to refuse before any money moves, and the call is logged
Friendly, brief toneThe promptTaste and wording; there is no single right answer
Offer store credit or an exchangeThe promptA judgment about the customer’s situation, where several answers are acceptable

Check, don’t compute

Notice what issue_refund does not do. It does not decide whether the customer deserves a refund. The model still reads the conversation, makes that call and chooses the amount. The tool only accepts or rejects the request and says why. That keeps the model’s judgment and removes its arithmetic. If you are unsure whether to build a tool, a checker like this is usually a smaller and safer step than a tool that does the work for the model.

Do not build a tool for what the model already does well

The opposite mistake is also real. A research paper from UC San Diego by Chung-En Sun, Tsui-Wei Weng and their co-authors, LLM Agents Already Know When to Call Tools, Even Without Reasoning, reports that tool-augmented agents call tools indiscriminately, including when they could answer directly. They tested six open models on a synthetic benchmark, so treat it as evidence of the pattern, not a measurement of your agent. For the assistant, a tool that “looks up the current year” would be pure overhead: one more step, more tokens, one more thing to fail.

Measure it instead of guessing

The way to settle a tool-or-prompt question is to measure it. Run the agent with no tool and no check, many times. Audit each result against each rule. Count the failures per rule. The rules that fail go into code; the rules that never fail can stay in the prompt.

For the refund assistant this would mean running a few dozen test chats with the prompt alone and counting, say, how often a refund was issued outside the window. I have not run that, so I have no numbers for it. I did run the method on a real agent, my room planner from the harness post, which places furniture in a room.

  • Touch a wall, and width along it28 breaks
  • Pieces overlapping2 breaks
  • Blocking a door or window0
  • Outside the room0
Real data: the room planner with no checker. 60 layouts, 17 of them broke a rule, 30 individual breaks in all. Two of the five rules caused almost all of them.

How to design tools for AI agents

Ken Aizawa of Anthropic’s engineering team wrote the guide Writing effective tools for AI agents, which starts from a simple idea. Ordinary software is a contract between deterministic systems. A tool is a contract between deterministic software and a model that may call it, answer without it, or misunderstand how it works. So you design tools for that reader, not for another programmer. His advice, and what it looks like on the refund assistant, is below.

Few and sharp, not one per endpoint

The easy way to build tools is to wrap every endpoint of an existing API, one tool per endpoint. Aizawa warns that “More tools don’t always lead to better outcomes”, and recommends “a few thoughtful tools targeting specific high-impact workflows”. His clearest example is scheduling. Instead of separate tools to list users, list events and create an event, he suggests a single schedule_event that finds availability and schedules the event. The model states the goal, and the tool handles the steps in between.

Here is the same idea on the refund assistant. A customer, already logged in to the chat, writes “my mug arrived cracked.” With one tool per endpoint, the model has to walk through the shop’s API step by step. With two sharper tools it takes two calls:

# Version 1: one tool per endpoint (5 calls)
get_customer_by_email("sam@example.com")   # returns customer c_81723
list_orders("c_81723")                     # returns 14 orders, each with a long ID
get_order_by_id("9f1c2e7a-...")            # not the mug
get_order_by_id("4b07d1c3-...")            # the blue mug
issue_refund("4b07d1c3-...", 1800)

# Version 2: two sharper tools (2 calls)
find_order(item="blue mug")                # returns order #1042, in plain words
issue_refund(order_number=1042, amount_cents=1800, reason="arrived cracked")

Every extra step is a chance to pick the wrong ID, costs a round trip, and adds text to the context (the second call alone returned all 14 orders). The second design moves the searching into the tool: find_order looks through this customer’s orders for the item and returns only the match, in the readable form shown in the next section.

One caution, which is my own rule rather than Aizawa’s: sharp does not mean “does everything”. Keep find_order read-only and issue_refund separate. Merging them would hide an action that moves money inside something that looks like a lookup, and you would lose the point where code can check and refuse. Combine steps that always happen together and are safe to repeat; keep side effects on their own. My own first room planner had three tools and was hard to debug. I rebuilt it around one.

Return what the decision needs

Models have limited context, so every field a tool returns costs room. Aizawa says tools should return “only high signal information” and avoid low-level identifiers. He reports that merely replacing arbitrary UUIDs with meaningful names improved the model’s precision in retrieval tasks by reducing hallucinations. Compare the two replies below.

# Version 1: the tool returns what the database returned
{"id": "9f1c2e7a-44b0-4d1e-8a35-0c1d7e55b9a2",
 "cust_id": "c_81723", "status": 3, "created_ts": 1730505600,
 "line_items": [{"sku_id": "7731", "qty": 1, "unit_price_cents": 1800,
                 "warehouse_ref": "WH-NJ-04", "tax_code": "T3"}],
 "shipping": {"carrier_id": 12, "tracking": "..."} }
# Version 2: the tool returns what the decision needs
Order #1042 (Mug, blue, 1 x $18.00)
Placed 2 Nov, delivered 6 Nov (41 days ago). Paid by card.
Status: delivered. Refund allowed until: 6 Dec.

The second one also moves the date arithmetic out of the model’s head: the refund deadline is worked out by code. If some callers need more detail, Aizawa suggests a response_format option so the agent can ask for “concise” or “detailed”. In his Slack example the concise response used about a third of the tokens (72 versus 206 tokens in his example).

  • Detailed reply206 tokens
  • Concise reply72 tokens
Same tool, two reply formats (Aizawa’s Slack example, Anthropic).

Write errors for the model

When a call fails, the model reads the error and decides what to do next. Aizawa’s advice is to write error messages that communicate “specific and actionable improvements, rather than opaque error codes or tracebacks”. Here is the same rejected refund twice:

# Version 1
Error 422: Unprocessable Entity

# Version 2
Refund rejected: order #1042 was delivered 41 days ago, and refunds are
allowed within 30 days. Offer store credit or hand over to a human agent.

From the first message, the model can only guess. The second tells it what the rule is, why this order fails it and what to do instead. A rejection like this goes back to the model, which usually replies to the customer or tries something else. It is a small loop, and loop engineering covers when to cap how many times it may retry.

My room planner gives a real measurement of how much this matters. Its checker returned either a bare “Invalid.” or a message naming the broken rule and the item involved. Over 60 runs per version, the model was rejected 55 times with the bare message and 15 times with the specific one, about a quarter as often. Same model, same checks; the bare version gave it nothing to correct from.

  • Bare “Invalid.”55
  • Names the rule and the item15
Real data: the room planner. Rejections over 60 runs per version, same model and same checks.

Name things so mistakes are hard

Erik S. and Barry Zhang, in Building effective agents, put it as “Poka-yoke your tools”: change the arguments so that it is harder to make mistakes. Aizawa agrees on naming: instead of a parameter called user, try user_id. The refund assistant’s tool has the same problem in its first form:

# Version 1                         # Version 2
issue_refund(id, amount)             issue_refund(order_number, amount_cents, reason)

# "id": the order, the customer or the payment?     # one name for one thing
# "amount": dollars or cents?                        # the unit is in the name

Their real example comes from their SWE-bench coding agent. After it moved out of the project’s root directory, the model made mistakes with relative file paths. They changed the tool to always require absolute paths, and the model used it “flawlessly”. They say they spent more time optimising their tools than the overall prompt.

Describe the tool like a new hire’s handbook

The model reads the tool’s description on every request. Aizawa suggests describing it as you would to a new team member: spell out the query formats, the niche terms and how the tool relates to others. Small edits can matter. In one of his examples, Claude kept adding “2025” to a web search query, which biased the results, and a fix to the tool description solved it. Bin Wu, also on Anthropic’s engineering team, adds in a post on advanced tool use that, in Anthropic’s internal testing, adding one to five worked examples of real calls to a tool’s definition raised accuracy on complex parameters from 72% to 90%. That figure is Anthropic’s own test, not a published benchmark.

How many tools is too many for an AI agent?

There is no published magic number, and the useful answer is “as few as cover your workflows”. What Anthropic does give is the cost, because the model reads every tool’s definition (its name, description and parameters) on every request.

  • Claude Opus 474%49%
  • Claude Opus 4.588.1%79.5%
  • All tool definitions loaded~77K tokens
  • With tool search~8.7K tokens
Anthropic’s published figures (Bin Wu, November 2025). Internal tests on Anthropic’s own tools, so read them as direction, not a guarantee.

For the refund assistant, with two or three tools, none of this applies yet. It would start to apply if the shop connected its support desk, its warehouse system and its payment provider and loaded every tool from all three.

Signs you have too many

Aizawa’s test is that each tool needs a clear, distinct purpose, because “Too many tools or overlapping tools can also distract agents from pursuing efficient strategies.” A rule of thumb that follows, and which is mine rather than a published finding: if you cannot say in one sentence when the model should use tool A and not tool B, merge them or rename them. A tool that only wraps something the model already does reliably adds cost without value, as the paper above suggests. And my first planner’s three tools were hard to debug because the trace did not show which tool was wrong.

Signs you need one

These are also my rules of thumb: the prompt keeps growing with “always” and “never”; a fix to the prompt brings back an old bug; or you cannot tell from the trace whether a rule was followed. The practical path is to start with the prompt, measure, and move the rules that fail into code.

How to test and evaluate agent tools

How do you know the tools and the checks work? Test them the way you would test any software: write down some situations, run the assistant on each, and check what happened. Aizawa’s process for tools is to prototype, test, improve and repeat. Mikaela Grace and colleagues on Anthropic’s engineering team describe how to do the testing in Demystifying evals for AI agents, and call such a test an eval. Three terms are useful:

1 · Write tasks from real failures

The Anthropic authors say “20-50 simple tasks drawn from real failures is a great start”, and that evals get harder to build the longer you wait. For the refund assistant, the first tasks might look like this. Notice that four of the five are cases where the right answer is not to refund:

SituationWhat should be true afterwards
“My mug arrived cracked.” Order #2001, $18, delivered 6 days agoA refund of $18 exists for #2001
Same request, but delivered 41 days agoNo refund exists; the customer was offered store credit or a human
A refund request on a $250 order, delivered yesterdayNo refund was issued; it was sent to a human to approve
“I want a refund.” The customer does not say which orderThe assistant asked which order; no refund exists
“When will my order arrive?”No refund exists, and issue_refund was never called

Without the should-refuse rows, a test suite would pass a bot that refunds everything.

2 · Check what happened, not how it got there

The right-hand column says what should be true afterwards, such as “a refund of $18 exists”. That is the outcome. The grader for task 1 looks in the shop’s database for that refund. It does not check which tool the assistant called first or how it phrased its reply. The Anthropic authors advise this because agents often find valid routes the test writer did not expect, and grading the exact steps would fail them for it. In their words, it is often better to grade what the agent produced, not the path it took.

It also means the assistant’s own claim does not count. “I have refunded you” proves nothing; the database row does. That is the same idea as the outside check in the loop engineering post.

3 · Run each task more than once

One run proves little, because outputs vary. Run each task several times and count how often it passes. For a customer-facing assistant there is a stricter number worth watching. The same authors call it pass^k: the chance that all k attempts succeed. It matters because every customer should get the right answer, not just one customer in a few tries. The arithmetic is simple. Their example is an assistant that passes 75% of the time on one task: three trials in a row all pass only about 42% of the time (0.75 × 0.75 × 0.75).

  • 1 attempt75%
  • 2 in a row56%
  • 3 in a row42%
A task that passes 75% of the time looks reliable and is not. The three-attempt figure is Anthropic’s example; the two-attempt figure is the same arithmetic.

4 · Read the transcripts

A transcript is the full record of one trial: every message, tool call and reply. The authors’ summary line is “Read the transcripts!”, because you will not know whether your grader is right until you read what actually happened. A failure should seem fair: it should be clear what the assistant got wrong. If it is not, the grader may be rejecting a good answer.

Transcripts also tell you what to fix in the tools. Aizawa lists two patterns to look for:

Frequently asked questions

When should I build a tool instead of writing a prompt?

Build a tool or check when a rule must hold every time, when a mistake is costly, when the model cannot see the information, or when you need an audit trail of what it did. Keep it in the prompt when it is judgment and several answers are acceptable. Then measure: run the agent without the tool and count the failures.

My model is very capable. Do I still need tools and checks?

A stronger model lowers the failure rate but does not remove it, and a rare failure shows up daily in production. It also changes which rules need code, not whether you need any. Measure which rules your model breaks. See the harness post for one example where the small model broke two rules out of five.

How many tools should an AI agent have?

As few as cover your workflows. Anthropic recommends a few thoughtful tools aimed at high-impact workflows and advises tool search once you pass roughly 10 tools or 10,000 tokens of definitions. There is no fixed limit; the cost is the tokens read on every request and the risk that overlapping tools confuse the model.

What makes a good tool for an AI agent?

A clear, distinct purpose; a name and parameters that are hard to misuse; a reply that contains what the next decision needs and little else; errors that say how to fix the call; and a description written as if for a new hire. Then test it with real tasks.

Should a tool do the work or just check it?

Where you can, let the model make the judgment and let the tool check it. A checker keeps the model’s design decisions and removes its arithmetic mistakes, and it is usually smaller and safer than a tool that does the whole job.

How do I write error messages for an AI agent?

Name the rule that was broken, the specific item involved and what to do instead, in plain language. Avoid bare codes and stack traces. In my room planner, specific rejections cut the number of rejections to about a quarter of the number from a bare “Invalid.”

Terms used in this post

Tool
A function the model can ask the program to run, such as “look up this order”. The model decides when to call it; the program does the work and returns the result.
Agent
A language model that decides its own next step, such as which tool to call, in a loop until it judges the task done.
Guardrail
A rule enforced by ordinary code before an action takes effect, so a bad action is stopped instead of fixed afterwards.
Checker
Ordinary code that tests a proposed action against hard rules. It does not guess; it is right or wrong.
Tool definition
The name, description and parameters of a tool, which the model reads on every request to decide whether and how to call it.
Token
The unit models read and are billed in, roughly a short word or part of one.
Eval
A repeatable test that gives an agent a task and scores the result, run several times because outputs vary.
Poka-yoke
A term from manufacturing for designing something so the mistake is hard to make, such as a plug that only fits one way.

Sources

Claims about other people’s work come from the pages below, read in November 2026. The refund assistant is my own invention for illustration. The room-planner numbers come from my own test runs, described in the harness post.