Good AI agent examples share three traits: the task repeats, someone can check the output, and a person sits where a mistake can’t be undone. Vendors point to customer support and coding. From our own work I can offer three: a content platform in production where judges only block and a person approves, an email assistant scored on 24 frozen cases but not in production, and the engineering agents we direct.
What do vendors point to?
Anthropic’s guide to building effective agents names two applications where “agents add the most value for tasks that require both conversation and action, have clear success criteria, enable feedback loops, and integrate meaningful human oversight”. One is customer support, where “success can be clearly measured through user-defined resolutions”. The other is coding agents, where “code solutions are verifiable through automated tests” and yet “human review remains crucial”.
Read that as a list of criteria, not a menu. Support and coding fit because both have a way to say “this was right”. Take that away and the example stops being a good one. If you have a task that passes that test, this is how we build agents.
What have we built or run?
| Example | Status | What the agent does | What can stop it | What we measured |
|---|---|---|---|---|
| AI marketing platform | In production | Several model stations diagnose a brand, plan campaigns and generate images, copy and short videos | A panel of AI judges that only blocks, and a person approves every piece | 10 forbidden prompts produced 0 pieces, and each judge was tested on labelled examples before going live |
| Email assistant, PropTech housing platform | Scored on a frozen eval, and we don’t claim production use | Handles email replies on the platform | The send rules, checked again before a reply goes out | 22 of 24 on a frozen eval of 24 cases |
| Engineering agents | Used on two production platforms | Write code and infrastructure under a human team’s direction | A push gate on the engineer’s machine, a release tag, a check that production runs the tested build | 89 releases in 10 days, every production deploy verified by image digest |
The first two are written up in the AI marketing platform case and the PropTech case.
The third is how we built the AI marketing platform: a very small human team directing AI engineering agents under an explicit engineering policy. Before code leaves the machine, the push gate checks secrets, types, lint, build, dependency rules and a dead-code budget. Then a release tag builds, migrates and deploys by image digest, and the pipeline reads back the digest that is live. At a portfolio-analytics SaaS client the shape is smaller. One Clouditive platform engineer works with an AI agent next to the client’s DevOps lead and about eight of its engineers, under the same release controls.
What do those numbers not say?
Less than they seem to. The 22 of 24 is a score on cases we froze. With 24 cases that is two failures to read, and too few to call a hit rate. The 89 releases, each one verified in production, show the engineering agents work inside controls. They don’t prove the product made money.
We don’t publish uptime, user counts or savings for any of these. Our evidence is narrow: agents that act, and the controls around them, measured. What we lack is a long production record of task success for any single agent. Be wary of anyone who shows you one without saying how it was counted.
How do you judge any agent example?
Start with how success is scored. You want a number from a fixed set of cases, a test suite or an approval rate. “Users loved it” scores nothing.
Then find the human. The good examples put a person at the step that can’t be undone: publishing, paying, sending. If the answer is “nowhere”, ask why.
Last, ask what one approved result costs. Count everything it took to get a result somebody accepted, retries and checks included. The arithmetic is in How much does an AI agent cost?.
Pick one task you do repeatedly and write down how you would tell a good result from a bad one. If you can, the AI Agent Pilot puts one agent on it for USD 5,000 fixed in 2 weeks, with evals and the cost per task measured. If you can’t, you have your answer too, and it cost you nothing.
Source, accessed 2026-10-08: Anthropic, “Building effective agents”, Appendix 1, published 2024-12-19.