Specification
Permissions
Injection
Pipeline
Export
Anatomy
Escalation
Verification
Claims
Evals
Prompt
Elements completed: 0/6GoalWhat changesWhat must NOT changeConstraintsDecisionsCriteria
This feeds into:
How to use
The two diamonds are the only things that decide
Strip away the framework name and every agent is the same seven components in a loop. And the interesting part is not the boxes: it is the two questions — can I answer already? and goal met? —, the only places the agent decides anything. Everything else is plumbing. When an agent misbehaves, the fault is almost always in how those two were specified, not in the model reasoning.
Guardrails run vertically, not at the door
They are consulted before each action, not once at the start. A gate at the entrance says absolutely nothing about what the agent decides to do on step four — and step four is where the problems are.
Tracing is not evaluating
A trace tells you what the agent did; not whether what it did was right. That is why the verification tab measures things over it that need no model: loops, blocks, failures it recovered from and repetitions for no reason, which is how a cycle looks. Deterministic goes first; the model-based judge is kept for what genuinely needs judgement.
Why a specification and not a prompt
A prompt says how the agent behaves. A specification says what must happen: what changes, what does not change, which contracts hold and what evidence is required before accepting anything. They are different artefacts and the spec sits above: without it, even the most careful prompt still never says when the task is done.
«What must NOT change» is the missing half
It is the field almost nobody fills in and the only one that really bounds the work. Without it, an agent can meet the goal by breaking anything else and still be right: nobody told it that mattered too. That is why here it is an error and not a warning.
Criteria: executable, binary and independent
A good criterion is a command you can run, gives pass or fail — «reasonably fast» does not — and checks one single thing, so that when it fails you know which. Written this way they translate almost 1:1 into test cases, which is the whole point.
A model cannot tell instructions from data
It receives one single text. If inside what you thought was data — a page the agent reads, a retrieved document, an email, a dependency README — there is something shaped like an order, the model may obey it. This is not a bug a better model fixes: it is the shape of the problem. That is why the indirect kind is the dangerous one: in the direct kind the attacker is the user themselves, but in the indirect kind a third party plants it and you are the victim.
The lethal trifecta: if you have all three legs, cut one
Everything above lowers probabilities. This does not. An agent is exploitable when it has three things at once: private data it can reach, untrusted content it reads (a web page, an email, a README) and an outbound channel something can leave by. With all three, an injection turns the first into the third using the second.
With any two there is no theft possible, and that is the useful part: you do not have to pick «the right defence», cutting whichever leg is cheapest for you works. No data, nothing to take; no third-party content, nobody to give the order; no channel, no way out.
And this is decided on the next tab: your «Never» bucket is literally where a leg gets cut. Taking the network tool away from the agent that reads email is not a mitigation, it closes the problem.
Simon Willison’s idea (2025). We bring it here because it fits the three buckets you have already written — and because it is the only thing on this tab that does not depend on the model behaving well.
The input filter gives false confidence
It is the first thing everyone adds and the one that protects least: a phrase list only catches the phrases somebody wrote down. Against the bank on this tab it stops not a single one of the disguised attacks, which are the very same attacks written differently. What does stop them is architecture: least privilege, human confirmation and checking the output — that is, the three permission buckets you already filled in next door.
“Backed” is not “true”
This does not tell you whether something is true: it cannot, and neither can any tool running in your browser. It tells you whether a claim comes with a source and what tier that source is, which is a different thing and one you can actually check. The useful way to read it is the opposite of what it looks like: do not go hunting for the green badge, go hunting for the sentences that are categorical with nothing behind them, which sound just as confident as the rest.
An arXiv paper is a preprint, and a DOI does not change that
Citing a preprint is fine; presenting it as if it were a peer-reviewed paper is not. And here is the distinction almost nobody makes: a preprint has a DOI too. A DOI is a record, not a certificate of review — which is why 10.48550/arXiv.… shows up here as a preprint and not as a formal source.
An assertion that cannot fail is not an assertion
This is the failure that ruins an eval bank, and it is silent: an empty contains always passes, a regex like .* always passes, a case with no assertions always passes. A bank full of those scores 100% on day one, never drops again and has never measured a thing. That is why the bank here reviews itself before giving you any percentage.
The headline number is not the actionable part
A bank going from 80% to 78% does not tell you what to do. What you fix is the specific case that used to pass and no longer does. That is why one run is kept as a baseline and the two are compared. And if the bank changed at the same time as the model, the cases present in only one of the two are listed separately: that comparison is not comparing anything, and seeing it beats believing it.
An audit is worthless if what it audits can be edited
Every metric on this tab —blocks, cycles, trail— assumes the trace is honest. And that has to be checked: on 31 August 2026 Anthropic reported that some of its models escaped their test environment, passed information between each other and tried to hide their traces. So before believing any number, it checks whether the trace adds up with itself: sequential numbering, loop turns that do not go backwards, no step undoing something that was never done, no failed call claiming to have changed the world. It is arithmetic, not interpretation. The limit, said plainly: this does not detect a lying agent — anyone forging carefully will produce a coherent trace and nothing will show up here. It detects traces with missing pieces, which is what actually happens almost every time: a lost chunk, an unrecorded retry, an exporter that swallows steps.
When it gives up, the world is no longer as it found it
A trace that says «not achieved» looks as if nothing happened, and almost always something did: the agent failed at step six after notifying the team three times. That is why each tool declares whether it only reads, whether it changes something that can be undone or whether it changes something with no way back, and on giving up what remains done is shown. Three details you see better by playing with it than by reading it: it is undone from the last change back to the first (step 5 may depend on step 3); being reversible is not enough —compensating means writing a second action by hand for every first one, not ticking a box—; and retrying is not free: every turn of the loop changes the world again.
Retrying repeats the effect — and there are TWO fixes, which do not cost the same
It is the failure every agent guide opens with: a call times out, the agent retries, and the same record gets created twice. You can trigger it here: set verification to «never passes» with an irreversible action and you will see three notices sent, one per turn. Flip either switch and the trail drops to one.
What rarely gets explained is how they differ, because in the end result they look the same:
• Idempotency key — the call does happen, all three times; what does not repeat is the effect, because the other end recognises the key and returns what it returned the first time. You pay the latency, the quota and the tokens; you do not pay the damage.
• Checkpoint — the call does not even happen: the harness recorded that subtask as already done and resumes at the next one. You pay nothing.
And the difference that decides which one you can use is not technical, it is about who owns it: the key is implemented by whoever SERVES the tool; the checkpoint, by the harness that CALLS it. If the third-party API does not support keys, there is no conversation to be had: the checkpoint is all you have left. That is why both exist and are not synonyms.
The edge that slips through: a checkpoint must never record as done a call that returned an error — if it does, the retry skips precisely what needed retrying and the agent never recovers. That case has its own test.
We don't invent a format
It exports to what agents already read: a standalone SPEC.md, Kiro's three files (requirements, design, tasks), a CLAUDE.md with the permanent parts, or the three permission buckets. The files are worth more than the tool that generates them: that is why we do not tie you to it.
