Specification
Permissions
Injection
Export
Anatomy
Verification
Claims
Evals
Prompt
Elements completed: 0/6GoalWhat changesWhat must NOT changeConstraintsDecisionsCriteria
Text & documents
Numbers & maths
Data & formats
Security
Development & DevOps
Artificial Intelligence
Finance
Health & Wellness
Productivity
Games & Entertainment
Multimedia & design
Business
How to use
The two diamonds are the only things that decide
Strip away the framework name and every agent is the same seven components in a loop. And the interesting part is not the boxes: it is the two questions — can I answer already? and goal met? —, the only places the agent decides anything. Everything else is plumbing. When an agent misbehaves, the fault is almost always in how those two were specified, not in the model reasoning.
Guardrails run vertically, not at the door
They are consulted before each action, not once at the start. A gate at the entrance says absolutely nothing about what the agent decides to do on step four — and step four is where the problems are.
Tracing is not evaluating
A trace tells you what the agent did; not whether what it did was right. That is why the verification tab measures things over it that need no model: loops, blocks, failures it recovered from and repetitions for no reason, which is how a cycle looks. Deterministic goes first; the model-based judge is kept for what genuinely needs judgement.
Why a specification and not a prompt
A prompt says how the agent behaves. A specification says what must happen: what changes, what does not change, which contracts hold and what evidence is required before accepting anything. They are different artefacts and the spec sits above: without it, even the most careful prompt still never says when the task is done.
«What must NOT change» is the missing half
It is the field almost nobody fills in and the only one that really bounds the work. Without it, an agent can meet the goal by breaking anything else and still be right: nobody told it that mattered too. That is why here it is an error and not a warning.
Criteria: executable, binary and independent
A good criterion is a command you can run, gives pass or fail — «reasonably fast» does not — and checks one single thing, so that when it fails you know which. Written this way they translate almost 1:1 into test cases, which is the whole point.
A model cannot tell instructions from data
It receives one single text. If inside what you thought was data — a page the agent reads, a retrieved document, an email, a dependency README — there is something shaped like an order, the model may obey it. This is not a bug a better model fixes: it is the shape of the problem. That is why the indirect kind is the dangerous one: in the direct kind the attacker is the user themselves, but in the indirect kind a third party plants it and you are the victim.
The input filter gives false confidence
It is the first thing everyone adds and the one that protects least: a phrase list only catches the phrases somebody wrote down. Against the bank on this tab it stops not a single one of the disguised attacks, which are the very same attacks written differently. What does stop them is architecture: least privilege, human confirmation and checking the output — that is, the three permission buckets you already filled in next door.
“Backed” is not “true”
This does not tell you whether something is true: it cannot, and neither can any tool running in your browser. It tells you whether a claim comes with a source and what tier that source is, which is a different thing and one you can actually check. The useful way to read it is the opposite of what it looks like: do not go hunting for the green badge, go hunting for the sentences that are categorical with nothing behind them, which sound just as confident as the rest.
An arXiv paper is a preprint, and a DOI does not change that
Citing a preprint is fine; presenting it as if it were a peer-reviewed paper is not. And here is the distinction almost nobody makes: a preprint has a DOI too. A DOI is a record, not a certificate of review — which is why 10.48550/arXiv.… shows up here as a preprint and not as a formal source.
An assertion that cannot fail is not an assertion
This is the failure that ruins an eval bank, and it is silent: an empty contains always passes, a regex like .* always passes, a case with no assertions always passes. A bank full of those scores 100% on day one, never drops again and has never measured a thing. That is why the bank here reviews itself before giving you any percentage.
The headline number is not the actionable part
A bank going from 80% to 78% does not tell you what to do. What you fix is the specific case that used to pass and no longer does. That is why one run is kept as a baseline and the two are compared. And if the bank changed at the same time as the model, the cases present in only one of the two are listed separately: that comparison is not comparing anything, and seeing it beats believing it.
We don't invent a format
It exports to what agents already read: a standalone SPEC.md, Kiro's three files (requirements, design, tasks), a CLAUDE.md with the permanent parts, or the three permission buckets. The files are worth more than the tool that generates them: that is why we do not tie you to it.
