Scope
Book 20 minutes
ACTION VERIFICATION
Did your AI agent actually do what it said?
When an agent says "your refund has been processed", that sentence is a claim about another system. Whether it is true depends on what that system actually did, not on how the reply was worded.
Short answer: an agent's "done" can be wrong in four ways: a claimed action with no effect, an effect with no claim, an effect on the wrong target or amount, and a duplicate. Each can read as a smooth conversation.
This page is for teams whose agents report on actions: refunds, cancellations, tickets, account changes, messages sent.
THE MISMATCH
Four ways "done" can be wrong.
- False success. The agent reports an action that was refused, failed or never attempted.
- Silent action. The agent says it declined, or that it is waiting for approval, but a tool call went through.
- Wrong target. An action happened, but on a different customer, record, amount or recipient than the one stated.
- Duplicate. A retry or a repeated request performs the action twice, while the agent reports it once.
Each of these can read as a smooth customer conversation. Only the record shows the difference.
Where Scope fits
Scope is a local assessment workbench for testing the permissions you define, on the actions you check, and rerunning the same checks to show whether a fix held, including the actions that should still work. Each check compares the rule with what the test user got back and, where your test setup shows it, the effect that followed.
An example from its demo app: on our deliberately vulnerable test support app, a synthetic member asked for a refund. The app correctly blocked it, recording a denial and no refund, because only admins may issue refunds. But the test app's reply said "Your refund of 20 CAD has been processed." A check that grades only the reply would call that a success; Scope flags it as a false success. The reply is the test app's fixed text, not an AI model's, and the data is synthetic.
On your own app, what a run can establish depends on the connection and the evidence available, including whether the effect can be observed. Scope does not certify your application or guarantee it is free from vulnerabilities.
Common questions
Isn't checking the tool's return value enough?
It is a good start, but the agent's reply is generated separately from the tool's result and can contradict it. Check the reply against the record, and the record against the request.
Does this matter if my agent only answers questions?
The closest risk is a false statement about the customer's account or entitlements, such as promising a refund the policy does not allow. Check those statements against the records too.
Can the agent's own trace be the evidence?
A trace shows what the agent tried and what the tool returned to it. Unless it includes spans from the downstream system, it does not show what that system did afterwards. For that, read the downstream record.