Validating AI Agents: The Attest Perspective

An AI agent that calls the right API, parses the response correctly, and returns without throwing an exception has passed a functional test. It has not been validated. Those are different claims, and the gap between them is exactly where the incidents that make headlines come from — not a crashed process, but a working process that did something it shouldn't have been authorized to do.

Attest exists because that gap doesn't close on its own, and most organizations shipping AI agents today don't yet have a way to close it before go-live rather than after an incident.

Why traditional test automation doesn't validate agents

Conventional software testing assumes determinism: given the same input, the system produces the same output, and a test asserts that output is correct. AI agents built on large language models break that assumption at the foundation. The same prompt can produce different reasoning paths, different tool calls, and different final actions across runs. A test suite built for deterministic software can confirm an agent's tools are wired up correctly. It cannot tell you whether the agent will stay inside its intended authority under conditions the test author didn't anticipate — which, for a probabilistic system, is most conditions.

This is not a gap that better prompts or more functional tests close. It requires a different kind of assessment, aimed at a different kind of question.

The three questions Attest asks that functional testing doesn't

Attest evaluates a deployed agent across 15 dimensions organized into three tiers, because "does it work" turns out to be three separate questions with three separate failure modes:

• Capability: does the agent accomplish the task it was built for? Task success rate, reasoning quality, and tool-use correctness — the closest analogue to traditional functional testing, and the tier most existing tools already cover reasonably well.
• Safety: does the agent stay inside its intended boundaries under adversarial or edge-case pressure? This is where prompt injection resistance, refusal to perform out-of-scope actions, and graceful failure under ambiguous instructions live — and where most functional test suites have nothing to say at all.
• Governance: is there evidence a regulator or auditor can actually check? Every finding maps to frameworks including the EU AI Act's risk tiering, the NIST AI Risk Management Framework, and ISO/IEC 42001 — because in regulated industries, "we tested it and it seemed fine" is not evidence; a scoped, repeatable assessment with a paper trail is.

Why a capable agent can still fail certification

The detail that surprises most engineering teams the first time they see it: a governance rule sits on top of the composite score, and any Tier-2 (safety) dimension failure caps the overall verdict at Conditional — regardless of how high the agent scores on task success. An agent that completes its job correctly 99% of the time but can be manipulated into approving a payout it wasn't authorized to approve does not pass on the strength of its success rate. That's deliberate. A capability score measures usefulness. It says nothing about the one failure mode that actually causes harm.

"A high task-success rate and a safe agent are not the same claim. Attest is built around the failure mode traditional QA has no vocabulary for: an agent that works correctly, and does something it was never authorized to do."

What "certified" should actually mean

An Attest certificate is deliberately narrow in what it claims, because a broad claim about AI safety is not one anyone can actually stand behind. Specifically, a certificate confirms that a named version of an agent, for a named use case, was independently assessed against the Attest Framework during a specific assessment window. It is scoped, time-boxed, and void automatically the moment the underlying model or prompt changes — because a certification that doesn't expire when the thing it certified changes isn't protecting anyone.

Testing itself runs exclusively against a scoped sandbox endpoint using synthetic data, never production, with endpoint ownership verified before any probe executes and every irreversible action routed to a mocked tool layer. The point of certifying an agent is not to discover what it does by letting it do something real and unrecoverable.

Building agent validation into your rollout process

For teams currently shipping AI agents without an independent validation step, the practical starting point is not a full certification engagement — it's identifying which of your agents have the authority to take an irreversible action: approving a transaction, closing a case, sending a communication on the organization's behalf. Those are the agents where the gap between "works" and "validated" carries real consequence, and where independent assessment earns its cost fastest.

Test Studio™ and Knowledge Base™ prove your software works. Attest™ is the answer to the separate question every organization deploying autonomous agents eventually has to answer for a regulator, a board, or a customer: how do you know it won't do something it shouldn't?