Artificial intelligence is rapidly moving beyond answering questions, generating reports, and identifying patterns.
We're giving it the ability to execute actions.
AI agents can interact with applications, change configurations, initiate security responses, and coordinate increasingly complicated workflows.
And that creates a question I believe every executive, technology vendor, and infrastructure operator should be asking:
Can you prove your AI will stop when it's supposed to?
Not whether the product documentation says it has an emergency-stop function.
Not whether a salesperson assures you that the system is safe.
And certainly not whether the vendor has experienced a particular failure before.
I want to know what happens when something goes wrong.
Because in critical infrastructure, the difference between a control that exists and a control that actually works can be enormous.
A Demonstration Isn't a Disaster
Imagine you're evaluating an AI automation platform for a hospital, utility, or major telecommunications network.
The vendor demonstrates impressive capabilities.
The system identifies threats, analyzes operational conditions, recommends corrective actions, and executes approved responses.
It has audit logs, access controls, human oversight, and an emergency-stop mechanism.
Everything looks wonderful.
Then you ask a few additional questions.
What happens when the AI loses communication with its policy controller?
What happens when the telemetry it's relying on becomes outdated?
What happens when an agent attempts to execute an action through an unauthorized interface?
And what happens when someone activates the emergency-stop function while the system is already executing a complicated sequence of operations?
Those aren't unreasonable questions.
They're fundamental engineering questions.
Now suppose the vendor tells you that these scenarios are extremely unlikely because its platform is highly available.
That answer doesn't necessarily disqualify the product.
But it certainly doesn't answer the question.
The fact that something hasn't failed doesn't establish that it cannot fail.
And the fact that a vendor hasn't tested a particular failure scenario doesn't automatically mean its product is unsuitable.
What matters is whether the vendor is willing to demonstrate how the system behaves when those conditions occur.
If they're willing to test, we have something to work with.
If they refuse meaningful testing, we have a much more serious problem.
The Industry Is Beginning to Address the Problem
Recent developments suggest that AI control and assurance are becoming increasingly important engineering disciplines.
Uber has published details about its Agent Guard architecture, describing how policy-enforcement mechanisms can be placed at agent and tool-execution boundaries.
Its approach includes specialized local models, parallel evaluation of guardrails, replay testing, and mechanisms for examining how agents and their controls behave.
That's useful production engineering experience.
But published architectural descriptions aren't the same as independently verified evidence of worst-case response times, availability during communications failures, or guaranteed enforcement under every operating condition.
Those claims require their own testing.
Meanwhile, the Responsible AI Institute announced its TrustX Assurance Lab on November 10, with an emphasis on evaluating autonomous AI controls under defined testing conditions.
The initiative is another indication that the industry recognizes the need for more systematic assurance.
However, the laboratory's initial work uses synthetic data and member-defined use cases, with broader member-environment testing planned for 2027.
That's progress, but it isn't proof that every autonomous system is ready for critical infrastructure.
And that's an important distinction.
We should recognize meaningful engineering progress without confusing it with universal proof of operational safety.
This Is What Digital Twins Are For
There's a common objection whenever someone proposes testing catastrophic failure scenarios.
"We can't do that in production."
And they're right.
I'm certainly not suggesting that we deliberately interrupt an operating hospital, disable a utility control network, or compromise a telecommunications system just to see what happens.
That's precisely why digital twins are so important.
A properly engineered digital twin can provide a controlled environment in which we reproduce relevant system configurations, dependencies, operational behaviors, and failure conditions.
We can simulate communications failures.
We can introduce corrupted or delayed telemetry.
We can test compromised identities, conflicting instructions, unavailable control services, and emergency-stop procedures.
We can evaluate what happens when multiple failures occur simultaneously.
And we can observe whether the system remains within its authorized operating boundaries.
All without deliberately placing real patients, customers, employees, or communities in danger.
Of course, a digital twin is only as useful as the accuracy of its models and assumptions.
If it doesn't adequately represent the dependencies and failure conditions of the real environment, its results may create false confidence.
That's why digital-twin testing must be accompanied by validation, documented limitations, and appropriate additional testing before production deployment.
But the larger point remains.
We have ways to test dangerous scenarios without creating actual disasters. We should be using them.
Failing Closed Isn't Always the Same as Failing Safely
Here's where things become particularly complicated.
In cybersecurity, we often talk about systems failing closed.
If authorization cannot be verified, the system denies the requested action.
That's generally a sensible security principle.
But in critical infrastructure, the consequences of denying an action depend on what that action actually does.
Consider a hospital.
If an AI loses communication with its central policy controller, should it immediately stop every operation it was managing?
What if those operations involve patient monitoring, medical equipment, or other essential services?
An indiscriminate shutdown could create a much more dangerous situation than the original communications failure.
The same principle applies to electrical grids, telecommunications networks, and emergency-response systems.
Sometimes the appropriate response is to preserve an existing safe operating state while refusing new or higher-risk instructions.
Sometimes it means transferring control to a local system operating under predetermined limits.
And sometimes immediate isolation is necessary.
The correct response depends on the system, its operating conditions, and the consequences of the available actions.
This is why safety cannot be reduced to a single emergency-stop button.
We need to define what the system is permitted to do when its normal control mechanisms are unavailable.
And those boundaries need to be established before the failure occurs.
Who Has the Authority to Say No?
One of the architectural principles we're examining at Atlas Global Technologies involves separating AI reasoning from operational enforcement.
Our proposed Talos 2.0 and Cognition Engine architecture is intended to use AI for analysis, correlation, and decision support while relying on independently governed mechanisms to authorize consequential actions.
The objective is to prevent an AI system from becoming the sole authority over the limits of its own behavior.
We're also examining how to test those boundaries through digital-twin environments, failure injection, and measurable enforcement requirements.
These are proposed engineering objectives, not demonstrated operational capabilities. Atlas remains in its prelaunch stage, and the architecture will require implementation, validation, and independent scrutiny.
We don't get to claim reliability simply because we've designed for it.
We have to prove it.
And the same standard should apply to everyone else.
The Question Every Customer Should Ask
I don't expect technology vendors to have anticipated every possible failure.
No system designer can honestly promise that.
And I don't believe the absence of a particular test result should automatically disqualify an otherwise promising solution.
But I do expect vendors to take legitimate questions about operational safety seriously.
I expect them to participate in meaningful testing.
I expect them to acknowledge limitations.
And I expect them to help customers understand what their systems will do when conditions deteriorate.
Because the more authority we give artificial intelligence, the greater our responsibility to understand the consequences of that authority.
The question isn't simply whether AI can automate an operation.
It's whether we can demonstrate that the automation behaves appropriately when the assumptions behind it are no longer true.
If a vendor is willing to test, we can have an engineering conversation.
If a vendor refuses to test, we have a trust problem.
And when the systems involved support hospitals, critical infrastructure, and the communities that depend on them, trust without evidence simply isn't good enough.
Join the discussion
Should customers require digital-twin failure testing before deploying autonomous AI into critical operations?
And should vendors be expected to demonstrate their emergency controls under realistic failure conditions rather than relying solely on documentation and assurances?
I'd be interested in hearing how other infrastructure, cybersecurity, and AI professionals approach that question.
Sources and further reading
- Uber: Securing AI Agents — Production architecture and policy enforcement for AI agents.
- Responsible AI Institute: TrustX Assurance Lab — November 10, 2026 announcement on autonomous AI control testing.
- External Control Layer: Specifications Comparison — Comparative analysis of agent-control frameworks and their failure behaviors.
- NIST: Challenges in Monitoring Deployed AI Systems — Post-deployment AI monitoring and assurance challenges.