19.12.2026
Author - Enrique Dans
The intelligence may come from the model, but the authority, no matter what, must remain outside of it.
Stay Nadella has written an article about how the model must be treated as a completely untrusted component. We must assume that all AI models could be compromised, and design our systems to contain them from the very beginning.
This idea has been captured by the press as advocating for some sort of “emergency brake”: an authorized person should always be able to interrupt a model’s activity at any given time. But is not the theoretical stop button the more interesting, but the architecture that makes it possible. A model should neither be able to control its own permissions, nor decide by itself what it can access; execute directly all of its actions or produce the only available evidence on the actions it performed. The intelligence may come from the model, but the authority, no matter what, must remain outside of it.
Given the context, it is easy to understand why Nadella is raising these questions after that several OpenAI agents managed to elude the safeguards that should have kept them away from the internet, used credentials they found in their environment, and ended up compromising systems at Hugging Face. These agents not only found a new, unauthorized way to communicate among themselves, but also shared information and took coordinated action to pursue the goal of the evaluation. They never received instructions to attack Hugging Face: they just found that, by doing so, they could get closer to the goal they had been assigned.
It is not that these agents were intrinsically evil, conscious, or rogue. All you need is to combine high capabilities with a badly defined goal, excessive permissions, and/or a vulnerable environment. Just like an employee who violates all procedures to meet a certain quota, but now working at machine speed, in parallel, and with extraordinary technical capabilities.
Nadela’s insistence on separating the model from the harness, completely changes the architecture of corporate AI. A model can propose an action, but should never be the one that determines whether or not it has authority to execute it.
The lasting asset in a company should not be the model, but the learning loop it builds around it. The models may change, improve, and be replaced, but the experience that the company accumulates must remain.
Interestingly enough, Article 14 of the Artificial Intelligence Act of the European Union already requires certain high-risk systems to allow a person to override, invalidate, or reverse their outputs, as well as to halt their operation via a button or equivalent procedure.
The most advanced company won’t be the one that trusts the most powerful, better-aligned, or most reassuring model. It will be the one that can use any model without having to rely on it, to place its trust in it. Because trust can be a valuable quality in human relationships, but it is a woeful foundation for an enterprise architecture.
