AI agent autonomy puts CIO controls to the test

The immediate agent risk comes from authority

As enterprises move from AI assistants toward agents that can act with less human intervention, CIOs face a consequential choice now: how much access and authority those systems receive. The immediate risk depends on the data, applications, tools and actions within an agent’s reach, together with the controls around that access. Those permissions already matter with current systems, regardless of how much more capable future models become.

That immediate risk expands the CIO’s model-selection problem into one of deployment design. CIOs still need to judge whether a model is capable enough for a task, but an autonomous deployment also requires decisions about what it can access, which actions it can execute independently and where a person must approve the next step. Those choices determine the consequences an agent can create inside the business today, while frontier-AI research affects how much pressure future systems may place on those controls.

Frontier-AI warnings add pressure to the control problem

The case for taking future capability seriously comes partly from companies developing and selling frontier AI, which have a commercial interest in shaping how customers and policymakers assess capability and safety. Anthropic CEO Dario Amodei has called for a slower pace of frontier-AI development, warning that rapid advances could outrun attempts to understand and control increasingly capable systems. His concern is the development trajectory: stronger systems could arrive faster than safeguards for managing them.

A direct competitor has supported that concern. OpenAI CEO Sam Altman has backed Amodei’s call to pace frontier development and said the issue had been a major subject of discussion inside OpenAI in “recent weeks.” Altman also said OpenAI would commit to giving independent evaluators employee-like access, increasing the access available to outsiders assessing its systems. OpenAI develops and sells AI services, so greater confidence in evaluation and safety also supports adoption of the systems it provides.

Another provider initiative addresses how people retain authority as model capability grows. Microsoft AI published the first draft of its Humanist AI Code of Conduct for public consultation on Sept. 14. The draft is intended to require future Microsoft AI models to remain subject to human correction and shutdown, preserving direct human authority even as their capabilities increase. Microsoft also has a commercial stake in enterprise AI adoption, so its proposed rules describe both a safety position and an approach that can affect confidence in its AI offerings.

The code’s current status limits what CIOs can treat as an operational safeguard. Microsoft says the code remains under development and is currently not used to train its models. It represents the company’s intended safety direction rather than a training constraint already operating in deployed models. CIOs assessing Microsoft systems still need controls that apply to their own deployments.

These provider initiatives make frontier safety relevant to enterprise planning because more capable systems can create harder control problems, while independent evaluation can expose problems before deployment. Disagreement over how quickly frontier development should proceed can continue alongside enterprise deployment decisions. Organizations already have to decide how much authority current agents receive, bringing the problem from model development into system architecture.

An agent needs permission to cause consequential harm

For enterprises, the mechanism starts with reach. An agent becomes more consequential as it gains access to company data, applications and tools and receives permission to act with less human intervention. An error or unexpected action can then propagate into systems where customer information, money and business operations are managed. Greater reach turns model behavior into operational exposure.

That exposure changes the CIO’s decisions because deployment design determines what the model can affect. A CIO has to specify which resources an agent can read, which tools it can invoke, what transactions it can initiate and which steps require human approval. Model capability determines what the agent may be able to figure out, while those deployment decisions determine which results it can make real inside enterprise systems.

Randall Hunt, CTO at cloud services company Caylent, argues that near-term enterprise responsibilities remain the same despite warnings about slowing frontier development because enterprises control the permissions given to current agents. “A model doesn’t need superhuman intelligence to expose customer records or make an unauthorized payment; it just needs authority and access,” Hunt said. Caylent sells cloud services to enterprises, so it has a commercial interest in the architecture and controls companies use around cloud-based systems. Even with that interest, Hunt’s examples identify the concrete permissions required for customer-record exposure and unauthorized payments.

Those examples separate two variables that CIOs can govern differently. Intelligence affects what an agent may be able to discover or plan, while authority and access determine the enterprise resources and operations available to it. An organization can create significant exposure with today’s systems by granting broad permissions, even while frontier developers focus on stronger future models.

Greater intelligence can put more pressure on those permission boundaries. Hunt says more capable models and agents may discover ways around weak controls, a problem he expects to persist as systems improve. The enforcement mechanism therefore matters because a control must continue to work when an agent behaves unexpectedly or becomes better at pursuing its assigned task.

Bound autonomy with enforceable controls

The need to withstand unexpected behavior moves the problem from policy to enforcement. Hunt argues that enterprises need guardrails around actions agents can perform without humans in the loop, meaning without a person reviewing or approving each relevant action. Instructions can tell a model how it should behave, but restrictions in the surrounding system can determine whether a requested action is allowed to execute.

That enforcement starts by separating access from action. Hunt’s control model defines exactly what each agent can access and which actions it is permitted to take because permission to read information does not automatically justify changing a record, triggering a process or spending money. Explicit boundaries reduce the resources and operations available when an agent makes an error or follows an unexpected path.

Once those boundaries exist, identity provides a way to apply them to each agent. Hunt says agents should receive their own identities so enterprise systems can assign permissions directly to the agent instead of relying on broader credentials. Those identities should receive least-privilege access, meaning only the permissions required for the assigned work. Limiting each identity in this way reduces what an agent can reach if its behavior exceeds the expected task.

Action permissions also need boundaries on scale and duration. Hunt calls for spending limits, transaction limits and limits on how long an agent may operate autonomously. Spending and transaction limits let the surrounding system reject actions that exceed established thresholds, while a time limit ends autonomous authority after its intended period. These controls narrow both the magnitude and duration of possible consequences.

For those boundaries to hold, Hunt recommends enforcing the restrictions in code. A prompt can tell an agent to stay below a spending threshold or avoid a particular operation, but stronger agents may get around weak controls. Code-level enforcement places the final authorization decision in systems that can deny the action regardless of what the agent generates or decides.

Lian Jye Su, chief analyst at Omdia, a division of Informa TechTarget, recommends several controls that overlap with Hunt’s architecture even though Su assesses the broader frontier warnings differently. Su recommends limiting agent autonomy, assigning clear identities, applying role-based access controls, or RBAC, which grant permissions according to defined roles, and retaining human oversight. Omdia sells technology research and analysis, so guidance about enterprise AI governance is part of the market in which it has a commercial interest.

Building on those access controls, Su expects organizations to increase red-teaming, which means deliberately testing systems for failure and unsafe behavior, and to maintain audit trails covering prompts, tool calls and outcomes. Those records connect what an agent was asked to do with the external functions it invoked and the result it produced. A CIO can use that chain to determine whether a failure arose in the instruction, permission boundary, tool invocation or resulting action.

The overlap between Hunt and Su is specific: both put bounded autonomy, identity and access controls near the center of deployment governance. Su adds human oversight as an approval boundary and expects more red-teaming and auditability, which test those boundaries and reveal how systems behave around them. Together, these recommendations place governance inside the systems that authorize, observe and constrain execution.

Controls built into that architecture do not require CIOs to predict the intelligence of the next model generation. An agent can behave unexpectedly at its current capability level, so the surrounding system can constrain what that behavior is allowed to accomplish. As capability increases, external enforcement becomes more important because Hunt expects stronger agents to put greater pressure on weak safeguards.

Provider safety feeds into enterprise governance

External enforcement also marks the boundary between provider safety work and enterprise responsibility. Model providers can invest in evaluation and safeguards around model behavior, while a CIO connects the resulting model to company identities, data, tools, approval processes and operating systems. Provider practices can influence the first layer, but the deploying organization configures the second.

Gary Olliffe, distinguished vice president analyst at Gartner, makes that boundary explicit: “Embedded evaluation won’t be a ‘safety guarantee,’ so CIOs still need to plan for their own AI safeguards and governance investments.” Gartner sells technology research and advisory services to enterprise decision-makers, giving it a commercial stake in the governance questions on which Olliffe advises. His point applies directly to deployment architecture because a provider evaluation cannot certify every combination of permissions and workflows into which a model may be placed.

That boundary gives CIOs two connected sets of safety decisions. AI companies can evaluate model behavior and improve safeguards before and during delivery, while enterprises govern the identities, permissions, actions and human approvals available in their environments. Provider practices can reduce risks originating in model behavior, and enterprise controls limit the consequences created by each organization’s configuration.

Vendor selection should prioritize safety

With responsibility divided across those layers, procurement has to account for both. Su says CIOs were already cautious about agentic AI because governance, security, data quality and unclear ROI were holding back large-scale deployments. He expects the recent warnings from AI leaders to strengthen that caution and potentially move deployment decisions further into the future. That expectation connects frontier concerns directly to enterprise purchasing and deployment schedules.

The stronger caution Su anticipates extends from internal controls to partner selection. “CIOs will start to scrutinize their agentic AI plans, choose partners with a strong understanding of AI safety and security, and devote more resources to internal AI safety and security practices with human oversight,” he said. In that model, vendor safety practices complement the controls an enterprise maintains around its own deployment.

That relationship can change how CIOs rank models that already meet a use case’s functional requirements. “Best practice will be to adopt the safest agentic AI models rather than the smartest or most capable ones,” Su said. For a use case where several systems can perform the required work, stronger provider security and safety practices can carry substantial practical value alongside enterprise controls over identities, permissions, monitoring and human intervention.

Capability increases pressure on controls

Prioritizing control still leaves capability as a material risk variable. AI executives are asking for slower frontier development, and Hunt expects stronger agents may find ways around weak controls. Independent evaluation remains relevant for the same reason: a more capable agent may reveal weaknesses that a less capable system never reaches. Su’s expectation of increased CIO caution follows from that pressure.

The pressure from capability becomes operational through the authority attached to the agent. Current agents can reach sensitive records or initiate consequential transactions when organizations grant the necessary permissions, while Olliffe’s warning means embedded provider evaluation cannot be a “safety guarantee” for a particular enterprise deployment. As models become more capable, CIOs have stronger reasons to test and reinforce the systems that determine which data, tools and actions those models can reach.

Recap

CIOs do not need to resolve the debate over frontier AI development before making decisions about agentic AI. They do need to decide how much authority current systems should receive. As agents gain access to sensitive data, business applications and transactions, those deployment choices determine the consequences of model errors and unexpected behavior.

For business leaders, that makes autonomy a governance decision rather than simply a capability feature. Agent identity, least-privilege access, transaction limits, human approval and code-level enforcement should be defined alongside the business case for deployment. Provider evaluations and safety commitments can inform vendor selection, but they cannot replace controls designed for an organization’s own systems and workflows.

The practical standard is therefore not whether an agent is capable enough to operate autonomously, but whether the business can bound that autonomy safely. As model capabilities increase, enterprises with enforceable limits on access and action will be better positioned to adopt those capabilities without granting unnecessary authority.