How much autonomy should AI agents have

The agent-autonomy question is really an authority question

Requiring human approval for every five-cent purchase can make a transactional AI agent inefficient. Giving the same agent permission to buy a $2,000 analyst report creates greater financial exposure. The key question is how much authority an agent should receive relative to the judgment it has demonstrated.

Agents can be designed to pay for APIs, data, computing resources, and content. AWS describes AgentCore Payments as a way for agents to discover and pay for APIs, Model Context Protocol servers, and paywalled content during a task. LangChain describes middleware for the same capability.

Moving money is straightforward once payment access exists. Delegating the decision to spend creates a separate governance problem.

Capability, judgment, and authority are three different things

Executives evaluating agents need to separate three concepts. Capability is what an agent can technically do. Judgment is how reliably it decides what to do. Authority is what surrounding systems permit it to do.

A traditional application can execute a payment when developer-written conditions are met. An agent can have a broader role: choose a service, assess its value to a goal, evaluate the price, and propose or initiate a purchase. The agent then has discretion over decisions that application logic might otherwise specify in advance.

Capability alone is insufficient evidence of decision quality. A system can perform an action yet fail to recognize consequences that matter to its operator.

Authority is separate because an organization can restrict what happens after the model makes a decision. An agent may decide that an expensive purchase is worthwhile while payment infrastructure rejects it. It can also stay within its authorized budget and spend that budget poorly.

This distinction changes how autonomy should be governed. Improvements in model capability require separate evidence of judgment. Access to a system establishes permission, while the surrounding infrastructure determines how much real-world consequence an agent’s decisions can create.

Put hard boundaries outside the model

Consider a research agent preparing a competitive analysis. It discovers a paid industry report, a market-data API, and several paywalled articles. Approval for every small transaction can consume some of the time saved through delegation. Unrestricted purchasing can expose the organization to uncontrolled spending.

A third design gives the agent bounded delegated authority. The instruction might be: spend up to $10 for this task, spend no more than $2 with any single provider, buy only research-related services, stop when the authorization expires, and return the receipts.

AWS describes AgentCore Payments as supporting this architecture. AWS has a commercial interest in adoption of AgentCore Payments, so this is a vendor claim. AWS describes an agent’s spending session as having a maximum amount and an expiration time, with those limits enforced deterministically outside the model. AWS also describes transactions as producing logs and audit trails.

Here, “deterministically” means a fixed rule decides whether an action is allowed rather than leaving that decision to the language model. The model can propose an action based on its context and objective. A separate policy layer decides whether that action falls within the organization’s authorization.

This separation also matters for prompt injection, an attack in which malicious or untrusted input manipulates the instructions an agent follows. An injection could alter what the model attempts to purchase. An independently enforced transaction limit can still reject a purchase outside its configured authorization, provided that control remains outside the model’s reach.

Delegated spending policies can cover a total amount, permitted purposes, approved providers, expiration, and receipt requirements. They also create requirements for identity, authorization, revocation, and audit records. Within those permissions, the agent retains discretion over what to buy.

The principle extends beyond payments. When the consequence of an action can be bounded mechanically, infrastructure can enforce that boundary independently of the model’s reasoning. Better prompting can influence behavior. Deterministic enforcement defines which proposed actions the surrounding system will authorize.

Guardrails limit failure; judgment needs separate evidence

An agent can stay inside every resource boundary and still make poor decisions. Compliance with a spending limit shows that the limit worked. Useful purchasing decisions require separate evidence.

Anthropic’s Jack Clark describes a limitation in current systems: “There’s a certain absence of valuable, intuitive creativity in today’s AI systems.” He adds, “Though they’re extraordinarily capable engineers, they [also] seem to have a certain property of rote, formulaic thinking that might prevent them [from] being good researchers.”

Clark’s observation illustrates the distinction between technical capability and judgment. Anthropic develops AI systems and therefore has a commercial stake in how the capabilities and limitations of such systems are understood. Clark’s characterization should be read in that context.

Safety metrics and performance metrics answer different questions. A spending cap can establish whether an agent exceeded its financial authorization, and an audit trail can record what it purchased. Evaluations must determine whether the agent recognized useful opportunities, selected appropriate resources, and made effective trade-offs.

Guardrails bound the dimensions they are designed to constrain. Judgment requires separate evidence. An organization needs to test whether an agent uses its discretion effectively in the work it plans to delegate.

Broad measures of model capability are therefore an incomplete basis for granting authority. Engineering performance demonstrates engineering performance. Research judgment requires evidence from research tasks. Authority should follow evidence from the context in which consequential decisions will occur.

Match authority to the consequences of being wrong

A practical governance approach classifies actions by their consequences. An organization can grant wider discretion when actions are inexpensive, observable, and easy to reverse. Controls can tighten as actions become more costly, less observable, or harder to undo. Human approval can remain part of execution when consequences exceed the organization’s delegated limits.

This creates a deployment path based on evidence rather than a binary label such as “autonomous.” A research agent could begin with a small task-level budget and narrow provider permissions. The organization could expand those permissions after evaluations and production results show that the agent selects useful resources, follows the intended purpose, and produces results worth the cost.

The same logic applies beyond purchasing. For an API call, infrastructure change, outbound communication, or access decision, the relevant question is how much consequence the agent can create before an independent control intervenes. Organizations can change those permissions without changing the underlying model.

That separation is operationally useful. A model upgrade can change capability, while authority remains independently controlled. Organizations can adjust it according to observed performance and the consequences they are prepared to authorize.

Key takeaways for leaders

• Separate capability, judgment, and authority: An agent’s ability to perform an action does not establish that it can make good decisions about when to act. Leaders should grant authority based on demonstrated judgment in the specific context where decisions will have consequences.
• Put hard boundaries outside the model: Use deterministic infrastructure to enforce spending limits, approved purposes, provider restrictions, expiration, and other authorization rules. This lets agents exercise discretion without relying on the model itself to enforce critical safeguards.
• Evaluate judgment separately from guardrails: Staying within a spending limit shows that the control worked, not that the spending was useful. Leaders should use task-specific evaluations to determine whether agents make effective choices and trade-offs within their delegated authority.
• Match authority to the consequences of failure: Give agents greater discretion when actions are inexpensive, observable, and reversible, and tighten controls as consequences increase. Expand authority gradually as evaluations and production results demonstrate reliable judgment.