Why AI agent governance is becoming the next enterprise priority

Strong AI governance is essential for controlling access to sensitive enterprise data

AI agents become valuable only when they can work with real business information. That creates a new challenge. The agent cannot make reliable decisions if it lacks access to the data it needs, but unrestricted access creates unacceptable business risk. The real objective is not to give agents more data. It is to give them the right data at the right time under strict control.

Google Cloud highlighted this issue with an internal example involving two generations of tensor processing units (TPUs), codenamed Viperfish and Ghostfish. If an employee asks an AI agent to calculate the financial conversion ratio between the two systems, the agent cannot invent an answer. It needs access to Google’s internal pricing rate card. That rate card is highly sensitive corporate information, so the question is no longer whether the AI is intelligent enough. The question is whether the organization can safely authorize the AI to use that information.

This changes how executives should think about enterprise AI. The competitive advantage does not come only from deploying more powerful language models. It comes from building governance that allows those models to operate inside the business without exposing confidential information. Identity management, permissions, audit trails, and observability become business capabilities rather than purely technical functions. Every action performed by an AI agent should be attributable, reviewable, and governed according to corporate policy.

The next generation of AI adoption will be determined less by model performance and more by organizational trust. If leadership cannot explain who gave an agent access to sensitive information, why it received that access, and what it did with the data, large-scale deployment becomes difficult regardless of how capable the underlying AI may be.

Michael Gerstenhaber, Vice-President of Product Management for Agent Platform at Google Cloud, summarized this position clearly. He said, “It’s only through identity, permissioning, audit and observability that I’ll ever be comfortable giving my virtual employee access to sensitive data – because that’s how we treat real employees.”

AI agents should be designed to be secure by default with centralized policies that prevent unauthorized data exfiltration

Security cannot be added after AI agents are deployed. It has to be built into every layer of the system from the beginning. As organizations give AI agents more autonomy, they also increase the potential impact of mistakes or malicious behavior. The solution is to assume that errors will happen and design systems that prevent those errors from becoming security incidents.

Google Cloud’s approach is based on centralized governance. Administrators define organization-wide policies that apply to every AI agent regardless of the task it performs. This creates consistent security across the enterprise instead of relying on individual application developers to implement their own protections. If an AI agent attempts to perform an action outside its approved permissions, the policy blocks it automatically.

The company’s architecture supports this through several coordinated components, including an agent gateway, dedicated agent registries, skills libraries, and a Model Context Protocol (MCP) registry. These components create multiple control points before an agent reaches sensitive systems. Even if an agent builds an incorrect workflow or receives misleading instructions, enterprise policies remain the final authority over what it can actually access.

This approach also improves operational consistency. Sensitive databases, confidential financial records, and regulated information should always be accessed through approved processes. Standardizing these interactions reduces operational risk, simplifies compliance audits, and makes security policies easier to maintain as AI deployments expand across the business.

Executives should view this as a governance investment rather than simply a cybersecurity investment. As AI agents become integrated into finance, operations, engineering, customer support, and legal functions, centralized policy management becomes essential for maintaining control at enterprise scale. Organizations that establish these foundations early will be able to expand AI adoption with greater confidence and fewer operational disruptions.

Michael Gerstenhaber, Vice-President of Product Management for Agent Platform at Google Cloud, emphasized this philosophy by stating, “An employee should have to show good judgement, and that means an agent should have to show good judgement.” He added that while employees should be accountable for malicious attempts to exfiltrate data, organizations should also make it “very, very hard to exfiltrate data” even when malicious intent is absent.

AI governance must differentiate between managing digital worker identities and human identities

Many organizations describe AI agents as digital employees. That is a useful way to think about their role, but it is not a useful way to manage their risk. AI agents and people behave differently, operate at different speeds, and create different security challenges. Governance must reflect those differences from the start.

A human employee has natural limits. An AI agent does not. It can perform thousands of tasks simultaneously, interact with multiple systems at once, and scale almost instantly across the organization. This changes the potential impact of a single permission decision. A permission that seems reasonable for one task could create unnecessary exposure when multiplied across hundreds or thousands of automated activities.

This is why access management must become much more granular. Instead of giving an AI agent broad access because it supports a particular department, organizations should authorize only the exact resources required to complete a specific task. An agent may receive temporary access to one confidential document while remaining blocked from other sensitive files, even if the human manager supervising the work has broader clearance. This follows the principle of least privilege, where systems receive only the minimum access necessary to perform their responsibilities.

Business leaders should also recognize that AI identity management is becoming its own discipline. Existing identity and access management platforms will need to evolve to support non-human identities at enterprise scale. Policies must cover how agents are created, authenticated, monitored, updated, and eventually disabled when they are no longer needed. Governance frameworks designed only for employees will not provide sufficient visibility or control over autonomous software.

The organizations that succeed with agentic AI will be those that treat AI identities as first-class enterprise assets. Every AI agent should have a clearly defined identity, explicit permissions, continuous monitoring, and a complete audit history. This creates accountability while allowing organizations to scale automation with confidence.

Michael Gerstenhaber, Vice-President of Product Management for Agent Platform at Google Cloud, highlighted this distinction by explaining that agents are “infinitely scalable” while people are not, and that agents “are not afraid of getting fired.” He argued that these differences require organizations to assign permissions differently because the level of judgment expected from AI systems is fundamentally different from that expected of employees.

Robust observability and independent security monitoring are critical for trusted AI agent operations

As AI agents become more autonomous, organizations need clear visibility into every important decision they make. Without that visibility, it becomes difficult to investigate errors, demonstrate compliance, or understand why an agent produced a particular outcome. Observability is therefore becoming a core requirement for enterprise AI.

Google Cloud argues that organizations should capture the complete execution history of an AI agent. This includes the sequence of actions it performed, the tools it selected, the permissions it used during each request, and the information it accessed. This detailed record allows administrators to audit decisions, identify failures, and improve agent behavior over time. It also creates evidence that supports internal governance and external regulatory requirements.

Michael Gerstenhaber drew on his previous experience at observability company Datadog to explain how distributed tracing techniques can be applied to AI workflows. These techniques allow organizations to reconstruct an agent’s execution path from beginning to end. However, he also emphasized that presenting this information requires careful design. Technical teams may need detailed traces, while business leaders require clear summaries that explain what happened without unnecessary complexity. Effective observability serves both audiences.

Google Cloud also separates security monitoring from the agent itself through Model Armor. According to the company, Model Armor operates out of band, meaning it monitors interactions independently of the engineers who developed the agent. It analyzes activity between the application programming interface (API) that generates the initial prompt and the inference process that produces the final output. This independent monitoring helps detect prompt injection attacks and toxic content while reducing the risk that internal changes to the agent could weaken security controls.

For executives, independent monitoring should be viewed as part of enterprise risk management. Regulatory expectations around AI transparency are increasing, and customers increasingly expect organizations to explain how AI systems make decisions. Strong observability improves operational resilience, supports compliance efforts, and builds confidence among employees, customers, and regulators. As AI becomes responsible for more business-critical processes, the ability to explain its actions will become just as important as the quality of its outputs.

Michael Gerstenhaber, Vice-President of Product Management for Agent Platform at Google Cloud, emphasized that organizations must make observability useful for business users as well as technical teams, noting that “The difficulty is not scaring the person who’s trying to interpret it.”

AI agents can continuously improve during runtime, reducing the need for traditional decommissioning

Most enterprise software follows a familiar lifecycle. It is deployed, updated periodically, and eventually replaced. AI agents introduce a different operating model. Rather than retiring an agent when its performance declines, organizations can improve its behavior continuously while it remains in production.

Google Cloud argues that the foundation model powering an AI agent remains unchanged after release, but the agent’s operational behavior does not have to remain static. By collecting observability data from real interactions, organizations can identify poor decisions, incorrect responses, or inefficient workflows. Human reviewers or automated evaluators powered by large language models can then provide corrective feedback that updates the agent’s memory and influences future decisions without modifying the underlying foundation model.

This creates a continuous improvement cycle. Every interaction becomes an opportunity to refine performance, increase precision, and reduce recurring errors. Instead of waiting for major software releases or retraining cycles, organizations can improve agents incrementally as business requirements evolve. This is particularly valuable in environments where regulations, internal policies, or operational processes change frequently.

For executives, this changes how AI investments should be managed. The focus shifts from periodic system replacement toward continuous operational optimization. Success depends on establishing strong governance around feedback quality, evaluation standards, and change management. Organizations also need clear approval processes to ensure that runtime improvements remain aligned with business objectives, compliance requirements, and organizational policies.

Continuous learning should not be interpreted as unrestricted self-modification. Business leaders still need governance over what feedback is accepted, who can authorize behavioral changes, and how those changes are validated before influencing future decisions. Maintaining human oversight remains essential, particularly for AI systems supporting regulated industries or high-impact business processes.

Michael Gerstenhaber, Vice-President of Product Management for Agent Platform at Google Cloud, said, “You don’t even have to decommission it, really.” He explained that observability traces, combined with feedback from human managers or automated large language model “judges,” allow agents to improve their runtime behavior even though “the model itself is trained within the same generation.”

Governed autonomous AI agents enable “elastic intelligence,” transforming how enterprises scale operational capacity

The long-term objective of enterprise AI is not simply to automate individual tasks. It is to increase the organization’s ability to execute complex work without being constrained by available human capacity. Google Cloud describes this vision as “elastic intelligence,” where governed AI agents can expand operational capability as business demand changes.

This requires organizations to move beyond assigning detailed instructions for every activity. Instead, employees define business objectives while AI agents determine how to complete approved tasks within established governance and permission frameworks. The quality of this approach depends on trust. Organizations must be confident that agents will operate within approved boundaries before giving them greater autonomy.

Google Cloud is developing Gemini Spark, a personal AI agent intended to operate with a higher degree of autonomy. Rather than requiring constant supervision, Spark is designed to carry out work based on broader business goals. This reflects a broader industry direction in which AI agents become persistent contributors to business operations instead of tools that respond only to individual prompts.

For executives, this represents a significant shift in workforce strategy. AI agents should not be viewed solely as productivity tools that reduce manual effort. They can become part of the organization’s operating model, supporting functions across finance, customer service, engineering, sales, and operations. As these capabilities mature, leadership teams will need governance frameworks that address accountability, performance measurement, security, and compliance alongside traditional workforce planning.

The organizations that gain the greatest value from autonomous AI will be those that establish trust before pursuing scale. Governance, identity management, observability, and centralized policy enforcement are not barriers to innovation. They are the foundations that make broader AI adoption possible. Without those capabilities, organizations will struggle to delegate increasingly complex work to autonomous systems regardless of how advanced the underlying models become.

Michael Gerstenhaber, Vice-President of Product Management for Agent Platform at Google Cloud, described this vision by saying, “For that to happen, you have to get comfortable giving it permissions upfront so that it can run autonomously.” He added that once this trust is established, enterprises gain a workforce that “never gets bored, never sleeps and can do a lot of highly complex work on my behalf all the time.”

Main highlights

• Build governance before scaling AI: AI agents can only deliver business value if organizations trust them with sensitive data. Leaders should prioritize identity, permissions, auditing, and observability as core capabilities for enterprise AI deployment.
• Make security the default operating model: Autonomous AI requires centralized policies that consistently enforce access controls across every agent. Embedding security into the platform reduces the risk of data leakage as AI adoption grows.
• Treat AI identities differently from human identities: AI agents have unique risk profiles because they can scale rapidly and operate continuously. Leaders should implement granular, task-based permissions and dedicated governance for non-human identities.
• Invest in observability to build trust: Organizations need clear visibility into how AI agents make decisions and access data. Strong observability and independent monitoring improve compliance, simplify investigations, and increase executive confidence in autonomous systems.
• Shift from replacement to continuous improvement: AI agents can become more effective through ongoing feedback without replacing the underlying foundation model. Leaders should establish structured review and governance processes that continuously improve agent performance while maintaining oversight.
• Use governance to unlock autonomous AI at scale: Long-term value comes from allowing trusted AI agents to execute business objectives with greater autonomy. Organizations that establish strong governance foundations today will be better positioned to scale AI-driven operations safely and efficiently.