Why smarter agentic AI starts with smaller models

Context matters more than model size

The largest AI model is rarely the best answer to every enterprise problem. The real constraint is context. An AI system needs to understand the customer, the process, past actions, business rules, and the decision it is being asked to make. Without this information, even a powerful model has to infer what is missing.

Ricky Thakrar, Head of Sales and Account Management at Zoho, described the company’s approach as “smaller, smarter, safer” at the CIO 100 Leadership Live New York event. The principle is straightforward. Use smaller models when the task allows it. Put more intelligence into the architecture around the model. Verify important decisions before an error becomes part of a workflow.

This changes how executives should think about AI investment. More model capacity does not automatically produce better business outcomes. Frontier models such as OpenAI’s GPT series, Anthropic’s Claude, and Google’s Gemini have broad reasoning capabilities. According to Thakrar, some of that capacity compensates for missing context. When a system does not know a company’s account history or internal processes, the model has to make more inferences.

A strong context layer reduces that burden. It gives the model relevant enterprise data before asking it to reason. A smaller model can then perform a narrow task because the system has already supplied much of what it needs to know. As Thakrar put it, “The intelligence moves from the model into the architecture around it.”

This has implications beyond cost. The architecture surrounding the model can contain knowledge specific to the company: business rules, customer history, permissions, workflow states, validation steps, and feedback from employees. That layer can become proprietary capability. The underlying model, by contrast, may be available to every competitor using the same AI provider.

The executive priority should therefore be context before model scale. Start by defining what information a task requires, where that information resides, and how reliably it can be retrieved. Then select enough model capacity to complete the task. Buying more intelligence before solving the context problem can increase spending without fixing the underlying weakness.

The same principle applies to safety. Validation is more useful when it occurs before an AI-generated decision becomes an irreversible action. Agentic systems can do more than produce text. They can update records, trigger workflows, contact customers, or interact with other software. The architecture must therefore control what the agent can do, what information it can use, and where a human or deterministic check is required.

This is why Thakrar argues that the model is the easier part of enterprise AI. The harder task is building a system in which models, enterprise context, business processes, and human judgment work together. That system can improve as employees add knowledge and correct errors. For enterprises, that accumulated context is likely to matter more than repeatedly switching to the newest model.

Zoho’s churn agent shows the cost of missing business context

One missing business rule was enough to damage trust in one of Zoho’s early AI agents within a week. The problem was that the agent had data but did not understand what that data meant inside Zoho’s business processes.

Zoho developed a churn-management agent for its account management team. When a customer subscription became inactive, the system gathered information from account notes, meeting recordings, and Zoho’s data-enrichment tools. It then summarized possible reasons for the apparent churn and scheduled a call.

The workflow looked useful. The underlying interpretation was wrong in important cases.

Zoho’s CRM records a bundle of products as a single line item. When an existing customer moves individually purchased products into that bundle, the old individual subscriptions change to inactive. The first version of the agent treated those inactive records as evidence of customer churn. In reality, the customer had consolidated products rather than left.

The error had an immediate organizational effect. “Within a week my team no longer trusted that agent,” Thakrar said. Zoho corrected the bundle problem in the second version, but another missing piece of context soon appeared.

Some customers initially buy Zoho products for pilots or sandbox environments. When they progress to a live deployment, they shut down the pilot versions. The CRM again records subscriptions becoming inactive. The underlying event can indicate customer progression, but an agent focused on the subscription status can interpret it as churn.

A third case made the problem clearer. Account managers sometimes determine that one Zoho product is not the right fit for a customer and recommend another product. The resulting churn is deliberate. It is not evidence that the account is unexpectedly at risk. Correct classification therefore requires knowledge about the account manager’s intent.

These failures identify the central problem with enterprise AI: data access is not the same as business understanding. A database can show that a subscription became inactive. Determining why it became inactive may require knowledge spread across CRM records, conversations, product configurations, sales processes, and employee decisions.

Trust also has an economic value. Almost a year into the churn-agent initiative, Thakrar said his team was still checking the agent’s work independently. The system had made employees more aware of potential churn, but the expected productivity gain had not yet arrived. If an employee must repeat the research to confirm an AI output, automation has not removed much work.

That lesson should affect how executives measure agent deployments. Initial accuracy is only one metric. Teams should also measure false-positive rates, the share of outputs employees verify manually, time spent on verification, and whether trust improves after corrections. An agent that technically completes a task but triggers a second manual process may simply move work rather than eliminate it.

Zoho’s examples were failures of missing context and business rules. A more capable model cannot reliably infer an internal CRM convention it has never been given. The practical response is to capture those conventions, feed them into the workflow, and use employee corrections to improve the context available to future decisions.

Thakrar’s current objective is broader: create an AI-assisted customer journey from sales through account management in which customer context survives the handoff. Information gathered during the sales process should remain available and useful after the customer becomes an account.

For executives, this is the more important AI investment. Before automating a business decision, map the information humans actually use to make it. Include exceptions, process transitions, historical decisions, and intent. Enterprise agents become valuable when they understand the operational meaning of the data they process.

Intelligent routing can cut AI costs by matching the model to the task

A 3-billion-parameter model can reduce costs by as much as 95% for some workloads, according to Ricky Thakrar, Head of Sales and Account Management at Zoho. His point is not that smaller models are universally better. It is that most enterprise tasks do not need frontier-level computing power.

The cost problem starts when companies send every request to their most capable model. Frontier models provide broad reasoning capacity, but that capacity comes at a higher price. Thakrar warned that companies can consume a full year’s planned token budget within months when they use these models indiscriminately.

Routing addresses this problem. It is the process of selecting the right model for each workload. A simple classification or validation task might go to a small model. A difficult task that requires substantial reasoning can go to a frontier model. Model capacity becomes a resource assigned according to need rather than a default setting.

Context makes this approach practical. A model often needs additional reasoning power because key information is missing or poorly structured. If the system supplies relevant customer history, process state, business rules, and task-specific instructions first, a smaller model may be able to produce the required result reliably.

Thakrar said Zoho has observed cases among clients where using a 3-billion-parameter model could save 95%. “You don’t always have to pay the frontier guys for every task,” he said. The 95% figure is an observation from Zoho’s client work, not a general benchmark for all enterprise workloads. Actual savings will depend on model pricing, task volume, context size, latency requirements, and how often requests must be escalated to a larger model.

For executives, routing should therefore be treated as both an engineering and financial control. The objective is not to maximize the percentage of requests handled by small models. It is to use the least expensive model that can consistently meet the required level of accuracy, speed, and risk control.

That requires measurement. Enterprises need to compare model performance by task, monitor failure rates, and define when a request should move to a more capable model. High-risk decisions may justify higher model costs even when a smaller model performs adequately on average. Low-risk, repetitive tasks offer a stronger case for aggressive cost optimization.

This approach also reduces dependence on a single AI provider. An effective routing layer can direct workloads across different models based on their capabilities, cost, availability, or enterprise requirements. That gives companies more control as model performance and pricing change.

The business case is clear. Model scale should follow workload complexity. Companies that establish this discipline can reserve expensive reasoning capacity for the tasks that need it and reduce the cost of routine AI operations.

The agent harness can matter more than the underlying model

A language model can generate an answer. An enterprise agent must do more. It may need to retrieve records, use software tools, remember prior activity, follow permissions, validate results, and execute an action. The infrastructure that coordinates these functions is often called the agent harness or orchestration layer.

Thakrar argues that this layer has greater practical importance than raw model capability. “It’s the process around the model, which matters way more than the model itself,” he said.

The harness determines what information reaches the model and what happens after the model produces a result. It can connect an agent to CRM systems, databases, internal applications, memory, tools, execution environments, and security controls. It can also restrict which actions an agent is allowed to perform and require verification before consequential actions proceed.

This distinction matters because language models are probabilistic. The same system can produce different responses to similar inputs, particularly when a task requires interpretation or judgment. Enterprise processes often contain steps where this variability is acceptable and other steps where it is not.

Thakrar separates these workloads into deterministic and non-deterministic elements. Machines are well suited to tasks such as reading, organizing, checking, and validating structured information. Judgment, synthesis, and decisions involving ambiguous context are less deterministic. Those stages require stronger controls and, in some cases, human review.

The architecture should reflect that distinction. A company does not need an AI model to freely reason through every stage of a workflow. Fixed business rules and conventional software can handle steps where the correct action is known in advance. AI can then focus on the parts that genuinely require interpretation. This reduces unnecessary model calls and creates clearer points for testing and control.

According to Thakrar, benchmark tests have shown that a less powerful model with a superior harness can outperform a much larger model operating through a weaker harness.

The underlying principle remains important for executive planning. Replacing one model with a newer model may improve a benchmark score. It does not automatically fix weak data retrieval, incorrect business rules, poor tool integration, missing permissions, or badly designed workflows. Those are architecture problems.

A well-designed harness also creates a more durable enterprise asset. Models can be changed as prices and capabilities improve, while the company retains the workflows, context, controls, and system integrations it has developed. This reduces the strategic importance of committing to any one model provider.

Governance belongs in this layer as well. An agent that can modify a customer record or trigger another system introduces different risks from a chatbot that only generates text. Enterprises need explicit permissions, validation rules, audit records, and escalation paths. Critical errors should be intercepted before an action becomes difficult to reverse.

For the C-suite, the investment priority should therefore extend beyond model procurement. The larger engineering task is to build the infrastructure that controls how models receive context, use tools, make decisions, and take action. Model capability will continue to improve. The quality of the enterprise system around that model will determine how much of that capability becomes reliable business value.

Human judgment belongs at the points where AI faces real ambiguity

Full automation was the wrong target for Zoho’s stakeholder-mapping agent. The system could process meeting information and external data quickly, but it struggled with the part of the task that required judgment. Zoho therefore changed the workflow instead of trying to eliminate the human role.

Ricky Thakrar, Head of Sales and Account Management at Zoho, described the agent as part of the company’s sales process. It uses context from an initial customer meeting together with third-party enriched data, such as a LinkedIn profile. It then assesses probabilities and proposes a stakeholder map identifying the people likely to influence a purchase.

The initial objective was complete automation. “The initial goal was just to eliminate that task completely from the human workflow,” Thakrar said. The agent would create the stakeholder map, save it, and leave it ready for the sales team.

The problem was nuance. Job titles do not have consistent meanings across companies. Two people with the same title may have very different authority, budget control, technical influence, or access to senior decision-makers. Meeting dynamics can provide additional information that is difficult to capture in structured data. An experienced salesperson may recognize these signals because they understand the conversation and the organization involved.

Zoho could have continued supplying more data and instructions in an attempt to automate these judgments. Instead, it redesigned the process. The AI produces a proposed stakeholder map. A human reviews it, makes corrections, and explains why those changes are required.

This division of work preserves much of the efficiency benefit. Thakrar said the agent’s maps were usually close enough to save employees time. Human review then handles the smaller set of decisions where contextual judgment matters most. The corrections also create richer context that can improve subsequent AI outputs.

For executives, the important design question is not whether a process can be fully automated. It is where uncertainty becomes material to the business. A low-confidence classification may be acceptable when it only changes how information is organized. The same level of uncertainty may be unacceptable when an agent is identifying a key decision-maker, changing a customer record, approving a transaction, or initiating an external action.

Human review should therefore be placed according to consequence and uncertainty. Deterministic steps with explicit rules are strong candidates for automation. Ambiguous decisions with meaningful commercial, financial, legal, or reputational consequences may require approval or review.

This approach also provides a practical way to improve agents. Human corrections can reveal information that was absent from the original context, expose recurring failure patterns, and identify business rules that should become part of the system. The organization should capture that feedback systematically rather than treating each correction as an isolated event.

There is a management constraint to consider. Human oversight has a cost. Requiring approval for every AI action can remove much of the productivity benefit, as Zoho’s churn-agent experience showed. The goal is therefore targeted intervention: automate predictable work, identify high-value uncertainty, and require people to focus on decisions where their judgment changes the outcome.

Thakrar calls this a “human harness.” For the C-suite, the broader lesson is clear. Human involvement should be an intentional part of agent architecture. The strongest workflows use people where judgment creates measurable value and use automation everywhere else that can be controlled reliably.

Specialization turns successful AI experiments into lower-cost production systems

Once an agent can reliably perform a narrow task, continuing to use a large general-purpose model may be unnecessary. Zoho’s third architectural pillar, specialization, addresses that cost and control problem.

Thakrar defines specialization as moving a proven process from a frontier model to a smaller, narrower model for production. Frontier models can be useful during development because they offer broad reasoning capabilities. They help teams establish whether an AI-enabled workflow can work before engineers optimize the underlying system.

Production changes the economics. A stable enterprise task may run thousands or millions of times. Once its inputs, outputs, exceptions, and quality requirements are understood, broad model capability can become an expense that delivers little additional value.

The objective is to capture the judgment and pattern recognition of subject-matter experts within a more focused system. Thakrar advocates using open-weight or open-source models that enterprises can train or fine-tune for specific workloads and deploy in their own data centers. This can give organizations tighter control over model behavior, infrastructure, and sensitive enterprise information.

Specialization should not, however, be interpreted as a requirement to train a model from the ground up for every process. Fine-tuning, task-specific instructions, retrieval of enterprise context, deterministic rules, and smaller existing models can all play different roles. The appropriate approach depends on workload volume, performance requirements, infrastructure costs, data sensitivity, and the frequency with which the task changes.

There is also an operational trade-off. Running models internally can provide greater control, but it transfers responsibility to the enterprise. The company must manage infrastructure, deployment, monitoring, security, model updates, evaluation, and specialist skills. For some workloads, buying model capacity through an external provider may remain more economical. Ownership creates value only when the workload and strategic importance justify the additional operating burden.

Thakrar places particular emphasis on the orchestration layer because it contains the company’s workflows and accumulated knowledge. “The true enterprise bet is to keep that orchestration layer, which is your IP and knowledge, in house,” he said. He also argued that enterprises should aim to “run, train, and host their own models.”

Executives should distinguish that strategic recommendation from a universal requirement. Smaller enterprises, low-volume workloads, and rapidly changing applications can favor managed services. High-volume or sensitive workloads may create a stronger case for greater internal control.

The more durable point is ownership of business knowledge. Customer context, process logic, expert corrections, validation rules, and workflow design can be more specific to the enterprise than the underlying model. Preserving control of those assets reduces the risk that an organization’s AI capability becomes inseparable from one external model provider.

Specialization can therefore serve three objectives: lower inference costs, greater operational control, and retention of enterprise knowledge. It is most valuable after a workflow has been proven and measured. First establish that the agent performs useful work. Then determine how much model capability that work actually requires. The production system can be optimized around that evidence rather than around the assumption that a larger model is always better.

Routing, harness, and specialization form a practical architecture for enterprise AI

Enterprise AI performance is not determined by the model alone. Zoho’s experience points to three architectural controls that matter more in production: routing, harness, and specialization. Together, they determine which model handles a task, what context and controls surround it, and whether a proven workload should continue using an expensive general-purpose model.

Ricky Thakrar, Head of Sales and Account Management at Zoho, argues that constrained, context-rich systems can outperform expensive models added to fragmented enterprise processes. His position follows directly from Zoho’s experience with its own agents. A capable model could not compensate for missing customer context in the churn workflow. More complete architecture was required.

Routing is the first control. It sends each workload to a model suited to its complexity. Routine tasks can run on smaller models, while difficult or uncertain requests can move to more capable systems. This matters financially. Thakrar said Zoho has seen cases among clients where a 3-billion-parameter model could reduce costs by as much as 95%. He also warned that indiscriminate use of frontier models can consume a planned annual token budget within months.

The 95% figure should not be treated as an industry-wide saving. Enterprises should test accuracy, latency, failure rates, and total cost before deciding where smaller models are appropriate.

The second control is the harness, also called the orchestration layer. It connects the model to enterprise data, memory, software tools, guardrails, and execution systems. It also determines which parts of a process are handled through fixed rules, which require AI reasoning, and where human approval is necessary.

This layer is critical for agentic AI because agents can take actions. A system that changes CRM records, sends communications, or triggers another application requires stronger controls than a system that only drafts text. Permissions, validation, logging, escalation, and human review therefore need to be designed into the workflow before deployment.

Specialization is the third control. Once an agent has demonstrated that it can perform a defined task reliably, the organization can evaluate whether the workload still needs a frontier model. A smaller, task-focused model may deliver the required performance at lower cost. Enterprises can also fine-tune or operate models around their own data, processes, and expert knowledge where the economics and security requirements justify doing so.

Thakrar connects specialization to ownership of intellectual property. “The true enterprise bet is to keep that orchestration layer, which is your IP and knowledge, in house,” he said. His position is that enterprises should ultimately seek the capability to run, train, and host their own models.

That recommendation requires a business case. Self-hosting can improve control over data, deployment, and model behavior, but it also creates infrastructure, security, monitoring, and talent costs. Managed models may remain the better choice for many workloads. What matters strategically is retaining control of the enterprise-specific context, rules, evaluations, and workflows that determine how the models are used.

The three pillars also create a clear operating sequence. First, give the system enough context to understand the task. Route the task to the least expensive model capable of meeting the required performance. Use the harness to constrain execution, validate important outputs, and introduce human judgment where uncertainty has material consequences. Once the process is stable, assess whether specialization can reduce cost or increase control.

This changes what the C-suite should measure. Model benchmark scores are not enough. Leaders need workload-level measures of cost, accuracy, latency, human review, error rates, and successful task completion. They should also track whether agents actually remove work. Zoho’s churn-agent experience showed why: an agent can technically generate the required output while employees still repeat its research because they do not trust it.

The strategic advantage therefore sits increasingly in system design. Models will continue to improve and become more interchangeable for many tasks. Enterprise context, workflow design, human feedback, validation rules, and operating knowledge are harder to replicate.

That is the central logic behind Zoho’s “smaller, smarter, safer” position. Use model capacity selectively. Put enterprise intelligence into the surrounding architecture. Verify consequential decisions before they become actions. Routing, harness, and specialization provide the technical structure for putting those principles into production.

Recap

The enterprise AI constraint is no longer access to powerful models. It is the ability to turn those models into reliable systems. More parameters cannot fix missing business context, weak workflows, poor controls, or employees who do not trust the output.

That changes the investment priority. Build the context layer first. Route each workload to the least expensive model that meets its requirements. Put validation, permissions, and human judgment at the points where mistakes carry real consequences. Once a workload is proven, determine whether a smaller, specialized model can run it at lower cost.

Executives should also measure outcomes beyond model accuracy. Track cost per completed task, error and escalation rates, human verification time, latency, and work actually removed from the process. An agent that produces an answer but forces employees to check everything again has not delivered meaningful automation.

The durable asset is the enterprise knowledge surrounding the model. Customer context, business rules, workflows, expert corrections, and orchestration are specific to the company. Models will change. Providers will change. That institutional knowledge should remain under enterprise control.

The goal is not maximum AI capability in every workflow. It is enough capability, supplied with the right context and governed by the right architecture. That is how agentic AI becomes smaller, smarter, safer, and more useful to the business.