How companies are cutting costs in the AI token rush

The surge in generative AI usage is driving unprecedented token consumption and costs

Generative AI has become one of the fastest-growing technologies in corporate use today. Companies across industries are embedding it into their products, workflows, and decision-making systems. But with adoption comes scale, and cost. Tokens, the small data units AI models use to process language and reasoning, measure both the scale and price of this revolution. Every prompt, document, or command generates thousands, sometimes millions, of these tokens. The result is a dramatic spike in infrastructure spend that many executives did not anticipate.

Sundar Pichai, CEO of Google, described tokens as the “fundamental units of data our models process.” Google now handles around 3.2 quadrillion tokens a month. That’s a massive operational load, even for one of the most capable AI infrastructure systems in the world. For enterprises still experimenting with AI, this kind of scale quickly turns into budget strain. Some organizations have already faced multi-hundred-million-dollar token bills simply because usage was not tightly monitored across teams. The trend shows that AI cost control now depends as much on architecture and process as on innovation.

C-suite leaders need to recognize that token management is a core business issue. Left unchecked, token consumption grows exponentially as employees, partners, and systems rely more on automated reasoning. Companies allocating capital to AI must invest in usage visibility, model efficiency, and predictive budgeting tools. Minimizing unnecessary token generation is becoming just as critical as managing cloud storage or bandwidth. Those who understand and optimize this layer early will control AI economics far better as competition accelerates.

Switching to lower-cost AI models can offer substantial cost savings without sacrificing performance

Every corporation wants frontier-level AI performance, but not every use case needs it. Task complexity should match model capability. Executives are learning that the difference between a high-end and a cost-optimized model often lies in marginal accuracy, while cost differences can be enormous. Selecting the proper model for each application has become a strategic lever for operational efficiency.

Sundar Pichai explained this clearly when he introduced Gemini 3.5 Flash, a less expensive model that offers near-frontier performance at under half the price of Google’s flagship systems. Businesses can mix Flash with more advanced models as needed, allowing consistent performance while cutting token expenses significantly. Deepak Seth, Senior Director Analyst at Gartner, noted that organizations often use models trained on vast literary datasets when a simpler model could handle the task flawlessly. This type of overprovisioning wastes computational resources, and money.

Steven Dickens, Principal Analyst at Hyperframe Research, illustrated this shift from personal experience. He relies on Amazon’s low-cost model Quick for day-to-day tasks, describing the subscription as a “great personal ROI.” By relying on smaller models for general tasks and stronger systems only where complex reasoning or synthesis is needed, enterprises can reduce AI costs while maintaining, or even improving, agility.

For C-suite executives, the lesson here is strategic resource allocation. AI is not one-size-fits-all. Just as with energy sources or manufacturing tools, choosing the right model for the right job can dramatically improve return on investment. Cost-effective models like Gemini Flash or Amazon Quick represent a more sustainable path to scale AI while keeping budgets under control. The most efficient AI systems of the next decade will not be the largest, but the best matched to their purpose.

Enhancing infrastructure through hardware optimization and caching can significantly lower the token load

The next phase in controlling AI costs goes beyond model selection, it’s about optimizing the systems that support them. Tokens are consumed not just by the model itself but by every data interaction between applications, APIs, and enterprise systems. Companies are discovering that inefficient data flow leads to a massive increase in token usage. Optimizing infrastructure to reduce redundancy and streamline access can dramatically lower overall expenses without affecting output quality.

Dheeraj Pandey, CEO of DevRev, explained that this challenge mirrors the early stages of cloud adoption, where uncontrolled scaling led to chaotic spending before standardization stabilized costs. DevRev’s approach aims to solve this by inserting a memory layer between AI agents and enterprise systems like Salesforce and ERP databases. This layer stores high-frequency query data and manages it through cheaper CPUs rather than costly GPUs. The result is a major decrease in the number of tokens generated during routine operations.

Song Pang, CTO of NetBrain, described a similar approach in network management. His team uses traditional computing to pre-map a network layout before sending specific data segments to an AI model. This selective token feeding ensures that only core reasoning tasks rely on AI, minimizing unnecessary computational overhead.

For executives, this points to a clear strategy: optimize from the ground up. Caching, intelligent memory management, and hybrid workflows reduce wasted processing cycles. The current wave of generative AI can only scale sustainably if companies invest in smarter infrastructure that handles data with precision rather than brute force. Efficiency at this level translates directly into predictable, controlled AI budgeting, a concern every boardroom now faces.

Improved prompt efficiency is a key factor in reducing token consumption

Optimizing prompt design is one of the simplest yet most effective ways to reduce AI costs. Every time an employee interacts with a generative AI model, the wording, clarity, and precision of that prompt directly determine how many tokens the model consumes to generate a useful response. The less efficient the prompt, the higher the cost and computation required to reach a result. Training teams to write focused, structured prompts is now an essential skill for any enterprise using AI at scale.

ManpowerGroup provides a strong example here. Max Leaming, the company’s Head of Data Science and AI Solutions, reported that their internal AI labor-market tool initially required around ten follow-up questions for each query to reach satisfactory answers. After concerted efforts to train users on prompt optimization, that number dropped to an average of four. The result was higher productivity, faster response times, and dramatically fewer tokens consumed per interaction.

For executives, this signals an opportunity that doesn’t require massive infrastructure investment. Prompt literacy and internal education can deliver substantial cost reduction. As AI becomes more integrated into daily workflows, employees who understand how to communicate clearly with models will directly impact operational efficiency.

C-suite leaders should treat prompt efficiency as a form of business process optimization. Better prompt design reduces friction, cuts expenses, and enhances the value of every AI interaction. In an environment where token consumption equates to real cost, efficiency begins with how humans communicate with machines.

Leveraging local and on-premise AI solutions can help mitigate rising cloud-based token costs

As the cost of cloud-based AI services increases, enterprises are exploring local and on-premise approaches to regain control over computation, data, and expenditure. By deploying AI models directly onto local hardware or corporate data centers, organizations can reduce dependence on third-party cloud infrastructure and avoid constant metering of token-based services. This shift provides both cost and security advantages, particularly in industries that handle sensitive or regulated data.

Recent developments show where this transformation is heading. During GTC Taipei, NVIDIA and Microsoft announced RTX Spark, a desktop AI system capable of running 120-billion-parameter models locally on Windows devices. The intention, as Microsoft CEO Satya Nadella stated, is to deliver “unmetered intelligence to every home and desk.” Running models locally in this manner removes many per-token fees, resulting in a more predictable cost structure and faster system responses.

At the same time, major hardware vendors such as HPE and Dell are expanding on-premise AI solutions that enterprises can install in their own facilities or regional data centers. This trend is strengthened by increasing concerns around data sovereignty and geopolitical instability in certain regions. However, Max Goss, Senior Director Analyst at Gartner, cautions that while localized and multi-vendor AI setups can mitigate financial and operational risks, they cannot completely eliminate exposure to global supply chain and infrastructure disruptions.

For executives, the decision to localize AI must be driven by a balance of cost predictability, operational control, and regulatory compliance. Local processing allows companies to own more of their AI workflows and build capacity at their own pace. While the upfront investment is significant, long-term savings and independence from fluctuating cloud pricing make it a strategic consideration for organizations deploying AI at scale.

Forward-deployed engineering teams are essential for integrating cost efficiency into AI architectures

Effectively managing AI economics requires technical expertise at the operational level. Forward-deployed engineers (FDEs) are becoming a key part of this framework. Positioned within customer environments, these engineers design and refine AI systems that align with cost, performance, and business goals. Their role is to ensure that AI workloads are architected to minimize wasteful token consumption without limiting capability.

Taimur Rashid, Managing Director at AWS’s Generative AI Innovation Center, explained that FDEs are instrumental in helping customers architect systems with built-in cost-awareness. These teams adapt use cases and model choices to the organization’s specific budget and revenue structure. Rashid also noted that while token consumption may be high in some business scenarios, it becomes acceptable when the AI-generated value offsets the expense. The key is intentional design, knowing when to spend and when to optimize.

For C-suite leaders, the growing importance of FDEs signals a shift from reactive AI adoption to proactive cost engineering. Rather than waiting for usage to drive expense reports, organizations can embed technical expertise into their deployment cycle from the start. This approach ensures accountability, sustainability, and alignment with broader commercial goals.

Companies that empower forward-deployed engineering teams will be better positioned to scale AI efficiently. With these teams guiding architectural decisions, enterprise leaders can ensure that AI innovation delivers measurable returns.

Shifting success metrics from token counts to business outcomes reflects a maturing AI cost-management strategy

As organizations gain a clearer understanding of how generative AI impacts budgets and productivity, the focus is moving from raw usage metrics toward measurable business outcomes. Counting tokens may offer transparency, but it does not capture the broader value AI creates through automation, innovation, or customer engagement. The most forward-thinking companies are beginning to assess AI performance through results, speed, quality, accuracy, and revenue impact, rather than counting the units of processing involved.

Deepak Seth, Senior Director Analyst at Gartner, highlighted this evolution, noting that companies are starting to adopt outcome-based pricing instead of token-based billing. This model allows executives to judge success by real-world performance metrics, such as time saved, customer satisfaction improvements, or increased productivity. As companies begin to understand the hidden costs of token-heavy operations, the transition to outcomes becomes not just strategic but necessary to ensure fair value exchange between AI providers and enterprises.

For C-suite leaders, this shift represents a critical step toward AI maturity. Token efficiency remains important for cost control, but outcome efficiency defines competitive advantage. Organizations that evaluate AI investments through business results can align technology spending with tangible returns. This approach fosters more stable financial planning and prioritizes systems that directly advance operational objectives.

In practice, moving to outcome-based assessment changes how AI deployments are designed and managed. It encourages tighter collaboration between technical and business teams to define clear success criteria before implementation. The companies that structure their AI strategies around measurable impact, rather than computational volume, will drive better margins and achieve more predictable performance across their digital operations.

In conclusion

Generative AI is moving fast, and the cost of keeping pace is becoming impossible to ignore. Token efficiency has emerged as the new frontier of AI strategy. The smartest companies are learning that efficiency is not just a technical metric, it’s a leadership issue that shapes margins, agility, and long-term scalability.

For decision-makers, the way forward requires balancing innovation with financial discipline. That means choosing models based on business fit, not hype. It means investing in infrastructure that reduces waste, training teams to prompt with precision, and empowering technical leaders to architect with cost in mind. Most importantly, it means measuring AI success by the results it drives, not the tokens it burns.

Those who act now will set the benchmarks for sustainable AI operations. The rest will be forced to adapt under pressure. In this next phase of AI adoption, control, clarity, and outcome focus will separate those leading the transformation from those reacting to it.