AI has reached the memory wall and context is the next priority

AI inference is shifting from a compute bottleneck to a context management bottleneck

For years, the AI conversation was simple: buy more GPUs. That made sense when training large models was the biggest challenge. Today, the problem is changing. AI is moving from answering individual questions to running persistent, multi-step workflows. These systems remember previous interactions, make decisions over time, and continue work across multiple sessions. That changes what limits performance.

The new constraint is context. Context is the information an AI system keeps available so it can continue a conversation, complete a task, or coordinate a sequence of actions without starting over. Every larger context window, every additional model call, and every piece of retained state increases the amount of data that must be stored and retrieved quickly. As enterprises adopt AI agents for business operations, customer service, software development, and internal workflows, this demand continues to grow.

Jeff Harthorn, AI Applied Research Lead at Solidigm, argues that this has become the defining infrastructure question. According to Harthorn, improvements in GPU cost per floating-point operation (FLOP), model architectures, and inference engines have been significant. However, the growth of context, and especially persistent state that survives across sessions, has been even faster.

This shift has practical business implications. Many organizations still evaluate AI infrastructure primarily by GPU count or compute capacity. Those metrics remain important, but they no longer provide the full picture. If an AI system cannot efficiently manage its context, adding more compute delivers diminishing returns because expensive accelerators spend more time waiting for data or rebuilding information they already processed.

Ace Stryker, Director of AI and Ecosystem Marketing at Solidigm, points out that several industry trends are driving this change simultaneously. Context windows continue to expand. Agentic AI systems routinely execute dozens or even hundreds of connected model calls. Enterprises also increasingly require inference state to persist for governance, compliance, auditing, and future reuse. Together, these trends create data growth that existing memory architectures were never designed to support.

For executive teams, this represents an important shift in investment priorities. Competitive advantage will increasingly depend on how efficiently organizations manage AI context, not simply on how much compute they own. Companies that recognize this early can improve infrastructure utilization, lower operating costs, and support more advanced AI applications without scaling hardware at the same pace as demand.

AI inference requires a storage architecture that differs fundamentally from that used in AI training

Training and inference solve different problems. They should not be expected to run efficiently on identical infrastructure.

Training is largely sequential. Massive datasets move through the system, models learn from that data, and storage is optimized for high-throughput, write-heavy operations. The traditional architecture, GPU memory, fast local NVMe storage, and large network-attached storage, works well for this purpose because the workload is predictable and data flows in large blocks.

Inference operates very differently. Once a model is deployed, its value depends on how quickly it can respond while maintaining context. Instead of processing large batches of data, inference repeatedly accesses smaller pieces of information with very low latency requirements. This includes key-value (KV) cache, which stores previously computed attention information so the model does not need to recalculate it, and retrieval data used to supplement model responses with external knowledge. Both must be available almost immediately if the system is to deliver fast and consistent results.

The challenge is that neither existing GPU memory nor traditional storage was designed for this role. GPU high-bandwidth memory (HBM) is extremely fast but expensive and physically limited. Conventional network storage provides capacity but cannot consistently deliver the latency required for active inference workloads. The result is an architectural gap between the fastest and slowest storage layers.

Jeff Harthorn, AI Applied Research Lead at Solidigm, believes this middle layer is where much of today’s infrastructure innovation is happening. According to Harthorn, the systems below GPU memory are increasingly being asked to perform tasks they were never designed to handle, creating opportunities for entirely new infrastructure approaches.

One visible consequence is unnecessary recomputation. Before an AI model begins generating new output, it processes existing context during what is known as the pre-fill stage. If previously generated KV cache cannot be retrieved quickly, the model recreates it from scratch. Those GPU cycles produce no new business value. They simply restore information that already existed.

This is why infrastructure leaders are beginning to focus on efficiency metrics beyond raw performance. Instead of measuring only tokens generated per dollar, many are evaluating goodput, the amount of useful work completed per dollar after eliminating wasted computation. Improving storage architecture can increase goodput without requiring additional GPUs, making it one of the most effective ways to improve infrastructure economics as AI deployments continue to scale.

A dedicated context memory tier is emerging between GPU memory and traditional bulk storage

The AI infrastructure stack is evolving because existing memory and storage layers are no longer enough for modern inference workloads. A new context memory tier is emerging between GPU memory and traditional network storage. Its purpose is straightforward: store and deliver inference context quickly enough that AI systems can continue working without repeatedly rebuilding information they already processed.

This layer is designed to hold data such as key-value (KV) cache and retrieval information. KV cache contains intermediate results generated during inference that allow a model to retain context efficiently, while retrieval data provides external information that the model can access during generation. Both types of data must be available with very low latency if AI applications are expected to deliver consistent performance.

The industry is beginning to formalize this architecture. Nvidia refers to this approach as CMX, while storage vendors including Solidigm are developing SSD products specifically optimized for context-heavy inference workloads. Rather than viewing storage as a single destination for data, infrastructure planners are increasingly treating context storage as its own operational layer with distinct performance requirements.

Jeff Harthorn, AI Applied Research Lead at Solidigm, believes this development is driven by workload demands rather than by any single vendor’s product strategy. He notes that the rapid growth in inference context is creating enough pressure for an entirely new infrastructure category to emerge. As happened with previous storage technologies, this category is expected to develop its own software ecosystem, operational standards, and deployment practices.

Ace Stryker, Director of AI and Ecosystem Marketing at Solidigm, says organizations designing new AI data centers can no longer assume storage exists in only two places. According to Stryker, storage must now exist across at least three layers to support the growing context memory requirement, and he expects this structure to become a permanent part of enterprise AI infrastructure.

For executives, this changes long-term infrastructure planning. The decision is no longer limited to selecting GPUs and expanding storage capacity. Organizations must determine where persistent inference context will reside, how quickly it can be accessed, and how efficiently that layer integrates with the rest of the AI platform. Businesses that account for this now are likely to achieve better infrastructure utilization as AI workloads become increasingly persistent and stateful.

Inefficient context storage leads to unnecessary GPU recomputation, reducing overall system efficiency

Many organizations measure AI infrastructure by GPU utilization. That remains an important metric, but utilization alone does not reveal whether those GPUs are performing useful work. If storage cannot deliver context data quickly enough, valuable compute resources spend time recreating information instead of generating new results.

This issue appears during the inference pre-fill stage, when the model processes all relevant context before producing output. If the required KV cache is unavailable in a fast storage layer, the model regenerates that context. The result is additional GPU activity without any improvement in the quality of the final response. The system consumes more power, increases latency, and reduces overall throughput while delivering the same business outcome.

Jeff Harthorn, AI Applied Research Lead at Solidigm, explains that a meaningful share of GPU cycles can be spent on this repeated pre-fill process. His observation reframes infrastructure planning. The problem is not always insufficient compute capacity. In many cases, the underlying issue is that storage cannot provide previously generated context fast enough for GPUs to use it efficiently.

This is encouraging organizations to evaluate AI infrastructure differently. Rather than focusing only on the number of tokens generated per dollar, attention is shifting toward “goodput,” which measures useful output per dollar after accounting for wasted computation. Goodput provides a more practical view of infrastructure efficiency because it reflects whether expensive compute resources are producing new business value rather than repeating previous work.

For business leaders, this has direct financial implications. GPUs are among the most expensive components of modern AI infrastructure. Every cycle spent rebuilding existing context increases operating costs without increasing productivity. Improving context storage reduces unnecessary recomputation, shortens response times, and allows organizations to extract more value from the compute resources they already own. In many environments, improving storage efficiency may deliver greater returns than simply expanding GPU capacity.

Storage is becoming a strategic component of AI infrastructure rather than a commoditized asset

Storage has traditionally received far less attention than compute. Organizations often selected storage based primarily on capacity and cost per gigabyte because it represented a relatively small portion of overall infrastructure spending. That approach is becoming less effective as AI inference workloads become more dependent on fast, reliable access to persistent context.

Modern AI systems require storage that actively contributes to application performance. If storage cannot deliver context data consistently and with low latency, GPUs remain underutilized, response times increase, and infrastructure costs rise. Storage is no longer simply where data is kept. It is becoming an active part of the inference pipeline.

Ace Stryker, Director of AI and Ecosystem Marketing at Solidigm, argues that this changes how organizations should think about infrastructure investments. According to Stryker, storage quality now has a direct impact on return on investment because poor storage performance can reduce the efficiency of expensive AI hardware. Infrastructure decisions that once focused almost entirely on compute must now include careful evaluation of storage performance.

Another important consideration is memory economics. High-bandwidth memory (HBM) and DRAM deliver excellent performance but are expensive and limited in capacity. Expanding these memory technologies indefinitely is neither practical nor cost-effective for many enterprise deployments. A dedicated SSD-based context layer provides additional capacity for inference state while reducing dependence on more expensive memory resources.

Stryker also notes that following Nvidia’s recommended architecture for many inference use cases allows organizations to rely more heavily on SSDs instead of expanding DRAM capacity. This can lower infrastructure investment while still supporting high-performance inference workloads.

For executive teams, the broader message is clear. AI infrastructure should no longer be evaluated as isolated hardware components. Compute, memory, networking, and storage must be optimized together. Organizations that continue treating storage as a commodity risk limiting the performance of their most valuable AI investments.

SSDs supporting AI inference must deliver predictable latency, high density, and robust network integration

Not all high-performance storage is equally suited for AI inference. Traditional performance benchmarks often emphasize maximum throughput or average latency. Inference workloads require a different set of priorities, with consistency becoming one of the most important performance characteristics.

AI inference systems depend on predictable response times because orchestration software continuously schedules workloads across GPUs and storage resources. If storage occasionally experiences long delays, even when average performance appears strong, those delays can disrupt workload scheduling and reduce overall system efficiency. This is why tail latency, the slowest response times experienced under load, has become a critical metric for inference infrastructure.

Jeff Harthorn, AI Applied Research Lead at Solidigm, emphasizes that reliable and observable storage performance is now more valuable than simply transferring the highest possible volume of data. Training workloads rewarded peak throughput. Inference workloads reward consistency because stable latency enables more efficient resource allocation across the entire AI system.

Scale also changes the storage equation. Large AI deployments are increasingly constrained by power availability as much as by hardware cost. As data centers continue to expand, storage density and power efficiency become major operational considerations. Metrics such as watts per petabyte are becoming increasingly important because they directly affect how much AI infrastructure can be deployed within existing power budgets.

Modern inference architectures also depend on advanced networking technologies that allow storage resources to remain accessible with minimal latency. Technologies such as NVMe over Fabrics and Remote Direct Memory Access (RDMA) support faster communication between compute and storage resources, while future support for Compute Express Link (CXL) is expected to further improve memory sharing and resource utilization across AI systems.

For business leaders, these technical capabilities translate into measurable business outcomes. Storage that delivers predictable performance improves GPU utilization, supports more consistent application responsiveness, and simplifies infrastructure operations. As AI deployments continue to grow, selecting storage based on sustained operational performance rather than peak benchmark results will become an increasingly important competitive advantage.

Organizations must proactively plan for the transition to a context memory tier

AI infrastructure is entering a period where architectural decisions will have long-term consequences. The standards, software interfaces, deployment models, and operational practices for context memory are still developing. Organizations that begin planning now will be better positioned to scale AI systems efficiently as these technologies mature.

The transition is not only about deploying new hardware. It also requires infrastructure teams to rethink how AI applications are designed, monitored, and managed. Persistent context is becoming a core operational requirement for enterprise AI, particularly as organizations deploy agents capable of handling complex workflows over extended periods. This means storage architecture, orchestration software, networking, and memory management must work together as an integrated system.

Jeff Harthorn, AI Applied Research Lead at Solidigm, believes the industry’s next major question is not whether organizations need more compute. Instead, the focus should be on using existing compute more efficiently. Better context management enables GPUs to spend more time generating useful output and less time recreating information that already exists. As AI adoption accelerates, improving efficiency may deliver greater business value than simply increasing hardware capacity.

Solidigm is participating in industry standards efforts, research collaborations, and partner initiatives that are helping define how this new infrastructure layer will operate. The broader point extends across the industry. Infrastructure vendors, hyperscale cloud providers, AI platform developers, and enterprise customers are collectively shaping the standards that will support future inference workloads.

For executive leadership, this is an opportunity to think beyond today’s deployment requirements. AI infrastructure investments are expected to remain in place for years, making architectural flexibility an important consideration. Planning for a dedicated context memory tier today can reduce future migration costs, improve infrastructure utilization, and support increasingly sophisticated AI applications without requiring proportional increases in compute resources.

Organizations should also recognize that AI infrastructure planning is becoming more strategic than tactical. Decisions about storage architecture now influence operational efficiency, application performance, governance capabilities, and long-term scalability. Companies that incorporate context memory into their infrastructure roadmaps early will be better prepared to support the next generation of enterprise AI while maximizing the value of their existing technology investments.

Recap

Enterprise AI is entering a new phase. The conversation is no longer centered on adding more GPUs or deploying larger models. Those investments still matter, but they are no longer enough to deliver the next wave of performance and efficiency.

As AI systems become more autonomous, persistent, and integrated into core business processes, context becomes a critical infrastructure resource. Organizations that can store, retrieve, and manage context efficiently will be able to extract more value from their existing compute investments while supporting more capable AI applications. Those that continue to rely on architectures built primarily for model training may find that infrastructure costs rise faster than business value.

This is also an opportunity to rethink how AI infrastructure is evaluated. Success should not be measured solely by hardware capacity or benchmark performance. It should be measured by how effectively the entire platform works together to deliver useful business outcomes. Storage, networking, memory, and compute are becoming increasingly interconnected, and weaknesses in one layer can limit the performance of the entire system.

For executive teams, the priority is clear. AI infrastructure planning should shift from short-term hardware procurement to long-term architectural strategy. Decisions made today about storage, context management, and data movement will influence scalability, operating costs, governance, and the ability to support increasingly sophisticated AI workloads over the coming years.

The organizations that move first will not necessarily be the ones with the most compute. They will be the ones that build infrastructure capable of using every resource more efficiently. As AI continues to evolve, that efficiency will become one of the strongest competitive advantages an enterprise can develop.