AI Memory Stack: Investor Implications
In a Reuters piece dated 30‑08‑2026, Bernstein analysts highlighted that artificial‑intelligence workloads are generating a growing memory bottleneck, pushing demand beyond traditional high‑bandwidth memory (HBM) into conventional DRAM, NAND flash and broader storage solutions.
Training large AI models remains highly compute‑intensive; limited memory bandwidth constrains both model size and computing speed. While HBM continues to play a central role, large‑scale training also depends on system DRAM, local solid‑state drives (SSDs) and networked storage for datasets, caching, and checkpointing.
Inference presents a distinct profile. The initial “prefill” stage, where a model processes a prompt and emits its first token, is largely compute‑bound, keeping GPUs and HBM critical. The subsequent “decode” stage, however, is memory‑bound: models retain prior tokens in a key‑value (KV) cache, and the memory required for this cache grows with context length and the number of concurrent users. Bernstein noted that in large deployments KV caches could consume more memory than the model weights themselves, potentially limiting the number of users an AI service can support.
These pressures are prompting the emergence of new memory tiers, such as Compute Express Link (CXL) memory, Nvidia’s “Storage Next” initiative, and CMX context storage, alongside existing HBM, conventional DRAM and SSDs. The goal of these technologies is to balance performance, capacity and cost as AI systems demand ever‑larger memory pools.
Retrieval‑augmented generation (RAG) further widens memory needs; building RAG databases requires substantial SSD or HDD capacity together with system DRAM, with DRAM playing an increasingly larger role when those databases are queried. Agentic AI could intensify requirements still more, as autonomous agents retain intermediate results, interact with external tools and exchange outputs, rapidly expanding KV‑cache demand and raising the need for CPUs and conventional memory in traditional servers.
Memory manufacturers are also pursuing high‑bandwidth flash, a technology that aims to combine HBM‑like bandwidth with NAND’s higher capacity and lower cost, though Bernstein cautioned that technical hurdles remain significant.
Based on these trends, Bernstein assigned Outperform ratings to Samsung Electronics, SK hynix, Micron Technology, SanDisk, Seagate Technology and Western Digital, while rating Kioxia as Underperform.