Overview

Barclays analyst Tom O'Malley highlighted that tokens‑per‑watt has become one of the most critical yardsticks of success for AI chips, driven by constrained data‑center space and demand for computing tokens outpacing supply. Leading hardware providers, hyperscalers, and component suppliers are all focused on extracting maximum performance within fixed power budgets.

Company‑Specific Highlights

  • AMD announced enhancements to its transcendental engine, adding a tensor data mover to boost both compute capability and energy efficiency.
  • Microsoft introduced a new architecture named Software Defined Local Access Dataflow, which leverages explicit software orchestration to lower total cost of ownership through improved energy efficiency.
  • Nvidia discussed a speculative decode approach and unveiled the NVHBM memory technology, claiming up to 30% higher bandwidth, 15% lower HBM power consumption, and 25% more die area freed compared with standard HBM4E.
  • Cerebras is pursuing disaggregated inference via partnerships with AMD and AWS, which O'Malley said could deliver an order‑of‑magnitude improvement in tokens‑per‑second‑per‑kilowatt.
  • OpenAI presented its Jalapeño chip, keeping prefill, draft, and decode functions on a single die rather than splitting them across separate hardware.
  • Google is adopting a dual‑sourcing strategy: one line of AI chips is built with MediaTek, while an inference‑focused variant uses Marvell.
  • Broadcom and Marvell maintain defensible positions thanks to their SerDes expertise and robust supply‑chain capabilities, even as industry attention shifts toward a chip‑on‑tile model.

System‑Level Trends

O'Malley noted a shift from optimizing individual components to maximizing full‑system performance, with racks now treated as the primary unit of compute. Strategies differ between training and inference chips, and the current debate centers on disaggregated inference versus integrated designs.

Memory Architecture Developments

Memory capacity, bandwidth, and power efficiency emerged as key themes. Beyond Nvidia's NVHBM, O'Malley suggested the next major memory shift could involve 3D DRAM, with Samsung and Cerebras exploring stacked configurations, though these efforts remain in very early stages due to thermal and power‑delivery challenges. Interest in CXL memory expansion persists, but the analyst remains cautious until concrete deployment evidence appears.

Implications

The collective focus on improving tokens‑per‑watt underscores an industry‑wide drive to deliver higher AI compute throughput without expanding power consumption, influencing chip design roadmaps, supply‑chain strategies, and future memory technology investments.