Hot Chips 2026 ran August 24–25, and if the program had one theme, it was memory. The Sunday tutorials spent a full morning on “feeding AI’s voracious hunger for data,” with talks on HBM base dies, advanced packaging, and 3D DRAM inference accelerators. Tuesday morning’s Memory session belonged to Samsung, with a talk titled “Samsung LPDDR5X-PIM: World’s First LPDDR based Processing in Memory (PIM) Solution for AI Inference,” presented by Samsung’s Karam Hwang. On August 29, Chips and Cheese published Chester Lam’s detailed analysis of the design, and it is worth reading in full.
How LPDDR5X-PIM Works
The idea is to push multiply-accumulate (MAC) units into LPDDR5X chips without breaking compatibility with standard memory controllers:
- The chip has 16 banks, each with its own PIM block containing a MAC tree plus register files: a 1,024-bit instruction register (up to 64 instructions), a 4-kbit source register holding activation vectors, and a 2-kbit scale register.
- Compute is activated through special row addresses. Sending mode-control rows switches the chip between states — closer to memory-mapped I/O (MMIO) than to an instruction stream — toggling between single-bank and multi-bank modes.
- Writes broadcast across all 16 banks, which makes the programming model SIMD-like, and Address Align Mode derives source-register indices from column addresses so that controller reordering does not break the setup.
In other words, this is not an NPU bolted next to the memory; it is the DRAM itself doing an extra layer of arithmetic without any change to the external interface.
Context helps here: the same Memory session also hosted the XCENA MX1, a CXL computational memory device, and the Sunday tutorial ran through HBM base dies, advanced packaging, and 3D DRAM accelerators for generative inference. Compute-capable memory is no longer one vendor’s experiment; it is becoming the industry’s default answer to the bandwidth wall.
The Headline Numbers: 8x Internal Bandwidth
Each PIM block sustains four INT8 or FP8 MACs per data clock, doubling with 4-bit weights; a full package reaches 2.4 TOPS, and eight chips stack up to 9.6 INT8 TOPS — roughly the level of Intel’s Meteor Lake laptop NPU. The real story is bandwidth: the combined internal compute bandwidth across the 16 banks is 614 GB/s, exactly eight times the 76.8 GB/s available through ordinary external accesses. For a workload that streams a weight matrix once per token, that difference is the entire product. “Move the compute to the data” is a slogan, but here the arithmetic checks out — the catch is written in the next section.
Three Engineering Realities
The analysis does not stop at the spec sheet, and the limits are stated plainly. First, PIM compute cannot run alongside ordinary memory accesses, so software has to isolate PIM regions; interleaving is limited, and locks or OS-level blocking are required. Second, PIM accesses must be uncacheable and non-speculative, which runs directly against CPU caching, prefetching, and out-of-order execution. Third, PIM blocks cannot exchange data with one another. Each of these is as much a software-model problem as a hardware one: the memory controller, the OS, and the allocator all have to know which address ranges are special, and standard toolchains assume none of that. The author’s verdict: without deeper system changes — dedicated compute commands, cache-coherent memory controllers, perhaps CPU instructions in the spirit of rep macb — the adoption barrier stays high.
What It Means for AI Inference Hardware
LLM inference is the canonical bandwidth-bound workload: moving weights dominates energy and latency relative to the arithmetic itself, which is precisely why memory vendors are pushing compute toward the data. Two signals stand out here. First, moving PIM from HBM and the server room down to LPDDR signals a target of on-device inference — phones and laptops running models locally rather than calling a data center. Second, TOPS at the level of a Meteor Lake NPU says the role is assistive compute, not a replacement for the main NPU. Pairing the PIM talk with a CXL computational memory device in the same session points at the broader direction: memory is becoming a place where compute lives, and the software model — not the silicon — will decide who can actually use it. For developers, the near term is a watching brief: isolation and uncacheable access requirements mean this will not behave like a transparent accelerator, and until tooling and the ecosystem mature, real gains depend on whether an application can be redesigned around this programming model.
Sources
- Hot Chips 2026: Samsung’s Processing-in-Memory (PIM) — Chips and Cheese
- Hot Chips 2026 Program — Hot Chips
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.