Samsung puts MAC units in LPDDR5X banks for 614 GB/s in-memory AI compute
- Samsung LPDDR5X-PIM places a compute block in each of 16 DRAM banks, allowing the blocks to use 614 GB/s of aggregate internal bandwidth versus 76.8 GB/s for regular accesses across two banks.
- Each bank's PIM block combines a MAC tree with register files: a 1024-bit instruction store holds 64 16-bit instructions, a 4 kbit source register holds activation vectors, and attached DRAM supplies model weights.
- The MAC arrays support INT8, FP8, and 4-bit formats; Samsung's figures imply 2.4 TOPS per package with 4-bit weights, while eight 16 GB chips would total 9.6 INT8 TOPS and 128 GB of memory.
- Samsung keeps the standard LPDDR5X protocol by reserving special row addresses to expose compute operations, while retaining compatibility with a normal memory controller.
Hacker News opinions
I'm glad Samsung is still pursuing PIM, after its earlier HBM2 PIM work around ISCA 2021 or 2022. I still don't see the killer application, though, and adoption is unlikely without one.
I think LLMs are the killer app. Running MACs directly in RAM could offload CPU work and cut the huge external-memory bandwidth demand, so a larger model might run just by adding enough RAM.
My concern is whether using a PIM memory region effectively discards its cache residency. The article raises cache-miss offsets, but Samsung does not quantify their impact or give a solution.
If the working set fits in cache, PIM is irrelevant. Its value is for workloads that are constrained by memory bandwidth beyond what cache can hold.
Regular DRAM could already do in-memory computation with suitable CPU instructions, such as a hypothetical "rep macb." There was an effort to make existing x86 string instructions work this way on SDR SDRAM controllers, but the tradeoffs did not favor it then.
A 64 GB DIMM that can run a 32B model at 30 tokens per second would sell. That is the practical threshold I care about.
Moving compute into a RAM bank reminds me of ISA extended-RAM cards in the 1980s, some of which had CPUs that took over or augmented the system. The bottleneck is different now, but avoiding a slow bus is the same basic motivation.