The Latency Arbitrage: How Speculative Decoding and Tree Attention Shattered the Autoregressive Memory Wall
A first-principles systems breakdown of speculative decoding, the memory-bandwidth wall, and how tree-based parallel verification (EAGLE-3, MTP, SSD) broke the single-token autoregressive bottleneck without losing a single bit of model precision.