SemiAnalysis's report dissects the underlying architecture of AI large model inference, identifying four stages: prefill, midfill, decode attention, and decode expert. It highlights that memory bandwidth holds greater economic value than capacity.

Research firm SemiAnalysis released an in-depth technical report on September 22, systematically dissecting the underlying architecture of AI large model inference services. The report points out that inference service is not a monolithic entity but a "Token Factory" composed of four working stages: Prefill, Midfill, Decode Attention, and Decode Experts.

This analytical framework emphasizes that each stage has distinctly different demands for compute power, memory bandwidth, and network, and conflating them will forfeit the structural advantages of the Mixture of Experts (MoE) model architecture. The report also concludes that in many inference scenarios, memory bandwidth generates more value than memory capacity, and the optimal use for accelerator HBM is to host currently active batch data, rather than storing idle context. The report uses NVIDIA's Blackwell series GPUs as a case study for its deductions, and its conclusions have direct guidance for hardware designers, cloud service providers, and model deployers.