The Big LLM Architecture Comparison

Modern LLM architectures primarily optimize for efficiency and scale, consistently moving from Multi-Head Attention to Grouped-Query Attention or Multi-Head Latent Attention. Mixture-of-Experts (MoE) is a dominant trend, enabling massive models like DeepSeek-V3 and Kimi K2 to achieve high capacity with sparse, efficient inference. Training stability is enhanced through specific RMSNorm placements (e.g., Post-Norm, QK-Norm) in models like OLMo 2 and Gemma 3, and advanced optimizers like Muon in Kimi 2. Long-context capabilities are improved by techniques such as sliding window attention (Gemma 3) and the Gated DeltaNet + Gated Attention hybrid (Qwen3-Next). Further innovations include per-layer embeddings for device efficiency (Gemma 3n), multi-token prediction for faster decoding (Qwen3-Next), and experimental positional encoding approaches like NoPE (SmolLM3), reflecting a focus on modular enhancements.