Introduction: Seven years on from the original GPT, a nagging question persists: beneath the dazzling benchmarks and impressive demos, are Large Language Models truly innovating at their core? As new “flagship” architectures emerge, one can’t help but wonder if we’re witnessing genuine paradigm shifts or merely sophisticated polish on a well-worn foundation. This column will cut through the marketing jargon to assess the true nature of recent architectural “advancements.”

Key Points

• The fundamental Transformer architecture remains stubbornly entrenched, with “innovations” primarily focusing on efficiency rather than conceptual breakthroughs.
• Current architectural refinements like Multi-Head Latent Attention (MLA) and Mixture-of-Experts (MoE) are sophisticated engineering plays to manage scale and cost, not reinventions of intelligence.
• The inherent opacity in LLM development—varying datasets, training regimes, and hyperparameters—makes it exceedingly difficult to isolate the true impact of architectural changes, fueling skepticism about their purported superiority.

In-Depth Analysis

For years, the tech world has lauded the rapid evolution of Large Language Models, yet a critical eye reveals a less dramatic truth: the foundational block, the Transformer, remains remarkably unchanged. While we’ve seen a shift from absolute to rotational positional embeddings, Multi-Head Attention giving way to Grouped-Query Attention (GQA), and GELU yielding to SwiGLU, these are, at best, incremental refinements. They’re akin to swapping out a car’s carburetor for fuel injection—an important efficiency gain, yes, but still the same internal combustion engine.

DeepSeek V3’s touted Multi-Head Latent Attention (MLA) is a prime example. Positioned as a memory-saving marvel, MLA compresses key and value tensors, primarily benefiting the KV cache during inference. While the original DeepSeek-V2 paper suggests MLA might even slightly outperform standard MHA—a notable claim—it’s fundamentally an optimization around a known memory bottleneck, not a rethinking of how attention itself works. It’s smart engineering, a clever workaround, but not a conceptual leap in how models learn or reason. The very fact that GQA, another efficiency-focused alternative, is its primary comparator underscores this point: the game is efficiency, not entirely new functionality.

Similarly, the resurgence of Mixture-of-Experts (MoE) in architectures like DeepSeek V3 is less a novel discovery and more a re-application of a technique known for decades. MoE layers balloon a model’s total parameter count, offering increased capacity, but cleverly activate only a small subset of “experts” per token during inference to maintain efficiency. This allows for immense “sparse” models, but it’s a trade-off: huge capacity at training time for controlled computational cost at inference. It’s a scaling hack, albeit an effective one, designed to push the limits of model size without fully breaking the bank on compute. The “groundbreaking” aspect isn’t the architectural primitive itself, but rather its current scale and the engineering required to manage its complexity. We are, it seems, simply becoming better at packing more into the same fundamental container.

Contrasting Viewpoint

While my skepticism points to the evolutionary, rather than revolutionary, nature of these architectural tweaks, proponents would quickly argue that “incremental” progress at this scale is revolutionary. When dealing with models encompassing hundreds of billions of parameters, even minor efficiency gains translate into monumental savings in compute, energy, and real-world deployment costs. They would assert that MLA’s ability to reduce KV cache memory usage, or MoE’s capacity to dramatically increase model parameters without a proportional increase in inference costs, are precisely what unlocks new capabilities and broader accessibility for AI. For a practical enterprise or a cloud provider, a 10-20% gain in efficiency due to architectural choices can mean the difference between a viable product and an economic non-starter. They might even argue that the current focus on scaling within the Transformer paradigm is a necessary and logical step, pushing the limits of a proven architecture before attempting entirely new, and far riskier, foundational designs.

Future Outlook

Looking ahead 1-2 years, I predict more of the same, albeit with increasing sophistication. Expect continued iterative improvements focused squarely on efficiency: denser packing of parameters, more intelligent expert routing in MoE systems, novel memory hierarchies, and perhaps more specialized attention mechanisms tailored for specific tasks. The biggest hurdle to true architectural “revolution” remains the sheer cost and computational complexity of developing and validating entirely new foundational primitives. Moving beyond the Transformer would require a paradigm shift on par with its original inception, and frankly, the incentives are still heavily skewed towards optimizing what already works at scale. Furthermore, the industry still grapples with the black box problem of attributing performance gains—is it the data, the scale, or the architectural nuance? Until we can definitively answer that, we’ll continue polishing the chrome on our reliable old engine.

For more context, see our deep dive on [[The Economics of LLM Training]].

Further Reading

---

Original Source: LLM architecture comparison (Hacker News (AI Search))

阅读中文版 (Read Chinese Version)