Qwen4's Architecture: A Preview of What's to Come
Alibaba’s Qwen team has released Qwen3.8-Flash-Next, an open-weight preview of the architecture they intend to use for Qwen4. This model carries a staggering 125B parameters but activates only 6B for each token it produces.
Key Takeaways:
- Qwen3.8-Flash-Next aims to reduce inference costs for agentic jobs with long contexts, which require significant computational power.
- The architecture includes:
- A new sparse attention scheme targeting micro-blocks.
- A gated residual mechanism for better layer interaction.
- Dropping batch-size warmup in the training process.
- A unique 51B parameter embedding system that helps offload onto memory-constrained accelerators, a common challenge due to export controls.
- While Alibaba claims cost savings, the true capabilities of Qwen4 remain to be seen.
License and Accessibility:
The model is available on Hugging Face under a qwen-community license. However, concerns have been raised about Alibaba’s plans to charge commercial users for access, potentially impacting the open-source nature of the model. The European AI Act's treatment of monetized components under an open-source license could play a significant role in this debate.
Additional Notes:
- Some aspects of the model's performance are measured using internal benchmarks and GPT-4o, raising questions about transparency.
- Thomson Reuters has already integrated Qwen into its operations, highlighting the growing reliance on these large language models.