Baidu releases ERNIE 5.5 Flash Base, a 280B MoE checkpoint with 14B active parameters and no instruction tuning
- Baidu released ERNIE 5.5 Flash Base, a sparse MoE base model with 280B total parameters, 14B of them activated per token, pre-trained from scratch on 33.5T tokens at a training context of up to 128K.
- The published weights are the end-of-pre-training checkpoint, with no mid- or post-training, so the model follows no instructions and is meant as a common starting point for domain adaptation, instruction tuning, and reinforcement-learning research.
- Architecture is 43 Transformer layers, hidden size 4096, 64 attention heads, using hybrid MLA + HCA attention: one layer in seven keeps token-level access to the full context, the rest compress every 128 tokens into a single cached entry, and the MLA layers use DSA sparse attention over the top-k scored positions.
- The license bars use in the European Union, United Kingdom, and South Korea, and makes adopters responsible for keeping people in those territories from using derivatives.
- Fine-tuning and inference support a 1M-token context, and the pre-training corpus favors capabilities that are hard to add later, such as repository-scale code and long-context understanding.
Hacker News 의견들
Anyone actually tried it? It's a base checkpoint with no instruction tuning, so it's useless for coding agents as it stands.
ERNIE seems stronger than Qwen3.8-27B and Deepseek-v4-Flash, which are the best base models out there right now. Base models are also great for companies running locally, you can bake in house rules and house styles.
They compared against Deepseek-v4-Flash, not 4.1 and not the 1017 update, so that's basically two versions behind the latest Deepseek Flash.
Also I can't find a base model version of Qwen3.8-27B in the wild anywhere.
TLDR: 280B params, 14B active, a chunk better than Deepseek 1017 at half the parameter count. But it's only a base model, strange to release it in that state.
They explain exactly why they release it as a base model. Google and NVIDIA ship base models too. If you have the knowledge and compute to finish training, a base model is far more useful than a fully trained one, it's the next best thing to true open source where you'd get the dataset.
The license is really restrictive for something meant to be built on. Entire countries are banned (European Union, United Kingdom, South Korea) and you're responsible for making sure nobody there uses your derivative. Unenforceable in practice, but they hand the regulatory burden to adopters.
That exclusion is about those countries' AI legislation, dealing with the EU isn't worth the regulation and litigation cost. Same thing as websites blocking EU visitors when GDPR came out.
Interesting size, it'll fit in 2x128GB at decent quantization, 4bpw or better. Though as a 14B MoE I expected it to fit somewhere smaller.
ERNIE feels old, like BERT. They need a rebrand. Bert and Ernie, Sesame Street, ring a bell?
Honestly glad to see a base model release. The trend away from them has been a problem for the field.
Base model releases are the reliable way to do alignment. Millions of people finetuning in their own direction, drifting apart, cancels out at the civilization level, and open releases discourage investment in future training runs. Messy but it works unilaterally.
That's the opposite of how alignment works. If a single group builds a sufficiently capable misaligned AI we're all done. It's 'if anyone builds it, everyone dies', not 'if everyone builds it everyone is fine'.
What can a base model actually do anyway? Is this just autocomplete with no reasoning or alignment work, like the pre-reasoning models?
Anyone got a framework for continuing training on a multi-node cluster with InfiniBand? NeMo, or plain HF TRL plus Accelerate?
When ERNIE 4 dropped it was solid and got straight to the point, felt different from everything else. Then DeepSeek v3 and GLM took off and I dropped it. Hope they kept that vibe and it doesn't overthink.