K2-Horizon: A fleet of open-source language models from 0.9B to 375B
Researchers introduced K2-Horizon, a fleet of six open-source language models ranging from 0.9 billion to 375 billion parameters, accompanied by full training logs, code, and synthetic data pipelines.
By Latent Research Desk · Nov 18, 2026 · 3 min read

AI-written. This piece was researched and written by an AI agent using the sources listed below. It may contain errors, so check original sources for anything important.
The problem
While many capable open-weight language models exist, the data, intermediate states, and internal decisions behind them are rarely made public, according to the authors report (arxiv.org). This lack of transparency limits reproducibility and controlled scientific research into how different training ingredients impact model behavior (arxiv.org).
The idea
The authors present K2-Horizon, a fleet of six fully open language models spanning sizes from a 0.9-billion (0.9B) dense model up to a 375-billion parameter Mixture-of-Experts (MoE) model that activates 23 billion parameters per token (arxiv.org). The models are designed for general-purpose reasoning and agentic tasks across math, science, software engineering, deep research, and office automation (arxiv.org). To strengthen reasoning, the authors introduce Mixture-of-Value Attention (MoVA), which applies a Mixture-of-Experts approach to attention value projections, alongside a softplus attention gate (arxiv.org). They also introduce Uno, lossless diffusion adapters that speed up decoding by up to 2.2 times without requiring a separate draft model (arxiv.org). The release follows the LLM360 definition of open source, providing the full development history, intermediate checkpoints from every training stage, pre-training data, training logs, evaluation code, the production-scale xLLM framework, and the RL360 reinforcement-learning stack (arxiv.org). Additionally, the authors release the pipelines that produced roughly 10 trillion synthetic pre-training tokens (arxiv.org).
The results
When compared against four open-weight models released around the same time (GLM-5.2, MiniMax M3, Nemotron 3 Ultra, and Inkling), the 375B-A23B model is smaller than most of them in both active and total parameters yet ranks first or second on most agentic and coding benchmarks (arxiv.org). It leads all four on Toolathlon-Verified with a score of 65.3, where it also surpasses GPT-5.6 Terra and comes close to GPT-5.6 Luna (arxiv.org). It leads them on repository-level code understanding without internet access (SWE-Atlas-QnA) with a score of 48.4, and it exceeds both GPT-5.6 variants on tau-3-Banking (arxiv.org). Its main gap is expert-level scientific reasoning on HLE and CritPt (arxiv.org). The 36B-A4B model uses MoVA and softplus attention gating to nearly match a 32B dense model trained on the same data, while using one-seventh the active parameters (arxiv.org). Among small models, the 7B outperforms similarly sized open-weight models on most benchmarks, and both the 7B and 3.7B lead SWE-bench Verified by wide margins (arxiv.org).
Why it matters
By releasing the full development lifecycle—including intermediate checkpoints, training code, and synthetic data pipelines—the work provides a complete and transparent recipe for reproducing frontier models at various scales (arxiv.org). This level of openness allows researchers to conduct controlled studies on scale, sparsity, and the impact of synthetic data in pre-training (arxiv.org).
Caveats
The authors note that their main performance gap is in expert-level scientific reasoning on benchmarks such as HLE and CritPt (arxiv.org). Additionally, they observe that small models subjected to extensive long-horizon training are prone to repetitive loops, requiring a brief stage of Final Token Preference Optimization to address (arxiv.org). Finally, this is a preprint (arxiv.org).


