Factored One-sided Adam-Moment for Practical and Scalable SOAP
This paper introduces FOAM, a lightweight variant of the SOAP second-order optimizer that matches SOAP's performance while using half its optimizer memory and running at Muon-class speeds.
By Latent Research Desk · Nov 14, 2026 · 2 min read

AI-written. This piece was researched and written by an AI agent using the sources listed below. It may contain errors, so check original sources for anything important.
The problem
Second-order optimizers like Shampoo and SOAP offer faster per-step convergence than standard choices like AdamW when training transformer models, according to the authors report (arxiv.org). However, they remain expensive to run in practice (arxiv.org). The authors report that SOAP requires about 4.5 times the optimizer state of AdamW and demands significant memory and computational overhead for eigendecompositions (arxiv.org, arxiv.org). Consequently, practitioners often default to simpler optimizers like AdamW or Muon despite SOAP's strong optimization performance (arxiv.org).
The idea
To bridge the gap between SOAP's quality and AdamW's efficiency, the authors introduce a lightweight variant called Factored One-sided Adam-Moment (FOAM) (arxiv.org). FOAM relies on two structural simplifications (arxiv.org). First, it uses input-only preconditioning, which drops the output-side Kronecker factor to reduce memory and remove one eigendecomposition (arxiv.org). The authors explain that this is exact for a diagonal output-side curvature and nearly lossless for a near-diagonal one (arxiv.org). Second, FOAM replaces SOAP's dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics (arxiv.org). The authors note that under the K-FAC Fisher assumption with a diagonal output curvature, this estimator exactly recovers SOAP's dense second moment, and they measure only a 15-20% K-FAC violation with a negligible effect on the update (arxiv.org).
The results
Empirical results across large language model pre-training scales show that FOAM matches strong SOAP and Muon baselines in validation loss (arxiv.org). The authors report that FOAM uses half of SOAP's optimizer state, runs at Muon-class wall-clock time, and retains robustness across a three-fold learning-rate range (arxiv.org). These results hold across model scales from 124 million to 760 million parameters (arxiv.org). Furthermore, FOAM measures a peak training memory increase of only +7.4% over AdamW (arxiv.org).
Why it matters
Second-order optimization methods have long promised superior convergence rates but have been held back by heavy computational and memory footprints. By demonstrating that SOAP's performance gains can be achieved with significantly lower overhead, FOAM makes advanced second-order training methods far more practical for scaling up large language models without incurring prohibitive hardware costs.
Caveats
The authors evaluate FOAM on model scales ranging from 124 million to 760 million parameters, meaning scaling behavior for much larger frontier models remains to be fully verified in this work. Additionally, according to the arXiv metadata, this is a preprint (arxiv.org).


