Sequence compression with test-time regression in efficient models

The problem

Sequence compression is a fundamental operation in modern efficient large model architectures, yet it remains underexplored according to the authors (arxiv.org). For instance, block-wise sparse attention compresses token blocks into summary vectors to select relevant blocks, while multimodal large language models merge neighboring visual features into compact tokens (arxiv.org). These compressors act as critical information bottlenecks because once information is discarded, downstream tasks cannot recover it (arxiv.org). Most current models rely on simple mechanisms like parameter-free mean pooling, learned projections over concatenated tokens, or attention-weighted pooling (arxiv.org). The authors argue that these default methods share a fundamental structural limitation: they aggregate tokens using weights that ignore inter-token relationships (arxiv.org). Mean pooling uses uniform weights, linear projection uses fixed position-dependent weights, and attention-weighted pooling scores tokens independently (arxiv.org). Consequently, redundant tokens reinforce one another while distinctive information is diluted (arxiv.org).

The idea

The authors introduce TTR-Compressor, a sequence compression operator based on test-time regression (arxiv.org). Rather than computing a weighted average, the method lets input tokens define a ridge-regression problem, taking its fitted coefficients as the compressed representation (arxiv.org). The authors state that TTR-Compressor constructs a self-supervised ridge-regression problem directly from input tokens without external labels, using its closed-form solution as the compressed representation (arxiv.org). The outer training loop learns which regression problem produces useful representations, while the inner closed-form solution determines how to fit each input (arxiv.org). By explicitly capturing inter-token correlations through the Gram matrix in the regression solution, TTR-Compressor fits correlated tokens jointly, meaning redundant evidence has a diminishing rather than purely additive influence (arxiv.org). The method introduces only $2d$ learnable parameters per compressor, making it a drop-in replacement that incurs minimal overhead (arxiv.org).

The results

The authors evaluate TTR-Compressor on two applications with different compression requirements: block-wise sparse attention and visual token merging (arxiv.org). In block-wise sparse attention, where the summary must act as a faithful retrieval key, TTR-Compressor improves RULER retrieval task performance over attention-based compressors by a large margin (arxiv.org). In visual token merging, where the summary must preserve a faithful representation, TTR-Compressor consistently outperforms the widely used multi-layer perceptron (MLP) merger across diverse compression ratios, matching its performance with $55.60%$ fewer visual tokens (arxiv.org).

Why it matters

These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models (arxiv.org). By reframing token summarization as an instance-adaptive optimization problem with an efficient dual closed-form solution, TTR-Compressor overcomes the structural limitations of conventional pooling and projection-based compression (arxiv.org).

Caveats

The authors note that this work is presented as a preprint (arxiv.org). The text details various mathematical formulations and evaluations on specific tasks such as RULER and visual merging, but does not explicitly list broader failure modes or limitations outside of stating that conventional methods ignore inter-token correlations (arxiv.org).