TB-MathModel benchmarks mathematical modeling agents with containerized verifiers

Mathematical modeling challenges require converting messy, open-ended real-world scenarios into assumptions, mathematical formulations, code, and actionable recommendations. While traditional benchmarks test isolated math puzzles or rely on language models to grade free-form written reports, researchers introduced TB-MathModel to turn entire collegiate mathematical modeling problems into verifiable, containerized agent tasks. By combining reproducible reference solutions from prize-winning human teams with programmatic physics and logic checks, the benchmark provides a machine-checkable way to assess whether multimodal AI agents can solve complex, long-horizon modeling challenges without generating physically impossible answers.

The problem

According to the authors of the paper, evaluating artificial intelligence on mathematical modeling exposes a fundamental gap in current benchmarks. Traditional math benchmarks, such as GSM8K or MATH, measure formal deduction on closed-form word problems where a single number or expression is undeniably correct. Operations-research datasets evaluate whether an algorithm can solve an already-formulated optimization problem, but they do not test how an agent chooses abstractions, makes simplifying assumptions, or conducts sensitivity analyses.

Conversely, existing modeling evaluations fall into two extremes. Some assess narrow solver fragments, while others instruct a large language model to grade long essay-style reports. The authors argue that language model judges reward fluent, well-written prose without confirming whether the underlying calculations are accurate or physically possible. Furthermore, real mathematical modeling problems do not have a single unique ground truth. Different human teams make different modeling assumptions, meaning an automated evaluation cannot simply penalize any deviation from a single reference solution or punish an agent when its model achieves a mathematically superior result.

The idea

To bridge this gap, the researchers built TB-MathModel, an agentic benchmark based on 18 complete problems from the 2023 to 2025 editions of two premier competitions: the China Undergraduate Mathematical Contest in Modeling (CUMCM) and the international Mathematical Contest in Modeling (MCM). The dataset includes verbatim problem prompts, official data attachments, and 21 figures cropped from the original PDFs, which contain essential visual information such as geometric trajectory curves.

Agents are placed inside an isolated, containerized terminal environment with these materials and must generate a single structured JSON artifact. To evaluate the output fairly and rigorously, TB-MathModel implements three central design choices:

1. Multi-paper reference panels: Rather than treating one human response as absolute truth, the benchmark compares an agent's numeric answers against reproductions of human papers that won Outstanding awards.
2. Direction-aware, one-sided scoring: Each metric has a designated semantic direction, such as target values, exact values, or directional objectives (where higher or lower numbers indicate better performance). Under this system, an agent is never penalized for finding a more optimal solution than the reference paper.
3. Two-layer feasibility verification: The benchmark subjects outputs to a programmatic gate across six decidable invariant forms, supported by 624 kernel-checked proof obligations written in the Lean 4 interactive theorem prover. If an agent outputs numbers that violate physical constraints—such as geometric boundaries or conservation laws—a hard gate zeroes out the score.

The results

Testing seven multimodal model and harness configurations across three random seeds revealed notable differences in performance and efficiency, according to the benchmark evaluation. The strongest entries reached a multi-reference O-Eval of 64.58%, while an open model achieved a score of 59.81% at 1/100 the cost, incurring an evaluation cost of $2.54. Varying only the harness moved the O-Eval score by at most 0.07 percentage points. Variations across random seeds reached up to 2.9 percentage points, indicating that differences between closely ranked models on the leaderboard were not statistically separated.

The authors also highlighted a case study on a problem known as "dragon-dance." In that instance, the models generated answers that contradicted the benchmark's original anchor solution. Further inspection revealed that the human anchor was flawed and physically infeasible, while the models' answers were correct—an outcome that allowed the benchmark creators to fix their anchor before publication.

Why it matters

TB-MathModel demonstrates that open-ended mathematical modeling can be measured automatically without relying on subjective language model judges. By establishing executable criteria for feasibility alongside directional scoring, the benchmark creates a repeatable framework for testing how well AI systems reason over long horizons, write and execute code, interpret multimodal data files, and adhere to physical constraints.

Because the benchmark logs full execution traces and verifies outputs programmatically, the authors suggest the platform can also serve as a reproducible environment for gathering agent trajectories to train and fine-tune future reasoning models.

Caveats

According to the authors, several limitations remain. The work is currently an unvetted preprint. The benchmark is restricted to 18 tasks spanning three contest years, which limits broader statistical sampling across historic modeling styles. Additionally, constraining agents to a structured JSON schema creates tension with the open-ended nature of modeling: human competitions involve rich written discussions and secondary plots, whereas the automated verifier scores only the designated numeric JSON fields and ignores auxiliary spreadsheets, images, or written reports generated during execution.