[Paper] ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization

Source: arXiv - 2602.15983v1

Overview

Large language models (LLMs) are getting good at turning natural‑language problem statements into optimization code, but they often produce “silent failures”: the generated code runs without crashing yet solves the wrong mathematical formulation. The paper ReLoop proposes a two‑pronged solution—structured generation and behavioral verification—that dramatically reduces these hidden errors and makes LLM‑driven optimization pipelines far more trustworthy.

Key Contributions

• Structured Generation Pipeline – a four‑stage reasoning chain (understand → formalize → synthesize → verify) that mirrors how human modelers build optimization problems, with explicit variable‑type reasoning to catch formulation bugs early.
• Behavioral Verification Framework – a lightweight, solver‑based perturbation test that checks whether the generated model behaves as expected, without needing ground‑truth code.
• IIS‑Enhanced Execution Recovery – when verification flags an error, the system automatically extracts an Irreducible Inconsistent Subsystem (IIS) to pinpoint and repair the faulty constraints.
• Comprehensive Empirical Evaluation – experiments on five LLM families (foundation, SFT, RL) across three benchmark suites show correctness improvements from 22.6 % to 31.1 % and a jump to 100 % successful execution.
• RetailOpt‑190 Dataset – a new collection of 190 compositional retail‑optimization scenarios that expose multi‑constraint interactions where LLMs typically stumble, released for the community.

Methodology

1. Understanding – the LLM parses the natural‑language description, extracts entities (variables, parameters) and their types (continuous, integer, binary).
2. Formalizing – it builds a symbolic representation of the objective and constraints, explicitly linking each term to the previously identified variables.
3. Synthesizing – the model translates the symbolic form into concrete code (e.g., Pyomo, JuMP) while preserving the type annotations.
4. Self‑Verification – before execution, the system runs simple sanity checks (e.g., dimension consistency, bound feasibility).

If the model passes these checks, behavioral verification kicks in: the generated optimization problem is solved repeatedly while systematically perturbing parameters (e.g., demand, cost). The resulting solution trajectories are compared against expected monotonic or feasibility patterns derived from the problem statement. Deviations trigger the IIS diagnostic, which isolates the offending constraints for automatic repair or human‑in‑the‑loop correction.

Results & Findings

Across five LLMs (including GPT‑4‑style, fine‑tuned, and RL‑trained variants) and three benchmark suites, ReLoop consistently lifted both correctness and execution rates. The behavioral verifier alone contributed the biggest single boost on problems where a single constraint was mis‑specified, while the structured pipeline shone on deeply nested, multi‑stage retail scenarios.

Practical Implications

• Safer AI‑assisted Modeling – developers can now rely on LLMs to draft optimization models for supply‑chain, scheduling, or finance tasks without fearing silent logical bugs that would otherwise surface only after costly downstream analysis.
• Rapid Prototyping – the four‑stage chain can be wrapped into IDE plugins or CI pipelines, turning a natural‑language spec into production‑ready code in minutes while automatically flagging hidden errors.
• Debug‑as‑a‑Service – the IIS‑based diagnostics give developers concrete, actionable feedback (e.g., “constraint C3 mixes binary and continuous variables”), reducing the time spent hunting down subtle formulation mistakes.
• Dataset‑Driven Benchmarking – RetailOpt‑190 provides a realistic testbed for any company building LLM‑driven decision‑support tools, encouraging more robust evaluation beyond toy examples.
• Cross‑Domain Applicability – although demonstrated on linear/integer programming, the verification ideas extend to mixed‑integer nonlinear, stochastic, or even reinforcement‑learning‑based optimization pipelines.

Limitations & Future Work

• Scalability of Verification – behavioral tests involve multiple solves; for very large‑scale models (e.g., millions of variables) the overhead could become prohibitive.
• Coverage of Perturbation Rules – the current perturbation heuristics are handcrafted for the benchmark domains; automatically deriving appropriate perturbations for arbitrary problem classes remains open.
• Residual Correctness Gap – even with ReLoop, only ~31 % of generated models are semantically correct, indicating that deeper reasoning or external knowledge bases may be needed.
• Human‑in‑the‑Loop Integration – future work could explore tighter UI/UX loops where developers intervene on IIS diagnostics in real time, potentially boosting correctness further.

Overall, ReLoop marks a significant step toward trustworthy, LLM‑powered optimization, turning what was once a risky “code‑gen” shortcut into a reliable component of modern decision‑automation pipelines.

Authors

• Junbo Jacob Lian
• Yujun Sun
• Huiling Chen
• Chaoyu Zhang
• Chung-Piaw Teo

Paper Information

• arXiv ID: 2602.15983v1
• Categories: cs.SE, cs.AI, cs.LG, math.OC
• Published: February 17, 2026
• PDF: Download PDF