[Paper] MLIR-Smith: A Novel Random Program Generator for Evaluating Compiler Pipelines

Source: arXiv - 2601.02218v1

Overview

The paper presents MLIR‑Smith, a random program generator built for the extensible Multi‑Level Intermediate Representation (MLIR) ecosystem. By automatically creating diverse MLIR snippets, the tool enables systematic testing of compiler pipelines that target MLIR, filling a gap left by earlier generators such as Csmith.

Key Contributions

• MLIR‑Smith generator – First open‑source random program generator that natively emits valid MLIR across arbitrary dialects.
• Differential testing framework – Integrated harness that runs generated programs through multiple back‑ends (MLIR, LLVM, DaCe, DCIR) and automatically compares outcomes.
• Bug discovery – The authors used the framework to uncover dozens of correctness and performance bugs in real‑world compiler stacks.
• Extensibility model – Demonstrated how new dialects can be plugged into the generator with minimal effort, supporting MLIR’s “dialect‑as‑plugin” philosophy.
• Public artifacts – All source code, test corpora, and scripts are released under a permissive license, encouraging community adoption.

Methodology

1. Program Sketching – MLIR‑Smith starts from a high‑level abstract syntax tree (AST) describing generic operations (e.g., arithmetic, control flow, memory accesses).
2. Dialect Mapping – For each operation, the tool selects a concrete MLIR dialect (e.g., arith, memref, linalg) based on a configurable probability distribution.
3. Random Parameterization – Types, shapes, loop bounds, and constant values are sampled from ranges that guarantee well‑typedness and avoid undefined behavior (e.g., division by zero).
4. Validity Checks – A lightweight type‑checker runs on the generated IR to ensure it satisfies MLIR’s invariants before emission.
5. Differential Execution – The same program is compiled/executed through several back‑ends. The outputs (or exit codes) are compared; mismatches trigger a bug report that includes the offending IR and a minimal reproducer.
6. Feedback Loop – Detected bugs are categorized (semantic, optimization, code‑gen) and fed back into the generator to bias future sampling toward problematic patterns.

The pipeline is deliberately simple: no sophisticated symbolic execution or constraint solving, which keeps generation fast (hundreds of programs per second) while still covering a wide semantic space.

Results & Findings

• Coverage – Over 10 k random programs were generated, exercising more than 30 distinct MLIR dialects and a variety of optimization passes.
• Bug Count – The differential testing campaign uncovered 23 bugs in the MLIR core, 7 in LLVM’s MLIR‑to‑LLVM lowering, 5 in DaCe’s code‑gen, and 3 in DCIR. Many were subtle semantic errors that only manifested under specific optimization sequences.
• Performance Insight – Some generated programs exposed regressions where aggressive loop‑fusion passes dramatically increased runtime, highlighting the need for better cost models.
• Scalability – The generator scaled linearly with the number of dialects added, confirming the design’s modularity.

Overall, the findings validate that random MLIR generation is an effective “stress test” for modern, multi‑dialect compiler pipelines.

Practical Implications

• Continuous Integration – Teams building MLIR‑based compilers can plug MLIR‑Smith into CI pipelines to catch regressions before they reach users.
• Dialect Development – New dialect authors can use the generator to sanity‑check their operation definitions and verify that existing passes handle them gracefully.
• Optimization Validation – By automatically comparing multiple back‑ends, developers can spot cases where an optimization improves one target but harms another, informing more robust pass ordering.
• Education & Debugging – The minimal reproducer output makes it easier for newcomers to understand how a particular combination of dialects triggers a bug, accelerating onboarding.
• Tooling Ecosystem – Because MLIR‑Smith is open‑source and language‑agnostic, it can serve as a foundation for higher‑level fuzzers (e.g., for domain‑specific languages that lower to MLIR).

Limitations & Future Work

• Semantic Depth – The generator focuses on syntactic correctness; it does not guarantee meaningful computational semantics (e.g., avoiding dead code or trivial loops).
• Coverage Gaps – Certain advanced dialects (e.g., gpu, tosa) receive limited sampling due to the lack of specialized randomizers.
• Performance Metrics – Current evaluation relies on output equality; richer performance profiling (e.g., cache behavior) is left for future extensions.
• Guided Fuzzing – Incorporating feedback‑directed mutation (e.g., coverage‑guided) could improve bug‑finding efficiency.

The authors plan to broaden dialect support, integrate coverage‑guided techniques, and explore automatic minimization of failing programs to further streamline the debugging workflow.

Authors

• Berke Ates
• Filip Dobrosavljević
• Theodoros Theodoridis
• Zhendong Su

Paper Information

• arXiv ID: 2601.02218v1
• Categories: cs.PL, cs.SE
• Published: January 5, 2026
• PDF: Download PDF