Benchmarks in Leipzig

Source: Hacker News

Authors:Andrei Balakin, Miklós Bóna, Marie-Charlotte Brandenburg, Clara Briand, Veronica Calvo Cortes, Shelby Cox, Jesus A. De Loera, Danai Deligeorgaki, Hannah Friedman, Tim Gehrunger, Chiara Giardino, Stephen Griffeth, Baran Hashemi, Elena Hoster, Alexander Ivanov, Nupur Jain, Aryaman Jal, Leonie Kayser, Joris Koefler, Kevin Kühn, Mario Kummer, Felix Lotter, René Marczinzik, Victor S. Miller, Alejandro Morales, Greta Panova, Gianni Petrella, Nathan Pflueger, Lakshmi Ramesh, Nikolas Rieke, Carlos Rodriguez, Andrea Rosana, Flavio Salizzoni, Otto T.P. Schmidt, Sven Ulf Schmitz, Lina Maria Simbaqueba Marin, Luca Sodomaco, Christian Stump, Bernd Sturmfels, Alexander Taveira Blomenhofer, Simon Telen, Philipp Tuchel, Emil Verkama, Carl Felix Waller, Julian Weigert, Annette Werner, Nathan Williams, Claudius Zibrowius

[View PDF](https://arxiv.org/pdf/2606.05818)
[HTML (experimental)](https://arxiv.org/html/2606.05818v1)
        Abstract:Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers. Most of the work was done during the 3-day workshop *Benchmarks in Leipzig* with 35 participants at the Max Planck Institute for Mathematics in the Sciences in Leipzig, Germany. We present the resulting collection of 100 questions. We evaluated these questions in three stages: a single attempt by five state-of-the-art LLMs, followed by a 20-runs-per-model evaluation with three of these models, and finally a 3-run attempt with two heavy-thinking models. After Stage 1, 41 questions remained completely unsolved; after Stage 2, this count dropped to 16; and we concluded Stage 3 with only 2 unsolved questions. This demonstrates that the mathematical reasoning capabilities of LLMs are becoming impressive.




          
      Comments:
      8 pages including 8 benchmark statistics tables + 20 pages appendix containing the 100 Leipzig Benchmark questions
    

      Subjects:
      
        History and Overview (math.HO); Artificial Intelligence (cs.AI); Algebraic Geometry (math.AG); Combinatorics (math.CO); Representation Theory (math.RT)
    
      Cite as:
      [arXiv:2606.05818](https://arxiv.org/abs/2606.05818) [math.HO]
    
    
       
      (or 
          [arXiv:2606.05818v1](https://arxiv.org/abs/2606.05818v1) [math.HO] for this version)
      
    
    
       
                    [https://doi.org/10.48550/arXiv.2606.05818](https://doi.org/10.48550/arXiv.2606.05818)
          
          
              arXiv-issued DOI via DataCite (pending registration)

        
      
    


  

Submission history

From: Christian Stump [view email]
[v1] Thu, 4 Jun 2026 07:59:08 UTC (38 KB)