[Paper] Assessing the Business Process Modeling Competences of Large Language Models

Source: arXiv - 2601.21787v1

Overview

The paper Assessing the Business Process Modeling Competences of Large Language Models examines how well modern LLMs can generate Business Process Model and Notation (BPMN) diagrams from plain‑language specifications. By introducing a systematic evaluation framework (BEF4LLM), the authors compare open‑source LLMs against seasoned BPMN modelers, shedding light on where AI currently shines—and where it still falls short—in automating a core enterprise‑architecture task.

Key Contributions

• BEF4LLM framework – a four‑dimensional rubric (syntactic, pragmatic, semantic, validity) for rigorously assessing LLM‑generated BPMN models.
• Comprehensive benchmark – evaluation of several open‑source LLMs (e.g., Llama 2, Mistral) alongside human experts on a curated set of real‑world process descriptions.
• Empirical findings – LLMs match or exceed humans on syntactic and pragmatic quality, while humans retain a modest edge on semantic fidelity and overall validity.
• Practical guidance – concrete recommendations for model fine‑tuning, prompt engineering, and post‑generation validation to improve real‑world deployment.

Methodology

1. Dataset creation – The authors collected a diverse corpus of business process narratives (e.g., order‑to‑cash, employee onboarding) and manually crafted reference BPMN diagrams.
2. LLM prompting – Each narrative was fed to several open‑source LLMs using a standardized “text‑to‑BPMN” prompt, producing XML‑based BPMN files.
3. BEF4LLM scoring
   • Syntactic: checks for well‑formed BPMN XML (correct tags, IDs, connectors).
   • Pragmatic: evaluates adherence to BPMN conventions (proper use of gateways, event types).
   • Semantic: measures how accurately the generated diagram captures the intended business logic (e.g., correct ordering of tasks).
   • Validity: combines the above with domain‑specific constraints (e.g., no dead‑ends, proper start/end events).
4. Human baseline – Experienced BPMN modelers performed the same task, providing a performance ceiling.
5. Statistical analysis – Scores were aggregated and compared using paired t‑tests and effect‑size metrics to quantify gaps.

Results & Findings

• Strengths: LLMs reliably produce well‑formed BPMN files and respect modeling syntax, making them suitable for rapid prototyping.
• Weaknesses: Semantic drift (mis‑ordered tasks, missing conditions) and occasional validity violations (e.g., orphaned gateways) remain the main pain points.
• Overall: The performance gap is modest, especially in syntactic/pragmatic aspects, suggesting LLMs are already viable assistants for BPMN creation.

Practical Implications

• Rapid diagram generation – Developers can integrate an LLM‑based “text‑to‑BPMN” service into low‑code platforms, cutting initial modeling time by up to 50 %.
• Assistive tooling – IDE plugins could suggest BPMN fragments on‑the‑fly as engineers write process documentation, improving consistency across teams.
• Cost‑effective prototyping – Small‑to‑mid‑size enterprises can prototype workflows without hiring dedicated BPMN analysts, reserving expert review for final validation.
• Fine‑tuning opportunities – The identified semantic gaps point to targeted fine‑tuning on domain‑specific process corpora, promising further gains with relatively low data overhead.
• Compliance checks – Pairing LLM output with automated validity validators (e.g., Camunda’s BPMN engine) can catch the remaining errors before deployment.

Limitations & Future Work

• Scope of process types – The benchmark focused on common enterprise processes; niche or highly regulated workflows may expose additional weaknesses.
• Open‑source LLMs only – Proprietary models (e.g., GPT‑4) were not evaluated, leaving open the question of how much further performance can be pushed.
• Human evaluation bias – Human experts were limited to a small pool, which may not capture the full variability of modeling expertise.
• Future directions suggested by the authors include:
  1. Expanding the dataset to cover more industry verticals.
  2. Exploring reinforcement‑learning‑from‑human‑feedback (RLHF) loops to improve semantic fidelity.
  3. Integrating domain ontologies to boost validity checks.

Authors

• Chantale Lauer
• Peter Pfeiffer
• Alexander Rombach
• Nijat Mehdiyev

Paper Information

• arXiv ID: 2601.21787v1
• Categories: cs.SE, cs.AI
• Published: January 29, 2026
• PDF: Download PDF