[Paper] Enhancing Performance Insight at Scale: A Heterogeneous Framework for Exascale Diagnostics

Source: arXiv - 2605.03561v1

Overview

The paper presents a new, heterogeneous diagnostics framework that lets developers and system operators extract performance insights from exascale machines—systems with millions of concurrent threads—without drowning in telemetry overhead. By combining a low‑latency C++ API with GPU‑accelerated analysis, the authors demonstrate that massive execution traces can be ingested and processed in seconds, opening the door to real‑time performance tuning on today’s largest supercomputers.

Key Contributions

• High‑throughput C++ ingestion API – pulls in telemetry from 100 k MPI ranks on the Aurora system in under 10 seconds.
• GPU‑accelerated diagnostics layer – delivers up to 314× speedup over pure‑CPU processing for trace analysis at the same scale.
• Topology‑aware outlier mapping – automatically correlates logical performance anomalies to physical Slingshot interconnect coordinates, pinpointing network congestion across 22 racks.
• Tri‑dimensional performance model – “re‑materializes” iterative behavior from raw traces, enabling quantitative predictions of speedup (e.g., 32.28 % for a GAMESS workload on Frontier).
• Open integration hooks – a clean C++/Python interface that lets external tools plug in custom analytics or machine‑learning models without rewriting the core infrastructure.

Methodology

1. Data Capture – The existing hpcanalysis framework collects per‑rank telemetry (timings, counters, network metrics) during a run.
2. C++ Ingestion Layer – A thin, lock‑free C++ API streams these records directly into a shared memory buffer, avoiding costly file I/O and serialization.
3. GPU‑Accelerated Processing – The buffered data is transferred to the GPU where a set of CUDA kernels perform common diagnostics (e.g., histogramming, correlation, outlier detection). Because kernels operate on millions of records in parallel, the analysis time collapses from minutes to seconds.
4. Topology Mapping – The framework queries the system’s topology service to translate logical rank IDs into physical node and interconnect coordinates, then visualizes hotspots on a rack‑level map.
5. Tri‑dimensional Modeling – A three‑axis model (time, iteration, resource usage) is built from the trace, allowing the system to “re‑play” the computation and estimate how changes (e.g., load‑balancing, communication pattern tweaks) would affect overall runtime.

All steps are orchestrated through a high‑level Python driver, keeping the workflow approachable for developers who are not GPU‑programming experts.

Results & Findings

The authors also show that the topology‑aware mapping correctly identified a bottleneck in the Slingshot fabric that was previously invisible to standard profiling tools, leading to a configuration change that reduced overall runtime by ~5 %.

Practical Implications

• Real‑time performance steering – System administrators can now run diagnostics during a production job and intervene before a slowdown becomes costly.
• Scalable toolchain integration – Because the API is language‑agnostic, existing CI pipelines (e.g., for HPC code regression testing) can embed these diagnostics without massive refactoring.
• Network‑aware optimization – Mapping outliers to physical interconnect locations enables targeted hardware tuning (e.g., routing policy changes) that would otherwise require exhaustive manual probing.
• Accelerated research cycles – GPU‑driven analysis reduces turnaround from days to minutes, allowing developers to iterate on algorithmic changes (e.g., load‑balancing strategies) much faster.
• Cross‑system portability – The framework was validated on both Aurora (Intel Xeon + Slingshot) and Frontier (AMD EPYC + HPE Cray network), suggesting it can be adopted on other exascale or near‑exascale platforms with minimal effort.

Limitations & Future Work

• GPU memory ceiling – Extremely large traces (multi‑TB) still need to be chunked, which introduces modest overhead; future work will explore out‑of‑core GPU processing and streaming kernels.
• Model generality – The tri‑dimensional performance model is currently tuned for iterative scientific codes (e.g., quantum chemistry); extending it to irregular, event‑driven workloads will require additional feature engineering.
• Hardware dependency – While the API is portable, the current speedup numbers rely on NVIDIA‑class GPUs; evaluating performance on AMD or Intel GPUs is left for later studies.
• User‑level tooling – The paper provides a prototype Python driver, but a full‑featured UI (e.g., web dashboard) is still under development.

Overall, the framework marks a significant step toward making exascale performance diagnostics as routine as unit testing is for software developers today.

Authors

• Dragana Grbic

Paper Information

• arXiv ID: 2605.03561v1
• Categories: cs.DC, cs.PF
• Published: May 5, 2026
• PDF: Download PDF