[Paper] Acceleration of Parallel Tempering for Markov Chain Monte Carlo methods

Source: arXiv - 2512.03825v1

Overview

The paper presents a high‑performance implementation of Parallel Tempering (PT) for Markov Chain Monte Carlo (MCMC), targeting modern multi‑core CPUs and GPUs. By coupling the classic Metropolis‑Hastings sampler with PT and accelerating it with OpenMP and CUDA, the authors achieve order‑of‑magnitude speed‑ups, making it feasible to tackle much larger and more complex statistical‑physics models (e.g., protein folding, Ising lattices) in reasonable time.

Key Contributions

• Parallel PT implementation using two mainstream parallel programming models: OpenMP (CPU) and CUDA (GPU).
• Performance benchmarks showing up to 52× speed‑up on a 48‑core CPU and 986× speed‑up on a high‑end GPU.
• Reference baseline for future quantum‑computing implementations of PT‑MCMC, providing a concrete classical performance target.
• Open‑source‑ready design that isolates the PT logic from the underlying hardware, facilitating reuse in other scientific codes.

Methodology

1. Algorithmic foundation – The authors start from the Metropolis‑Hastings MCMC algorithm and embed it within a Parallel Tempering framework. PT runs several replicas of the system at different temperatures; periodically, neighboring replicas attempt to swap states, allowing low‑temperature chains to escape local minima.
2. Parallelization strategy –
   • CPU (OpenMP): Each replica is assigned to a separate thread. The swap step is synchronized using lightweight barriers, while the Metropolis updates are embarrassingly parallel within each replica.
   • GPU (CUDA): Replicas are mapped to CUDA blocks; each thread within a block handles a subset of the lattice or particle coordinates. The swap operation is performed via atomic operations and shared‑memory reductions to keep latency low.
3. Implementation details – The code is written in C++ with templated kernels to support arbitrary energy functions. Memory layout is optimized for coalesced access on the GPU, and the authors employ pinned host memory to speed up CPU‑GPU data transfers.
4. Benchmark suite – Synthetic Ising‑model instances of varying size (from 32×32 up to 1024×1024 spins) and a simplified protein‑folding lattice model are used to evaluate scaling behavior and raw throughput.

Results & Findings

• Swap acceptance rates remain comparable to the serial baseline, confirming that parallelism does not degrade the statistical quality of the samples.
• GPU memory footprint grows linearly with the number of replicas, but fits comfortably on modern cards up to 64 replicas.
• The CUDA version outperforms the CPU version by an order of magnitude even when the CPU uses all available cores, highlighting the advantage of massive data‑parallelism for the Metropolis updates.

Practical Implications

• Accelerated scientific simulations: Researchers can now run PT‑MCMC on realistic system sizes (e.g., large protein lattices, high‑resolution Ising grids) within hours instead of days, opening the door to more exhaustive parameter sweeps and Bayesian inference tasks.
• Integration into existing toolchains: Because the implementation is built on standard OpenMP and CUDA APIs, it can be dropped into existing C/C++ simulation packages (e.g., LAMMPS, GROMACS) with minimal refactoring.
• Cost‑effective scaling: For teams without access to large HPC clusters, a single workstation equipped with a modern GPU can achieve performance comparable to a small CPU cluster, reducing both hardware and energy costs.
• Benchmark for quantum algorithms: The reported classical speed‑ups provide a concrete yardstick for emerging quantum PT‑MCMC proposals, helping the community assess when quantum advantage becomes realistic.

Limitations & Future Work

• Memory bandwidth bound on GPUs: Scaling stalls once the number of replicas exceeds the GPU’s memory bandwidth, suggesting that future work could explore mixed‑precision or compression techniques.
• Model specificity: The benchmarks focus on lattice‑based energy functions; extending the approach to continuous‑space molecular dynamics or more complex potentials may require additional kernel optimizations.
• Swap synchronization overhead: While negligible for modest replica counts, the barrier synchronization could become a bottleneck for hundreds of replicas; asynchronous or hierarchical swapping schemes are a promising avenue.
• Quantum comparison: The authors plan to implement a quantum PT‑MCMC version, but the current work does not yet provide a direct performance comparison or error analysis for quantum hardware.

Authors

• Aingeru Ramos
• Jose A Pascual
• Javier Navaridas
• Ivan Coluzza

Paper Information

• arXiv ID: 2512.03825v1
• Categories: cs.DC
• Published: December 3, 2025
• PDF: Download PDF