[Paper] A Workflow-Oriented Framework for Asynchronous Human-AI Collaboration in Hybrid and Compute-Intensive HPC Environments

Source: arXiv - 2605.03743v1

Overview

The paper introduces a workflow‑oriented framework that lets human experts and AI systems collaborate asynchronously in high‑performance computing (HPC) environments. By decoupling human checkpoints from the heavy compute jobs, the framework keeps massive HPC resources busy while still allowing critical human judgment in defence‑grade AI pipelines.

Key Contributions

• Asynchronous checkpointing: Enables workflows to pause for human input without suspending the underlying HPC jobs, eliminating idle compute slots.
• Hybrid‑infrastructure support: Seamlessly integrates SLURM‑managed clusters, local workstations, and public cloud resources under a single orchestration layer.
• Container‑aware execution: Works with both native binaries and Docker/Singularity containers, simplifying reproducibility across heterogeneous systems.
• Domain‑specific extensions: Tailored APIs for defence and security use‑cases where human oversight (e.g., threat validation, policy compliance) is mandatory.
• Portability demonstration: Real‑world validation on the MareNostrum 5 supercomputer, showing minimal code changes when moving between on‑prem and cloud back‑ends.

Methodology

1. Workflow definition – Users describe a pipeline as a directed acyclic graph (DAG) of tasks. Certain nodes are marked as human‑gate tasks.
2. Scheduler abstraction – The framework translates the DAG into SLURM job scripts (or equivalent cloud batch jobs) while launching a lightweight “orchestrator” service on a local or cloud VM.
3. Non‑blocking pause – When a human‑gate task is reached, the orchestrator records the task’s state, notifies the human via a web UI or CLI, and immediately frees the compute node to continue downstream tasks that do not depend on the pending input.
4. State persistence – All intermediate artefacts (model checkpoints, logs, metadata) are stored in a shared object store (e.g., Ceph, S3) so that later human input can be merged without re‑running expensive steps.
5. Resumption – Once the expert supplies the required data (e.g., label correction, policy flag), the orchestrator injects the new artefact back into the DAG, triggers any dependent tasks, and updates the job scheduler accordingly.

The design deliberately avoids tight coupling between the HPC scheduler and the human UI, making the approach agnostic to the underlying compute platform.

Results & Findings

• Resource utilization: In a 48‑hour model‑training run on MareNostrum 5, the asynchronous checkpoint reduced idle node time by ≈ 32 % compared with a naïve “pause‑the‑job” approach.
• Turn‑around time: End‑to‑end latency from human decision to downstream task start dropped from ≈ 2 h (blocking) to ≈ 15 min (asynchronous).
• Portability: The same workflow YAML file was executed on a local workstation (4‑core CPU) and on an Azure Batch pool with no code changes, demonstrating true hybrid capability.
• User satisfaction: A small pilot with defence analysts reported a 4.5/5 usability rating for the web‑based checkpoint UI, citing clear status visibility and minimal disruption to their analysis flow.

Practical Implications

• Defence & security AI pipelines – Teams can now embed expert validation steps (e.g., target identification, rule‑based policy checks) without stalling massive training jobs, preserving both security compliance and compute efficiency.
• MLOps for compute‑intensive models – Data scientists building large language models, climate simulations, or physics‑informed neural nets can integrate “human‑in‑the‑loop” quality gates (data curation, bias audits) without sacrificing cluster throughput.
• Cost savings – By keeping HPC nodes busy while waiting for human input, organizations avoid paying for idle time on on‑prem or cloud‑burst resources, which can translate into significant operational expense reductions.
• Cross‑platform reproducibility – The container‑aware abstraction ensures that the same vetted workflow runs on a university cluster, a government supercomputer, or a commercial cloud, easing collaboration across agencies.

Limitations & Future Work

• Scalability of the orchestrator – The current prototype relies on a single orchestrator instance; scaling to thousands of concurrent human‑gate tasks will require a distributed coordination layer.
• Security boundaries – While the framework supports secure object stores, integrating fine‑grained access controls for classified data across hybrid clouds remains an open challenge.
• User interaction modalities – The study focused on a web UI; future work will explore voice, AR/VR, or automated decision‑support bots to further reduce latency.
• Generalization beyond SLURM – Extending native support to other schedulers (PBS, LSF, Kubernetes) is planned to broaden applicability across non‑HPC environments.

Bottom line: By decoupling human oversight from heavyweight compute, this workflow framework offers a pragmatic path for developers and AI engineers to embed critical expert judgment into large‑scale HPC AI projects without sacrificing performance or cost efficiency.

Authors

• Sergio Mendoza
• Cedric Bhihe
• Natalia Zamora
• David Modesto
• Jose Martin Bugallo Batalla
• Jesus Gomez Canovas
• Rafel Palomo Avellaneda
• Miguel Perez Espinosa

Paper Information

• arXiv ID: 2605.03743v1
• Categories: cs.DC, cs.AI, cs.HC, cs.SE
• Published: May 5, 2026
• PDF: Download PDF