[Paper] AI4EOSC: a Federated Cloud Platform for Artificial Intelligence in Scientific Research

Source: arXiv - 2512.16455v1

Overview

The paper presents AI4EOSC, a federated cloud platform that stitches together multiple European e‑Infrastructure sites to give scientists a single, reproducible environment for the entire AI/ML workflow—from interactive model development to large‑scale training on GPUs and seamless deployment across the cloud continuum. By abstracting the underlying heterogeneity, AI4EOSC aims to make AI‑driven research more transparent, portable, and collaborative.

Key Contributions

• Federated Architecture – A unified service layer that aggregates compute, storage, and AI services from geographically distributed e‑Infrastructure providers.
• End‑to‑End ML Lifecycle Support – Integrated tooling for data annotation, experiment tracking, GPU‑accelerated training, federated learning, and multi‑target deployment (edge, cloud, HPC).
• Reproducibility & Traceability – Automated provenance capture, container‑based packaging, and versioned model registries to ensure experiments can be reproduced across sites.
• Extensible Service Catalog – Plug‑in model providers, dataset repositories, and storage back‑ends, allowing communities to tailor the platform to domain‑specific needs.
• User‑Friendly Interfaces – Interactive development environments (JupyterLab, VS Code Server) and web dashboards that hide the complexity of the underlying federation.
• Open‑Source Reference Implementation – A publicly available codebase and deployment scripts that demonstrate how to spin up the platform on existing research infrastructures.

Methodology

The authors built AI4EOSC on top of existing standards (OpenID Connect for identity, OIDC‑compatible OAuth for authorization, and the European Open Science Cloud (EOSC) APIs). The platform consists of three logical layers:

1. Federation Layer – Registers and monitors remote sites, exposing a common catalogue of compute (CPU/GPU), storage, and AI services via a central broker.
2. Orchestration Layer – Uses Kubernetes (with federation extensions) to schedule containers, manage GPU allocation, and enforce policy (e.g., data locality, quota).
3. User Experience Layer – Provides web‑based portals and APIs that let users launch Jupyter notebooks, submit training jobs, track experiments (via MLflow‑compatible metadata), and deploy models through serverless functions or container registries.

The team evaluated the platform on a testbed of four European research clouds, measuring deployment time, job turnaround, and reproducibility across sites. They also conducted user‑studies with domain scientists to assess usability.

Results & Findings

• Deployment Consistency – A full ML pipeline (data ingest → notebook → GPU training → model registry) could be reproduced on any of the four sites with ≤ 5 % variation in runtime, confirming the effectiveness of container‑based isolation and the federation broker.
• Performance Overhead – The additional abstraction layer added an average of 2–3 % latency for job submission and 1 % for data transfer, which the authors deem negligible compared to the benefits of portability.
• User Satisfaction – Surveyed researchers reported a 30 % reduction in time spent on environment setup and a 25 % increase in confidence that results could be shared and reproduced.
• Scalability – The platform successfully coordinated simultaneous training jobs on 8 GPUs across three sites, demonstrating that federated scheduling can handle modest multi‑site workloads without bottlenecks.

Practical Implications

• Accelerated AI Research – Developers can focus on model innovation rather than wrestling with heterogeneous cloud credentials, VM images, or GPU provisioning.
• Cross‑Institution Collaboration – Teams spread across Europe (or beyond) can share notebooks and trained models without manual data movement, fostering reproducible science.
• Cost‑Effective Resource Utilization – The broker can route jobs to under‑utilized sites, balancing load and potentially lowering compute costs for research projects.
• Edge‑to‑Cloud Deployments – By exposing deployment options from edge devices to large cloud clusters, AI4EOSC enables real‑time inference use‑cases (e.g., remote sensing, IoT analytics) within the same managed environment.
• Template for Other Domains – The modular service catalog and open‑source stack can be adapted for fields like genomics, climate modeling, or industrial IoT, lowering the barrier for AI adoption in any data‑intensive science.

Limitations & Future Work

• Geographic Scope – The current evaluation is limited to four European sites; broader global federation may expose latency and policy challenges not yet addressed.
• Data Governance – While authentication is standardized, fine‑grained data‑access policies across jurisdictions remain an open problem.
• Federated Learning Maturity – Support for privacy‑preserving federated learning is prototype‑level; more robust algorithms and security audits are needed.
• Automation of Resource Negotiation – Future work includes smarter, policy‑driven scheduling that can automatically negotiate quotas and pricing across participating clouds.

Overall, AI4EOSC demonstrates that a well‑engineered federated cloud can make AI research more reproducible, collaborative, and scalable—an enticing prospect for developers looking to bring cutting‑edge ML into scientific workflows without the usual infrastructure headaches.

Authors

• Ignacio Heredia
• Álvaro López García
• Germán Moltó
• Amanda Calatrava
• Valentin Kozlov
• Alessandro Costantini
• Viet Tran
• Mario David
• Daniel San Martín
• Marcin Płóciennik
• Marta Obregón Ruiz
• Saúl Fernandez
• Judith Sáinz-Pardo Díaz
• Miguel Caballer
• Caterina Alarcón Marín
• Stefan Dlugolinsky
• Martin Šeleng
• Lisana Berberi
• Khadijeh Alibabaei
• Borja Esteban Sanchis
• Pedro Castro
• Giacinto Donvito
• Diego Aguirre
• Sergio Langarita
• Vicente Rodriguez
• Leonhard Duda
• Andrés Heredia Canales
• Susana Rebolledo Ruiz
• João Machado
• Giang Nguyen
• Fernando Aguilar Gómez
• Jaime Díez

Paper Information

• arXiv ID: 2512.16455v1
• Categories: cs.DC, cs.AI
• Published: December 18, 2025
• PDF: Download PDF