[Paper] Transformed Latent Variable Multi-Output Gaussian Processes

Source: arXiv - 2605.05133v1

Overview

The paper introduces Transformed Latent Variable Multi‑Output Gaussian Processes (T‑LVMOGP), a new way to make Gaussian‑process models work when you have thousands—or even tens of thousands—of correlated output variables. By blending neural‑network embeddings with a variational inference scheme, the authors achieve both scalability and expressive power, opening the door for high‑dimensional tasks such as climate forecasting and spatial transcriptomics.

Key Contributions

• Scalable multi‑output GP architecture that handles >10k outputs without resorting to overly restrictive low‑rank kernels.
• Deep kernel construction: inputs and output‑specific latent variables are projected into a shared embedding space via a Lipschitz‑regularised neural network.
• Stochastic variational inference tailored to the transformed latent space, enabling efficient training on large datasets.
• Empirical validation on real‑world benchmarks (global climate modeling, zero‑inflated spatial transcriptomics) showing superior predictive accuracy and faster runtimes versus state‑of‑the‑art baselines.
• Open‑source implementation (released with the paper) that integrates with popular GP libraries (e.g., GPyTorch, GPflow).

Methodology

1. Latent Variable Augmentation – Each output dimension (d) is assigned a low‑dimensional latent vector (\mathbf{z}_d). These vectors capture output‑specific characteristics (e.g., geographic location for climate sensors).

2. Neural Embedding – A neural network (f_{\theta}(\cdot)) takes the concatenation of the input (\mathbf{x}) and the latent (\mathbf{z}d) and maps them to an embedding (\mathbf{h}{x,d}=f_{\theta}([\mathbf{x},\mathbf{z}_d])). The network is regularised with a Lipschitz penalty to keep the mapping smooth and well‑behaved.

3. Deep Kernel – A standard stationary kernel (e.g., RBF) is applied in the embedding space:

   [ k_{d,d’}(\mathbf{x},\mathbf{x}’) = k_{\text{RBF}}(\mathbf{h}{x,d}, \mathbf{h}{x’,d’}). ]

   This yields a flexible, data‑driven multi‑output kernel that can capture complex cross‑output relationships.

4. Stochastic Variational Inference (SVI) – The joint posterior over the GP function values and the latent vectors is approximated with a factorised variational distribution. Mini‑batch stochastic gradients (via the reparameterisation trick) update both the GP parameters and the neural network weights, keeping memory usage linear in the batch size rather than the total number of outputs.

5. Training Pipeline – The authors implement the model in PyTorch, leveraging automatic differentiation and GPU acceleration for the neural embedding, while the GP computations are handled by GPyTorch’s scalable kernel tricks (e.g., inducing points).

Results & Findings

• Predictive quality improves consistently across metrics (RMSE, NLL, MAE), especially on tasks with strong inter‑output correlation.
• Computational efficiency scales roughly linearly with the number of outputs, thanks to the stochastic variational scheme and the compact latent representation.
• Ablation studies show that the Lipschitz regularisation stabilises training and prevents over‑fitting of the neural embedding, while a latent dimension of 5–10 is sufficient for most benchmarks.

Practical Implications

• Large‑scale sensor networks: Engineers can now build GP‑based predictive models for thousands of IoT devices (e.g., smart‑city air‑quality stations) without sacrificing the ability to capture spatial or temporal dependencies.
• Geoscience & climate analytics: Researchers can replace cumbersome ensemble methods with a single probabilistic model that yields calibrated uncertainties for each grid cell.
• Bioinformatics pipelines: In spatial transcriptomics, T‑LVMOGP handles zero‑inflated count data and provides smooth expression maps, facilitating downstream tasks like cell‑type deconvolution.
• Integration with existing stacks: Because the model is built on PyTorch/GPyTorch, developers can plug it into existing ML pipelines, benefit from GPU acceleration, and combine it with downstream deep‑learning components (e.g., classifiers).
• Uncertainty‑aware decision making: The GP framework naturally supplies predictive variances, enabling risk‑aware automation (e.g., adaptive sampling for environmental monitoring).

Limitations & Future Work

• Latent dimensionality selection still requires manual tuning; an automated Bayesian non‑parametric prior could make the model more adaptive.
• Kernel choice is limited to stationary kernels in the embedding space; extending to non‑stationary or spectral kernels may capture richer dynamics.
• Interpretability of latent vectors is indirect; future work could enforce structured priors (e.g., geographic coordinates) to make the latent space more semantically meaningful.
• Scalability beyond 20k outputs: While the method scales well up to ~15k outputs, memory consumption of the inducing point set becomes a bottleneck; hierarchical inducing schemes are a promising direction.

TL;DR: T‑LVMOGP merges neural embeddings with variational Gaussian processes to deliver a scalable, expressive multi‑output model that works on real‑world, high‑dimensional problems—making it a compelling tool for developers building uncertainty‑aware systems at scale.

Authors

• Xiaoyu Jiang
• Xinxing Shi
• Sokratia Georgaka
• Magnus Rattray
• Mauricio A Álvarez

Paper Information

• arXiv ID: 2605.05133v1
• Categories: cs.LG
• Published: May 6, 2026
• PDF: Download PDF