[Paper] Aligning LLMs Toward Multi-Turn Conversational Outcomes Using Iterative PPO

Source: arXiv

Abstract

Optimizing large language models (LLMs) for multi‑turn conversational outcomes remains a significant challenge, especially in goal‑oriented settings like AI marketing or sales agents who facilitate transactions via messaging platforms. The difficulty stems from sparse, long‑horizon rewards and the discrepancy between response‑level planning and token‑level generation.

In this technical note, we propose a formal reduction of the multi‑turn RL problem into a sequence of single‑turn RLHF‑style problems. This is achieved by setting a learned multi‑turn Q‑function as the reward model for the single‑turn problem. We demonstrate and prove a key insight: solving this single‑turn RL problem with standard token‑level PPO is equivalent to a policy improvement step within the multi‑turn problem.

This insight naturally leads to Iterative PPO, a batch online policy iteration algorithm that alternates between fitting Q‑functions from logged conversation trajectories and improving the policy. A major practical advantage is that Iterative PPO directly leverages stable, off‑the‑shelf single‑turn RLHF tools, making it straightforward to implement. Our method occupies a middle ground between fully online and fully offline approaches, retaining the adaptability of online updates while gaining the stability benefits of offline training.