On the Convergence of Experience Replay in Policy Optimization: Characterizing Bias, Variance, and Finite-Time Convergence

Abstract

Experience replay is a core ingredient of modern deep reinforcement learning, yet its benefits in policy optimization are poorly understood beyond empirical heuristics. This paper develops a novel theoretical framework for experience replay in modern policy gradient methods, where two sources of dependence fundamentally complicate analysis: Markovian correlations along trajectories and policy drift across optimization iterations. We introduce a new proof technique based on auxiliary Markov chains and lag-based decoupling that makes these dependencies tractable. Within this framework, we derive finite-time bias bounds for policy-gradient estimators under replay, identifying how bias scales with the cumulative policy update, the mixing time of the underlying dynamics, and the age of buffered data, thereby formalizing the practitioner's rule of avoiding overly stale replay. We further provide a correlation-aware variance decomposition showing how sample dependence governs gradient variance from replay and when replay is beneficial. Building on these characterizations, we establish the finite-time convergence guarantees for experience-replay-based policy optimization, explicitly quantifying how buffer size, sample correlation, and mixing jointly determine the convergence rate and revealing an inherent bias-variance trade-off: larger buffers can reduce variance by averaging less correlated samples but can increase bias as data become stale. These results offer a principled guide for buffer sizing and replay schedules, bridging prior empirical findings with quantitative theory.