No Weighted-Regret Learning in Adversarial Bandits with Delays · arXivDesk
2103.04550Mar 8, 2021Accepted to JMLR. This is an extended journal version of the preliminary conference paper "Online EXP3 Learning in Adversarial Bandits with Delayed Feedback" published in Neurips 2019
No Weighted-Regret Learning in Adversarial Bandits with Delays
Ilai Bistritz, Zhengyuan Zhou, Xi Chen, Nicholas Bambos, Jose Blanchet
Consider a scenario where a player chooses an action in each round t out of T rounds and observes the incurred cost after a delay of dt rounds. The cost functions and the delay sequence are chosen by an adversary. We show that in a non-cooperative game, the expected weighted ergodic distribution of play converges to the set of coarse correlated equilibria if players use algorithms that have "no weighted-regret" in the above scenario, even if they have linear regret due to too large delays. For a two-player zero-sum game, we show that no weighted-regret is sufficient for the weighted ergodic average of play to converge to the set of Nash equilibria. We prove that the FKM algorithm with n
Nearby in the stack
dimensions achieves an expected regret of
O(nT43+nT31D31)
and the EXP3 algorithm with
K
arms achieves an expected regret of
O(logK(KT+D))
even when
D=∑t=1Tdt
and
T
are unknown. These bounds use a novel doubling trick that, under mild assumptions, provably retains the regret bound for when
D
and
T
are known. Using these bounds, we show that FKM and EXP3 have no weighted-regret even for
dt=O(tlogt)
. Therefore, algorithms with no weighted-regret can be used to approximate a CCE of a finite or convex unknown game that can only be simulated with bandit feedback, even if the simulation involves significant delays.