Sequential decision making under uncertainty: efficient Q-learning frameworks

Abstract

This thesis focuses on the development of efficient, convergent algorithms for solving problems in dynamic programming, reinforcement learning, and multi-agent learning. The work begins with novel first-order iterative schemes derived from a computationally expensive second-order method that approximates the Bellman equation using smooth functions. These new schemes retain the global convergence property while being more computationally efficient and easier to implement. Next, the thesis proposes a Weighted Smooth Q-Learning (WSQL) algorithm to address overestimation and underestimation biases in Q-learning and double Q-learning, respectively. By incorporating a weighted blend of mellowmax and log-sum-exp operators, WSQL achieves stability and theoretical convergence guarantees. The third part of the thesis introduces off-policy two-step Q-learning algorithms—both standard and smooth variants—that improve convergence and robustness without relying on importance sampling. Finally, the thesis extends these techniques to the multi-agent setting, proposing a multi-step minimax Q-learning algorithm for solving two-player zero-sum Markov games. Theoretical analysis ensures boundedness and almost sure convergence of the algorithms under suitable assumptions. Across all contributions, the proposed methods are validated through comprehensive numerical experiments on benchmark problems, demonstrating their e!ectiveness, robustness, and practical utility. Keywords: Reinforcement Learning, Q-learning, Bellman Equation, Value Iteration, Two-Player Zero-Sum Games, Stochastic Approximation, Smooth Operators.