MAE Seminar: From Robust Control to Robust Machine and Reinforcement Learning - A Unifying Theory via Performance and Risk Tradeoff

Abstract: Robustness is a fundamental concept in systems science and engineering. It is a critical consideration in all inference and decision-making problems, both single agent and multiagent ones. It has surfaced again in recent years in the context of machine learning (ML), reinforcement learning (RL) and artificial intelligence (AI). We describe a novel and unifying theory of robustness for all these problems emanating form the fundamental results we obtained in my research group some 25 years ago on robust output feedback control for general systems (including nonlinear, HMM and set-valued). In the first part of this lecture, I will summarize this theory and the universal solution it provides consisting of two coupled HJB equations. These results are a sweeping generalization of the transformational control theory results on the linear quadratic gaussian problem obtained by Jacobson, Speyer, Doyle, Glover, Khargonekar, Francis, in the 1970s and 1980s. Our results rigorously established the equivalence of three seemingly unrelated problems: the robust output feedback control problem, a partially observed differential game, and a partially observed risk sensitive stochastic control problem. In the second part of this lecture, I will start by showing the “four block” view of this problem and show for the first time a similar formulation of the so-called robust (or adversarially robust) ML problem. Thus we have a rigorous path to analyze robustness and attack resiliency in ML. I will show several examples. I will also describe how using an exponential criterion in deep learning explains the convergence of stochastic gradients despite over-parametrization (Poggio 2020). Then I will describe our most recent results on robust and risk sensitive reinforcement learning. Here the emergence of exponential of an integral criterion from our earlier theory is essential. We show how all forms of regularized RL can be derived from our theory, including KL and Entropy regularization, relation to probabilistic graphical models and distributional robustness. The deeper reason for this unification emerges: it is the fundamental tradeoff between performance optimization and risk minimization in decision making, via duality. This connects to Prospect Theory. I will close with open problems and future research.

Bio: John S. Baras is a Distinguished University Professor, holding the Lockheed Martin Chair in Systems Engineering and a permanent joint appointment with the Institute for Systems Research (ISR) and the Department of Electrical and Computer Engineering at the University of Maryland College Park. He received his doctorate in applied mathematics from Harvard University in 1973, and he has been with UMD since then. From 1985 to 1991, he was the founding director of the ISR. Since 1992, he has been the director of the Maryland Center for Hybrid Networks, which he co-founded. He is a fellow of IEEE (Life), SIAM, AAAS, NAI, IFAC, AMS, AIAA, member of the National Academy of Inventors and a foreign member of the Royal Swedish Academy of Engineering Sciences. Major honors and awards include the 1980 George Axelby Award from the IEEE Control Systems Society, 2006 Leonard Abraham Prize from the IEEE Communications Society, 2017 IEEE Simon Ramo Medal, 2017 AACC Richard E. Bellman Control Heritage Award, and 2018 AIAA Aerospace Communications Award. In 2016 he was inducted in the University of Maryland A. J. Clark School of Engineering Innovation Hall of Fame. In June 2018 he was awarded a doctorate honoris causa by his alma mater the National Technical University of Athens, Greece. His research interests include systems, control, optimization, autonomy, machine learning, artificial intelligence, communication networks, applied mathematics, signal processing and understanding, robotics, computing systems, formal methods and logic, network security and trust, systems biology, healthcare management and model-based systems engineering. He has been awarded 20 patents and honored with many awards as innovator and leader of economic development.