Foundations and Trends in Machine Learning最新文献

英文中文

Model-based Reinforcement Learning: A Survey 基于模型的强化学习:综述

Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Foundations and Trends in Machine Learning

Pub Date : 2023-01-01 DOI: 10.1561/2200000086

Thomas M. Moerland, Joost Broekens, Aske Plaat, Catholijn M. Jonker

Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is an important challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This monograph surveys an integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps: dynamics model learning and planning-learning integration. In this comprehensive survey of the topic, the authors first cover dynamics model learning, including challenges such as dealing with stochasticity, uncertainty, partial observability, and temporal abstraction. They then present a systematic categorization of planning-learning integration, including aspects such as: where to start planning, what budgets to allocate to planning and real data collection, how to plan, and how to integrate planning in the learning and acting loop. In conclusion the authors discuss implicit model-based RL as an end-to-end alternative for model learning and planning, and cover the potential benefits of model-based RL. Along the way, the authors draw connections to several related RL fields, including hierarchical RL and transfer learning. This monograph contains a broad conceptual overview of the combination of planning and learning for Markov Decision Process optimization. It provides a clear and complete introduction to the topic for students and researchers alike.

顺序决策，通常形式化为马尔可夫决策过程(MDP)优化，是人工智能领域的一个重要挑战。解决这个问题的两个关键方法是强化学习(RL)和规划。这本专著调查了这两个领域的整合，更广为人知的是基于模型的强化学习。基于模型的强化学习主要有两个步骤:动态模型学习和计划学习集成。在这个主题的全面调查中，作者首先介绍了动态模型学习，包括处理随机性、不确定性、部分可观察性和时间抽象等挑战。然后，他们提出了计划-学习整合的系统分类，包括以下方面:从哪里开始计划，为计划和实际数据收集分配哪些预算，如何计划，以及如何将计划整合到学习和行动循环中。最后，作者讨论了隐式基于模型的强化学习作为模型学习和规划的端到端替代方案，并涵盖了基于模型的强化学习的潜在好处。在此过程中，作者将几个相关的强化学习领域联系起来，包括分层强化学习和迁移学习。这本专著包含了马尔可夫决策过程优化的规划和学习相结合的广泛概念概述。它为学生和研究人员提供了一个清晰而完整的主题介绍。

{"title":"Model-based Reinforcement Learning: A Survey","authors":"Thomas M. Moerland, Joost Broekens, Aske Plaat, Catholijn M. Jonker","doi":"10.1561/2200000086","DOIUrl":"https://doi.org/10.1561/2200000086","url":null,"abstract":"Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is an important challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This monograph surveys an integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps: dynamics model learning and planning-learning integration. In this comprehensive survey of the topic, the authors first cover dynamics model learning, including challenges such as dealing with stochasticity, uncertainty, partial observability, and temporal abstraction. They then present a systematic categorization of planning-learning integration, including aspects such as: where to start planning, what budgets to allocate to planning and real data collection, how to plan, and how to integrate planning in the learning and acting loop. In conclusion the authors discuss implicit model-based RL as an end-to-end alternative for model learning and planning, and cover the potential benefits of model-based RL. Along the way, the authors draw connections to several related RL fields, including hierarchical RL and transfer learning. This monograph contains a broad conceptual overview of the combination of planning and learning for Markov Decision Process optimization. It provides a clear and complete introduction to the topic for students and researchers alike.","PeriodicalId":47667,"journal":{"name":"Foundations and Trends in Machine Learning","volume":"10 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135799442","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}