具有随机回报的分布稳健机会约束马尔可夫决策过程

IF 1.6 2区 数学 Q2 MATHEMATICS, APPLIED Applied Mathematics and Optimization Pub Date : 2024-07-26 DOI:10.1007/s00245-024-10167-w
Hoang Nam Nguyen, Abdel Lisser, Vikas Vikram Singh
{"title":"具有随机回报的分布稳健机会约束马尔可夫决策过程","authors":"Hoang Nam Nguyen,&nbsp;Abdel Lisser,&nbsp;Vikas Vikram Singh","doi":"10.1007/s00245-024-10167-w","DOIUrl":null,"url":null,"abstract":"<div><p>A Markov Decision Process (MDP) is a framework used for decision-making. In an MDP problem, the decision maker’s goal is to maximize the expected discounted value of future rewards while navigating through different states controlled by a Markov chain. In this paper, we focus on the case where the transition probabilities vector is deterministic, while the reward vector is uncertain and follow a partially known distribution. We employ a distributionally robust chance constraints approach to model the MDP. This approach entails the construction of potential distributions of reward vector, characterized by moments or statistical metrics. We explore two situations for these ambiguity sets: one where the reward vector has a real support and another where it is constrained to be nonnegative. In the case of a real support, we demonstrate that solving the distributionally robust chance-constrained Markov decision process is mathematically equivalent to a second-order cone programming problem for moments and <span>\\(\\phi \\)</span>-divergence ambiguity sets. For Wasserstein distance ambiguity sets, it becomes a mixed-integer second-order cone programming problem. In contrast, when dealing with nonnegative reward vector, the equivalent optimization problems are different. Moments-based ambiguity sets lead to a copositive optimization problem, while Wasserstein distance-based ambiguity sets result in a biconvex optimization problem. To illustrate the practical application of these methods, we examine a machine replacement problem and present results conducted on randomly generated instances to showcase the effectiveness of our proposed methods.</p></div>","PeriodicalId":55566,"journal":{"name":"Applied Mathematics and Optimization","volume":"90 1","pages":""},"PeriodicalIF":1.6000,"publicationDate":"2024-07-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Distributionally Robust Chance-Constrained Markov Decision Processes with Random Payoff\",\"authors\":\"Hoang Nam Nguyen,&nbsp;Abdel Lisser,&nbsp;Vikas Vikram Singh\",\"doi\":\"10.1007/s00245-024-10167-w\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p>A Markov Decision Process (MDP) is a framework used for decision-making. In an MDP problem, the decision maker’s goal is to maximize the expected discounted value of future rewards while navigating through different states controlled by a Markov chain. In this paper, we focus on the case where the transition probabilities vector is deterministic, while the reward vector is uncertain and follow a partially known distribution. We employ a distributionally robust chance constraints approach to model the MDP. This approach entails the construction of potential distributions of reward vector, characterized by moments or statistical metrics. We explore two situations for these ambiguity sets: one where the reward vector has a real support and another where it is constrained to be nonnegative. In the case of a real support, we demonstrate that solving the distributionally robust chance-constrained Markov decision process is mathematically equivalent to a second-order cone programming problem for moments and <span>\\\\(\\\\phi \\\\)</span>-divergence ambiguity sets. For Wasserstein distance ambiguity sets, it becomes a mixed-integer second-order cone programming problem. In contrast, when dealing with nonnegative reward vector, the equivalent optimization problems are different. Moments-based ambiguity sets lead to a copositive optimization problem, while Wasserstein distance-based ambiguity sets result in a biconvex optimization problem. To illustrate the practical application of these methods, we examine a machine replacement problem and present results conducted on randomly generated instances to showcase the effectiveness of our proposed methods.</p></div>\",\"PeriodicalId\":55566,\"journal\":{\"name\":\"Applied Mathematics and Optimization\",\"volume\":\"90 1\",\"pages\":\"\"},\"PeriodicalIF\":1.6000,\"publicationDate\":\"2024-07-26\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Applied Mathematics and Optimization\",\"FirstCategoryId\":\"100\",\"ListUrlMain\":\"https://link.springer.com/article/10.1007/s00245-024-10167-w\",\"RegionNum\":2,\"RegionCategory\":\"数学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"MATHEMATICS, APPLIED\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Applied Mathematics and Optimization","FirstCategoryId":"100","ListUrlMain":"https://link.springer.com/article/10.1007/s00245-024-10167-w","RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"MATHEMATICS, APPLIED","Score":null,"Total":0}
引用次数: 0

摘要

马尔可夫决策过程(Markov Decision Process,MDP)是一种用于决策的框架。在马尔可夫决策过程问题中,决策者的目标是在马尔可夫链控制的不同状态下,最大化未来奖励的预期贴现值。在本文中,我们关注的是过渡概率向量是确定的,而奖励向量是不确定的,并且遵循部分已知分布的情况。我们采用分布稳健的机会约束方法对 MDP 进行建模。这种方法需要构建以矩或统计度量为特征的奖励向量的潜在分布。我们探讨了这些模糊集的两种情况:一种是奖励向量具有真实支持,另一种是奖励向量受限为非负。在有真实支持的情况下,我们证明了求解分布稳健的机会约束马尔可夫决策过程在数学上等价于矩数和(\phi \)-发散模糊集的二阶圆锥编程问题。对于 Wasserstein 距离模糊集,它变成了一个混合整数二阶圆锥编程问题。相比之下,在处理非负报酬向量时,等价优化问题则有所不同。基于矩的模糊集会导致一个共正优化问题,而基于瓦瑟斯坦距离的模糊集则会导致一个双凸优化问题。为了说明这些方法的实际应用,我们研究了一个机器替换问题,并展示了随机生成实例的结果,以展示我们提出的方法的有效性。
本文章由计算机程序翻译,如有差异,请以英文原文为准。

摘要图片

查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
Distributionally Robust Chance-Constrained Markov Decision Processes with Random Payoff

A Markov Decision Process (MDP) is a framework used for decision-making. In an MDP problem, the decision maker’s goal is to maximize the expected discounted value of future rewards while navigating through different states controlled by a Markov chain. In this paper, we focus on the case where the transition probabilities vector is deterministic, while the reward vector is uncertain and follow a partially known distribution. We employ a distributionally robust chance constraints approach to model the MDP. This approach entails the construction of potential distributions of reward vector, characterized by moments or statistical metrics. We explore two situations for these ambiguity sets: one where the reward vector has a real support and another where it is constrained to be nonnegative. In the case of a real support, we demonstrate that solving the distributionally robust chance-constrained Markov decision process is mathematically equivalent to a second-order cone programming problem for moments and \(\phi \)-divergence ambiguity sets. For Wasserstein distance ambiguity sets, it becomes a mixed-integer second-order cone programming problem. In contrast, when dealing with nonnegative reward vector, the equivalent optimization problems are different. Moments-based ambiguity sets lead to a copositive optimization problem, while Wasserstein distance-based ambiguity sets result in a biconvex optimization problem. To illustrate the practical application of these methods, we examine a machine replacement problem and present results conducted on randomly generated instances to showcase the effectiveness of our proposed methods.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
CiteScore
3.30
自引率
5.60%
发文量
103
审稿时长
>12 weeks
期刊介绍: The Applied Mathematics and Optimization Journal covers a broad range of mathematical methods in particular those that bridge with optimization and have some connection with applications. Core topics include calculus of variations, partial differential equations, stochastic control, optimization of deterministic or stochastic systems in discrete or continuous time, homogenization, control theory, mean field games, dynamic games and optimal transport. Algorithmic, data analytic, machine learning and numerical methods which support the modeling and analysis of optimization problems are encouraged. Of great interest are papers which show some novel idea in either the theory or model which include some connection with potential applications in science and engineering.
期刊最新文献
Null Controllability of Coupled Parabolic Systems with Switching Control Pullback Measure Attractors for Non-autonomous Fractional Stochastic Reaction-Diffusion Equations on Unbounded Domains Longtime Dynamics for a Class of Strongly Damped Wave Equations with Variable Exponent Nonlinearities On the Local Existence of Solutions to the Fluid–Structure Interaction Problem with a Free Interface A Stochastic Non-zero-Sum Game of Controlling the Debt-to-GDP Ratio
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1