{"title":"Physics-Informed Neural Networks with Trust-Region Sequential Quadratic Programming","authors":"Xiaoran Cheng, Sen Na","doi":"arxiv-2409.10777","DOIUrl":null,"url":null,"abstract":"Physics-Informed Neural Networks (PINNs) represent a significant advancement\nin Scientific Machine Learning (SciML), which integrate physical domain\nknowledge into an empirical loss function as soft constraints and apply\nexisting machine learning methods to train the model. However, recent research\nhas noted that PINNs may fail to learn relatively complex Partial Differential\nEquations (PDEs). This paper addresses the failure modes of PINNs by\nintroducing a novel, hard-constrained deep learning method -- trust-region\nSequential Quadratic Programming (trSQP-PINN). In contrast to directly training\nthe penalized soft-constrained loss as in PINNs, our method performs a\nlinear-quadratic approximation of the hard-constrained loss, while leveraging\nthe soft-constrained loss to adaptively adjust the trust-region radius. We only\ntrust our model approximations and make updates within the trust region, and\nsuch an updating manner can overcome the ill-conditioning issue of PINNs. We\nalso address the computational bottleneck of second-order SQP methods by\nemploying quasi-Newton updates for second-order information, and importantly,\nwe introduce a simple pretraining step to further enhance training efficiency\nof our method. We demonstrate the effectiveness of trSQP-PINN through extensive\nexperiments. Compared to existing hard-constrained methods for PINNs, such as\npenalty methods and augmented Lagrangian methods, trSQP-PINN significantly\nimproves the accuracy of the learned PDE solutions, achieving up to 1-3 orders\nof magnitude lower errors. Additionally, our pretraining step is generally\neffective for other hard-constrained methods, and experiments have shown the\nrobustness of our method against both problem-specific parameters and algorithm\ntuning parameters.","PeriodicalId":501162,"journal":{"name":"arXiv - MATH - Numerical Analysis","volume":"82 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - MATH - Numerical Analysis","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.10777","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Physics-Informed Neural Networks (PINNs) represent a significant advancement
in Scientific Machine Learning (SciML), which integrate physical domain
knowledge into an empirical loss function as soft constraints and apply
existing machine learning methods to train the model. However, recent research
has noted that PINNs may fail to learn relatively complex Partial Differential
Equations (PDEs). This paper addresses the failure modes of PINNs by
introducing a novel, hard-constrained deep learning method -- trust-region
Sequential Quadratic Programming (trSQP-PINN). In contrast to directly training
the penalized soft-constrained loss as in PINNs, our method performs a
linear-quadratic approximation of the hard-constrained loss, while leveraging
the soft-constrained loss to adaptively adjust the trust-region radius. We only
trust our model approximations and make updates within the trust region, and
such an updating manner can overcome the ill-conditioning issue of PINNs. We
also address the computational bottleneck of second-order SQP methods by
employing quasi-Newton updates for second-order information, and importantly,
we introduce a simple pretraining step to further enhance training efficiency
of our method. We demonstrate the effectiveness of trSQP-PINN through extensive
experiments. Compared to existing hard-constrained methods for PINNs, such as
penalty methods and augmented Lagrangian methods, trSQP-PINN significantly
improves the accuracy of the learned PDE solutions, achieving up to 1-3 orders
of magnitude lower errors. Additionally, our pretraining step is generally
effective for other hard-constrained methods, and experiments have shown the
robustness of our method against both problem-specific parameters and algorithm
tuning parameters.