Pub Date : 2023-11-13DOI: 10.48550/arxiv.2311.07541
Kovács, György, Fazekas, Attila
Addressing the reproducibility crisis in artificial intelligence through the validation of reported experimental results is a challenging task. It necessitates either the reimplementation of techniques or a meticulous assessment of papers for deviations from the scientific method and best statistical practices. To facilitate the validation of reported results, we have developed numerical techniques capable of identifying inconsistencies between reported performance scores and various experimental setups in machine learning problems, including binary/multiclass classification and regression. These consistency tests are integrated into the open-source package mlscorecheck, which also provides specific test bundles designed to detect systematically recurring flaws in various fields, such as retina image processing and synthetic minority oversampling.
{"title":"mlscorecheck: Testing the consistency of reported performance scores and\u0000 experiments in machine learning","authors":"Kovács, György, Fazekas, Attila","doi":"10.48550/arxiv.2311.07541","DOIUrl":"https://doi.org/10.48550/arxiv.2311.07541","url":null,"abstract":"Addressing the reproducibility crisis in artificial intelligence through the validation of reported experimental results is a challenging task. It necessitates either the reimplementation of techniques or a meticulous assessment of papers for deviations from the scientific method and best statistical practices. To facilitate the validation of reported results, we have developed numerical techniques capable of identifying inconsistencies between reported performance scores and various experimental setups in machine learning problems, including binary/multiclass classification and regression. These consistency tests are integrated into the open-source package mlscorecheck, which also provides specific test bundles designed to detect systematically recurring flaws in various fields, such as retina image processing and synthetic minority oversampling.","PeriodicalId":496270,"journal":{"name":"arXiv (Cornell University)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136353162","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-13DOI: 10.48550/arxiv.2311.07366
Blesch, Kristin, Wright, Marvin N.
This paper introduces $textit{arfpy}$, a python implementation of Adversarial Random Forests (ARF) (Watson et al., 2023), which is a lightweight procedure for synthesizing new data that resembles some given data. The software $textit{arfpy}$ equips practitioners with straightforward functionalities for both density estimation and generative modeling. The method is particularly useful for tabular data and its competitive performance is demonstrated in previous literature. As a major advantage over the mostly deep learning based alternatives, $textit{arfpy}$ combines the method's reduced requirements in tuning efforts and computational resources with a user-friendly python interface. This supplies audiences across scientific fields with software to generate data effortlessly.
{"title":"arfpy: A python package for density estimation and generative modeling\u0000 with adversarial random forests","authors":"Blesch, Kristin, Wright, Marvin N.","doi":"10.48550/arxiv.2311.07366","DOIUrl":"https://doi.org/10.48550/arxiv.2311.07366","url":null,"abstract":"This paper introduces $textit{arfpy}$, a python implementation of Adversarial Random Forests (ARF) (Watson et al., 2023), which is a lightweight procedure for synthesizing new data that resembles some given data. The software $textit{arfpy}$ equips practitioners with straightforward functionalities for both density estimation and generative modeling. The method is particularly useful for tabular data and its competitive performance is demonstrated in previous literature. As a major advantage over the mostly deep learning based alternatives, $textit{arfpy}$ combines the method's reduced requirements in tuning efforts and computational resources with a user-friendly python interface. This supplies audiences across scientific fields with software to generate data effortlessly.","PeriodicalId":496270,"journal":{"name":"arXiv (Cornell University)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136353306","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-13DOI: 10.48550/arxiv.2311.07369
Chataing, Nicolas, Dolan, Stephen, Scherer, Gabriel, Yallop, Jeremy
We propose a new language feature for ML-family languages, the ability to selectively *unbox* certain data constructors, so that their runtime representation gets compiled away to just the identity on their argument. Unboxing must be statically rejected when it could introduce *confusions*, that is, distinct values with the same representation. We discuss the use-case of big numbers, where unboxing allows to write code that is both efficient and safe, replacing either a safe but slow version or a fast but unsafe version. We explain the static analysis necessary to reject incorrect unboxing requests. We present our prototype implementation of this feature for the OCaml programming language, discuss several design choices and the interaction with advanced features such as Guarded Algebraic Datatypes. Our static analysis requires expanding type definitions in type expressions, which is not necessarily normalizing in presence of recursive type definitions. In other words, we must decide normalization of terms in the first-order lambda-calculus with recursion. We provide an algorithm to detect non-termination on-the-fly during reduction, with proofs of correctness and completeness. Our termination-monitoring algorithm turns out to be closely related to the normalization strategy for macro expansion in the `cpp` preprocessor.
{"title":"Unboxed data constructors -- or, how cpp decides a halting problem","authors":"Chataing, Nicolas, Dolan, Stephen, Scherer, Gabriel, Yallop, Jeremy","doi":"10.48550/arxiv.2311.07369","DOIUrl":"https://doi.org/10.48550/arxiv.2311.07369","url":null,"abstract":"We propose a new language feature for ML-family languages, the ability to selectively *unbox* certain data constructors, so that their runtime representation gets compiled away to just the identity on their argument. Unboxing must be statically rejected when it could introduce *confusions*, that is, distinct values with the same representation. We discuss the use-case of big numbers, where unboxing allows to write code that is both efficient and safe, replacing either a safe but slow version or a fast but unsafe version. We explain the static analysis necessary to reject incorrect unboxing requests. We present our prototype implementation of this feature for the OCaml programming language, discuss several design choices and the interaction with advanced features such as Guarded Algebraic Datatypes. Our static analysis requires expanding type definitions in type expressions, which is not necessarily normalizing in presence of recursive type definitions. In other words, we must decide normalization of terms in the first-order lambda-calculus with recursion. We provide an algorithm to detect non-termination on-the-fly during reduction, with proofs of correctness and completeness. Our termination-monitoring algorithm turns out to be closely related to the normalization strategy for macro expansion in the `cpp` preprocessor.","PeriodicalId":496270,"journal":{"name":"arXiv (Cornell University)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136353309","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Safety assurance is of paramount importance across various domains, including automotive, aerospace, and nuclear energy, where the reliability and acceptability of mission-critical systems are imperative. This assurance is effectively realized through the utilization of Safety Assurance Cases. The use of safety assurance cases allows for verifying the correctness of the created systems capabilities, preventing system failure. The latter may result in loss of life, severe injuries, large-scale environmental damage, property destruction, and major economic loss. Still, the emergence of complex technologies such as cyber-physical systems (CPSs), characterized by their heterogeneity, autonomy, machine learning capabilities, and the uncertainty of their operational environments poses significant challenges for safety assurance activities. Several papers have tried to propose solutions to tackle these challenges, but to the best of our knowledge, no secondary study investigates the trends, patterns, and relationships characterizing the safety case scientific literature. This makes it difficult to have a holistic view of the safety case landscape and to identify the most promising future research directions. In this paper, we, therefore, rely on state-of-the-art bibliometric tools(e.g., VosViewer) to conduct a bibliometric analysis that allows us to generate valuable insights, identify key authors and venues, and gain a birds eye view of the current state of research in the safety assurance area. By revealing knowledge gaps and highlighting potential avenues for future research, our analysis provides an essential foundation for researchers, corporate safety analysts, and regulators seeking to embrace or enhance safety practices that align with their specific needs and objectives.
{"title":"The Last Decade in Review: Tracing the Evolution of Safety Assurance\u0000 Cases through a Comprehensive Bibliometric Analysis","authors":"Sivakumar, Mithila, Belle, Alvine Boaye, Shan, Jinjun, Adesina, Opeyemi, Wang, Song, Chechik, Marsha, Fokaefs, Marios, Shahandashti, Kimya Khakzad, Odu, Oluwafemi","doi":"10.48550/arxiv.2311.07495","DOIUrl":"https://doi.org/10.48550/arxiv.2311.07495","url":null,"abstract":"Safety assurance is of paramount importance across various domains, including automotive, aerospace, and nuclear energy, where the reliability and acceptability of mission-critical systems are imperative. This assurance is effectively realized through the utilization of Safety Assurance Cases. The use of safety assurance cases allows for verifying the correctness of the created systems capabilities, preventing system failure. The latter may result in loss of life, severe injuries, large-scale environmental damage, property destruction, and major economic loss. Still, the emergence of complex technologies such as cyber-physical systems (CPSs), characterized by their heterogeneity, autonomy, machine learning capabilities, and the uncertainty of their operational environments poses significant challenges for safety assurance activities. Several papers have tried to propose solutions to tackle these challenges, but to the best of our knowledge, no secondary study investigates the trends, patterns, and relationships characterizing the safety case scientific literature. This makes it difficult to have a holistic view of the safety case landscape and to identify the most promising future research directions. In this paper, we, therefore, rely on state-of-the-art bibliometric tools(e.g., VosViewer) to conduct a bibliometric analysis that allows us to generate valuable insights, identify key authors and venues, and gain a birds eye view of the current state of research in the safety assurance area. By revealing knowledge gaps and highlighting potential avenues for future research, our analysis provides an essential foundation for researchers, corporate safety analysts, and regulators seeking to embrace or enhance safety practices that align with their specific needs and objectives.","PeriodicalId":496270,"journal":{"name":"arXiv (Cornell University)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136353318","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-13DOI: 10.48550/arxiv.2311.07520
Maliyov, Ivan, Yin, Jia, Yao, Jia, Yang, Chao, Bernardi, Marco
Nonequilibrium dynamics governed by electron-phonon (e-ph) interactions plays a key role in electronic devices and spectroscopies and is central to understanding electronic excitations in materials. The real-time Boltzmann transport equation (rt-BTE) with collision processes computed from first principles can describe the coupled dynamics of electrons and atomic vibrations (phonons). Yet, a bottleneck of these simulations is the calculation of e-ph scattering integrals on dense momentum grids at each time step. Here we show a data-driven approach based on dynamic mode decomposition (DMD) that can accelerate the time propagation of the rt-BTE and identify dominant electronic processes. We apply this approach to two case studies, high-field charge transport and ultrafast excited electron relaxation. In both cases, simulating only a short time window of ~10% of the dynamics suffices to predict the dynamics from initial excitation to steady state using DMD extrapolation. Analysis of the momentum-space modes extracted from DMD sheds light on the microscopic mechanisms governing electron relaxation to steady state or equilibrium. The combination of accuracy and efficiency makes our DMD-based method a valuable tool for investigating ultrafast dynamics in a wide range of materials.
{"title":"Dynamic mode decomposition of nonequilibrium electron-phonon dynamics:\u0000 accelerating the first-principles real-time Boltzmann equation","authors":"Maliyov, Ivan, Yin, Jia, Yao, Jia, Yang, Chao, Bernardi, Marco","doi":"10.48550/arxiv.2311.07520","DOIUrl":"https://doi.org/10.48550/arxiv.2311.07520","url":null,"abstract":"Nonequilibrium dynamics governed by electron-phonon (e-ph) interactions plays a key role in electronic devices and spectroscopies and is central to understanding electronic excitations in materials. The real-time Boltzmann transport equation (rt-BTE) with collision processes computed from first principles can describe the coupled dynamics of electrons and atomic vibrations (phonons). Yet, a bottleneck of these simulations is the calculation of e-ph scattering integrals on dense momentum grids at each time step. Here we show a data-driven approach based on dynamic mode decomposition (DMD) that can accelerate the time propagation of the rt-BTE and identify dominant electronic processes. We apply this approach to two case studies, high-field charge transport and ultrafast excited electron relaxation. In both cases, simulating only a short time window of ~10% of the dynamics suffices to predict the dynamics from initial excitation to steady state using DMD extrapolation. Analysis of the momentum-space modes extracted from DMD sheds light on the microscopic mechanisms governing electron relaxation to steady state or equilibrium. The combination of accuracy and efficiency makes our DMD-based method a valuable tool for investigating ultrafast dynamics in a wide range of materials.","PeriodicalId":496270,"journal":{"name":"arXiv (Cornell University)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136353325","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-13DOI: 10.48550/arxiv.2311.07523
Yang, Yewon, Geiser, Achim
Measuring the total charm cross section is important for the comparison to theoretical predictions of the highest precision available for charm today, which are completely known up to NNLO QCD for the total inclusive cross sections. These are also independent of charm fragmentation, while practical measurements of charm hadrons in a fiducial phase space are not. Recently the LHC experiments have reported non-universality of charm fragmentation, which shows that e.g. charm baryon-to-meson ratios are not universal in different collision systems, and that the related production fractions also depend on transverse momentum. This breaks the charm fragmentation universality that was assumed until recently for the extrapolation of experimental measurements to the full total charm cross section phase space. A proposal is made how to address this non-universality in a data driven way without the need to implement any particular non-universal fragmentation model. As a practical example, this method is applied to the extrapolation of published LHC measurements of $D^0$ production at $sqrt{s}=5$ TeV to the corresponding total charm cross section, which fully accounts for charm fragmentation non-universality for the first time. The result, $8.43 ^{+1.05}_{-1.16}(text{total})$ mb, differs substantially from the one assuming charm fragmentation universality, but still compares well to theoretical QCD predictions up to NNLO.
{"title":"A novel phenomenological approach to total charm cross section\u0000 measurements at the LHC","authors":"Yang, Yewon, Geiser, Achim","doi":"10.48550/arxiv.2311.07523","DOIUrl":"https://doi.org/10.48550/arxiv.2311.07523","url":null,"abstract":"Measuring the total charm cross section is important for the comparison to theoretical predictions of the highest precision available for charm today, which are completely known up to NNLO QCD for the total inclusive cross sections. These are also independent of charm fragmentation, while practical measurements of charm hadrons in a fiducial phase space are not. Recently the LHC experiments have reported non-universality of charm fragmentation, which shows that e.g. charm baryon-to-meson ratios are not universal in different collision systems, and that the related production fractions also depend on transverse momentum. This breaks the charm fragmentation universality that was assumed until recently for the extrapolation of experimental measurements to the full total charm cross section phase space. A proposal is made how to address this non-universality in a data driven way without the need to implement any particular non-universal fragmentation model. As a practical example, this method is applied to the extrapolation of published LHC measurements of $D^0$ production at $sqrt{s}=5$ TeV to the corresponding total charm cross section, which fully accounts for charm fragmentation non-universality for the first time. The result, $8.43 ^{+1.05}_{-1.16}(text{total})$ mb, differs substantially from the one assuming charm fragmentation universality, but still compares well to theoretical QCD predictions up to NNLO.","PeriodicalId":496270,"journal":{"name":"arXiv (Cornell University)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136353327","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-13DOI: 10.48550/arxiv.2311.07145
Nasirov, Avazbek, Kayumov, Bakhodir
The evaporation residue (ER) cross section of 3n and 4n channels related to the synthesis of superheavy element (SHE) with the charge number $Z=119$ in the $^{51}$V+$^{248}$Cm reaction has been calculated by the dinuclear system (DNS) model as a sum of the partial cross sections of the corresponding channels. The angular momentum distribution of the compound nucleus is estimated by the dynamical trajectory calculations of the capture probability which is considered as the DNS formation probability. The fusion probability decreases by the increase of the DNS angular momentum due to its influence on the intrinsic fusion barrier $B_{rm fus}^*$. The range $alpha_2=60^{circ} div 70^{circ}$ of the orientation angle of the axial symmetry axis of the deformed target nucleus $^{248}$Cm is favorable for the formation of the compound nucleus. The fusion probability decreases at around $alpha_2=90^{circ}$ since the number of the partial waves contributing to the capture decreases. Therefore, it is important to calculate the capture cross section dynamically. The 4n channel cross section of the SHE synthesis is larger than the 3n channel cross section maximum value of the ER cross section is 12.3 fb at $E_{rm c.m.}$=232 MeV.
{"title":"Optimal colliding energy for the synthesis of superheavy element $Z$=119","authors":"Nasirov, Avazbek, Kayumov, Bakhodir","doi":"10.48550/arxiv.2311.07145","DOIUrl":"https://doi.org/10.48550/arxiv.2311.07145","url":null,"abstract":"The evaporation residue (ER) cross section of 3n and 4n channels related to the synthesis of superheavy element (SHE) with the charge number $Z=119$ in the $^{51}$V+$^{248}$Cm reaction has been calculated by the dinuclear system (DNS) model as a sum of the partial cross sections of the corresponding channels. The angular momentum distribution of the compound nucleus is estimated by the dynamical trajectory calculations of the capture probability which is considered as the DNS formation probability. The fusion probability decreases by the increase of the DNS angular momentum due to its influence on the intrinsic fusion barrier $B_{rm fus}^*$. The range $alpha_2=60^{circ} div 70^{circ}$ of the orientation angle of the axial symmetry axis of the deformed target nucleus $^{248}$Cm is favorable for the formation of the compound nucleus. The fusion probability decreases at around $alpha_2=90^{circ}$ since the number of the partial waves contributing to the capture decreases. Therefore, it is important to calculate the capture cross section dynamically. The 4n channel cross section of the SHE synthesis is larger than the 3n channel cross section maximum value of the ER cross section is 12.3 fb at $E_{rm c.m.}$=232 MeV.","PeriodicalId":496270,"journal":{"name":"arXiv (Cornell University)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136353426","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-13DOI: 10.48550/arxiv.2311.07187
Chen, Junqing, Jin, Bangti, Liu, Haibo
We propose a novel iterative numerical method to solve the three-dimensional inverse obstacle scattering problem of recovering the shape of the obstacle from far-field measurements. To address the inherent ill-posed nature of the inverse problem, we advocate the use of a trained latent representation of surfaces as the generative prior. This prior enjoys excellent expressivity within the given class of shapes, and meanwhile, the latent dimensionality is low, which greatly facilitates the computation. Thus, the admissible manifold of surfaces is realistic and the resulting optimization problem is less ill-posed. We employ the shape derivative to evolve the latent surface representation, by minimizing the loss, and we provide a local convergence analysis of a gradient descent type algorithm to a stationary point of the loss. We present several numerical examples, including also backscattered and phaseless data, to showcase the effectiveness of the proposed algorithm.
{"title":"Solving Inverse Obstacle Scattering Problem with Latent Surface\u0000 Representations","authors":"Chen, Junqing, Jin, Bangti, Liu, Haibo","doi":"10.48550/arxiv.2311.07187","DOIUrl":"https://doi.org/10.48550/arxiv.2311.07187","url":null,"abstract":"We propose a novel iterative numerical method to solve the three-dimensional inverse obstacle scattering problem of recovering the shape of the obstacle from far-field measurements. To address the inherent ill-posed nature of the inverse problem, we advocate the use of a trained latent representation of surfaces as the generative prior. This prior enjoys excellent expressivity within the given class of shapes, and meanwhile, the latent dimensionality is low, which greatly facilitates the computation. Thus, the admissible manifold of surfaces is realistic and the resulting optimization problem is less ill-posed. We employ the shape derivative to evolve the latent surface representation, by minimizing the loss, and we provide a local convergence analysis of a gradient descent type algorithm to a stationary point of the loss. We present several numerical examples, including also backscattered and phaseless data, to showcase the effectiveness of the proposed algorithm.","PeriodicalId":496270,"journal":{"name":"arXiv (Cornell University)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136353442","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-13DOI: 10.48550/arxiv.2311.07311
Hong, Xudong, Ryzhova, Margarita, Biondi, Daniel Adrian, Demberg, Vera
Recently, large pre-trained language models (LLMs) have demonstrated superior language understanding abilities, including zero-shot causal reasoning. However, it is unclear to what extent their capabilities are similar to human ones. We here study the processing of an event $B$ in a script-based story, which causally depends on a previous event $A$. In our manipulation, event $A$ is stated, negated, or omitted in an earlier section of the text. We first conducted a self-paced reading experiment, which showed that humans exhibit significantly longer reading times when causal conflicts exist ($neg A rightarrow B$) than under logical conditions ($A rightarrow B$). However, reading times remain similar when cause A is not explicitly mentioned, indicating that humans can easily infer event B from their script knowledge. We then tested a variety of LLMs on the same data to check to what extent the models replicate human behavior. Our experiments show that 1) only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the $neg A rightarrow B$ condition. 2) Despite this correlation, all models still fail to predict that $nil rightarrow B$ is less surprising than $neg A rightarrow B$, indicating that LLMs still have difficulties integrating script knowledge. Our code and collected data set are available at https://github.com/tony-hong/causal-script.
最近，大型预训练语言模型(llm)已经展示了卓越的语言理解能力，包括零概率因果推理。然而，目前尚不清楚它们的能力在多大程度上与人类相似。我们在这里研究基于脚本的故事中事件$B$的处理，它因果地依赖于之前的事件$ a $。在我们的操作中，事件$A$在文本的前面部分被声明、否定或省略。我们首先进行了一个自定节奏的阅读实验，结果表明，当存在因果冲突($负a 右箭头B$)时，人类的阅读时间明显长于逻辑条件($ a 右箭头B$)。然而，当没有明确提到原因A时，阅读时间仍然相似，这表明人类可以很容易地从他们的脚本知识中推断出事件B。然后，我们在相同的数据上测试了各种llm，以检查模型在多大程度上复制了人类行为。我们的实验表明，1)只有最近的LLMs，如GPT-3或Vicuna，与人类在负A右B条件下的行为相关。2)尽管存在这种相关性，但所有模型仍然无法预测$nil right - row B$比$ - A right - row B$更令人惊讶，这表明llm仍然难以整合脚本知识。我们的代码和收集的数据集可在https://github.com/tony-hong/causal-script上获得。
{"title":"Do large language models and humans have similar behaviors in causal\u0000 inference with script knowledge?","authors":"Hong, Xudong, Ryzhova, Margarita, Biondi, Daniel Adrian, Demberg, Vera","doi":"10.48550/arxiv.2311.07311","DOIUrl":"https://doi.org/10.48550/arxiv.2311.07311","url":null,"abstract":"Recently, large pre-trained language models (LLMs) have demonstrated superior language understanding abilities, including zero-shot causal reasoning. However, it is unclear to what extent their capabilities are similar to human ones. We here study the processing of an event $B$ in a script-based story, which causally depends on a previous event $A$. In our manipulation, event $A$ is stated, negated, or omitted in an earlier section of the text. We first conducted a self-paced reading experiment, which showed that humans exhibit significantly longer reading times when causal conflicts exist ($neg A rightarrow B$) than under logical conditions ($A rightarrow B$). However, reading times remain similar when cause A is not explicitly mentioned, indicating that humans can easily infer event B from their script knowledge. We then tested a variety of LLMs on the same data to check to what extent the models replicate human behavior. Our experiments show that 1) only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the $neg A rightarrow B$ condition. 2) Despite this correlation, all models still fail to predict that $nil rightarrow B$ is less surprising than $neg A rightarrow B$, indicating that LLMs still have difficulties integrating script knowledge. Our code and collected data set are available at https://github.com/tony-hong/causal-script.","PeriodicalId":496270,"journal":{"name":"arXiv (Cornell University)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136353447","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sufficiently strong flow shear reduces turbulent transport in tokamak plasmas, thereby improving the prospects for fusion power plants. It is therefore of great importance to efficiently explore parameter space to find where strong plasma flow can be achieved. To this end, we propose a new, physically motivated quasi-linear model for estimating momentum transport from turbulence in the presence of toroidal flow shear and plasma shaping. The method gives good estimates of momentum transport for up-down asymmetric geometries as well as low magnetic shear and tight aspect ratio. The results are benchmarked with high-fidelity nonlinear GENE simulations, demonstrating that it provides a fast and accurate estimate of momentum transport.
{"title":"A new quasilinear model for turbulent momentum transport in tokamaks\u0000 with flow shear and plasma shaping","authors":"Sun, Haomin, Ball, Justin, Brunner, Stephan, Volcokas, Arnas","doi":"10.48550/arxiv.2311.07159","DOIUrl":"https://doi.org/10.48550/arxiv.2311.07159","url":null,"abstract":"Sufficiently strong flow shear reduces turbulent transport in tokamak plasmas, thereby improving the prospects for fusion power plants. It is therefore of great importance to efficiently explore parameter space to find where strong plasma flow can be achieved. To this end, we propose a new, physically motivated quasi-linear model for estimating momentum transport from turbulence in the presence of toroidal flow shear and plasma shaping. The method gives good estimates of momentum transport for up-down asymmetric geometries as well as low magnetic shear and tight aspect ratio. The results are benchmarked with high-fidelity nonlinear GENE simulations, demonstrating that it provides a fast and accurate estimate of momentum transport.","PeriodicalId":496270,"journal":{"name":"arXiv (Cornell University)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136353455","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：