A semi-automated approach to policy-relevant evidence synthesis: combining natural language processing, causal mapping, and graph analytics for public policy
Rory Hooper, Nihit Goyal, Kornelis Blok, Lisa Scholten
{"title":"A semi-automated approach to policy-relevant evidence synthesis: combining natural language processing, causal mapping, and graph analytics for public policy","authors":"Rory Hooper, Nihit Goyal, Kornelis Blok, Lisa Scholten","doi":"10.1007/s11077-024-09548-3","DOIUrl":null,"url":null,"abstract":"<p>Although causal evidence synthesis is critical for the policy sciences—whether it be analysis <i>for</i> policy or analysis <i>of</i> policy—its repeatable, systematic, and transparent execution remains challenging due to the growing volume, variety, and velocity of policy-relevant evidence generation as well as the complex web of relationships within which policies are usually situated. To address these shortcomings, we develop a novel, semi-automated approach to synthesizing causal evidence from policy-relevant documents. Specifically, we propose the use of natural language processing (NLP) for the extraction of causal evidence and subsequent homogenization of the text; causal mapping for the collation, visualization, and summarization of complex interdependencies within the policy system; and graph analytics for further investigation of the structure and dynamics of the causal map. We illustrate this approach by applying it to a collection of 28 articles on the emissions trading scheme (ETS), a policy instrument of increasing importance for climate change mitigation. In all, we find 300 variables and 284 cause-effect pairs in our input dataset (consisting of 4524 sentences), which are reduced to 70 unique variables and 119 cause-effect pairs after homogenization. We create a causal map depicting these relationships and analyze it to demonstrate the perspectives and policy-relevant insights that can be obtained. We compare these with select manually conducted, previous meta-reviews of the policy instrument, and find them to be not only broadly consistent but also complementary. We conclude that, despite remaining limitations, this approach can help synthesize causal evidence for policy analysis, policy making, and policy research.</p>","PeriodicalId":51433,"journal":{"name":"Policy Sciences","volume":null,"pages":null},"PeriodicalIF":3.8000,"publicationDate":"2024-09-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Policy Sciences","FirstCategoryId":"91","ListUrlMain":"https://doi.org/10.1007/s11077-024-09548-3","RegionNum":3,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PUBLIC ADMINISTRATION","Score":null,"Total":0}
引用次数: 0
Abstract
Although causal evidence synthesis is critical for the policy sciences—whether it be analysis for policy or analysis of policy—its repeatable, systematic, and transparent execution remains challenging due to the growing volume, variety, and velocity of policy-relevant evidence generation as well as the complex web of relationships within which policies are usually situated. To address these shortcomings, we develop a novel, semi-automated approach to synthesizing causal evidence from policy-relevant documents. Specifically, we propose the use of natural language processing (NLP) for the extraction of causal evidence and subsequent homogenization of the text; causal mapping for the collation, visualization, and summarization of complex interdependencies within the policy system; and graph analytics for further investigation of the structure and dynamics of the causal map. We illustrate this approach by applying it to a collection of 28 articles on the emissions trading scheme (ETS), a policy instrument of increasing importance for climate change mitigation. In all, we find 300 variables and 284 cause-effect pairs in our input dataset (consisting of 4524 sentences), which are reduced to 70 unique variables and 119 cause-effect pairs after homogenization. We create a causal map depicting these relationships and analyze it to demonstrate the perspectives and policy-relevant insights that can be obtained. We compare these with select manually conducted, previous meta-reviews of the policy instrument, and find them to be not only broadly consistent but also complementary. We conclude that, despite remaining limitations, this approach can help synthesize causal evidence for policy analysis, policy making, and policy research.
期刊介绍:
The policy sciences are distinctive within the policy movement in that they embrace the scholarly traditions innovated and elaborated by Harold D. Lasswell and Myres S. McDougal. Within these pages we provide space for approaches that are problem-oriented, contextual, and multi-method in orientation. There are many other journals in which authors can take top-down, deductive, and large-sample approach or adopt a primarily theoretical focus. Policy Sciences encourages systematic and empirical investigations in which problems are clearly identified from a practical and theoretical perspective, are well situated in the extant literature, and are investigated utilizing methodologies compatible with contextual, as opposed to reductionist, understandings. We tend not to publish pieces that are solely theoretical, but favor works in which the applied policy lessons are clearly articulated. Policy Sciences favors, but does not publish exclusively, works that either explicitly or implicitly utilize the policy sciences framework. The policy sciences can be applied to articles with greater or lesser intensity to accommodate the focus of an author’s work. At the minimum, this means taking a problem oriented, multi-method or contextual approach. At the fullest expression, it may mean leveraging central theory or explicitly applying aspects of the framework, which is comprised of three principal dimensions: (1) social process, which is mapped in terms of participants, perspectives, situations, base values, strategies, outcomes and effects, with values (power, wealth, enlightenment, skill, rectitude, respect, well-being, and affection) being the key elements in understanding participants’ behaviors and interactions; (2) decision process, which is mapped in terms of seven functions—intelligence, promotion, prescription, invocation, application, termination, and appraisal; and (3) problem orientation, which comprises the intellectual tasks of clarifying goals, describing trends, analyzing conditions, projecting developments, and inventing, evaluating, and selecting alternatives. There is a more extensive core literature that also applies and can be visited at the policy sciences website: http://www.policysciences.org/classicworks.cfm. In addition to articles that explicitly utilize the policy sciences framework, Policy Sciences has a long tradition of publishing papers that draw on various aspects of that framework and its central theory as well as high quality conceptual pieces that address key challenges, opportunities, or approaches in ways congruent with the perspective that this journal strives to maintain and extend.Officially cited as: Policy Sci