Rapid increase in data volume, compounded by the reproducibility crisis, has led to the need to automate both experimental and computational aspects of neuroscience investigations. Automating neuroscience investigations enables an unprecedented ability to record and inspect how results were achieved. Here we review some of our recent work to integrate provenance and reproducibility measures into a tool called NeuroManager that automates a standard computational neuroscience pipeline, unifying the experiment--data--modeling--analysis cycle and allowing the scientist to focus on model evolution. Through a flexible daily workflow that leverages servers, clusters, and clouds simultaneously, NeuroManager automates manual tasks including database access, job submission, simulation scheduling, and preservation of provenance.
{"title":"Provenance and Reproducibility in the Automation of a Standard Computational Neuroscience Pipeline","authors":"David B. Stockton, A. Prinz, F. Santamaría","doi":"10.1145/3322790.3330592","DOIUrl":"https://doi.org/10.1145/3322790.3330592","url":null,"abstract":"Rapid increase in data volume, compounded by the reproducibility crisis, has led to the need to automate both experimental and computational aspects of neuroscience investigations. Automating neuroscience investigations enables an unprecedented ability to record and inspect how results were achieved. Here we review some of our recent work to integrate provenance and reproducibility measures into a tool called NeuroManager that automates a standard computational neuroscience pipeline, unifying the experiment--data--modeling--analysis cycle and allowing the scientist to focus on model evolution. Through a flexible daily workflow that leverages servers, clusters, and clouds simultaneously, NeuroManager automates manual tasks including database access, job submission, simulation scheduling, and preservation of provenance.","PeriodicalId":192842,"journal":{"name":"Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116871950","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
M. Krafczyk, A. Shi, A. Bhaskar, D. Marinov, V. Stodden
Continuous integration (CI) is a well-established technique in commercial and open-source software projects, although not routinely used in scientific publishing. In the scientific software context, CI can serve two functions to increase reproducibility of scientific results: providing an established platform for testing the reproducibility of these results, and demonstrating to other scientists how the code and data generate the published results. We explore scientific software testing and CI strategies using two articles published in the areas of applied mathematics and computational physics. We discuss lessons learned from reproducing these articles as well as examine and discuss existing tests. We introduce the notion of a "scientific test" as one that produces computational results from a published article. We then consider full result reproduction within a CI environment. If authors find their work too time or resource intensive to easily adapt to a CI context, we recommend the inclusion of results from reduced versions of their work (e.g., run at lower resolution, with shorter time scales, with smaller data sets) alongside their primary results within their article. While these smaller versions may be less interesting scientifically, they can serve to verify that published code and data are working properly. We demonstrate such reduction tests on the two articles studied.
{"title":"Scientific Tests and Continuous Integration Strategies to Enhance Reproducibility in the Scientific Software Context","authors":"M. Krafczyk, A. Shi, A. Bhaskar, D. Marinov, V. Stodden","doi":"10.1145/3322790.3330595","DOIUrl":"https://doi.org/10.1145/3322790.3330595","url":null,"abstract":"Continuous integration (CI) is a well-established technique in commercial and open-source software projects, although not routinely used in scientific publishing. In the scientific software context, CI can serve two functions to increase reproducibility of scientific results: providing an established platform for testing the reproducibility of these results, and demonstrating to other scientists how the code and data generate the published results. We explore scientific software testing and CI strategies using two articles published in the areas of applied mathematics and computational physics. We discuss lessons learned from reproducing these articles as well as examine and discuss existing tests. We introduce the notion of a \"scientific test\" as one that produces computational results from a published article. We then consider full result reproduction within a CI environment. If authors find their work too time or resource intensive to easily adapt to a CI context, we recommend the inclusion of results from reduced versions of their work (e.g., run at lower resolution, with shorter time scales, with smaller data sets) alongside their primary results within their article. While these smaller versions may be less interesting scientifically, they can serve to verify that published code and data are working properly. We demonstrate such reduction tests on the two articles studied.","PeriodicalId":192842,"journal":{"name":"Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130379731","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
K. Chard, N. Gaffney, Matthew B. Jones, K. Kowalik, Bertram Ludäscher, J. Nabrzyski, V. Stodden, I. Taylor, M. Turk, C. Willis
We present and define a structured digital object, called a "Tale," for the dissemination and publication of computational scientific findings in the scholarly record. The Tale emerges from the NSF funded Whole Tale project (wholetale.org) which is developing a computational environment designed to capture the entire computational pipeline associated with a scientific experiment and thereby enable computational reproducibility. A Tale allows researchers to create and package code, data and information about the workflow and computational environment necessary to support, review, and recreate the computational results reported in published research. The Tale then captures the artifacts and information needed to facilitate understanding, transparency, and execution of the Tale for review and reproducibility at the time of publication.
{"title":"Implementing Computational Reproducibility in the Whole Tale Environment","authors":"K. Chard, N. Gaffney, Matthew B. Jones, K. Kowalik, Bertram Ludäscher, J. Nabrzyski, V. Stodden, I. Taylor, M. Turk, C. Willis","doi":"10.1145/3322790.3330594","DOIUrl":"https://doi.org/10.1145/3322790.3330594","url":null,"abstract":"We present and define a structured digital object, called a \"Tale,\" for the dissemination and publication of computational scientific findings in the scholarly record. The Tale emerges from the NSF funded Whole Tale project (wholetale.org) which is developing a computational environment designed to capture the entire computational pipeline associated with a scientific experiment and thereby enable computational reproducibility. A Tale allows researchers to create and package code, data and information about the workflow and computational environment necessary to support, review, and recreate the computational results reported in published research. The Tale then captures the artifacts and information needed to facilitate understanding, transparency, and execution of the Tale for review and reproducibility at the time of publication.","PeriodicalId":192842,"journal":{"name":"Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems","volume":"517 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123097066","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Cybersecurity, which serves to protect computer systems and data from malicious and accidental abuse and changes, both supports and challenges the reproducibility of computational science. This position paper explores a research agenda by enumerating a set of two types of challenges that emerge at the intersection of cybersecurity and reproducibility: challenges that cybersecurity has in supporting the reproducibility of computational science, and challenges cybersecurity creates for reproducibility of computational science.
{"title":"Initial Thoughts on Cybersecurity And Reproducibility","authors":"E. Deelman, V. Stodden, M. Taufer, Von Welch","doi":"10.1145/3322790.3330593","DOIUrl":"https://doi.org/10.1145/3322790.3330593","url":null,"abstract":"Cybersecurity, which serves to protect computer systems and data from malicious and accidental abuse and changes, both supports and challenges the reproducibility of computational science. This position paper explores a research agenda by enumerating a set of two types of challenges that emerge at the intersection of cybersecurity and reproducibility: challenges that cybersecurity has in supporting the reproducibility of computational science, and challenges cybersecurity creates for reproducibility of computational science.","PeriodicalId":192842,"journal":{"name":"Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116813616","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Andrea David, Mariette Souppe, I. Jimenez, K. Obraczka, Sam Mansfield, K. Veenstra, C. Maltzahn
Computer network research experiments can be broadly grouped in three categories: simulated, controlled, and real-world experiments. Simulation frameworks, experiment testbeds and measurement tools, respectively, are commonly used as the platforms for carrying out network experiments. In many cases, given the nature of computer network experiments, properly configuring these platforms is a complex and time-consuming task, which makes replicating and validating research results quite challenging. This complexity can be reduced by leveraging tools that enable experiment reproducibility. In this paper, we show how a recently proposed reproducibility tool called Popper facilitates the reproduction of networking experiments. In particular, we detail the steps taken to reproduce results in two published articles that rely on simulations. The outcome of this exercise is a generic workflow for carrying out network simulation experiments. In addition, we briefly present two additional Popper workflows for running experiments on controlled testbeds, as well as studies that gather real-world metrics (all code is publicly available on Github). We close by providing a list of lessons we learned throughout this process.
{"title":"Reproducible Computer Network Experiments: A Case Study Using Popper","authors":"Andrea David, Mariette Souppe, I. Jimenez, K. Obraczka, Sam Mansfield, K. Veenstra, C. Maltzahn","doi":"10.1145/3322790.3330596","DOIUrl":"https://doi.org/10.1145/3322790.3330596","url":null,"abstract":"Computer network research experiments can be broadly grouped in three categories: simulated, controlled, and real-world experiments. Simulation frameworks, experiment testbeds and measurement tools, respectively, are commonly used as the platforms for carrying out network experiments. In many cases, given the nature of computer network experiments, properly configuring these platforms is a complex and time-consuming task, which makes replicating and validating research results quite challenging. This complexity can be reduced by leveraging tools that enable experiment reproducibility. In this paper, we show how a recently proposed reproducibility tool called Popper facilitates the reproduction of networking experiments. In particular, we detail the steps taken to reproduce results in two published articles that rely on simulations. The outcome of this exercise is a generic workflow for carrying out network simulation experiments. In addition, we briefly present two additional Popper workflows for running experiments on controlled testbeds, as well as studies that gather real-world metrics (all code is publicly available on Github). We close by providing a list of lessons we learned throughout this process.","PeriodicalId":192842,"journal":{"name":"Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116178321","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Dylan Chapp, Danny Rorabaugh, Duncan A. Brown, E. Deelman, K. Vahi, Von Welch, M. Taufer
The PRIMAD model with its six components (i.e., Platform, Research Objective, Implementation, Methods, Actors, and Data) provides an abstract taxonomy to represent computational experiments and promote reproducibility by design. In this paper, we employ a post-hoc assessment of the model applicability to a set of Laser Interferometer Gravitational-Wave Observatory (LIGO) workflows from literature sources (i.e., published papers). Our work outlines potential advantages and limitations of the model in terms of its levels of abstraction and means of application.
{"title":"Applicability Study of the PRIMAD Model to LIGO Gravitational Wave Search Workflows","authors":"Dylan Chapp, Danny Rorabaugh, Duncan A. Brown, E. Deelman, K. Vahi, Von Welch, M. Taufer","doi":"10.1145/3322790.3330591","DOIUrl":"https://doi.org/10.1145/3322790.3330591","url":null,"abstract":"The PRIMAD model with its six components (i.e., Platform, Research Objective, Implementation, Methods, Actors, and Data) provides an abstract taxonomy to represent computational experiments and promote reproducibility by design. In this paper, we employ a post-hoc assessment of the model applicability to a set of Laser Interferometer Gravitational-Wave Observatory (LIGO) workflows from literature sources (i.e., published papers). Our work outlines potential advantages and limitations of the model in terms of its levels of abstraction and means of application.","PeriodicalId":192842,"journal":{"name":"Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems","volume":"12 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-04-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134089758","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems","authors":"","doi":"10.1145/3322790","DOIUrl":"https://doi.org/10.1145/3322790","url":null,"abstract":"","PeriodicalId":192842,"journal":{"name":"Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134308748","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：