Some thoughts on best publishing practices for scientific software

Abstract

It is increasingly recognized that software is central to much of science, and that rigorous approaches to software development are important for making sure that science is based on a solid foundation (Wilson et al. 2014). While there has been increasing discussion of the software development practices that lead to robust scientific software (e.g., Jackson et al. 2011, Osborne et al. 2014, Wilson et al. 2014), figuring out how to actively encourage the use of these practices can be challenging. Poisot (2015) proposes a set of best practices to be included as part of the review process for software papers. These include automated testing, public test coverage statistics, continuous integration, release of code in citeable ways using DOIs, and documentation (Poisot 2015). These are all important recommendations that will help encourage the use of good practice in the development of scientific software (Jackson et al. 2011, Osborne et al. 2014, Wilson et al. 2014). Requiring these approaches for publication of an associated software paper should help improve the robustness of published software (automated testing, continuous integration), its ease of use (documentation, continuous integration), and the potential for the scientific community to build on and contribute to existing efforts. As part of thinking about these best practices, Poisot (2015) grapples with one of the fundamental challenges of scientific software publication: how do we review scientific software? Most scientists are not trained in how to conduct code reviews (Petre and Wilson 2014) and the time commitment to do a full review of a moderately sized piece of software is substantial. In combination, this would make it very difficult to find reviewers for software papers if reviewers were expected to perform a thorough code review. Poisot joins Mills (2015) in suggesting that this task could be made more manageable by requiring all software submitted for publication to have automated testing with reasonably high coverage. While Mills (2015) suggests that this will “encourage researchers to use this fundamental technique for ensuring code quality”, Poisot takes the idea a step further by suggesting that reviewers could then focus on reviewing the tests to determine if the software does what it is intended to do when provided with known inputs. This approach isn’t perfect. Tests are necessarily limited in the inputs that are evaluated and mistakes can occur in tests as well as in the code itself. However, reviewing tests to determine whether they are sufficient and whether the code produces correct outcomes in at least some cases is, I think, much more tenable than reviewing an entire codebase line by line. It is one of the most reasonable solutions I have seen to the challenge of reviewing software. While I agree with all of the major recommendations made in Poisot (2015), I think the ideas related to making software citeable will benefit from further discussion. While the benefits of citeability for scientific software are clear, it is less clear whether this citation should necessarily occur through the use of DOIs. As Poisot (2015) points out, there are advantages to using DOIs for scientific software. Many journals accept scholarly products with DOIs for inclusion in reference lists, which means that software gets acknowledged in the same way as papers. This helps provide credit to scientists developing software in a manner that is easily understood by academic reward structures. It also has the potential to make bibliometric analyses of software use more straightforward. However, many major scientific software products do not use DOIs for the software itself, preferring generic citations to the software (e.g., SymPy: SymPy Development Team