Avoiding Measurement Errors from Manipulating Data in Software

Abstract

In the age of digital technology, the act of calibrating a device nearly always requires the conversion of an analog signal into a digital representation that will be used and manipulated in software as a part of the calibration process. This conversion from analog to digital and the subsequent processing that occurs in the digital domain can introduce additional errors in the measurement. If the data types and methodologies are not properly controlled, the magnitude of these errors can add significant uncertainty to the calibration. The objective of this paper is to explore the various ways that software can introduce errors and uncertainty into measurements, with the purpose of raising awareness for developers about the choices that can be made when manipulating measurement data in software. This paper will investigate several sources of software error that apply across different programming environments, including excel, text-based, and graphical programming environments. The sources of error that will be discussed will include the following: •Rounding errors associated with datatype conversions and data truncation. •Numerical errors are related to the limitations of computers in representing numeric values. •Computational errors that can be introduced by common math functions and methodologies.