Validating Drilling States Classifiers with Suboptimal Datasets

The wide-scale deployment of analytics to support the well construction processes based on rig data has opened a host of opportunities to improve performance, quality, and safety at all levels in the offshore drilling industry. As automation and high-stakes decision making starts to rely more on these types of classifiers, a topic of consideration is the validation methods employed during their development to ensure accuracy and precision, requiring the best available methods to help data scientists evaluate their soundness, features and limitations, and explain to key stakeholders who may not be familiar with such techniques. In the particular case of drilling states determination from signal data, there may be cases where the ground truth records are either at lower resolution than desired, or where some degree of uncertainty on the labeling exist, techniques such as inter-rater reliability (IRR) or inter-rater agreement (IRA) can help to demonstrate consistency among observational decision provided by multiple sources and be used as a way to show the level of agreement between, for example, a proposed drilling state generator classifier using drillfloor data and existing IADC codes from available logs at the same time. This approach can be used to help decisions on further development of the particular classifier before committing to stricter model validation. This paper will show examples of these techniques applied to automatic generation of certain IADC codes using signal data vs log records, and how IRR/IRA can help inform the quality of the results.