Flight Examiners’ Methods of Ascertaining Pilot Proficiency

Abstract

Objective: To determine how flight examiners reason and which methods they use when assessing the competencies of pilots for continued accreditation purposes and type-rating training. Background: Early work on pilot performance assessment focused on measurement models, including the accuracy and reliability of the scores attributed to the human factors variables included. More recent studies investigated the nature of the evidence that flight examiners used. No previous studies were found on how flight examiners assess line pilots’ performance during flight training and examination. Method: This study employed methods typical for cognitive anthropology, combining ethnographic observations of debriefings and interviews, stimulated recall concerning debriefing, and modified think-aloud protocols of assessment of flight episodes. Twenty-three flight examiners from 5 regional airlines were observed and interviewed in 3 contexts. Results: The data revealed that flight examiners used the documentary method, where initial observations are treated as documentary evidence of underlying phenomena (e.g., situational awareness, decision making) while presupposing these phenomena for making and categorizing the observations. Flight examiners, using a variety of techniques, actively create situations for obtaining additional observations that further substantiate the presupposed underlying phenomena. Conclusion: Even when flight examiners use rating scales, their assessment method is based on categorization of facts and, therefore, shares similarities with medical diagnosis. Suggested quality improvement measures include increasing awareness of diagnostic error, developing diagnostic tools, and developing means to measure diagnostic errors.