Development and validation of technique to measure cyber situation awareness

Within the cyber security domain, specifically within the field of computer network defence, professional log analysts are employed to monitor organizations' networks in order to detect malicious activity and suggest necessary measures. A log analyst needs to perceive malicious activity, comprehend the impact and type of threat, and predict future consequences. In other words, they need good cyber situation awareness. Research about cyber situation awareness measurement is limited, especially when it comes to practical examples. The current paper describes the development and validation of a freeze-probe technique aiming to measure log analysts' situation awareness. Goal directed task analysis and hierarchical task analysis were used to develop a first version of a measurement technique. The measurement technique had the form of two questionnaires designed for the two different roles in log analysis. The validation was conducted in a realistic setting during an exercise involving five professionals, where the questionnaires were well received by the log analysts. Only smaller adjustments were suggested. The results suggest that the technique can be used to evaluate cyber situation awareness for log analysts, as well as function as a tool in log analysts' daily work to keep track of incidents.