How different levels of semantic segmentation affect human perception of driving scenes

As automated vehicles continue to advance, teleoperation has emerged as a critical support system for navigating complex and unpredictable environments that exceed the vehicles' current autonomous capabilities. A main issue in the implementation of teleoperation is latency caused by the high bandwidth required to transmit the video feed from the vehicle to the remote teleoperation station. A possible approach for addressing the latency problem is the transfer of lower-resolution or compressed videos between the vehicle and the teleoperation station. When applying semantic segmentation on the video feed, many pixels are mapped to a limited set of possible colors according to the types of objects that they represent. This concept has been commonly used in autonomous driving algorithms and has the potential to enable the transferring of smaller-sized videos thus reducing bandwidth. In this study, we examine how presenting semantically segmented driving scenes to humans affects their perception of the scene, and specifically, how it affects their hazard perception and situation awareness. We conducted two user studies comparing the effects of using different levels and types of semantic segmentation. Our results indicate that viewing partly segmented scenes, such that only a selected set of object types are colored, commonly achieves the same effect, and sometimes even outperforms a realistic view. Our study and its insights may pave the way for future research, development, and design of teleoperation systems of automated vehicles.