Understanding visual search in graphical user interfaces

Abstract

How do we find items within graphical user interfaces (GUIs)? Current understanding of this issue relies on studies using symbol matrices, natural scenes, and other non-GUI stimuli. To understand whether the effects discovered in those environments extend to mobile, desktop, and web interfaces, this paper reports on visual search performance and eye movements with 900 real-world GUIs. In an eye-tracking study, participants ($N=84$) were given a cue (textual or image) describing a target to find within a GUI. The study found that the type of GUI, the absence/presence of the target, and cue type affected search time more than visual complexity did. We also compared visual search to free-viewing in GUIs, concluding that these two tasks are distinctly different. Synthesis of the results points to a Guess-Scan-Confirm pattern in visual search: in the first few fixations, gaze is frequently directed toward the top-left corner of the screen, a pattern possibly related to the top-left being a statistically likely location of the target or of information that could aid in finding it; attention then gets more selectively guided, in line with the GUI’s structure and the features of the target; and, finally, the user must confirm whether the target has been identified or, instead, that no target is visible. The VSGUI10K eye-tracking dataset (10,282 trials) is released for study and modeling of visual search.