Multisensory Research最新文献

Scratch-and-sniff books combine olfactory cues with textual and visual narratives thus offering multisensory reading experiences. Despite their popularity since their emergence in the 1970s, these books have received little scholarly attention, particularly regarding their cognitive and developmental relevance. This paper presents a mixed-methods analysis of 334 scratch-and-sniff books, investigating how smell content varies by target age, conceptual complexity, and linguistic framing. Quantitative analyses reveal systematic differences in smell types, labelling strategies, and descriptive complexity depending on target readership age. Books targeted at younger readers tend to feature simple, explicitly labelled smells (e.g., "apple", "chocolate"), while those books targeting an older audience rely more on implicit references, humour, and/or socially transgressive smells (e.g., "fart", "smelly feet"). Qualitative readings show how olfactory cues are embedded in broader narrative structures that support inferencing, emotional resonance, and social norm acquisition. We argue that scratch-and-sniff books may be designed to use smell not only with the intention to entertain, but also to prompt crossmodal mapping, encourage inferential reasoning, and scaffold early engagements with sensory and social meaning. These findings indicate that scratch-and-sniff books systematically adapt their olfactory content and linguistic framing to developmental stages, suggesting that smell may serve not only to entertain, but also to scaffold emerging sensory, cognitive, and social capacities in early reading.

Tactile slip is a common sensation that is interesting from the perspective of brain perception since it is susceptible to illusions. Here we create the illusion of slipping in a particular direction despite a net zero movement using a simple haptic interface consisting of only two moving parts. We further show that by superimposing net zero motions of different frequencies we can also control the illusion of movement speed. The latter extends previous observations where differential local skin displacement has been reported to cause illusions of surface roughness.

When searching for an object in my pocket, I can clearly perceive the edges, shape, and hardness of the items I touch - regardless of which part of my fingers makes contact and despite large sensorimotor noise. What accounts for these perceptual invariants? Vincent Hayward introduced key elements to investigate this question, by (i) analyzing mechanical spatiotemporal invariants emerging from the constraints of skin-body-environment interactions, alongside corresponding neural invariants, as well as (ii) introducing the underlying concept of a plenhaptic function encompassing all haptic effects that could arise from interaction with the environment. This article presents this framework and some key insights into the nature of human haptics it has enabled. Specifically, it reviews evidence that haptic neural sensing relies on distributed dynamic effects produced within the skin and musculoskeletal system, describes how these effects may be encoded in the brain, and demonstrates how this approach can inform the design of versatile and effective haptic sensing for robots.

Haptic perception plays an integral role in how we interact with and make sense of the physical world. As research in this domain continues to expand across disciplines, conceptual ambiguities in terminology and theoretical frameworks persist. In this review, I address several of these ambiguities by proposing explicit accounts of notions often used implicitly in haptic research. Following a brief overview of the physiological and mechanical foundations of skin-object interactions and the somatosensory system, I introduce working definitions for terms such as haptic cues, stimuli, and perceptual dimensions, along with related assumptions and misconceptions. I argue that the distinction between distal and proximal stimuli or cues - more commonly applied in visual perceptual psychology - can serve as a helpful framework in haptic research for distinguishing physical (object-level) and mechanical (interaction-level) inputs. I then review some of the most commonly studied haptic dimensions, including stimulus magnitude, shape, material and texture, motion and time, weight, and size, along with the cues thought to inform them. This is followed by an examination of the notion of stability in haptic perception - often referred to as constancy or invariance - and its relevance for understanding how the sensory system creates consistent and behaviorally relevant percepts under variable conditions and cues. The specific phenomenon of perceptual and sensory metamers is explored as a particularly compelling example of cue integration and perceptual stability. I then address the various levels at which haptic stimuli and cues can be manipulated in research. Finally, I consider implications for haptic technologies and virtual environments as well as future directions. This conceptual framework aims to reduce terminological ambiguity and advance a clearer account of how stable haptic percepts are achieved.

Olfaction is an important yet relatively often overlooked sense, relevant to cognition, emotion, mental health, and ageing. Research into sensory domains such as interoception indicates discrepancies between behaviourally assessed accuracy and self-report measures, such as confidence. Initial evidence suggests a similar discrepancy between behavioural and self-report measures may also be present for olfaction. This study employed a multidimensional framework to examine the relationships among behavioural accuracy, self-reported confidence, and metacognitive insight for olfactory processing, using the Sniffin' Sticks Test across three olfactory tasks: threshold, discrimination, and identification. Participants exhibited varying metacognitive insight into their olfactory performance, with insight being significantly lower for the threshold task. In a complementary analysis, accuracy and confidence were not related in the threshold task, but were aligned during both the discrimination and identification tasks. These results suggest that individuals who have the most sensitive olfactory threshold performance do not display corresponding high levels of confidence, supporting the finding of reduced insight for this type of olfactory task specifically. Across tasks, behavioural accuracy was related in all tasks, as was self-reported confidence. However, metacognitive insight was not associated across tasks. Together, these results highlight that metacognitive insight for olfactory ability may differ depending on the type of olfactory task. These initial findings require replication with alternative methods to account for differences in task performance, and further investigation to ascertain patterns of results in old age where there can be a decline in olfactory function. A more comprehensive understanding of olfactory processing across accuracy, self-report, and insight measures, across a range of different olfactory tests, may provide further insight into the relationship between specific aspects of olfactory ability and how these might relate to mental health and neurological conditions.

The publication of Barry Stein and Alex Meredith's The Merging of the Senses in 1993 was a hugely influential event during the development of my own research career, as an experimental psychologist, as I am sure it was for so many others. At the time, I was embarking on the study of crossmodal links in spatial attention in neurologically normal adult humans. The body of neurophysiological research summarized in Stein and Meredith's book helped to draw people's attention to the importance of spatiotemporal coincidence to spatial behaviours (such as orienting). Cognitive neuroscientists have sometimes struggled to demonstrate similar phenomena in awake humans while at the same time Bayesian accounts have come to provide a popular alternative explanation for the way in which multisensory integration operates under many conditions. A growing awareness of the importance of considering not only spatiotemporal factors but also the semantic meaning and crossmodal correspondences that help to solve the multisensory binding problem has also emerged in the literature, as has a realization of the importance of context effects. Nevertheless, for those cognitive psychologists, like myself, interested in evaluating the implications for human spatial attention and multisensory perception, the book certainly galvanized a generation of young researchers to move beyond the unisensory approach to psychology that had seemingly become entrenched in the literature.

A classic definition of multisensory integration (MI) has been proposed as 'the presence of a (statistically) significant change in the response to a crossmodal stimulus complex compared to unimodal stimuli'. However, this general definition did not result in a broad consensus on how to quantify the amount of MI in the context of reaction time (RT). In this brief note, we argue that numeric measures of reaction times that only involve mean or median RTs do not uncover the information required to fully assess the effect of MI. We suggest instead novel measures that include the entire RT distributions functions. The central role is played by relative entropy (a.k.a. Kullback-Leibler divergence), a statistical concept in information theory, statistics, and machine learning to measure the (non-symmetric) distance between probability distributions. We provide a number of theoretical examples, but empirical applications and statistical testing are postponed to a later study.

One of the most extensively studied constructs in multisensory research is the temporal window of integration. Its extent has been variously estimated by measuring the temporal boundaries within which stimuli in different sensory modalities are perceived as simultaneous or elicit multisensory integration effects. However, there is ample evidence that these two approaches produce distinct psychometric outcomes, as the widths of the windows they yield differ even when estimated with equivalent designs and stimuli. In fact, these two estimates can sometimes even be negatively correlated. What is more, the perception of synchrony has been found to be neither necessary nor sufficient for the occurrence of multisensory illusions. This suggests that subjective simultaneity and integration phenomena are dissociable, undermining the conclusions of studies that use them interchangeably. Failing to disentangle the temporal windows in which they occur has led to contradictory findings and considerable confusion in basic research that has started extending to other domains. In clinical studies, for example, this confusion has affected work ranging from neuropsychological conditions (such as schizophrenia, mild cognitive impairment, dyslexia, and autism) to more general health factors (such as obesity and inflammation); in applied research, it is seen in studies using virtual reality, human-computer interfaces, and warning systems in vehicles. In this brief review, we discuss the importance of distinguishing these two constructs. We propose that, while the temporal boundaries of integration phenomena are aptly described as the temporal window of integration (TWI), the temporal boundaries of simultaneity judgements should be referred to as the temporal window of synchrony (TWS).

In this narrative historical review, we both summarize and critically evaluate the experimental literature that has emerged over the last century or so investigating the various ways in which the addition of music influences people's perception of, and response to, film. While 'sensation transference', whereby the mood of the background music carries over to influence the viewer's feeling about the film content, has often been documented, background music can also affect a viewer's visual attention, their interpretation, and their memory for whatever they happen to have seen. The use of sound in film (no matter whether its use is diegetic or non-diegetic - that is, part of the recounted story or not) is interesting inasmuch as simultaneously presented auditory and visual inputs do not necessarily have to be integrated perceptually for crossmodal effects to occur. The literature published to date highlights the multiple ways in which music affects people's perception of semantically meaningful film clips. Nevertheless, despite the emerging body of rigorous scientific research, the professional addition of music to film would still appear to be as much an art as a science. Furthermore, a number of potentially important questions remain unresolved, including the extent to which habituation, sensory overload, distraction, film type (i.e., fictional or informational), and/or context modulates the influence of background music. That said, this emerging body of empirical literature provides a number of relevant insights for those thinking more generally about sensory augmentation and multisensory experience design. Looking to the future, the principles uncovered in this work have growing relevance for emerging domains such as immersive media, virtual reality, multisensory marketing, and the design of adaptive audiovisual systems.

This paper critically reviews the literature on mid-level audiovisual crossmodal correspondences, that is, those associations that emerge between structured, often dynamic stimuli in vision and audition. Unlike basic correspondences (involving perceptually simple, or unitary, features) or complex ones (involving semantically rich combinations of stimuli), mid-level correspondences occur between temporally and/or spatially patterned stimuli that are perceptually structured but are typically not inherently meaningful (e.g., melodic contours and moving shapes). Taken together, the literature published to date suggests that such correspondences often rely on structural or analogical mappings, reflecting shared spatiotemporal organization across the senses rather than the direct similarity of low-level features or emotional content. Drawing on evidence from developmental, comparative, and experimental studies, we discuss the possible mechanisms underpinning these mappings - including perceptual scaffolding, amodal dimensions, and metaphorical mediation - and outline open questions regarding their perceptual, cognitive, and neural bases. We also evaluate key methodological approaches and provide suggestions for future research aiming to understand the hierarchy of crossmodal correspondences across levels of perceived stimulus complexity. Besides advancing theoretical models, our paper offers practical insights for domains such as multimedia design and crossmodal art.