Cognitive Science最新文献

In music performance contexts, vocalists tend to gesture with hand and upper body movements as they sing. But how does this gesturing relate to the sung phrases, and how do singers’ gesturing styles differ from each other? In this study, we present a quantitative analysis and visualization pipeline that characterizes the multidimensional co-structuring of body movements and vocalizations in vocal performers. We apply this to a dataset of performances within the Karnatak music tradition of South India, including audio and motion tracking data of 44 performances by three expert Karnatak vocalists, openly published with this report. Our results show that time-varying features of head and hand gestures tend to be more similar when the concurrent vocal time-varying features are also more similar. While for each performer we find clear co-structuring of sound and movement, they each show their own characteristic salient dimensions (e.g., hand position, head acceleration) through which movement co-structures with singing. Our time-series analyses thereby provide a computational approach to characterizing individual vocalists’ unique multimodal vocal-gesture co-structuring profiles. We also show that co-structuring clearly reduces degrees of freedom of the multimodal performance such that motifs that sound alike tend to co-structure with gestures that move alike. The current method can be applied to any multimodally ensembled signals in both human and nonhuman communication, to determine co-structuring profiles and explore any reduction in degrees of freedom. In the context of Karnatak singing performance, the current analysis is an important starting point for further experimental study of gesture-vocal synergies.

The theory of embodied simulation posits that semantic processing related to actions involves the simulation of sensorimotor experiences, similar to action recognition, which also activates the action observation network. Visual and action experiences obtained through vision and proprioception can facilitate the processing of action verbs via this simulation process. However, the differential effects of these two types of action representations on the processing of action verbs remain to be explored. This study uses an action–language priming paradigm and three behavioral experiments to explore how visual and action experiences from different perspectives affect sensorimotor simulation in action verb processing. Experiment 1 studied how action image perspectives (first-person vs. third-person) and image-word congruency affect action verb priming. Experiment 2 examined the role of the action agent in perspective priming. Experiment 3 investigated that motor experience congruency, jointly activated by visual perspective and personal pronouns, influences action verb processing. Experiment 1 showed faster action verb processing with the first-person perspective (1PP) during prime–target incongruency and non-mirrored conditions, indicating better action control and prediction, enhancing sensorimotor simulation. Experiment 2 found faster responses with the 1PP during incongruency, with no effect from the action agent on sensorimotor simulation. Experiment 3 showed faster reaction times for prime–target congruency than incongruency, with no effect of perspective congruency. These results show that action verb processing involves simulating sensorimotor experiences from specific perspectives, emphasizing the key role of action experience and offering new evidence for action verb representation theories.

As part of the multimodal language system, gestures play a vital role for listeners, by capturing attention and providing information. Similarly, disfluencies serve as a cue for the listeners about one's knowledge on a topic. In two studies, the first study with natural and the second study with controlled stimuli, we asked whether the combination of gestures and speech disfluencies would affect how listeners made feeling-of-another's-knowing (FOAK) judgments regarding speakers’ knowledge states. In Study 1, we showed participants videos of speakers providing navigational instruction. We manipulated the speakers’ use of gestures and speech disfluencies, whereas facial expressions, words, and additional visual cues (e.g., background, clothes, objects) naturally occurred. We found that fluent speech elicited higher FOAK ratings than disfluent speech, but no significant effect was found for gestures. In the follow-up Study 2, we examined the same disfluency-gesture interaction in a more controlled setting using video stimuli with an actress controlling for background, intonation, and word choice, as well as iconic and beat gesture types as gesture subcategories. Participants also filled out the Gesture Awareness Scale. Results were identical with the first study, in which only the disfluent speech received significantly lower FOAK ratings, revealing no effects of gesture use or type. These findings suggest that individuals may use certain communicative cues more than others, particularly in the context of assessing others’ knowledge.

Certain recurrent features of language characterize the way a whole language system is structured. By contrast, others target specific categories of items within those wider systems. For example, languages tend to exhibit consistent order of heads and dependents across different phrases—a system-wide regularity known as harmony. While this tendency is generally robust, some specific syntactic categories appear to deviate from the trend. We examine one such case, the order of adjectives and genitives, which do not exhibit a typological tendency for consistent order with respect to the noun. Instead, adjectives tend to follow and genitives precede the noun. Across two silent gesture experiments, we test the hypothesis that these category-specific ordering tendencies reflect cognitive biases that favor (i) conveying objects before properties that modify them, but (ii) conveying expressions of possession before possessors. While our hypothesis is thus that these biases are semantic in nature—they impact preferences for how concepts are ordered—the claim is that they may have downstream effects on conventionalized syntax by contributing to an over-representation of postnominal adjectives and prenominal genitives. We find that these biases affect gesture order in contexts where no conventionalized system is in place. When a system is in place, participants learn that system, and category-specific biases do not impact their learning. Our results suggest that different contexts reveal distinct types of cognitive biases; some are active during learning and others are active during language creation.

Goal-directed tasks unfold in hierarchies of larger and smaller sub-tasks, and pursuing them jointly implies that participants must agree on whether they are continuing an ongoing sub-task (horizontal transition) or switching to the next sub-task (vertical transition). Previous research indicates that humans employ short and efficient coordination markers as procedural conventions to distinguish horizontal (e.g., in English, with yeah and uh-huh) and vertical transitions (with okay, all right). However, it remains unclear (1) whether such words serve as potentially universal coordination devices and (2) which properties make some markers more suitable for horizontal versus vertical transition contexts. We hypothesized that horizontal transitions in ongoing sub-tasks are associated with higher dual-tasking interference between verbal coordination and the nonlinguistic task, therefore, constraining the lexicality of coordination markers. In our experimental study, we assessed how speakers of three typologically diverse languages (Swiss French, Vietnamese, and Shipibo-Konibo; N = 232) used coordination markers to navigate a joint LEGO-building task. We found that in each language, coordination markers comprise a system of transition-specific conventions and that participants strategically deployed markers with minimal lexical and acoustic forms (uh-huh, mm) and repetitions in horizontal transitions, while more lexicalized markers (e.g., okay) in vertical transitions. Our findings suggest that (1) coordination markers are potentially universal linguistic devices for navigating joint activities and (2) the forms of coordination markers might be shaped by the constraints of their primary interaction context (here, horizontal and vertical transitions). Our study provides new evidence of how interactional settings might selectively shape language use through the forces of convergent language evolution.

According to the noisy channel framework of sentence processing, communication can succeed even when the input is corrupted because comprehenders rationally infer the speaker's intended meaning based on the prior probability of the literal interpretation and the probability that the input has been corrupted by noise. To test whether and under what conditions comprehenders consider word exchanges as a possible source of corruption, we ran five experiments on processing three types of simple German sentences: subject-before-object sentences (SO), object-before-subject sentences (OS), and passive sentences. Critical sentences had implausible meanings, but could be “repaired” by exchanging function words or by exchanging nouns. Experiments 1 through 4 presented sentences along with yes-no questions to probe interpretation. Implausible SO and passive sentences consistently elicited few nonliteral interpretations, whereas many nonliteral interpretations were given to implausible OS sentences. This was true regardless of whether word exchanges had to cross a main verb or an auxiliary, and it was more pronounced if the overall proportion of implausible sentences was low. We conclude that when answering yes-no questions, word exchanges are considered with function words of the same syntactic category, but not with nouns, and only when they result in a more likely syntactic structure. Experiment 5 showed that when explicitly asked to correct implausible sentences, comprehenders use noun exchanges frequently. We propose that the results for both yes-no questions and explicit corrections follow if the prior probability assigned to implausible sentences differs between tasks.

Perceptions of source credibility may play a role in major societal challenges like political polarization and the spread of misinformation as citizens disagree over which sources of political information are credible and sometimes trust untrustworthy sources. Cognitive scientists have developed Bayesian Network models of how people integrate perceptions of source credibility when learning from information provided by sources, but these models do not involve the crucial source characteristic in politics: bias. Biased sources make claims that align with a particular political agenda, whether or not they are true. We present a novel Bayesian Network model which integrates perceptions of a source's bias as well as their expertise. We demonstrate the model's validity for predicting how people will update beliefs and perceptions of bias and expertise in response to testimony across two studies, the second being a preregistered conceptual replication and extension of the first.

Recent models of sequential narratives suggest that readers form predictions about upcoming panels as they read. However, previous work has considered these predictions only in terms of currently viewed information. In the current studies, we investigate to what extent readers are using information from un-fixated panels in comic stories. Using the moving-window paradigm, we studied whether reading behavior was disrupted when upcoming panels were unavailable to the reader, in short comic strips (Experiment 1) and multipage comics (Experiment 2). Both studies showed the greatest disruption to reading when all peripheral information was removed, but such changes persisted when only partial peripheral information was available. The results indicate that readers are making use of information from at least two panels ahead of the current fixation location. We consider these findings in relation to the PINS model of comic reading, and how the role of peripheral information might be further explored.

Conversational alignment, also known as accommodation, entrainment, interpersonal synchrony, and convergence, is defined as the tendency for interlocutors to exhibit similarity in their communicative behaviors. There have been many theories and explanations set forth as to why alignment occurs and, accordingly, the mechanisms that underlie it. To date, however, alignment research has been largely siloed, with different research teams often examining alignment through the lens of a single theoretical account. Considering causal mechanisms in tandem offers a more holistic and nuanced understanding of the dynamic nature of alignment, its purposes, and its consequences. Accordingly, we propose the Mechanistic Framework of Alignment (MFA), a qualitative conceptual model that integrates existing theories of conversational alignment into one unified framework. To explain this framework, we first review five alignment mechanisms, discussing the underlying assumptions, contributions, and supporting evidence for each. We then introduce two overarching factors—conversational goal and alignment type—that are critical for understanding when and how these mechanisms give rise to aligned behavior. Illustrative examples demonstrate how the relative weightings of each mechanism interact with these contextual variables. Finally, we conclude with directions for how future research can extend and refine this framework and how the MFA can support future work in this area.

Certain speech sounds are consistently associated with visual properties such as shape and size, a phenomenon known as crossmodal correspondences. Well-established examples demonstrate that the vowel /u/ is often linked to rounder and larger objects, while /i/ is associated with more angular and smaller ones. However, most previous research utilized English pseudowords, leaving a gap in our understanding of how these correspondences manifest in tonal languages. The current study extends the investigation to Mandarin Chinese, a tonal language, to examine the roles of vowels, consonants, and lexical tones in sound–shape and sound–size correspondences. Participants heard consonant-vowel-tone syllables and rated each on a 5-point scale with rounder/more angular shapes or larger/smaller icons at opposite ends. The results confirmed the established vowel effect: /u/ was associated with rounder and larger patterns than /i/. Results for consonants demonstrated that the voiced–unvoiced contrast predicted sound–shape judgments, while the aspirated–unaspirated contrast, which is less prominent in English, influenced sound–size judgments. Lexical tones also revealed systematic effects, with Tone 1 (flat), Tone 2 (rising), Tone 3 (falling–rising), and Tone 4 (falling) progressively matched from rounder to more angular shapes, while Tones 1 and 2 were linked to larger sizes than Tones 3 and 4. These phonemic features reliably predicted crossmodal correspondences even when controlling for acoustic properties, suggesting robust mappings between phonemic and visual representations. This study highlights the common vowel effects across Mandarin and English while revealing unique influences of consonants and lexical tones, underscoring the role of language experience in shaping crossmodal correspondences.