The current study examined the inter-brain coherence (IBC) between 34 dyads of fathers and infants 7–9 months of age using functional near-infrared spectroscopy (fNIRS). We specifically focused on father–infant IBC to broaden the empirical base beyond the mother–infant connections, as the former has received limited attention. There were three conditions: a baseline condition and two task conditions when the infant and the adult participant jointly listened to maternal storytelling in Cantonese in infant-directed speech (IDS) and adult-directed speech (ADS). Father–infant IBC was compared with stranger–infant IBC in the same experimental settings. Our results found that father–infant IBC was greater in the baseline and ADS conditions but not in the IDS condition, compared to stranger–infant IBC. Further, stranger–infant dyads showed greater IBC in the IDS condition than in the ADS condition, with no significance in father–infant IBC between the two speech conditions. These results identified different inter-brain connection mechanisms between the two dyads. The IBC pattern in stranger–infant dyads is driven by neural entrainment to mothers' speech, whereas father–infant IBC is more resistant to mothers' behaviours in the co-presence of both parents.