Recently, interpreters' management of rapport is increasingly being investigated. Yet little attention has been directed towards the role of the interpreter's non-verbal behaviour when managing rapport and to the influence of video mediated forms of interpreting on the use of non-verbal behaviour. Therefore, this study proposes a multimodal micro-interactional framework for analysing interpreters' management of rapport challenge in both onsite (OSI) and video remote interpreting (VRI) interaction. The paper introduces a multimodal coding scheme based on Spencer-Oatey's Rapport Management Theory (2008), which is then applied to a dataset of video recorded interpreter-mediated interactions to examine how interpreters employ verbal, paraverbal, and non-verbal resources to multimodally address rapport challenge. Data were collected from simulated interactions involving professional public service interpreters and role-players adopting the role of primary participants in a reception centre for asylum seekers. The findings reveal that in OSI interpreters use a wide range of non-verbal resources when conveying rapport challenges, whereas VRI imposes constraints on non-verbal communication, often necessitating more disruptive verbal strategies to manage rapport. The study underscores the importance of a multimodal approach to interpreting research, highlighting how non-verbal behaviours significantly contribute to the management of interpersonal relations in interpreter-mediated talk.