Multimodal Autonomous Verbal Assessment With Visual Inspection

Abstract

This paper proposes an autonomously assessing system for the verbal examination of candidates. The system uses audio-video inputs and processes them to detect the candidate’s spoken answer, and compares it to the model answer in the dataset with the corresponding question. The semantic similarity score will be calculated and used to recommend the next question from the database using various types of recommendation systems discussed in the paper. Additionally, the system employs video analysis techniques to detect and prevent modern malpractices like multiple faces and reading from notes during the examination process. The proposed system aims to improve the efficiency and fairness of verbal examinations by eliminating human bias and accurately evaluating the candidate’s understanding of the subject. The system performance will be evaluated using a dataset of spoken answers and the results will demonstrate its effectiveness in improving the efficiency and fairness of the verbal examination process.