Solving Complex Background Problem Using RetinaNet for Sign System for Indonesian Language (SIBI) Gesture-to-Text Translator

Abstract

SIBI is the standardized sign language system offi-cially used in Indonesia. The application of SIBI is often found to be a hindrance because there are too many gestures that must be memorized. A mobile-based application is needed as gesture-to-text translator. From Rakun et al., Skin Color Segmentation was used as a method to segment hand and facial features using greenscreen background as dataset (3.367% of WER and 80.180% of SAcc). When this application is used, the gesture video is recorded on complex background but performed poorly (135.180% of WER and 0% of SAcc score). The computational time using Skin Color Segmentation is 0.013 s per frame. OpenPose was used to locate hand and facial position. OpenPose can give better performance (6.312% of WER and 69.293% of SAcc score) compared to Skin Color Segmentation but cannot be implemented on mobile application. The computational time using OpenPose is 0.410 s per frame. The focus of this study is to find a model that can locate hand and facial position on complex background and also can be implemented on mobile application. The model we use is RetinaNet. RetinaNet is proven to locate hand and facial position much better (4,100% of WER and 78,990 % of SAcc score) than Skin Color Segmentation and OpenPose. The computational time using RetinaNet is 0.038 s per frame.