A multimodal user interface using the webinos platform to connect a smart input device to the Web of Things

Abstract

We propose an innovative overlay architecture among heterogeneous devices that works on different operating systems and an application based on standard web technologies that leverages on such architecture to perform a broad range of functions using audio commands. To achieve this, a secure web runtime platform allows connecting different devices such as an Android mobile device, a single board computer and a Web of Things (WoT) sensor/actuator in a secure and independent manner. Methods and functions have been created in order to capture the audio from the microphone on personal computers and on mobile phones, and to safely transfer files to a single board computer, in which a speech recognition engine is embedded. Therefore, by recording the audio command from one of these devices and performing voice command identification using the speech engine, it is possible to perform different actions previously defined by the user on one of his devices. This work demonstrates how open source platforms can interconnect and operate with proprietary architecture in a complementary and secure manner.