Controller free hand interaction in Virtual Reality

Abstract

This paper discusses a low-cost mixed reality application wherein the movement of hands will be captured using the phone's rear camera and replicated to the objects in the virtual world. In this, a client-server communication channel over UDP Sockets will be set up where the Virtual Reality App in the phone will be the client for a Machine Learning server. The client (virtual reality app on the phone) will capture the live feed of the movement of hands which will be sent to the ML Server for processing. The ML Server will process the live feed and detect the position of hands, which will be passed back to the client (virtual reality app on the phone), where this movement of hands will be replicated in the virtual world.