Youth making machine learning models for gesture-controlled interactive media

Machine learning (ML) technologies are ubiquitous and increasingly influential in daily life. They are powerful tools people can use to build creative, personalized systems in a wide variety of contexts. We believe ML has vast potential for young people to use to make creative projects, especially when used in conjunction with programming. This potential is understudied. We know little about what projects youth might create, or what computational practices they could engage in while building them. We combined a beginner-level ML modeling toolkit with a beginning programming tool and then investigated how young people created and remixed projects to incorporate custom ML-based gestural inputs. We found that (1) participants were able to build and integrate ML models of their own gestures into programming projects; (2) the design of their gestures ranged from coherent to disjoint with respect to the narratives, characters, and actions of their interactive worlds; and (3) they tested their projects by assessing the programmed vs. modeled aspects of them as distinct units. We conclude with a discussion of how we might support youth in combining code and ML modeling going forward.