Towards the next generation of trustable, efficient and sustainable text-to-audio generative models

Abstract

Text-to-Audio (TTA) models are deep learning generative systems that generate audio samples from textual descriptions given as input to the models. The goal of this research is to leverage the opportunities of TTA models and advance them to address key challenges. Regarding opportunities, we proposed studies to investigate how these models can be easily integrated into music production practices as user-friendly sketching tools for audio samples, democratizing access to music creation without the need for sample libraries or instruments. They also offer significant potential for research and innovation. Deep learning models require large amounts of data to achieve good performance, but gathering data can be challenging. We demonstrated that by generating desired audio content through natural language, these models provide valuable training data for audio applications where data are not always massively available. However, several important challenges arise with these models, such as the rightful attribution of copyrighted data, the generation of deepfake content, and energy consumption, as shown in our studies. Through this comprehensive investigation, we aim to advance text-to-audio generative models by aligning their development with the needs of end users and society. Achieving this requires combining machine learning techniques with principles of ethics, sustainability, and trustworthiness.