Transformers in source code generation: A comprehensive survey

Abstract

Transformers have revolutionized natural language processing (NLP) and have had a huge impact on automating tasks. Recently, transformers have led to the development of powerful large language models (LLMs), which have advanced automatic code generation. This study provides a review of code generation concepts and transformer applications in this field. First, the fundamental concepts of the attention mechanism embedded into transformers are explored. Then, predominant automated code generation approaches are briefly reviewed, including non-learning code generation (e.g., rule-based), shallow learning (e.g., heuristic rules, grammar-based), and deep learning models. Afterward, this survey reviews pre-training and fine-tuning techniques for code generation, focusing on the application of efficient transformer methods such as parameter-efficient tuning, instruction tuning, and prompt tuning. Additionally, this work briefly outlines resources for code generation (e.g., datasets, benchmarks, packages) and evaluation metrics utilized in code generation processes. Finally, the challenges and potential research directions (e.g., multimodal learning) are investigated in depth.