RUITE: Refining UI Layout Aesthetics Using Transformer Encoder

Abstract

In the User Interface (UI) design process, designers sketch the UI design (low fidelity prototype) with minimal focus on visual appearances before converting them to higher fidelities. Contrary to low-fidelity, higher fidelity prototypes require better layout and aesthetic quality, during which designers adhere to design laws and conventions. Numerous research studies attempt to automate this transformation of low fidelity sketches to higher fidelities using Deep Neural Networks. However, these studies seldom focus on the layout quality and aesthetics of the generated higher fidelity prototype. As a solution, this paper proposes RUITE, a UI layout refinement engine that optimizes layouts using a Transformer Encoder. We trained RUITE by adding noise to misalign 35,369 UI layouts from the RICO dataset as input and the original aligned layout annotation as ground-truth. Upon evaluation with 4,421 misaligned UI layouts, RUITE provides 77% accuracy in aligning them. RUITE improves the existing research on transforming low-fidelity sketches to higher fidelities by beautifying generated UI layouts.