Assessing the Creativity of Designs at Scale

How best to assess the creativity of a large number of designed artifacts remains an open problem. The typical approach is to have a panel of experts answer likert questions about individual artifacts. This process typically requires a substantial amount of training to ensure the judges achieve an acceptable level of agreement. Consequently, the approach does not scale well as it is infeasible to have a panel of experts regularly evaluate the creativity of a large number of designs. The current work explores an alternative approach that uses both individual and pairwise judgements from novice crowd workers to support reliable and scalable assessment of creative designs. This approach, which we call TrueCreativity, can operate over a set of evaluations from a large number of judges and appropriately weights their evaluations based on their past reliability and agreement with other judges. We show that this approach produces results that strongly correlate with another measure of creativity.