Evaluating assistance of natural language policy authoring

The goal of the research study reported here was to investigate policy authors' ability to take descriptions of changes to policy situations and author high-quality, complete policy rules that would parse with high accuracy. As a part of this research, we investigated ways in which we could assist policy authors in writing policies. This paper presents the results of a user study on the effectiveness of providing syntax highlighting in a natural language policy authoring interface. While subjects liked the new interface, they showed no improvement in accuracy when writing rules. We discuss our results in terms of a three phase authoring process that users move through when authoring or modifying policies. We describe this process, discuss why and how our interface failed to support it and make recommendations to designers on how to better support this process.