Rules as policy data? Measuring and linking policy substance and legislative context

There is growing scholarly interest in analyzing changes in policies, laws, and regulations. Some concepts depart from the goal of identifying changes in policy substance. Other contributions have concentrated on the structural characteristics of laws and regulations containing these substantive changes. Extracting measures of policy substance from legislative texts is a challenging and time-consuming endeavor as it requires the manual assessment and coding of legal acts. The assessment of the structural characteristics of laws and regulations, by contrast, can be done applying automated natural language processing. An important critical question is, thus, whether we can combine these approaches and simplify the information extraction by inferring changes in the policy substance from the legislative context in which these changes are embedded. Examining more than 100 legal acts in the area of EU environmental and climate policy, we find that the measures capturing policy substance and the structural characteristics of legal acts context are not systematically linked. In other words: changes in the structural features of legal acts cannot be used as an approximation for changes in policy substance. We conclude by sketching out a research agenda when (and when not) to use the different concepts and related measurements.