Mind the Gap: Tracking Recommendations to Improve Program Effectiveness

Program evaluations offer essential information regarding all aspects of program operations. But too often, recommendations for program refinement and enhancement are quickly forgotten. This paper will illustrate an effective strategy to ensure that these recommendations are taken “off the shelf” and incorporated into future program design and evaluation activities. This paper describes the process evaluation “best practice” that has been used effectively in several jurisdictions to ensure theat program recommendations are well-formulated, strategic, and monitored throughout the evaluation cycle. It begins by describing the types of recommendations that truly make a differencestrategic recommendation that focus on measurable improvements in program deliveryrather than tactical recommendations that target minor adjustments to program implementation. Several examples from real word evaluations are highlighted in this paper, to demonstrate the different ways in which this technique can be modified to meet differing program goals. Specific tactics discussed include: creating a recommendation tracking database that organizes each recommendation by topic, sector, and category; classifying the current status of each recommendation; and the reporting interval for updating program recommendations. This approach has been well-received by program administrators and public service commissions. An example recommendation tracker template is also included in this paper so attendees have a take-away that provides immediate and lasting value. Tracking recommendations is an important way to demonstrate the value of conducting program evaluations, document the the progress each program is making over time, and ensure that the investments in EM&V activities are incorporated into future program designs. Introduction Program evaluations offer essential information regarding all aspects of program operations. This is especially true for process evaluations, which focus on providing essential and timely feedback regarding current program operations. A critical outcome of process evaluations, therefore, is to provide recommendations that will improve and enhance program operations. However, formulating recommendations that will provide both guidance to benefit an entire program portfolio or lead to improvements in overall program operations require 2020 Energy Evaluation Europe Conference — London, UK 2 thoughtful analysis. Unfortunately, many process evaluations include recommendations as an after-thought. They are hastily written, sometimes repeated verbatim for each evaluated program, and are not carefully tracked over time. The result is that these recommendations are read and maybe overlooked or forgotten, thus diluting the value of the overall process evaluation. This paper describes the ways in which the status of major recommendations from program or portfoliowide evaluations are monitored throughout the program cycle. This approach, currently used for statewide evaluations in Arkansas and Maryland, tracks the progress of the key recommendations during in each evaluation report. Monitoring and updating the implementation status of each major recommendations provides greater transparency regarding program operations and instills confidence that the previous evaluation findings are being incorporated into future program designs. This evaluation approach aligns exactly with the feedback loop developed by The National Action Plan for Energy Efficiency (2007) (see Figure 1). (Source: NAPEE 2007) Figure 1: Program Implementation Cycle With High-Level Evaluation Activities As Figure 1 shows, program evaluation is viewed as an ongoing process that provides information regarding changes in program direction and adjustments to program goals and objectives over time. Monitoring the ways in which the previous evaluation findings have been implemented in current program operations ensures that the critical takeaways from previous evaluations will be incorporated in current program design. Thus, tracking the progress of major recommendations offers a way for evaluators and implementers to assess the effectiveness of these program changes and identify new strategies to consider in evolving energy markets. Establishing the Rules of the Road Unlike impact evaluations, which rely on a set of protocols such as the International Performance Measure and Verification Protocols (IPMVP), only a few jurisdictions have developed specific protocols to for conducting process evaluations. In 2012 New York State recognized the importance of establishing Evaluation,