The purpose of energy efficiency programs is to eventually cease to exist because the market no longer needs them. Program administrators use programs to overcome barriers to energy efficiency, often through incentives, training, or information, and hope these barriers go away.
Historically, most energy efficiency programs have been classified as resource-acquisition programs targeting individual participants. Recognizing the limited reach of typical resource acquisition programs, many program administrators have begun shifting to market transformation initiatives that seek to create sustained structural changes in markets. Because these initiatives differ considerably from typical resource acquisition programs and involve distinct areas of uncertainty, it is crucial that evaluations be set up to accurately measure their impact. Let’s explore the key aspects of market transformation programs and what makes them different.
Overview of Market Transformation
The Midwest Market Transformation Collaborative defines market transformation as the “strategic process of intervening in a market to create lasting change that results in the accelerated adoption of energy efficient products, services, and practices.” Market interventions can accelerate increases in energy efficiency in a variety of ways, such as by influencing technological advancement, influencing the production and supply of higher-efficiency products and equipment, shifting the product mix in stocking practices, shifting design specifications and contractor bidding practices, expanding technical expertise and expanding workforce development, and accelerating building energy codes and appliance standards.
While resource acquisition and market transformation programs can use similar interventions (e.g., incentives and trade ally training), they differ in fundamental ways. First, market transformation programs target all consumers of a product or service rather than just program participants. Additionally, the timelines of the two program types are very different: resource acquisition programs typically focus on annual impacts, whereas market transformation programs often span 10 or more years. Market transformation programs do not expect observed market changes and energy savings until after at least a few years of deployment, while the intervention begins to create market lift. Despite a slow initial deployment period, market transformation can lead to significant sustained savings in the longer term. This lengthy horizon requires a long-term funding commitment and underlying policy and regulatory objectives that can tolerate a program that may not be cost-effective at the outset and that can prevent market disruption from retracted funding.
A key aspect of market transformation initiatives is that, because they aim to measure change in a market, they need to start from an initial baseline and should not be backward-looking, trying to claim credit for past actions. Market transformation strategy development begins with a detailed understanding of a market, including the product/service’s supply chain, the key drivers and decision influencers, and key barriers to higher energy efficiency. A market characterization typically includes an analysis of market size and identifies market actors and influencers in the supply chain, along with their relative market share.
Market transformation programs require a documented theory of change, outlining how the program’s interventions will reduce the identified barriers. This is often illustrated in a program theory/logic model. This allows for the program (and evaluators) to understand and measure the linkages between program activities and outputs and outcomes, hopefully resulting in market change.
Evaluation
Market transformation evaluations require a different approach than evaluating traditional (resource-acquisition) energy efficiency programs. Because market transformation relies on market analyses, not quantifications of annual savings built from the bottom up by calculating the savings from individual projects, there is additional (or different) uncertainty in the claimed savings.[1] Before implementing and evaluating market transformation programs, there needs to be a plan (and underlying regulatory framework) in place that specifies how market progress will be measured and how savings will be calculated.
Because the target is a market and not a single transaction, evaluators must look at multiple indicators to understand if the market intervention activities are resulting in the desired outcomes. Importantly, this needs to be set up from the start of the initiative based on the program logic so that changes can be compared to a baseline over time. Examples of market progress indicators include the market availability of certain types of equipment/services, market share of different equipment types, the promotion of equipment/services by contractors, and the level of different applicable barriers. Evaluators should conduct regular research to understand the changes to the market and measure market progress indicators. Should an evaluation indicate that the initiative is not progressing toward intended outcomes, the program logic will serve as a tool to trace back to underperforming activities and/or revisit the underlying rationale and assumptions.
Additionally, evaluators must consider the energy savings of the total market level. The methodology must account for savings that would have naturally occurred without the program’s intervention. It must also prevent any double counting from any overlapping existing resource acquisition programs.
Finally, evaluators should understand that markets are highly complex, constantly in flux, and involve countless interactions and influences among countless parties. Both the implementation and evaluation strategies of market transformation programs must be flexible and adaptable to respond to changes in the market. The complexity of markets also requires a high degree of engagement and collaboration to cultivate relationships, build momentum, align goals, avoid duplication of effort, and ensure the needs of diverse stakeholders are met.
Special thanks to Jennifer Holmes for many of the ideas used in this post!
[1] While the estimation of savings from resource acquisition programs is often described as more precise because they are based on engineering calculations, they are also subject to high levels of their own uncertainty, including sampling bias, measurement bias, methodological choices, and engineering subjectivity, to name a few.
