I spent last week in HOT San Diego at the AESP Annual Conference. It was 20 degrees above average, hanging out in the 85-90°F range of blazing afternoon temperatures. It felt like late June in the upper Midwest. Multiple locals noted that “this is not normal.” For sure.
As I noted from my LinkedIn feed, thank you, AESPeers, for your feedback and encouragement to keep the Energy Rant authentic, witty, and edgy. I heard from people I didn’t know or remember, and I learned that the Rant is forwarded and shared throughout our industry. With that, I commiserated with utility folks on the program implementation and delivery end of the spectrum. The other end of the spectrum would be the program evaluation. Too often, program evaluation is where innovation and savings go to die a painful, burning death at the stake. That’s what this week’s post is about.
Program Evaluator Modi Operandi
In my experience, program evaluators rarely seek to improve programs, portfolio delivery, or cost effectiveness. Their MO is that the beatings will continue until realization rates and attribution are sufficiently puny that they can justify their $400-per-hour value. After all, they are saving “ratepayer” funds.
My favorite quote, which I haven’t shared with readers in the last few years, is from Val Jensen, former Exelon Vice President. He said, “Programs are designed to be evaluated. They are not designed to be effective.” That was the pre-Covid, 2019 AESP Annual Conference in San Antonio. For more on attribution definitions and snaggles, read up from that 2019 series of posts here: Attribution on the Cheap, which is quite excellent, if you don’t mind my saying so. Click through to read about six flaws of Stifling Evaluation Dogma.
Measure Life and the ICU Dashboard
At this year’s AESP conference in San Diego, I chatted with a seasoned program expert from a large utility. He mentioned a couple of things that had me laughing out loud and rolling my eyes at the same time. He said they (the utility) argued with the evaluator over attribution, such as maybe a 6.0% attribution rate for data center projects (it’s probably 4.15% by now), or extending the shelf-life of retro-commissioning (RCx) projects due to monitoring-based commissioning (MBCx).
Monitoring-based commissioning systems gather trend data, reveal performance abnormalities, throw alarms, but do not necessarily fix anything. They are like an electrocardiogram, or better, maybe a bedside ICU dashboard that continuously monitors vitals, including heart rate, blood pressure, and oxygen levels. If an alarm blows, the medical experts pounce to save the patient – or the energy manager pounces to avoid a massive demand charge or heating or cooling excursion.
Figure 1 Vital Sign Monitoring Screen
For MBCx, savings are maintained beyond the proclaimed measure life of the typical RCx project, which may average three to five years. Although keeping grandpa’s ticker firing smoothly for three to five years, like an RCx measure, may seem good today, wouldn’t you like to see it fire for ten to fifteen years? That’s what MBCx can do. If MBCx can maintain energy efficiency and electric-load management (demand) in check for that long and forever, why not adorn RCx measures with MBCx surveillance with extended lifetimes to improve cost-effectiveness and market adoption? Because many evaluators think their job is to chop things down to size to demonstrate the value of their exorbitant fees.
Resource Acquisition Vs. Market Transformation
Let’s move on to market transformation. Many folks in our industry, and in some cases entire organizations, like the Northwest Energy Efficiency Alliance (NEEA), are chasing an aspirational goal: transforming markets to be more energy-efficient.
The energy efficiency world can be boiled down to two types of programs: 1) resource acquisition and 2) market transformation.
Resource acquisition (RA) is what most people are familiar with: upstream incentives (manufacturers), midstream incentives (distributors and retailers), and downstream incentives (consumers/customers). Cash incentives are directed at these entities to buy down and/or provide the service to mitigate barriers to efficiency adoption.
Resource acquisition programs are a bit like pet tricks for treats. Here’s an excerpt from a Rant from 2013 that makes me laugh to this day: “Let’s face it. For these consumer programs, people are treated like animals. Sit. Stay. Good boy. Here’s a biscuit. Stupid pet tricks. Speaking of stupid pet tricks, this has engineering nerd written all over it. Is Pavlovian response a free rider? If the customer has no clue what the savings are, isn’t that a free rider? What is the name of Pavlov’s dog? Just curious.”
NEEA defines market transformation (MT) as “the strategic process of intervening in a market to create lasting change in market behavior by removing identified barriers or exploiting opportunities to accelerate the adoption of all cost-effective energy efficiency as a matter of standard practice.” I have an analogy from President Eisenhower: “Leadership is the art of getting someone else to do something you want done because he wants to do it.” Market transformation is attractive and needed because it is less expensive per unit of energy savings, and portfolios need additional methods for achieving savings as the end of the lighting gravy train approaches.
As an analogy, RA is like a Nutrisystem diet, where customers pay for their measured and curated food, and MT is like a Keto diet, where customers are informed and educated to make it work for themselves. Succinctly, RA is selling fish, whereas MT is teaching fishing.
Swinging back to program evaluation, if you evaluate a fishing course like you evaluate fish buying, guess what happens? The fishing course ”fails” because there aren’t any fish to buy. Sound crazy? That’s what happened, according to my experienced industry insider with the big utility. The utility’s MT efforts were given an F because the spot where a transaction with a willing witness to attest to the exchange could not be found. 
Closing Out
There is a saying out there that goes something like, “utilities want to innovate as long as somebody else goes first.” That is a function of a regulated and tied-down industry, but it is also a function of obsolete and misappropriated program evaluation methodologies.