Anytime I listen to a net-to-gross (NTG) scrum, the philosophical gears in my brain ramp up to a smooth whir. This is what happened at ACEEE summer camp, also known as the ACEEE Summer Study on Energy Efficiency in Buildings. This post is inspired by a paper titled “Applying Gross Savings and Net Savings in an Integrated Policy Framework”, presented by Dan Violette (Navigant) and co-authored by Elizabeth Titus (Northeast Energy Efficiency Partnerships), Pam Rathbun (Tetra Tech), and our very own Teri Lutz.
Since Dan referenced the SEE Action Energy Efficiency Program Impact Evaluation Guide, for purposes of saying the authors didn’t want the ACEEE paper to deviate from it, I use that for reference. I should mention that discussing how to do NTG assessments would be on par with explaining how to build a nuclear weapon, the triggering mechanism, and an ICBM to deliver it to the adversary’s headquarters; i.e., not going to happen in 30 minutes of talking or the 12 pages of the ACEEE paper.
Terms
The following terms are taken from the SEE Action guide and abbreviated to save space. The italicized words are direct as quoted for discussion purposes below.
Gross savings = savings that result directly from program-related actions, regardless of why they participated.
Net savings = savings that are attributable to an energy efficiency program. Net savings account for free riders[1], participant and non-participant spillover[2], and market effects[3].
Case 1 – The Rueful Participant
Assume for example that a customer purportedly implements an efficiency measure, say LED lighting. He takes the money and runs, and thinks, “Man, the program paperwork was a journey through hell. I spent more money and resources chasing down what was eventual pocket change. I’m never going to do that again”. This project is “awarded” a NTG of 1.0.
The customer likes the lighting and associated non-energy benefits but still loathes the program. He does more projects throughout his facility and additional facilities. I assume every like-project this customer implements thereafter is attributed to the program as spillover because the original project was awarded a NTG of 1.0. Isn’t this a little perverted? At what point does the customer who hates the program lose the spillover status?
Case 2 – When is Christmas?
In the airport on the way home, I participated in ACEEE’s conference survey because there are some genuine improvements I’d like to see with the conference. One ubiquitous question was, how did you find out about the conference- email website, etc.? My answer was “other”, and in the form of a question: “How did you find out about Christmas?” This conference happens the week ending the Friday preceding the Friday of Labor Day weekend on even years. At this point, that’s just the way it is to me. When and how did I decode this? I have no idea.
In other words, ACEEE has created thousands of Pavlovian souls who by the end of the conference week are fatigued and looking forward to returning home, but a year later are looking forward to the next one – like Christmas.
Does this sell-job ever become a free rider? Comparing to programs, how can anyone possibly remove the engrained market effects, and the way of doing business that a long running portfolio develops? Like marketing and brand building, it’s a huge flywheel that takes tremendous energy to start, or conversely, to stop.
Case 3 – The Accidental Attribute[4]
People don’t know why they do things even if they think they know. The science of decision making is immense and fascinating. Do NTG panels include the following critical questions?
- What time of day did you decide to participate?
- Were you hungry at the time?
- Was your bladder uncomfortably full?
- Was your footing secure and level?
A month ago I wrote about decision overload, referencing favorable judicial leniency in the morning. In the above linked article, I found more astounding data on this study. Decision making by time of day is shown below. The discontinuities at the dotted lines are breaks in the judges’ day. Wow.
When people make decisions on empty bladders, late in the day, while famished and can’t decide what to make for dinner, their decisions are far more random.
Case 4 – The Mindless Attribute
In the philosophical exercise of pin the tail on the donkey (determining attribution), there are full free riders and partial free riders. By deduction, some participants must have acted solely due to the program. Really? I can only think of one type of decision that is based on one factor. That is, a reflexive avoidance of pain. E.g., I just got the jolt of my life from an electric fence. Hmm. I think I will let go.
NTG is Critical
Despite my flippant commentary, determining NTG is important for efficient allocation of capital, in the form of other peoples’ money. It is necessary to answer this question: should we keep spending money on this program or technology?
[1] Participant would have done the measure anyway, in absence of the program.
[2] Eligible energy users weren’t aware of the program, or didn’t want to mess with the bureaucratic process again, but somehow were magically affected by the program.
[3] Changes in supply and purchasing habits brought about by efficiency programs.
[4] Person who acts in accordance with the wishes of the program, as in 100% attributable.
Jeff, thanks so much for your insightful article about the benefits yet the downside of program evaluations. We can all relate to the case scenarios you described above. I like many others are bruised X Program evaluations that seem as if they are an exercise of devaluing the efforts of many well-intended programs and the hard-working efforts of the people and teams that implement them. In my experience it has felt more like a judicial system that just looked at all the evidence that was negative. Like your example of the flywheel, people have been influenced in so many ways it becomes very difficult especially so many months after the action to capture all the influencing factors that happened recent but also in the years past. Therein lies another problem, how long it takes for feedback to get back to the program implementer. When I worked in manufacturing, we used something called statistical process control and we had immediate feedback if production went out of compliance. Then we immediately corrected the problem and started production. Today’s programs are launched well before feedback from previous program arrives. I think we can all agree we are Desiring the same objective in Energy Efficiency, to make real impact, therefore evaluation needs to be more of a team exercise along with program design and provide near-instantaneous feedback. In similar fashion a much better job is going to need to transform this industry – realistically assess all attributable, not just some absolutes from a limited subset of questions that characterize the programs actions comma and data that is agreed to amongst both implementers and evaluators well in advance of the evaluation so program designs align with evaluation objectives