Skip to main content

Energy Program Evaluation Asylum

By August 19, 2013November 8th, 2021Energy Efficiency, Energy Rant

Last week, some colleagues and I attended the International Energy Program Evaluation Conference (IEPEC) in Chicago.  The conference was great with my favorite part: meeting people, getting to know them better, and building friendships.  Session content, as usual, spawned a theme in my mind.  This year’s theme: This business is crazy.

There are panel discussions of experts, you know, the thought leaders with 30+ years of experience in the business, and as I sat there listening, I thought to myself, “Isn’t this a rerun of something I attended in 2011 in Boston, and in 2009 in Portland, and in 2007 in Chicago?”  Indeed, Mimi Goldberg, a lifetime achievement award winner[1] herself, opined in the prelude to a question that this (discussion at hand) was the same discussion she had heard many times for eons [my words].  Many of these discussions include free riders, free drivers, spillover, and these sorts of things.  I will define these to refresh your memory and then move on to the insanity.

  • Free Rider: program participant that would have done the project regardless of the program – received incentives or services for something they would have done on their own.
  • Free Driver: end user implements a project as a result of program activity but doesn’t consume program dollars or services.
  • Spillover: essentially factors that contribute to free drivers, with very subtle differences

The industry spends millions and millions of dollars quantifying these three things, which contribute to net savings to gross savings ratios, or NTG.

  • Gross savings: Best estimate, verified savings per program rules[2], of projects enrolled in the program.
  • Net savings: Gross savings with a slice removed using black magic and lots of imprecise and very uncertain data – the portion of the gross savings that are attributable to the program.  Net savings essentially incorporates free riders, drivers, and spillover effects.

Net to gross (NTG) is a controversial, academic, ubiquitous, eternal football specially designed for thought leaders to kick around every time they get together, particularly in rooms filled with70 people.  When the conference is over, the football is restored like a classic car, and stored for safe keeping till the next thought leader match two years hence.

Consider a performance contracting project that was discussed briefly in one session.  The project included new boilers and envelope measures.  The natural gas savings were 50%, verified by energy bills and plotted against heating degree days.  Savings: crystal.  Unless the sashes had fallen out of the old windows and the walls were previously 18 gauge steel siding with no insulation, the vast majority of the savings probably came from the boiler plant overhaul.

The uniformed analysis of such a boiler plant upgrade would compare the rated operating efficiency of the old boilers (80%) against that of the new boilers, which were probably in the 90% plus range.  The incentive in most programs would be based on the capacity of the new boilers, and the rated efficiency difference between old and new.  However, the real savings come from the systemic efficiency of the old versus new plants as a whole.  This is a lot different than operating efficiency of the boiler.  Think of an 8 liter, V-12 engine on a moped being replaced with a 0.05 liter, one-cylinder engine.  That is what happened in this facility.  The operating efficiency of the V-12 is probably better than that of the 0.05 liter engine but it is massively oversized and wasteful for the application.  This is a baseline issue that greatly impacts the gross AND net savings for the project.

Many people are adamant about using minimum efficiencies from codes and standards for a retrofit/replacement, regardless of the operating circumstances.  Boilers are rated for a 24 year design life, designed for 30, and last for 60 years or more[3].  Also consider the popular refrigerator recycling programs.  The incentive is provided for killing and dismantling the old refrigerator.  Why?  Because the damn thing was built like a 1940s era farm tractor – it will never die – like boilers.  So, you’re telling me that just because something is at the end of its rated life it will be replaced anyway?

Then we have retrocommissioning (RCx) programs, which thankfully are gaining traction because the savings potential is enormous, clearly measureable, and when done right, enduring.  For these, the baseline is obviously as found.  If as found is ok for RCx, why is it not ok for retrofit and replacement, thought leader?  Nobody makes a savings adjustment on an RCx project because the chiller in use is beyond its useful life.

Another session included the presentation of an operations and maintenance program for industrial customers.  Savings were determined entirely by billing regression.  That may be fine, but what are the measures?  Crickets.  Was the equipment beyond its useful life? (who cares).  Here we have neither a baseline nor post implementation anything.  People were devouring the analysis, and I’m thinking, did a tree actually fall in the forest?

In the energy program evaluation asylum, we joust incessantly about ifs, buts, candy, and nuts while losing sight of or not seeing or even realizing the big picture – reasonably accurate energy impacts.

[1] An award IEPEC gives to one or two people every other year, at least, when the conference is held in the US.

[2] Not the same as actual savings because of “as compared to what?” (e.g. what is the basis for comparison).

[3] My SWAG.

Jeff Ihnen

Author Jeff Ihnen

More posts by Jeff Ihnen

Join the discussion 8 Comments

Leave a Reply