Skip to main content

Tales from the Energy Program Evaluation Asylum

By August 14, 2017November 6th, 2021Energy Rant

I’ve spent a lot of time reviewing papers written for the International Energy Program Evaluation Conference, or IEPEC, over the years. I dig the conference, which I like to call the asylum, as I did four years ago. People in the evaluation business can get pretty fired up. While I witnessed no verbal punches at this conference, one guy was practically shouting his comment and quaking during his peanut gallery monologue. Is there a question here, somewhere?

Here’s the thing: lighten up, y’all. Don’t take things so damn seriously.

On that note, there were very constructive conversations around climate change. I was extremely (and I use that word extremely sparingly) pleased to find the organization and industry is not simply ranting about knuckle draggers[1] on the other side. The conversations have finally progressed to engaging them. Imagine that! I’ll write more on this later in a separate post.

Here are a few subjects I thought worthy of bantering.

Equipment Useful Life

This topic is pretty simple. How long do you maintain your car for reliable driving before it becomes a cash incinerator and you send it to the scrap yard? At that point, it has reached the end of its useful life. The end of useful life is not when someone offloads their fine-running, leased Mini Cooper after three years of successful driving, and then Jeff swoops in and buys it at a 40% discount to a new model.

If cars were widgets subsidized by energy efficiency programs, the guy who leased, and presumably buys or leases a different new vehicle, would achieve savings relative to an alternative new vehicle. Is this fair?

For efficiency programs, everything is compared to some other new-equipment alternative. So we get into these sorts of philosophical, nutty arguments. What would have otherwise happened? It’s like writing the end of a story – whatever you want.

  • I could have used my three-year old car for another 17 years, and therefore, I should get savings compared to that car for 17 years and an alternative new car for three The new car hasn’t been developed yet, and I don’t even know what the fuel will be. Write your own story.
  • What is the end-of-life anyway? The motor seizes? The transmission falls off the bottom? The cost to repair is more than the value of the vehicle (but less than the cost of another car)?

Sacred cow alert: many stubborn people are adamant that any new purchase should be compared to some other new model, only.

Let’s go straight to lighting and the 37 Watt[2] per tube fluorescent fixture. There are millions, no zillions, of these fixtures in use yet today.

When we leave these fixtures, that are equivalent to a 1972 Cadillac Sedan DeVille, in place and deny the utility customer incentives to send the heap to the scrap mill, we are “stranding” thousands of megawatt hours for a single utility in many cases. Stranded savings total a whopping 70-75% of total opportunity, even in the Once Golden State of California.

That isn’t low-hanging fruit. It’s fruit that is falling off the wagon!

Statistical Sampling

Recently, in Drive-By Evaluation, I demonstrated with statistically valid samples how doing desk reviews alone, and not looking at projects installed in the field, gives misleading results and provides crumbs by way of improving programs, let alone accurately projecting savings.

The industry standard sampling plan is aimed at 90/10 outcomes, which is 90% chance that the sample results are within plus or minus 10% of the results of the entire population.

Alert: This has nothing to do with accuracy. Want a great example? Glad you asked.

Gas furnaces. Calculating savings from gas furnaces is straightforward. It is proportional to the difference in one over efficiency:


If the evaluator does not get natural gas billing data and account for baseline natural gas use (for water heating, clothes drying, cooking), the results are likely to be about 40% high. Why? Because home HVAC is always oversized by a bunch. You can find similar trends for any reputable impact evaluator.

Therefore, the evaluator can be totally off the mark and hit 90/10 because if they evaluated the entire population that way, they’d get the same wrong result. Congrats!

NILM and EM&V 2.0

What would an evaluation conference be without an overdose on acronyms and jargon-laden claptrap? NILM, aka non-intrusive load-monitoring, is a concept where you use crude periodic power reads at the building meter level and disaggregate[3] it into the various end-uses in a home or business. For example, a microwave might pull 750 Watts; a refrigerator, 120 Watts; an air conditioner 2,000 Watts; a coffee grinder 110 Watts. The concept of NILM would provide identification of these loads remotely using equipment, such as a smart meter, that is already in place.

These home energy loads run from seconds to minutes to hours at a time. NILM readings happen every 15 minutes or an hour. Can you recognize the problem yet? I have a wild imagination, but I cannot think of a comparable scenario with such an enormous range of loads and timescales of operation. It’s like measuring something with a yardstick that requires an electron microscope to see.

The objective of this fantasy is to determine operating patterns for substantial loads like refrigerators and lighting. The reality is, it might (might) be sufficient for one of the largest loads of the home – the central air conditioner.

There may have been a dozen papers published on this subject, and they all produced the same results: does not work. It can work but not with standard equipment in place.

[1] Their impression; not mine.

[2] Total fixture power divided by number of lamps.

[3] See Evaluation, Measurement and Verification 2.0

Jeff Ihnen

Author Jeff Ihnen

More posts by Jeff Ihnen

Leave a Reply