Skip to main content

Impact Evaluation – Catch Lucky

By April 1, 2013November 8th, 2021Energy Efficiency, Energy Rant

For many years of my adult life I thought, “courts and judges: who cares?”  I’ve learned that I do, big time.  Why?  Because courts can bend and twist laws and essentially rewrite them to the point that we might as well send legislators home, which in and of itself sounds like a good idea.  The danger is the courts are the backstop to protect us from unconstitutional law, and if they are creating law by fiat, there is no backstop.  There is no accountability.  They have the final say.  Energy efficiency impact evaluation is much the same, as evaluators declare what programs have actually accomplished.  There are parallels, especially with the final judgment.

One of the jobs of the evaluator is to understand the rules of program for eligibility, baselines, and a raft of other issues.  We have experienced evaluators injecting their own opinion into the evaluation process.  For example, Wal-Mart builds all their box stores with certain features.  “No savings for you” declared one evaluator – because this stuff would have been implemented anyway!  This is like the first amendment.  Vile ogres have the same right to petition government ogres as does everyone else.  We can’t pick and choose.  Evaluators can’t make their own laws either.

As I have blasted in this blog many times, I insist that programs be cost effective and serve as a net benefit to customers and a cheaper alternative to adding supply, transmission, and distribution systems.  Who’s going to do it?  You, Weinberg?  I have more responsibility than you can possibly fathom.[1]  To answer the question (I’ll answer the question, truthfully even), it’s program evaluators.

Evaluators are the supremes[2] of energy efficiency programs and program evaluation is the court case.  Like the supremes, evaluators can interpret and report on reality (the law) or they can generate their own reality, in which case everyone loses, except “special interests”, which may include the utility or the program implementers.

The Dumb Down Low Ball

My fear in this isn’t so much intentional misrepresentation but lack of depth of investigation and digging and technical competence/expertise on impacts.  Furthermore, what really concerns me is the Wal-Mart-ization of this critical element of energy efficiency programs.  Selecting a program evaluation team is like house shopping from the curb.  Evaluation service buyers can be juked into “buying” a fake town like the bad guys on Blazing Saddles.  The pretty façade can resemble a proposal and eventually the report itself.  Prop up a nice façade in front of a 1972 leaky trailer home and call it a Frank Lloyd Wright modern contemporary masterpiece.

As Wal-Mart learned, somebody will always undercut you.  Refer to Dollar General.  However, buying program evaluation is not the same as buying a box of Lucky Charms, which has been proven in blind taste tests to taste the same from Dollar General as it tastes out of Safeway.

Decent impact evaluation, particularly for commercial and industrial programs, requires substantial expense.  Does the evaluation buyer really want to know what’s going on, or do they want a report to show the regulators / legislature, er I mean the court, that everything is hunky dory?  I promise you, hunky dory is cheaper than even Dollar General.

Getting it Right Costs Money

Dollar General isn’t going to cut it.  It costs money, in some cases quite a bit, to do a decent job on impact evaluation for C&I programs.  The energy calculations performed by the program can be lousy and disheveled or may hardly exist.  A new building automation system that affects a half-million square feet of office space may receive a $100,000 incentive.  Evaluating this project and estimating the savings in the first place is a bit more complicated than doing the same for the Dollar General light bulb.  If the savings aren’t there, why are they not there?  This is a difficult and costly question to answer but if it isn’t answered correctly, send the legislature home and skip the evaluation altogether, because what is the point?  The point is (1) get the right answer and (2) improve the program and that may mean switch to a consultant that knows how to do custom calculations.

Getting Wrong to be Right Costs More Money

And another thing: programs with lousy impact estimates cost more to evaluate.  It takes longer to figure things out.  It takes more digging on site, more data collection, more interviews, and more analysis.  Then finally when it’s all pulled together, it may take several rounds with the utility/implementers to hash out the adjustments.  The hunky dory report is met with no resistance.

Suggestion:  Selecting an evaluation team should be based on qualifications, and the fee should be negotiated after selection based on needs and priorities.  Typical fees are 2-5% of total energy efficiency portfolio spending – a higher percentage for smaller portfolios and lower percentage for huge ones.

Look, someone will always sell cheaper plastic cutlery.  If buyers desire to pay $59 for Motel 6 and delude themselves to believe sitting on a stained chair at a Formica table with Canada Dry in a plastic cup with ice is the same as lounging with a Bombay martini at a Hilton lounge, you go man.

[1] That is correct – a Colonel Jessup rant.
[2] Supreme Court justices.
Jeff Ihnen

Author Jeff Ihnen

More posts by Jeff Ihnen

Leave a Reply