Tweening Around the Tworld!

Austin Energy’s Value of Solar Study presentation the other night drove home to me how few people actually think critically about almost any issue. A few of my fellow commissioners showed a strong grasp of the issues and asked penetrating questions. Nonetheless, almost all of this process demonstrated poor critical thinking skills from almost everyone.

This discussion on critical thinking would help all of us.

Here is its summary:

It is naive to expect social-science education, natural-science education, or education in general-at least in their present forms-to elevate critical thinking to something more than a pedagogical fashion that everyone applauds but few conceptualize very deeply. This leaves the skeptical community. We identify ourselves as champions of science and reason. But this is a broad mandate. We should avoid concentrating our skepticism too narrowly on the realms of superstition, pseudoscience, and the supernatural-for the ultimate challenge to a critical thinker is posed not by weird things but by insidiously mundane ones. If we hope to realize the promise of critical thought, it is important that skeptics affirm a multidimensional definition of critical thinking — reasoning skills, skeptical worldview, values of a principled juror — that exempts no aspect of social life.

Well, which one?

Take the quiz!

My results:

I am Iron Man

Iron Man 95% Green Lantern 80% Superman 70% Hulk 70% Spider-Man 65% Batman 50% Catwoman 45% Supergirl 43% The Flash 40% Robin 23% Wonder Woman 13%

Inventor. Businessman. Genius.

Austin Energy has commissioned a study to determine the economic value of generating large quantities of electric power from current solar technologies (known colloquially as the Value of Solar Study or VoSS). This is a laudable goal. The study is available here and its economic development counterpart is here. It does a very good job of enumerating different costs of solar energy generation and how solar can benefit a municipal utility.

The real measure of any study though is how reliably does it guide policy makers to choose wisely amongst their options? In this area, due to a single glaring methodological flaw, the VoSS is almost useless in guiding Austin City Council. Lets examine this flaw in some detail.

One of the largest economic benefits of solar and wind energy is the benefit of avoiding the cost of fossil fuels. While we can easily calculate this for the current year based upon the avoided cost of Natural Gas priced by the commodities futures market, this problem becomes much more complex over the projected 30 year lifetime of the solar generation hardware. The futures market only looks five years into the future. Therefore, we have to develop a model for years 6 through 30 of the study.

The one thing we know about the future is that we don’t know what the future holds. But we can estimate it.

The VoSS

Austin Energy provided the VoSS authors with a confidential proprietary model of future gas prices. Normally, this might be a problem. How can you critique that which you cannot see? There are some pretty simple questions that any study needs to be able to answer that, without disclosing proprietary details, can give policy makers some confidence in its results. They are:

1. How long has this model been in existence?
2. How many other organizations depend upon the output of this model to make their policy decisions?
3. Since the VoSS uses the model to predict prices for natural gas from 6 to 30 years into the future, we need to see how effective the model has been in predicting the present price of natural gas. Therefore, just using data from five years ago, did the model predict today’s prices on the natural gas market?
4. If the model missed predicting today’s price (as it almost certainly did), how much was it wrong by? In other words, what is the difference between the model and reality? For bonus credibility, what has been the model’s standard deviation of its error over the period of five years ago to the present?

All of these questions are asking for aggregate measures of the model’s predictive efficacy and if policy makers are placing ‘money bets’ on its predictions. Most folks who are knowledgeable about statistical methods would agree that the above questions do not disclose proprietary information about the construction of the model.

For Comparison: A Public Natural Gas Price Model

The below image is an example model output commissioned by the U.S. Government’s Energy Information Administration.

Please note the range of values, the blue band, predicted for natural gas prices. This is the kind of output that should be driving the conclusions made by our VoSS. Yet, the VoSS has no error bars in its price projections.

Any public discussion of long term investments must take into account the uncertain nature of the future.

Why are Models Complex and the Results Uncertain?

So that we can all understand the complexity a model must manage and, hence, difficulty it has in predicting the future, lets look in depth at issues the VoSS’s model must manage.

As we all dramatically saw in late August 2005, hurricanes, such as Katrina, can create havoc in both the oil and natural gas markets. For example, as a result of Katrina, I am told that 10% of the natural gas formerly produced in the Gulf of Mexico (GoM) is now ’shut in’ or stranded. The pipelines that used to deliver this gas on to shore are broken and, because the volumes of gas are so low, it is no longer economically viable to fix those pipelines. In economic modeling talk, Katrina was an exogenous event. What other exogenous events might an effective model consider?

1. The effects of demand destruction due to price increases/volatility on gas prices?
2. The effects of Liquified Natural Gas (LNG) plant construction on gas prices?
3. The effects of competition for LNG from western Europe on gas prices?
4. The historical high in the number of severe hurricanes in the GoM over the next 15 years?
5. The political risk of depending upon Qatari or Russian natural gas?
6. The reliability of uncharacterized ‘proven reserves’ in Qatar and Russia?
7. The shortage of rigs to drill replacement wells?
8. Do Qatar and Russia have enough rigs to meet the demand from Europe and the US?
9. Are there enough liquefaction plants being constructed near the stranded gas?
10. Our dependence upon politically unstable Nigeria?
11. Terrorist attacks on the gas infrastructure?

Lets remember that I am not a natural gas industry professional. Anyone who actually models the gas market for a living can imagine many more exogenous events. For example, Matt Simmons, a natural gas investment banker, contends that natural gas wells have a very different depletion curve than oil wells. That natural gas wells stop producing gas in a dramatic, sudden fashion. Mr. Simmons believes that the industry is incorrectly modeling gas well depletion and, hence, will be caught by surprise when the wells flare out.

As an example of the political risk we are taking by depending upon natural gas, the above picture lists the worldwide reserves of natural gas. Yes, it says we have 60 years of reserves at current consumptions levels. With both China’s and India’s appetite for energy growing at a phenomenal rate, that 60 year estimate will be wrong soon. In fact, there is a very good chance that energy consumption will double over the 30 years that VoSS covers. Is this challenge to gas pricing factored into the VoSS’s model?
It is truly a complex world out there. Reducing an investment decision to a single static number is wrong.

Perhaps after the VoSS hearing on Monday, June 26th, I may change my opinion but the Austin Energy consultant will have to work very hard to convince me otherwise. (We’ll see how good of a tap dancer he is.)

Summary

The VoSS addresses many issues and makes many good, sound economic assumptions. Yet, because it projects a single value for the costs of natural gas over the next 30 years, it is not a useful guide to policy or as a benchmark price for soliciting RFPs for purchasing solar derived energy. It is essentially useless to guide the City of Austin’s policy makers, the City Council. As I write this, I cannot in good conscience recommend to Council that they base policy decisions on this document.