by Peter Rose
Everyone complains about the Weather but nobody does anything about it!
Judging from many recent observations among the more objective blog-sites, such as Climate, Etc., this old American adage would seem to apply just as well to perceived cognitive bias in contemporary climatological research: Everyone talks about Cognitive Bias in Climate Research, but nobody does anything about it!
However, a branch of western science with which I am intimately familiar has been effectively and routinely countering cognitive bias for at least two decades now – Petroleum Geoscience. In the hope of sharing our experience with other scientists, I contacted Professor Curry a month ago, offering to share various effective methods we have developed for dealing with cognitive bias in geoscience data for our exploration plays and prospects. She graciously invited me to contribute the following note.
Daunting uncertainty routinely attends petroleum exploration ventures. Most key geotechnical parameters must be estimated. Cognitive bias flourishes in such circumstances. Where petroleum geoscience differs from academic or institutional science is that we are forced to see the results of our geotechnical predictions in the cold, hard light of day (or night), and they are prompt, frequent, often humbling, and even career-threatening. Perhaps petroleum geoscientists can be forgiven for wishing sometimes that their academic colleagues had to function under similar circumstances!
Cognitive bias – usually overoptimism – characterizes initial estimates of geotechnical parameters underpinning exploration projects. If left undetected, this leads to disappointing project outcomes; when such cognitive bias is endemic in a company’s annual portfolio of Exploration and Production (E & P) ventures, the portfolio underperforms, and the company’s annual performance suffers. Stockholders are unhappy; corporate heads roll.
So, beginning mostly in the 1990s, most now-surviving E & P companies started training their geotechnical staffs in:
- Using pragmatic operational techniques that detected and corrected cognitive bias;
- Adopting uniform software that standardized evaluation of all new ventures;
- Establishing trained “risk-teams” that reviewed all prospects before drilling; and
- Following all executed projects with post-audit reviews and measured E & P team performance in accurately forecasting geotechnical parameters.
The result (together with remarkable contemporaneous advances in seismic resolution) was a marked improvement in E & P performance in the late 1990s and 2000’s, continuing to the present day.
The five most prevalent cognitive biases in E & P work are:
Confirmation bias – Ignoring data that don’t fit our theories (or the desired outcome).
Overconfidence – Predictive ranges are too narrow, leading to many sad surprises.
Representativeness – False analogs or misleading spectacular examples.
Anchoring – Initial estimating and adjustment process is incomplete.
Motivational bias –Perceived personal self-interest influences technical estimates.
In their work on exploration ventures, today’s geoscientists routinely search for and develop three classes of technical estimates:
- Geological Chance of Success – the probability that recoverable reservoired petroleum will be found by the exploratory well, considering the evidence for the five independent geological conditions required for a subsurface accumulation to exist (Petroleum source rocks; Migration to the prospect; Reservoir rock presence; Trapping configuration; Subsequent preservation).
- Geotechnical Parameter Dimensions (assuming subsurface petroleum presence) – Estimates (as probabilistic ranges) of key factors such as reservoir volume, reservoir yield, flow rates, production decline rates, etc.
- Economic Parameters –Estimates (as probabilistic ranges) of project time, costs, product prices, interest rates, taxes, etc.
Estimates of all such parameters should be as unbiased and well-founded as professionally possible. The geoscientific goal is for the E & P portfolio to deliver its promised performance; the required professional ethic is one of objectivity and pride in predictive performance.
Geological chance of success predictions inherently involve informed subjective probability. Disciplined estimating requires marshaling of all facts bearing on the presence (or absence) of each of the five geological chance factors separately. Structured exercises in the application of multiple working hypotheses expand conceptual horizons. “Group wisdom” is brought into action, considering all pertinent subsurface geological evidence (including seismic amplitude data), consideration of data quality and quantity, and structured quasi-Delphi rounds, to generate an appropriate probability for each of the five geological chance factors. The lowest-probability factors (“critical chance-factors”) are identified, and then reinvestigated to further refine and qualify before the final prospect Geologic Chance of Success is determined. For costly ventures, the corporate “risk-team” reviews all geotechnical estimates before final approval for drilling. After drilling, geotechnical reasons for success or failure and consequential learnings are noted. The corporate “risk manager” compiles and communicates staff performance annually with respect to predictions of Geological Chance of Success and consequent learnings. Through the late 1990s and early 2000s, our Chance of Success predictions improved steadily.
Geotechnical parameter ranges governing the volume of recoverable oil and gas contained within the subject prospect are probabilistic, following the lognormal distribution. They are routinely plotted as a cumulative log-probability distribution, ordinarily a straight, upward sloping line (Fig 1). Accordingly, geoscientists have learned that if they design their predictive ranges for parameters such as Gross Rock Volume, Reservoir Net-to Gross, and Reservoir Yield so as to fit the lognormal expectation, they make much better predictions. This approach especially reduces the strong tendency to overestimate prospect recoverable volumes. Interestingly, the usual reason for such overestimation is not that the “high-side” estimate is too high – rather it is that the “low-side” estimate is too high, rendering the derived mean estimate of recoverable volumes larger than it should be. Reality-checks are provided by the lognormal distribution itself. For example, if the high end (= P1) of the initial porosity distribution is, say, 75% (= practically impossible), the slope of the distribution should be steepened so the P1 value becomes a rational upper limit, i.e., ~ 30%. Use of basin field-size distributions brings perspective to individual prospects in those basins or trends. Flow rates and production decline curves also fit characteristic harmonic or exponential patterns, and predictions made using such curves will result in greater predictive accuracy. In all cases, however, the corporate “risk-team” will vet the developed estimates of recoverable volumes, and report back after drilling results are known. Gradually, a team’s predictive performance is established and (hopefully) improves in the future.
Economic parameters are usually beyond the purview of geoscientists, and may be forecast using distributions based on past experience. The critical parameter for the corporate drilling portfolio is the expected net present value (ENPV) of the venture, which incorporates all estimates of chance, recoverable volumes, investments and production revenues over the life of the project, taking into account the time-value of money. Estimating is serious business.
As a lifetime geoscientist, I sincerely hope that some of these methods (Fig. 2) may prove useful to colleagues in other fields of science. It seems to me that all of the aforementioned techniques can, and should be, applied routinely throughout any disciplined scientific investigation. Post-audits may be the exception – the results of institutional scientists don’t seem to be tested for accuracy. But petroleum geoscientists deal with that cleansing task routinely; it has the effect of gradually eliminating incompetents and feather-merchants from the professional gene pool.
All Western Science seems to offer as counterparts to structured post-audits of research projects are 1) peer-review, and 2) the purifying effects of following published scientific validation or invalidation. The first seems badly compromised now; the second necessarily takes a long time (and an awful lot of publicly funded research).
Most experienced petroleum geoscientists understand Cognitive Bias – after all, they have experienced the business pressures that stimulate it, as well as the inevitable consequences of succumbing to it. They have seen, over their careers, more than a few geologic salesmen in action, trying to sell over-hyped and carelessly researched drilling prospects. That may be one reason why so many petroleum geoscientists agree with Professor Curry that Climatology needs to clean up its act – it’s damaging the rest of Western Science.
Biosketch. Dr. Peter R. Rose (Ph. D., Geology, University of Texas, Austin) has been a professional geologist for 55 years, specializing in Petroleum Geology, E&P Risk Analysis, and Mineral Economics. He is the author of more than 75 published papers. In 1998 he founded Rose & Associates, LLP. Pete retired in 2005; the firm continues as the global standard among consulting companies in that field, providing instruction, software and consulting services on an international scale. His 2001 book, Risk Analysis and Management of Petroleum Exploration Ventures, now in its 7th printing, is considered by many as the “Bible” on that topic, and has been translated into Chinese, Japanese, and Russian. In 2005 Pete was the 89th President of the American Association of Petroleum Geologists. In 2013, the Geological Society of London awarded him its prestigious Petroleum Group Medal for lifetime contributions to Petroleum Geology, the first American to be so recognized, and in 2014 the AAPG honored him with its Halbouty Outstanding Leadership Award.
JC note: As a result of some email exchanges, I invited Pete to do a guest post. As with all guest posts, keep your comments civil and relevant.
Filed under: Sociology of science, Uncertainty
