What is red teaming?

by Judith Curry
Last week, Scott Pruitt, EPA Administrator, stated he intended to form a ‘red team’ to debate climate science.  What exactly is ‘red teaming’, and how can this be implemented in a way that is useful for climate science and for policy makers?

Reactions to proposed climate red team
Some analyses of Pruitt’s proposal from the media:

• E&E News: Red teams gain prominence to question climate science
• NYTimes: EPA to Give Dissenters a Voice on Climate, No Matter the Consensus

Arguably the most interesting article on this comes from Joseph Majkut of the Niskanen Center:  Can a Red Team Exorcise the Climate Debate? Excerpts:
On the contrary, both climate scientists and advocates should see opportunity in a red team exercise. A properly-done red team exercise could both elevate the status of climate science in the Trump administration and among Republicans, and reset how we approach climate science as a nation.
Many climate skeptics suspect that the climate science community is caught up in political conformity that leans toward alarmism, and that alternative ideas about the causes and risks of climate change cannot break through peer review. Red teaming is designed to address such a situation. As Micah Zenko writes in his authoritative book Red Team: in institutions that are supposed to police themselves through internal processes, like the scientific community with peer-review, “even longtime analysts are susceptible to adopting the assumptions and biases of the institutions and subjects they are supposed to be objectively studying.” Whether climate science is caught in such a morass or not, many people in power think that it is. We have to find a way to unstick that belief if the climate debate is to move forward. A red team exercise is a fine way to do it. 
The strongest red team exercises have buy-in from all parties and give the red team resources to perform original analysis along a set of critical questions. They also ensure that the team has the right mixture of expertise so that its results will be considered credible to the institution they are looking to influence (in this case, climate science). Lastly, they give the red team sufficient independence to come to original and creative conclusions.
What is red teaming?
There are many pitfalls in  establishing and conducting a successful and useful red team exercise.  To avoid these pitfalls, the relevant policy makers, potential scientist participants, and journalists should have an open discussion on the objectives and guidelines.  And not to mention actually learn something from experts on red teaming.
There is an authoritative book on this topic by Micah Zenko entitled Red Team: How to Succeed by Thinking Like the Enemy.  I haven’t read it yet, but I just purchased it.
Text from an amazon.com review:  In reading this book an aspect of the subject which really stood out to me was that no matter your position of authority or social status, humility and critical thinking often go hand in hand, and that pride and its companion arrogance are often rewarded by a special kind of blindness.
A very interesting and useful online resource for red teaming is provided by the Red Team Journal. Red Team Journal was founded in 1997 to promote the practice of red teaming, alternative analysis, and wargaming. The site has influenced a generation of red teamers to think systematically and creatively about their assumptions, challenges, adversaries, and competitors.
Excerpts from the Red Team Journal About page:
Defined loosely, red teaming is the practice of viewing a problem from an adversary or competitor’s perspective. The goal of most red teams is to enhance decision making, either by specifying the adversary’s preferences and strategies or by simply acting as a devil’s advocate. 
Alternative analysis is the superclass of techniques of which red teaming may be considered a member. As with red teaming, these techniques are designed to help debias thinking, enhance decision making, and avoid surprise. According to Fishbein and Treverton, “alternative analysis seeks to help analysts and policy-makers stretch their thinking through structured techniques that challenge underlying assumptions and broaden the range of possible outcomes considered.” They further clarify the term by specifying that “Alternative analysis includes techniques to challenge analytic assumptions (e.g. ‘devil’s advocacy’), and those to expand the range of possible outcomes considered (e.g. ‘what-if analysis,’ and ‘alternative scenarios’).”
As one would expect, the quality of the output hinges inter alia on the quality and experience of the team, the team’s approach and toolset, and the overall context of the effort. An overconfident or culturally biased analyst or team will not benefit as much from these approaches as might an analyst or team that employs “actively open-minded thinking.”
Excerpts from Red teaming: a balanced view:
Clearly, not every red team is created equal. Superior red teams, for example, tend to

• View the problem of interest from a systems perspective;
• Shed the cultural biases of the decision maker and, as appropriate, adopt the cultural perspective of the adversary or competitor;
• Employ a multidisciplinary range of skills, talents, and methods;
• Understand how things work in the real world;
• Avoid absolute and objective explanations of behaviors, preferences, and events;
• Question everything (to include both their clients and themselves); and
• Break the “rules.”

One can argue that the best red teamers are born, not trained. It seems that some people have an instinctive ability to red team, while others—despite extensive training—can never escape the secure but confining pen of convention. In fact, this is perhaps the key characteristic of the inferior red team: an inability or unwillingness to color outside the lines. Inferior red teams also tend to

• Accept without question the client’s description of the problem;
• Embrace the biases inherent in their own values and culture;
• Adopt the first or most easily discerned answer;
• Defer to reputation and status; and
• Know it all.

The members of an inferior red team might include deferential technocrats and self-important experts.
Resistance. Not every decision maker wants a red team (or at least a candid red team). A red team can undermine a decision maker’s preferred strategies or call into question his or her choices, policies, and intentions. It takes a decision maker of solid integrity to sponsor, empower, and manage a superior red team. That said, a thoughtful decision maker also balances the costs and benefits of red teaming with the costs and benefits of advocacy, compromise, and consensus building. It is also important to note that not all resistance is harmful; it can represent valid interests, concerns, and risks of which the red team is simply unaware.
To Red Team or Not to Red Team. Nearly everyone can benefit from some form or degree of red teaming. Whether the “red team” is a highly structured, formal unit or a self-appointed devil’s advocate, almost every idea, concept, design, or plan benefits from healthy opposition and testing. Too much red teaming, however, can be as harmful as too little. No one wants a relentless contrarian gumming up every phase of a project. It won’t take long, in fact, for everyone to dismiss the contrarian as an annoyance.
Decision makers must be careful to apply red teaming judiciously. Among other factors, timing is especially important. Establishing a red team too early can lead to aimless dithering; establishing it too late can trigger fierce (and justifiable) resistance. Even so, the adage “better late than never” sometimes applies. (If one adage always applies to red teaming, it is “one size [doesn’t] fit all.”)
It is also important to consider and value the perspective of all client stakeholders. Not every problem has a distinct boundary delineated by a single, unbiased point of view. Often the overriding characteristic of a complex problem is the unclear, contradictory, and confusing tangle of relationships and concerns among the various stakeholders. The broader the problem, the greater the challenge. Indeed, this may explain why national-level initiatives rarely experience honest red teaming. Red teams must avoid serving as a shill for a single stakeholder when red teaming complex problems of this sort.
In short, the decision when and how to red team can be a surprisingly complex one. Dropping a red team into a highly charged political situation can undermine trust and erode hard-won consensus. Similarly, red teaming a decision during implementation can raise more questions than it answers, sabotage morale, and cause a decision maker to second-guess sound choices unnecessarily. On the other hand, aiming a seasoned red team at a problem or system at the right time with the proper mandate can steer a decision maker away from an otherwise pending catastrophe.
From Red teaming laws:
Red teaming is governed by informal and wholly unscientific laws based largely on human nature.   Sample laws selected by JC:
Red Teaming Law #1: The more powerful the stakeholders, the more at stake, the less interest in red teaming. This law trumps all other laws. 
Red Teaming Law #2: Skeptics make the best red teamers, especially when they’re skeptical of red teaming. 
Red Teaming Law #6: You don’t want a red team you can leash.
Red Teaming Law #7: If you’re apprehensive about red teaming, it probably means you need it. 
Red Teaming Law #9: Red teaming is not forecasting; red teaming is the art of challenging assumptions and exploring the possible.
Red Teaming Law #10: The inferior red teamer defers to reputation and status. The superior red teamer pokes arrogance in the eye. 
Red Teaming Law #12: What your customers won’t let you challenge sometimes points directly at their most critical vulnerability.
Red Teaming Law #17: The superior red teamer learns how things work in the real world, not just how they work on a diagram or presentation slide. 
Red Teaming Law #20: If you defeat the red team, you still have to defeat the enemy. 
Red Teaming Law #22: Unexpected surprise is what happens while you’re waiting for the expected surprise. Think tanks and pundits specialize in expected surprise.
Red Teaming Law #24: Tell the red team what you want, and it’ll confirm what you know. Tell the red team what the adversary wants, and it’ll uncover what you don’t want to know (but should). Good red teaming tastes more like medicine than candy. 
Red Teaming Law #25: The goal of a red team usually isn’t to find a needle in the haystack, it’s to help you see the haystack. 
Red Teaming Law #26: Never regard your adversary with contempt. The superiority you feel is not worth the surprise that invariably follows. 
Red Teaming Law #29: Your adversary is never completely wrong, and you are never more than partly right
Red Teaming Law #31: A red team without a blue team is like a blue team without a red. Blue and red represent a reciprocal system of perceptions; one should not be considered without the other. 
Red Teaming Law #34: In many ways, the art of red teaming is actually the art of asking the right questions, from the right perspective, at the right time. 
Red Teaming Law #35: Behind every successful red team stands a leader who will not bend to whim, coercion, or fear. We need this sort of leader as much as we need superior red teamers. 
Red Teaming Law #38: The status quo sticks like glue to assumptions, plans, and strategies. A good red team is a powerful solvent. 
Red Teaming Law #47: The apprentice red teamer asserts knowledge. The journeyman red teamer applies knowledge. The master red teamer asks questions. 
From Red Teaming: Feel the Power:
Perhaps because of this we tend to view red teams as neutral, unbiased, and inherently independent. Seldom is this so. First, no red team is free of its own biases. Second, no red team is ever hired or implemented by a customer who is free of his or her own biases. These biases intermingle to taint the effort. Sometimes the effect is minimal, and sometimes it’s significant; rarely is it acknowledged. Third, red teaming is usually avoided entirely when the “powers that be” don’t want an idea, plan, or system challenged, particularly when it took a tremendous amount of effort to build the coalition that supports the idea. This is the basis of RTJ Red Teaming Law #1 (“Jaunty Man”): “The more powerful the stakeholders, the more at stake, the less interest in red teaming.” 

Objectives?
Steve Koonin’s op-ed proposed that the red team critique the IPCC WG1 report, with a back-and-forth rebuttal with the blue team.  Joseph Majkut criticized this approach:
If the review is simply a back and forth, as has been suggested by Steve Koonin, there is little chance that the process will have the buy-in of the science community and the administration, and it would risk looking like a kangaroo court to the public (or the 21st century version of the Scope’s Monkey Trial).  
While I don’t really agree with Majkut here, I do think the red team needs to go ‘outside the box’ of merely critiquing  the IPCC Report.  The naive view of a consensus based on incontrovertible evidence is . . . naive. As I wrote in a previous post:
The disagreement is not so much about observational evidence, but rather about the epistemic status of climate models, the logics used to link the observational evidence into arguments, the overall framing of the problem and overconfident conclusions in the face of incomplete evidence and understanding.  The ‘multiple lines of evidence’ argument simply doesn’t work for a very complex problem, and there are multiple lines of evidence that lead to alternative conclusions.  
Why do scientists disagree about climate change?

• Insufficient and inadequate  observational evidence
• Disagreement about the value of different classes of evidence (e.g. global climate models)
• Disagreement about the appropriate logical framework for linking and assessing the evidence
• Assessments of areas of ambiguity & ignorance
• Belief polarization as a result of politicization of the science

IMO, the greatest rationale for a climate ‘red team’ is to overcome the framing bias on the whole issue of climate change (broadly defined) that was triggered by the UNFCCC treaty to avoid dangerous human caused climate change, which defined the mandate for the IPCC (and hence the rationale and funding for government spending on climate science).  I will have more to say on the ‘framing’ issue in a subsequent post.
What is needed is clarification of objectives and concerns from the Trump administration, the administrative framework for the exercise including selection of team members, and then the red team itself needs to decide how to frame the problem and their approach.
Steve Koonin proposed to look at the fundamental science aspects of climate change (as per IPCC WG1).  I think there is much ‘meat’ in WG1 for debate and a red team approach.  A more fruitful topic for a red team approach might be climate change impacts (the topic of WG2).  The policy responses (climate change, extreme weather, energy, water) might be very good targets for a red team approach, and could provide additional policy options for consideration.
Who’s red and who’s blue?
If this red team exercise had been conducted under the Obama administration, it would be very clear who is ‘red’ and who is ‘blue’.  Obama and his administration slagged off on anyone with a different perspective as a ‘denier,’  hence a red team would have been antithetical to Obama’s strategy on this issue.
Now  the Trump administrate is challenging the established perspective of Obama and the UNFCCC/IPCC. Red is the new blue. The good news is that red teaming on this issue is needed (something that would have been impossible under the Obama administration.)   However, the risk is that the over enthusiasm of the Trump administration for overthrowing pretty much everything from the Obama administration and the UNFCCC/IPCC will bias the proceedings and diminish the legitimacy of the outcome in influencing the national dialogue on this topic.  This means that the blue team needs to serve as a ‘red team’ on the red team.
There are some special challenges for the blue team — arrogance and appeals to authority won’t work here.  See above for characteristics of inferior red teams.
So what’s next?
A number of reporters have asked this, but I have heard nothing from anyone in the Trump administration on this issue.  So I have no inside information on any of this.
It seems that there is a lot of interest in this topic from journalists and in the twitosphere (not to mention apparently growing interest in the Trump administration).
I will be writing a series of blog posts on this issue, that will hopefully stimulate some thinking on how to approach this in a constructive way.  Next up will be a post on framing the red team effort.