Another bogus claim from Lewandowsky would hardly seem to warrant a blog post, let alone a bogus claim about people holding contradictory beliefs. The ability of many climate scientists to hold contradictory beliefs at the same time has long been a topic of interest at climate blogs (Briffa’s self contradiction being a particular source of wonder at this blog). Thus no reader of this blog would preclude the possibility that undergraduate psychology students might also express contradictory beliefs in a survey.
Nonetheless, I’ve been mildly interested in Lewandowsky’s claims about people subscribing to contradictory beliefs at the same time, as for example, the following:
While consistency is a hallmark of science, conspiracy theorists often subscribe to contradictory beliefs at the same time – for example, that MI6 killed Princess Diana, and that she also faked her own death.
Lewandowsky’s assertions about Diana are based by an article by Wood et al. entitled “Dead and Alive: Beliefs in Contradictory Conspiracy Theories”. A few months ago, I requested the supporting data from Wood. Wood initially promised to provide the data, then said that he had to check with coauthors. I sent several reminders without success and eventually without eliciting any response. I accordingly sent an FOI request to his university, accompanied by a complaint under any applicable university data policies. The university responded cordially and Wood immediately provided the data.
The most cursory examination of the data contradicted Lewandowsky’s claim. One can only presume that Lewandowsky did not carry out any due diligence of his own before making the above assertion.
A Subpopulation of Zero
Within the Wood dataset, only two (!) respondents purported to believe that Diana faked her own death. Neither of these two respondents also purported to believe that MI6 killed Princess Diana. The subpopulation of people that believed that Diana staged her own death and that MI6 killed her was precisely zero.
Lewandowsky’s signature inconsistency was completely bogus – a result that will come as no surprise to readers acquainted with his work.
Wood et al 2012
Lewandowsky appears to have uncritically relied on results published in Wood et al, 2012, which had stated in their running text:
People who believed that Diana faked her own death were marginally more likely to also believe that she was killed by a rogue cell of British Intelligence (r = .15, p = .075) and significantly more likely to also believe that she was killed by business enemies of the Fayeds (r = .25, p = .003).
Wood highlighted the inconsistency between conspiracy beliefs in their abstract which stated:
In Study 1(n= 137), the more participants believed that Princess Diana faked her own death, the more they believed that she was murdered.
However, as with the official MI6 example, neither of the two respondents purporting to believe that Diana had faked her own death, also believed that she had been murdered by Fayed’s business enemies or that she had been murdered by a rogue cell within MI-6.
Here’s how the correlations arose. Wood had used a 7-point Likert scale (ranging from Strong Disagree to Strong Agree) and people who expressed strong disagreement on one point were more likely to express strong disagreement on a related point. What Wood ought to have said is that participants who strongly disagreed that Diana faked her own death were more likely to strongly disagree that she was murdered. This does not imply people who believed that Diana faked her own death were “significantly more likely” to believe that she was also murdered by Fayed’s enemies.
This is very elementary.
Prior to posting, I wrote to Wood, explaining the problem as follows:
For example, in your Table I, you reported a “significant” correlation of .253 between the proposition that Diana faked her own death and the proposition that she was killed by Fayed’s enemies. I don’t know whether you examined the contingency table between these two propositions, but it is an important precaution and, in my opinion, conclusively contradicts the phenomenon that you had in mind. here is the contingency table (4 – neutral on the 7-point Likert scale):
Only a few respondents (6) purported to either believe that Diana had faked her own death (2) or that she had been killed by Fayed enemies (4) – and NONE believed both simultaneously. Your “significant correlation” arises not because people held inconsistent conspiracy beliefs, as you state, but because of differing confidence in their disbelief. Respondents were much more confident in their belief that Diana did not fake her own death than that she was not killed by Fayed’s enemies. In addition, respondents may express their disagreement with different degrees of emphasis. This is an entirely different phenomenon than believing in mutually inconsistent results.
Rather than conceding this seemingly obvious point, in a reasonably detailed response, Wood denied that his results were affected. First, he contested that non-normality was an issue, because Spearman correlations yielded similar or higher values than Pearson correlation. While true, this observation is obviously unresponsive to the fact that no respondents simultaneously believed that Diana had faked her own death and had been murdered by Fayed enemies (or a rogue cell or official MI6). Wood:
First, you raise the issue that there is some non-normality in the data. This is especially apparent in correlations involving the “faked death” item, which participants were highly sceptical about and therefore ended up with a restricted range of responses, as we noted in the text of the paper in our discussion of a potential floor effect. However, nonparametric tests give much the same results as we originally obtained – for instance, using either Spearman’s rho or Kendall’s tau-b renders the previously marginally significant correlation between “fake death” and “rogue cell” significant at the .01 level. Your use of scare quotes around “significant” is quite unwarranted – the relationships are indeed statistically significant, and if anything the use of Pearson correlations in the original paper understates, rather than overstates, their robustness.
Next, Wood challenged my analysis of the Faked Death/Fayed Enemies pairing as “felicitous”, arguing that a “more instructive” example was the correlation between the rogue cell and Fayed enemies combination.
The second point concerns the interpretation of the correlations. In fact you chose a rather felicitous example for the point you’re making… A more instructive example is the “rogue cell” / “business enemies” correlation, which, not being attenuated by general scepticism about the faked death claim, is more representative of the relationships among the different Diana theories, as is clear from Table 1 of the paper.
However, it was the Faked Death example that Wood et al highlighted in their Abstract and which Lewandowsky has drawn attention to. I didn’t pick it because of perceived weakness, but because Wood and Lewandowsky had themselves promoted it.
Wood also now claimed that because “so few” agreed with the faked death theory, it “does not seem reasonable to expect many people to give high endorsement to both theories”:
as noted previously, very few people expressed a high level of confidence at all in the “faked death” theory – in fact, it’s the least-agreed-with item in the scale. Given that so few agreed with the “faked death” example at all, it does not seem reasonable to expect many people to give high endorsement to both theories.
However, nowhere in Wood et al 2012 is there any explicit statement that only two respondents purported to believe in the Faked Death theory that was highlighted in the abstract. Had readers been aware that only two people purported to subscribe to this theory, then they would obviously not expect “many people to give high endorsement to both theories”. Unfortunately when zero people subscribed to both theories, one cannot justifiably assert that “In Study 1(n= 137), the more participants believed that Princess Diana faked her own death, the more they believed that she was murdered”,
Nor does Wood’s “more instructive” example help their cause. Only two respondents purported to agree that both Diana had been murdered by Fayed business enemies and by a rogue cell. But here’s how Wood et al 2012 had characterized the results:
Similarly, participants who found it likely that the Fayeds’ business rivals were responsible for the death of Diana were highly likely to also blame a rogue cell (r = .61, p < .001).
Given that only two respondents reported subscribing to both beliefs, it is obviously impossible to achieve p < .001 from such a miniscule sample. As with the other pairing, the correlation arises from people who disagree with both propositions, not from people who agree with both propositions. Although this latter point seems self-evident, Wood disputed it as followseven after I had pointed it out to him:
While this sample was generally sceptical about conspiracy theories in general, the fact that participants’ degrees of disbelief appear to stick together does not indicate that the correlations are simply an artefact of participants’ response styles. This explanation seems particularly implausible given the magnitude of the correlations that are not attenuated by floor effects (e.g., r = .61 for the “business enemies / rogue cell” correlation).
However, while two is more than zero, it is still far below the population necessary to arrive at statistically significant conclusions. Any “statistically significant” correlations arise for the reason set out in my email to Wood: “not because people held inconsistent conspiracy beliefs, as you state, but because of differing confidence in their disbelief”.
That Lewandowsky should make untrue statements will hardly occasion surprise among CA readers. However, drawing conclusions from a subpopulation of zero does take small population statistics to a new and shall-we-say unprecedented level.