by Judith Curry
“Is the point of research to make other professional academics happy, or is it to learn more about the world?” —Noah Grand, sociology professor, UCLA
“Science, I had come to learn, is as political, competitive, and fierce a career as you can find, full of the temptation to find easy paths.” — Paul Kalanithi, neurosurgeon and writer (1977–2015)
Vox has conducted a very interesting study and has written a long, insightful article: The 7 biggest problems facing science, according to 270 researchers. Excerpts:
In the past several years, many scientists have become afflicted with a serious case of doubt — doubt in the very institution of science.
As reporters covering medicine, psychology, climate change, and other areas of research, we wanted to understand this epidemic of doubt. So we sent scientists a survey asking this simple question: If you could change one thing about how science works today, what would it be and why?
We heard back from 270 scientists all over the world, including graduate students, senior professors, laboratory heads, and Fields Medalists. They told us that, in a variety of ways, their careers are being hijacked by perverse incentives. The result is bad science.
The scientific process, in its ideal form, is elegant: Ask a question, set up an objective test, and get an answer. Repeat.
But nowadays, our respondents told us, the process is riddled with conflict. Scientists say they’re forced to prioritize self-preservation over pursuing the best questions and uncovering meaningful truths.
Today, scientists’ success often isn’t measured by the quality of their questions or the rigor of their methods. It’s instead measured by how much grant money they win, the number of studies they publish, and how they spin their findings to appeal to the public.
“As long as things like publication quantity, and publishing flashy results in fancy journals are incentivized, and people who can do that are rewarded … they’ll be successful, and pass on their successful methods to others.”
Many scientists have had enough. They want to break this cycle of perverse incentives and rewards. They are going through a period of introspection, hopeful that the end result will yield stronger scientific institutions. In our survey and interviews, they offered a wide variety of ideas for improving the scientific process and bringing it closer to its ideal form.
Academia has a huge money problem
Their gripe isn’t just with the quantity, which, in many fields, is shrinking. It’s the way money is handed out that puts pressure on labs to publish a lot of papers, breeds conflicts of interest, and encourages scientists to overhype their work.
Grants also usually expire after three or so years, which pushes scientists away from long-term projects. Yet as John Pooley, a neurobiology postdoc at the University of Bristol, points out, the biggest discoveries usually take decades to uncover and are unlikely to occur under short-term funding schemes.
Some of our respondents said that this vicious competition for funds can influence their work. Funding “affects what we study, what we publish, the risks we (frequently don’t) take,” explains Gary Bennett a neuroscientist at Duke University. It “nudges us to emphasize safe, predictable (read: fundable) science.”
Finally, all of this grant writing is a huge time suck, taking resources away from the actual scientific work.
Too many studies are poorly designed. Blame bad incentives.
Scientists are ultimately judged by the research they publish. And the pressure to publish pushes scientists to come up with splashy results, of the sort that get them into prestigious journals.
Some of this bias can creep into decisions that are made early on. Many of our survey respondents noted that perverse incentives can also push scientists to cut corners in how they analyze their data.
“I have incredible amounts of stress that maybe once I finish analyzing the data, it will not look significant enough for me to defend,” writes Jess Kautz, a PhD student at the University of Arizona. “And if I get back mediocre results, there’s going to be incredible pressure to present it as a good result so they can get me out the door. At this moment, with all this in my mind, it is making me wonder whether I could give an intellectually honest assessment of my own work.”
Increasingly, meta-researchers (who conduct research on research) are realizing that scientists often do find little ways to hype up their own results — and they’re not always doing it consciously.
“The current system has done too much to reward results,” says Joseph Hilgard, a postdoctoral research fellow at the Annenberg Public Policy Center. “This causes a conflict of interest: The scientist is in charge of evaluating the hypothesis, but the scientist also desperately wants the hypothesis to be true.”
“I would make rewards based on the rigor of the research methods, rather than the outcome of the research,” writes Simine Vazire, a journal editor and a social psychology professor at UC Davis. “Grants, publications, jobs, awards, and even media coverage should be based more on how good the study design and methods were, rather than whether the result was significant or surprising.”
“We’ve gotten used to working away in private and then producing a sort of polished document in the form of a journal article,” Gowers said. “This tends to hide a lot of the thought process that went into making the discoveries. I’d like attitudes to change so people focus less on the race to be first to prove a particular theorem, or in science to make a particular discovery, and more on other ways of contributing to the furthering of the subject.”
“I think the one thing that would have the biggest impact is removing publication bias: judging papers by the quality of questions, quality of method, and soundness of analyses, but not on the results themselves,” writes Michael Inzlicht, a University of Toronto psychology and neuroscience professor.
JC note: New Scientist just published a relevant article Evolutionary forces are causing a boom in bad science.
Peer review is broken
The process frequently fails to detect fraud or other problems with manuscripts, which isn’t all that surprising when you consider researchers aren’t paid or otherwise rewarded for the time they spend reviewing manuscripts. That’s not to mention the problem of peer review bullying.
“We need to recognize academic journals for what they are: shop windows for incomplete descriptions of research, that make semi-arbitrary editorial [judgments] about what to publish and often have harmful policies that restrict access to important post-publication critical appraisal of published research.” —Ben Goldacre, epidemiology researcher, physician, and author
“The current peer review process embraces a concept that a paper is final,” says Nosek. “The review process is [a form of] certification, and that a paper is done.” But science doesn’t work that way. Science is an evolving process, and truth is provisional. So, Nosek said, science must “move away from the embrace of definitiveness of publication.”
One possible model already exists in mathematics and physics, where there is a long tradition of “pre-printing” articles. Studies are posted on an open website called arXiv.org, often before being peer-reviewed and published in journals. There, the articles are sorted and commented on by a community of moderators, providing another chance to filter problems before they make it to peer review.
And even after an article is published, researchers think the peer review process shouldn’t stop. They want to see more “post-publication” peer review on the web, so that academics can critique and comment on articles after they’ve been published. Sites like PubPeer and F1000Research have already popped up to facilitate that kind of post-publication feedback.
The bottom line is that traditional peer review has never worked as well as we imagine it to — and it’s ripe for serious disruption.
Too much science is locked behind paywalls
Many of our respondents urged their peers to publish in open access journals (along the lines of PeerJ or PLOS Biology). But there’s an inherent tension here. Career advancement can often depend on publishing in the most prestigious journals, like Science or Nature, which still have paywalls.
There’s also the question of how best to finance a wholesale transition to open access. After all, journals can never be entirely free. Someone has to pay for the editorial staff, maintaining the website, and so on. Right now, open access journals typically charge fees to those submitting papers, putting the burden on scientists who are already struggling for funding.
As a model, Cambridge’s Tim Gowers has launched an online mathematics journal called Discrete Analysis. The nonprofit venture is owned and published by a team of scholars, it has no publisher middlemen, and access will be completely free for all.
Bohannon reported that millions of researchers around the world now use Sci-Hub, a site set up by Alexandra Elbakyan, a Russia-based neuroscientist, that illegally hosts more than 50 million academic papers. “As a devout pirate,” Elbakyan told us, “I think that copyright should be abolished.”
One respondent had an even more radical suggestion: that we abolish the existing peer-reviewed journal system altogether and simply publish everything online as soon as it’s done.
Rachel Harding, a genetic researcher at the University of Toronto, has set up a website called Lab Scribbles, where she publishes her lab notes on the structure of huntingtin proteins in real time, posting data as well as summaries of her breakthroughs and failures. The idea is to help share information with other researchers working on similar issues, so that labs can avoid needless overlap and learn from each other’s mistakes.
Not everyone might agree with approaches this radical; critics worry that too much sharing might encourage scientific free riding. Still, the common theme in our survey was transparency. Science is currently too opaque, research too difficult to share. That needs to change.
Life as a young academic is incredibly stressful
The core point underlying all these suggestions, however, was that universities and research labs need to do a better job of supporting the next generation of researchers. Indeed, that’s arguably just as important as addressing problems with the scientific process itself. Young scientists, after all, are by definition the future of science.
“Many creative, hard-working, and/or underrepresented scientists are edged out of science because of these issues. Not every student or university will have all of these unfortunate experiences, but they’re pretty common. There are a lot of young, disillusioned scientists out there now who are expecting to leave research.”
JC note: on this topic, NYTimes has an article So many research scientists, so few openings as professors.
JC reflections
This paper provides a remarkable synthesis of many problems with the institution of academic research, summarizing many topics that have been the subject of previous CE posts (sociology of science tag).
The institutions and processes of science are not writ in stone – they evolve with new insights and wisdoms, and also with the technologies of the times. We are currently stuck with a late 20th century model for science, one that does not account for the current emphasis of academia on ‘flash’ publications in Nature/Science/PNAS, the politicization and associated biases in many research fields (notably climate science), and of course the internet. The root of the problem in 21st century science is viewing successful science as a ‘result’, rather than as a process.
It’s long past time to break the perverse incentives that rule academic research.
These days, I have little to no influence in academia, but here is my advice anyways. Stop judging your faculty hires and promotions based on the ‘flash’ criteria (e.g, Nature/Science publications and sexy press releases). Instead, focus on the mathematical and scientific rigor of the faculty members research, along with their potential to identify new and interesting research problems, which may or may not have eventual practical applications or generate media interest. Include in your reward system recognition for faculty members that make all of their research materials available online (published papers, data, code and notes that provide further information on research methods, rationale for experimental design, and ideas/methods that were tried and rejected). Bonus points for writing an online essay (scientific american style) that places the paper/results in a broader context, explains it to the lay public, discusses uncertainties, and where this might lead in the future.
I am totally in favor of open access publishing. There are many different models being tried – preprint archives with comments, post publication peer review, etc. The main point is the the internet affords a much more rational way for communicating science research among peers, and the gatekeeping of the elite journals, along with press embargoes and firewalls, impedes and slows scientific progress. How to facilitate this transition with the professional societies that publish journals is a key challenge. I am hoping that the American Meteorological Society might be open to some of these ideas – they were an early adopter in online publishing.
I am particularly concerned about the plight of young scientists. The stresses are horrendous, the rewards are much more difficult to discern – apart from the extreme difficulty and unlikelihood of landing a tenure track faculty position, the intellectual rewards of scientific research have been diminished by the perverse incentives of succeeding in academia. Again, this is an area that I have little influence on these days.
I think that a key issue is to prepare graduate students for careers in the private sector, and to grow the private sector applications of their research area. Some universities are doing a really good job at developing/promoting entrepreneurship at the undergraduate level (including Georgia Tech). However little to nothing is done for graduate students – in fact, graduate students are typically discouraged/prevented from engaging with faculty member start-up companies. I am thinking that professional societies could help with this, as well as the private sector companies (I have some ideas along these lines that I will flesh out in a future blog post).
We need to rethink how we approach graduate education and research, which will require changes to how all this is funded. We need 21st century approaches to this. Its pretty clear what the problems are, I look forward to hearing about your ideas for solutions.
Filed under: Sociology of science