What is medicine's 5 sigma? by Richard Horton, Published in "The Lancet" 11 April 2015

The following excerpt is taken from a magazine we've covered before, written by the editor himself. I'd link to the page, but I don't expect it to stay up for long.

 “A lot of what is published is incorrect.” I’m not allowed to say who made this remark because we were asked to observe Chatham House rules. We were also asked not to take photographs of slides. Those who worked for government agencies pleaded that their comments especially remain unquoted, since the forthcoming UK election meant they were living in “purdah”—a chilling state where severe restrictions on freedom of speech are placed on anyone on the government’s payroll. Why the paranoid concern for secrecy and non-attribution? Because this symposium—on the reproducibility and reliability of biomedical research, held at the Wellcome Trust in London last week—touched on one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with one of our greatest human creations. * The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”. The Academy of Medical Sciences, Medical Research Council, and Biotechnology and Biological Sciences Research Council have now put their reputational weight behind an investigation into these questionable research practices. The apparent endemicity of bad research behaviour is alarming. In their quest for telling a compelling story, scientists too often sculpt data to fi t their preferred theory of the world. Or they retrofi t hypotheses to fi t their data. Journal editors deserve their fair share of criticism too. We aid and abet the worst behaviours. Our acquiescence to the impact factor fuels an unhealthy competition to win a place in a select few journals. Our love of “significance” pollutes the literature with many a statistical fairy-tale. We reject important confirmations. Journals are not the only miscreants. Universities are in a perpetual struggle for money and talent, endpoints that foster reductive metrics, such as high-impact publication. National assessment procedures, such as the Research Excellence Framework, incentivise bad practices. And individual scientists, including their most senior leaders, do little to alter a research culture that occasionally veers close to misconduct. * Can bad scientific practices be fixed? Part of the problem is that no-one is incentivised to be right. Instead, scientists are incentivised to be productive and innovative. Would a Hippocratic Oath for science help? Certainly don’t add more layers of research red tape. Instead of changing incentives, perhaps one could remove incentives altogether. Or insist on replicability statements in grant applications and research papers. Or emphasise collaboration, not competition. Or insist on preregistration of protocols. Or reward better pre and post publication peer review. Or improve research training and mentorship. Or implement the recommendations from our Series on increasing research value, published last year. One of the most convincing proposals came from outside the biomedical community. Tony Weidberg is a Professor of Particle Physics at Oxford. Following several high-profi le errors, the particle physics community now invests great eff ort into intensive checking and rechecking of data prior to publication. By fi ltering results through independent working groups, physicists are encouraged to criticise. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. Weidberg worried we set the bar for results in biomedicine far too low. In particle physics, significance is set at 5 sigma—a p value of 3 × 10–7 or 1 in 3·5 million (if the result is not true, this is the probability that the data would have been as extreme as they are). The conclusion of the symposium was that something must be done. Indeed, all seemed to agree that it was within our power to do that something. But as to precisely what to do or how to do it, there were no firm answers. Those who have the power to act seem to think somebody else should act first. And every positive action (eg, funding well-powered replications) has a counterargument (science will become less creative). The good news is that science is beginning to take some of its worst failings very seriously. The bad news is that nobody is ready to take the first step to clean up the system. Richard Horton richard.h orton@lancet.com

The wolf-child phenomenon: a modern view

Anyone familiar with Mowgli of Kipling's Jungle Book has heard of such a thing as a wolf-child. The idea of a child being raised by wolves runs deep in human experience, dating back to the founding of Rome. But on more of an historical note, I recently came across this tidbit in a 1930 book by E. Stanley Jones, an incarnational missionary to British India:

A wolf-child, captured near where I live in India, had lived with wolves from the age of two to the age of eleven. It ran on all fours. Its knee joints were stiff and enlarged from running in this fashion. It would eat only raw meat, and when it was put on a more civilized diet, it took dysentery and died.

Now, Jones never said that he himself witnessed any of this, so we are forced to first examine this story for apparent veracity. Let's begin by listing the alleged facts:

1) Nine years previous to a given date, a two-year-old child had disappeared from society.
2) Shortly before the given date, a being was captured in the wild who met the description of a wolf-child.
3) This creature was identified as being the lost child, now age eleven.
4) The child was then brought back into society, whereupon it died of dysentery.

Now, notice the inferences made from these facts:
1) The child had lived among wolves for nine years.
2) The child had eaten nothing but raw meat for nine years.
3) The child could only walk on all fours, due to its being raised by wolves.
4) The child could only tolerate a diet of raw meat, due to being raised by wolves.

Well, these may or may not be true. All humans start out without the ability to walk upright, and some never attain it; others lose it quite early. Without studying a wolf-child in its natural habitat--something that has never been done by a scientist--it's impossible to say for sure which of these were true. In researching the subject of wolf-children, I found that one universal characteristic is that they lack human language--so Mowgli heads back to the fairy tales where he belongs. It's also rather common in a wolf-child narrative that the child spends somewhere between eight and fifteen years in the wild before making contact with society.

One glaring problem with the veracity of the wolf-child phenomenon is that it has never been studied scientifically. The closest anyone has come is in the case of a Los Angeles girl known as "Genie," who was kept locked in a bedroom by her father until she was 13 years of age. Although she learned some speech after being brought out into society, she has never progressed beyond a most rudimentary level of communication. And she was so traumatized by scientists squabbling over the exclusive rights to study her that she regressed as an adult to the autistic state in which they had found her.

Many 'wolf child' stories can be attributed to an autistic child who is abandoned in the wild but manages to survive until later discovery. His autism makes any transition back to civilised life all but impossible, and he often dies from the sudden change in diet. One should therefore not approach a 'wolf-child' account as showing what happens to someone who is raised in the wild; rather, it shows how animalistic autistic children are by nature, making it all the easier for them to live among animals than among humans.

The idea of a child being raised by wolves runs deep in the human psyche. But when examined in the light of science, it turns out to mostly just be a good story.