From the Neuroskeptic blog

 

Neuroskeptic is an important blog. A recent post addresses the publication process in science. You can read it also to understand how bad research gets published. We reviewers (and writers) don’t really know what to do with bad studies. Almost every bad study can get published somewhere.

Fixing Science’s Chinese Wall

By Neuroskeptic | December 21, 2013 5:59 am

In science we very often want certain things to be true.

Sometimes, this desire comes from noble reasons: we want a vaccine to work, because that would save many lives. Other times our motivations are less altruistic: we want our vaccine to work because then we will get lots of citations, a promotion, and a raise.

 

fixing_science

Scientists wanting things to be true is a fact, and a problem. Which is why science has evolved methods to prevent our desires from distorting the findings. For example, we use double blind trials to ensure that the researcher’s preferences and expectations can’t influence the outcome measurements. We use statistical inference with an agreed standard (p = 0.05) to determine the significance of findings, rather than leaving it up to individuals to pick what they want to consider signal and what noise.

These systems are designed to protect observations from the whims of the observers. As Feynman famously said“Science is a way of trying not to fool yourself… and you are the easiest person to fool.” It is rather like how in business, banks and other firms are required to have a ‘Chinese Wall‘ – a cordon around certain information to ensure that it can’t reach certain people. This is intended to prevent conflicts of interest influencing those branches of the company whose job is to provide impartial advice to the public or other departments.

However, there is a gap in science’s Wall between data and desire – the publication process. Whether the results of a particular experiment get published in a peer-reviewed journal depends upon at least three people:

  • the researcher(s) themselves, who decides which results to submit, in what form
  • the journal editor(s), who decides whether submissions are ‘interesting’
  • the peer reviewer(s), who advises the editor on whether submissions are ‘good’

All of these people are scientists, with their own agenda and desires. They get to decide the fate of scientific results – knowing what those results are. So the desires of researchers get to influence what’s published, and the result is publication bias. This influence would be considered unscientific if it were allowed to happen within the conduct of a given experiment – but once the results are in, we allow it.

There is an alternative. Journals ought to peer review and accept articlesbefore the work has been begun (preregistration with pre-peer review.) This ensures that science is published or not on the basis of the strength and originality of the methods, independent of what the results happen to be.

This would do for the later stages of the scientific process what double-blinding and randomization have done for the earlier, experimental stages. The Chinese Wall would be complete.

One Comment

  1. It appears to me that journal editors also have a responsibility to ensure that the Editorials they solicit are written by authors who are honest and as free from bias as is possible. An uncritically written editorial published in a prestigious scientific journal can be quite damaging to the reputation of the researcher(s) and to the cause of scientific progress. Another chink in The Chinese Wall?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>