Beware the peer reviewed study

By Susan Yoshihara, Ph.D. | September 17, 2015

A few years ago I testified before a State judiciary committee at which the chair raked my colleagues and I over the coals for not having enough “peer reviewed” studies to back our statements about the importance of the natural family. I returned two years later to the same committee and submitted in testimony a new, massive peer reviewed study to back our analysis. The chair (and later the media) panned it as biased.

As it turns out, the fickle chair’s jaded attitude, while unprofessional and smugly biased, was warranted. Peer reviewed studies, in psychology at least, may be largely unrepeatable–essentially unproven–even though they get published and cited to back a host of social policies.  Nature recently reviewed a study by the Replication  Project that found only 39 out of 100 were repeatable.

One reason is editorial bias:

The point [of the project] is not to critique individual papers but to gauge just how much bias drives publication in psychology. For instance, boring but accurate studies may never get published, or researchers may achieve intriguing results less by documenting true effects than by hitting the statistical jackpot; finding a significant result by sheer luck or trying various analytical methods until something pans out.

The losers are not just policy makers and the public, but scholars too. The peer review process is not about gatekeeping. Done right, it makes excellent scholars even better. It raises the technical and ethical standards of an entire field, and of scholarship — and therefore of society — more broadly.

Share on Facebook
Share on Twitter