The pearly gates of science

Peer review in context

This past summer, I worked on an ethics in research project with government researchers.¹ I’m very proud of our work and I look forward to sharing it with everyone here when it is cleared by agency internal review- a process that began about 4 months ago.

My experience inspired curiosity about the institutionalization of “peer review”, which is what this post is about. It’s not the most riveting end of the year essay, but it brought me peace. ‘Tis the season.

There is much disgruntlement about the peer review process across the research enterprise- a disgruntlement that I share. However, the underlying problem with peer review is the extent to which society has adopted a view that peer review is the Stamp of Truth denoting a rung on the ladder of knowledge progress.

Instead, peer review is better understood as one means to engage discussion and debate about research as part of the general social process of learning and knowledge creation. It’s a conversation that often has the feel of a negotiation.

In the 1970s, the then-CEO of the AAAS, William Carey, argued that the problem with peer review is that people “ask too much of it.” Speaking to Congress about the peer review process for NSF funding (a bit more on that below) Carey explains:

I also think that we should disabuse ourselves of the mystique which has grown up around peer review. It is not a fail-safe procedure, nor one that should silence disagreements on the theory that there is something sacramental about it.

There is disagreement as to the origins of peer review which matters for understanding its limitations and influences.

Sociologists of science, Harriet Zuckerman and Robert Merton, trace the “Ingredients of the referee system” to the early years of the Philosophical Transactions of the Royal Society in the 1600s. At the time, the Transaction editor wavered between the ad hoc use of refereeing as a journal selling point and a laissez- faire attitude of "sit penes authorem fides (let the author take responsibility for it): We only set it downe, as it was related to us, without putting any great weight upon it."

The idea that peer review is somehow inherent to Enlightenment and science itself is the view held by Elsevier, for example, the world’s largest scholarly publisher.

However, historians argue that this origin story is mistaken. Peer review is a product of the professionalization of scientists and is not inherent to science itself.

In its early years, the Transactions was not concerned with ‘truth’ and made efforts to distance itself from anything that suggested as much. Noah Moxham and Aileen Fyfe explain,

The Transactions was not supposed to be a repository of officially sanctioned knowledge, but of interesting or intriguing phenomena that were worthy of further consideration

Rather than growing up alongside science, refereeing (the antecedent term to peer review) grew up alongside expanding editorial responsibilities of the scholarly publication business and lent to an air of prestige:

refereeing process could be seen (internally) as protecting the Society’s reputation and finances and (externally) as a mechanism for generating expert evaluation of research

Use of refereeing for prestige harnessed the process as a means of gatekeeping and awarded an elite few with significant influence over what matters for publication and hence, research disciplines.

At least until the 1960s, Transactions and its Royal Society,

were only intermittently concerned with anything that might be termed the ‘reliability of scientific research’ the practices used prior to the mid-19th century are of no relation to this later period.

Yet, by the mid 1970’s/1980’s peer was fully institutionalized as a mark of scientific virtue.

Historian of science, Melinda Baldwin, asks how such an about-face was made possible. She argues that the full institutionalization of peer review originated as a mechanism of maintaining scientific autonomy from policymaker oversight.

Of note, the coalescing around the term ‘peer review’ rather than the previous refereeing or reviewing, served to signal who the acceptable people to review research- that is, only ones ‘peers.’

Baldwin’s analysis centers on the transcripts of 6 days of National Science Foundation (NSF) Congressional oversight hearings in 1975. The hearings open: “The subject of these hearings is, ‘peer review.’”

The hearing featured testimony from Republican Congressmen Robert Bauman and John Conlan, and Democratic Senator William Proxmire who were all critical of the way NSF made funding awards. All three wanted NSF peer review reports to be transparent and accessible. Conlan and Proxmire regarded the practice as an “incestuous” system.

Baldwin concludes:

The 1975 peer review controversies are most noteworthy not because of their effect on the actual practice of refereeing at funding bodies but because of their implications for public conceptions of peer review. The hearings provided a moment for stakeholders with different views about science and its funding to debate what the practice of peer review was for. What emerged at the hearings was a general (though not unanimous) consensus that the NSF—and any other organization—had to rely on external referees in order to judge “good science” properly. At the hearings, peer review was cast as a process that was crucial to the way science worked, one that had to be preserved and defended in order for science to work properly in the future.

Internal agency review and peer review are ostensibly aimed at similar objectives. The Office of Management and Budget describes peer review as follows:

Peer review is one of the important procedures used to ensure that the quality of published information meets the standards of the scientific and technical community.

Agency review however, has at least one added layer. It is on the look out for “influential scientific information” defined as

scientific information the agency reasonably can determine will have or does have a clear and substantial impact on important public policies or private sector decisions.

Agencies have leeway in identifying that which is influential in accordance with OMB’s “information quality” guidelines. Information quality encompasses utility, objectivity, and integrity- each with regulatory definitions (in this case, the term integrity is not referring to scientific integrity).

Influential scientific information is subject to increased scrutiny. Writes OMB, “The more important the information, the higher the quality standards to which it should be held,….”

Here are a few of my take aways:

Peer review is as much about gatekeeping of ideas as it is about quality of reporting, and this seems particularly so where the government is involved. The more that research outcomes challenge reigning ideas and policies, the higher the hurdles of quality.

This also suggests that work that upholds reigning ideas and policies, receives the least scrutiny because it is politically expedient to not ask too many questions about it.

The ideal of objectivity is always important in research. But, it is possible for the objectivity dimension of information quality to be interpreted as requiring an impossible standard of non-bias. Sarewitz is worth quoting on this point,

Even the most apparently apolitical, disinterested scientist may, by virtue of disciplinary orientation, view the world in a way that is more amenable to some value systems than others. That is, disciplinary perspective itself can be viewed as a sort of conflict of interest that can never be evaded.

I have a draft post on this issue for the future.

Peer review has value to the extent that review and critique of writing and thinking tends to make that writing and thinking stronger. However, while it is frequently the first debate about research (for better or worse), it is not necessarily the best and it should not be the last. The more broad the engagement and debate the better.

Finally, we need to talk to students about the scholarly literature differently than we do. Too frequently, students and thus, society, come to understand the value of research in terms of a perceived unfailing peer review process. Students are empowered as intellectuals and citizens when they are encouraged to remember that research and publication is produced by mere mortals that have bills to pay and work within institutions that place demands on them.

1. I’m being intentionally vague. Sorry.