Bias

Undisclosed conflicts of interest are a serious problem in the climate change literature

A flurry of commentary on bias indicates the depths of the conflicts of interest problem in climate change research.

Yesterday, Patrick Brown at The

Breakthrough Institute

had a great write up walking us through a cascade of bias in extreme weather attribution studies, the body of literature supporting it, and the media reporting on it.

Patrick names the following biases: Occurrence Bias, Choice Bias, Publication Bias, and Media Coverage Bias.

Patrick’s work comes on the heals of an essay series “Weather Attribution Alchemy” by

Roger Pielke Jr.

at The Honest Broker. In his most recent post to the series, Roger names: Selection bias, Misrepresentation of the statistics, and the Fallacy of begging the question.

Bias is a distortion of science that results in unreliable knowledge. Distorted science is socially, economically, and politically costly to society.

Here, I will situate these findings of bias in the extreme weather event attribution literature under the broader heading of researcher conflicts of interest (COI), a major topic of concern for research ethics.

Bias as an outcome of researcher conflicts of interest is relevant here for two reasons.

First the methods, organizations, and researchers that advance extreme weather event attribution are inherently tied to litigation and advocacy.

Second, the situation is symptomatic of a far deeper and widespread problem of poor author conflicts of interest disclosure practices throughout the climate change science literature.

Let’s take this in pieces.

---

Dennis Thompson, a political scientist and ethics expert, provides an often used definition of conflicts of interest in a 1993 comment in the New England Journal of Medicine:

A conflict of interest is a set of conditions in which professional judgment concerning a primary interest (such as a patient's welfare or the validity of research) tends to be unduly influenced by a secondary interest (such as financial gain).

Thompson describes the meaning of a primary interest:

The primary interest is determined by the professional duties of a physician, scholar, or teacher…In their most general form, the primary interests are the health of patients, the integrity of research, and the education of students.

And he describes the meaning of a secondary interest noting that the problem is not its existence, but the extent to which it displaces the primary interest:

The secondary interest is usually not illegitimate in itself, and indeed it may even be a necessary and desirable part of professional practice. Only its relative weight in professional decisions is problematic.

Researcher conflicts of interest can induce the researcher into biased decision making and research misconduct (e.g. cherry picking data) that result in distorted research results.

When found out after the fact conflicts of interest undermine public trust in science.

Disclosure of researcher conflicts of interest describes the context in which research takes place. Disclosure creates transparency and accountability, fosters public trust, and enables meta-analysis of a body of work to understand external influences on the direction and outcome of research areas.

Importantly, the existence of a conflict of interest does not suggest anything about the moral character of the researcher, nor does it suggest anything about the integrity of the research.

While there is a clear funding effect associated with researcher conflicts of interest, there are many potential reasons for the funding effect. For example, sponsors may halt publication if findings do not support their preferred outcome. This is a problem that Ben Caldecott referenced in the Financial Times regarding industry funding of ESG research at Oxford University.

Historically, a researcher’s financial conflicts of interest have been the main focus because they are easy to quantify and name. In more recent years though, ethicists have been vocal about the need to improve disclosure of non-financial conflicts of interest such as unpaid witness testimonies and board membership to advocacy organizations relevant to the research being reported.

---

Managing conflicts of interest, usually through disclosure and sometimes through full separation of people from activities, has been an issue in the legal professions for centuries. In the 1970s, Congress created legislation to better manage conflicts of interest among the government’s small army of expert advisory committees.

But researcher conflicts of interest in science only became a topic of discussion for the public and Congress in response to series of scandals in the rapidly growing biomedical research field in the 1980s as the relationship between industry and academics increased in response to changes in intellectual property laws, and the growth of the biotechnology industry, more generally.

In the midst of this public uproar came the 1995 opinion of Appeals Court Judge Alex Kosinski in response to the Supreme Court remand in Daubert v. Merrell Dow Pharmaceuticals, a significant case for considering scientific evidence in court.

Kosinski posited that research performed in response to ongoing litigation deserved more scrutiny than research performed separate from any particular case:

That an expert testifies based on research he has conducted independent of the litigation provides important, objective proof that the research comports with the dictates of good science. For one thing, experts whose findings flow from existing research are less likely to have been biased toward a particular conclusion by the promise of remuneration; when an expert prepares reports and findings before being hired as a witness, that record will limit the degree to which he can tailor his testimony to serve a party's interests.

Scholars in the diverse field of science studies argued forcibly against Kosinski’s distinction. Science studies is a collection of research areas that study the processes and structures of science, and decisions about the production and use of science.

These scholars argued that the conflicts of interest caused by ‘litigation related research’ is no different than other conflicts of interest. Researchers may be influenced by large range of factors, including the anticipation of litigation… or regulation, or really, the promise of future work of any kind.

Moreover, the creation of the dichotomy fails to acknowledge the very messy world and social processes of science. Sponsors of research can fuel the creation of entire areas of study and a handful of sponsored influential researchers can steer the direction of research across a population of researchers.

The image below is a comment in a popular journal of reporting climate change modeling results. The goal of the comment is to steer researchers. The authors provide no COI statement.

Historically, medical journals did not require COI disclosures with editorials and comments either. This changed when it became apparent that researchers were using these venues for marketing ideas supportive of their sponsors. This needs to change in climate science, too.

---

Taking a birds eye view however, extreme weather attribution is really just one piece of a larger climate risk analytics business that goes undisclosed in the literature. What went on in biomedical research and toxicology in the 1980s and 1990s is going on in climate science today.

There are major industry, regulatory, and litigation interests in this analytical activity. But you wouldn’t know it by reading author disclosure statements.

Regularly, almost daily, I come across papers in the climate change literature that should be accompanied by conflicts of interest disclosure but are not.

Take for example this paper using SSP5-8.5 to generate tropical cyclone risk. No COI is disclosed. On its surface, it looks like a bunch of academics generating yet another modeled future risk about hurricanes.

This paper reads as an industry paper to me.

The authors contribute to scenario development for the central banking coalition Network for Greening the Financial System, and the maintain a model that NGFS uses. This is in addition to an author with a past, though significant, relationship with a major reinsurer and he is the inventor of several patents held by his former reinsurance employer.

The content of the paper provides a way of classifying risk that is relevant to the NGFS and reinsurance turn towards risk management of ecosystem services.

The paper should have a disclosure statement likely a lengthy one. The choice of emission scenario SSP5-8.5- which is discredited among many researchers for use in policy decision making- could reflect bias from the authors’ conflicts of interest. For instance, part of the justification of the scenario relies on a paper by researchers who also have significant undisclosed conflicts of interests.

It is possible that there is a funding effect around scenario choice. Better disclosure practices would allow for meta-analysis revealing who studies what, how, and with what funding sources.

---

No one dies from a “cascade” of bias in climate risk modeling. This is unlike conflicts of interest in medical or toxicology research that can lead to bias, misconduct, and unreliable knowledge that causing patient death or severe illness.

Nonetheless, there are phenomenal economic, political, and social costs associated with this literature from misguided public understanding to dysfunctional politics to wasted resources.

Politicization of the climate literature is well documented. However, the relationship of the politicization to business interests is less discussed.

Because author disclosure practices in the climate change literature are so abysmal, it is difficult to produce meta-analysis to understand the effects of external pressure on science. The norms of professional ethics across the climate change research discipline needs major improvement.