Uncovering and Removing Data Bias in Healthcare

Good data will train good algorithms in healthcare. But what if the data used to train an algorithm isn’t telling the whole story? What happens when that data delivers faulty or biased decisions for health systems, insurance companies or government agencies using it to predict risk or determine treatment protocols in population health management? Data scientists and clinicians alike are talking about the problem of data bias that has enormous consequences on healthcare and human lives.

Why Diverse Data Matters

Computer systems are using artificial intelligence and machine learning to diagnose skin cancer, identify a stroke on a CT scan, and even detect potential cancers on a colonoscopy. These digital diagnosticians can make care better, faster and cheaper. But they are not always trained with data representing the whole population.

For example, if algorithms for reading chest X-rays are trained with data from primarily male patients, the results are not as accurate when applied to chest X-rays of female patients. Skin-cancer detection algorithms trained primarily on light-skinned individuals do worse at detecting skin cancer affecting darker skin.

Effective data sets must be based on a diverse range of race, gender or geography in order to avoid biased algorithms. But data scientists acknowledge that diverse data alone won’t guarantee elimination of data bias. Ultimately, people are responsible for the collection and use of data, raising concerns about lack of diversity among developers and funders of AI tools causing implicit bias. There is also discussion of data bias occurring when problems are framed from the perspective of majority groups, using implicitly biased assumptions about data, and using outputs of AI tools to perpetuate biases, either inadvertently or explicitly. 

COVID-19 Reveals Disparities in Healthcare

Provisional data from the National Center for Health Statistics clearly illustrated the disproportionate burden of COVID-19 deaths among racial and ethnic minority groups, particularly Hispanic or Latinx, non-Hispanic Black, and non-Hispanic American Indian or Alaska Native people. 

Underlying health and social inequities put many racial and ethnic minority groups at increased risk of getting sick, having more severe illness, and dying from COVID-19. Inequities in social determinants of health—including where people live, access to healthcare and insurance, occupation, income and education—are interrelated and influence a wide range of health and quality-of-life risks and outcomes.

A study of selected U.S. states and cities with data on COVID-19 deaths by race and ethnicity showed that 34% of deaths were among non-Hispanic Black people, though this group accounts for only 12% of the total U.S. population.

Data Bias is Often Invisible

Kadija Ferryman, PhD, industry assistant professor at the NYU Tandon School of Engineering is a cultural anthropologist who has studied and advised on the implications and causes of data bias in healthcare. She observes that beyond overt bias such as Black people being denied COVID-19 testing, there are less obvious ways bias manifests structurally through informatics and data.

“Machines like barometers that measure lung function capacity for example, have racial corrections built into those very instruments. Those readings can then be used in all sorts of data to potentially build algorithms. With COVID-19 data, that already has racial bias built into it because of racial corrections that have to do with someone's respiratory function. There's a racial correction made there, often invisibly, that can have cascading effects on the care they receive,” said Ferryman.

Another researcher in data bias, Ziad Obermeyer, MD, is an associate professor with the School of Public Health at the University of California, Berkeley. “The most concerning part of a lot of the biases is how little we see them. For example, we look a lot at COVID-19 statistics by geography, but of course, if some geographies have less access to COVID-19 testing, then we're not going to see all of the people in that county, in that city, in that state that have COVID-19 but never make it onto our radar screen. And likewise, we often prioritized where help is needed by looking at past data on utilization who uses the hospital.” 

Data Can Identify Social Barriers to Care

Melissa Kotrys, chief executive officer at the Health Current health information exchange, acknowledges that social determinants of health have been an issue for decades. “Now that we've got a reasonable handle on the exchange of clinical information, what other information can we bring to bear to improve the health and well-being of individuals and communities? I think the sky's the limit when it comes to how we can use data to identify what barriers are, what challenges are, what gaps are and how we can fill those."

Kotrys flagged the disconnect she sees in data sharing, made particularly stark amidst the COVID-19 crisis. "We always have to keep an eye toward: What is that outcome we are trying to achieve and what is it we are trying to advance? And then make sure policy decisions are in alignment with that." 

But making change without adequate information will be challenging. "I work with a lot of federal data, and what's happening is the data doesn't have key variables for us at the local level, state level, county level," said Deborah Duran, PhD, senior advisor to the director for data science, data analytics and data systems at the National Institute on Minority Health and Health Disparities.

"Even today, there still aren't states that are collecting race and ethnicity data on COVID-19 vaccines and cases. That absence of information can make it more difficult to acknowledge and subsequently address gaps in care.” But Duran warns that technology is not inherently neutral. As decision-makers rely more heavily on machine learning and analytics, she cautioned that they must be aware of both their own biases and those that might be baked into the system.

Watch Uché Blackstock, MD, founder and chief executive officer of Advancing Health Equity, talk with HIMSS TV about strategies to mitigate racial bias in digital health tools.

Who Sits at the Algorithm Table?

Researchers are recognizing the role of personal bias in the process of building algorithms. Medical devices are typically judged by technical experts after release into the market. Ferryman sees this as an opportunity to identify bias in the product or device by having people from multiple constituencies involved throughout the process of development, but especially in the “post-market” period.

She brings up the example of a machine learning algorithm designed to measure diabetic retinopathy. “The tool is so powerful because it allows what used to be done only by a specialist to now be done in a primary care physician’s office. But we know there are already biases in the clinical work about who is even offered these kinds of diagnostic tests.” She observed that although the algorithm works well and with accuracy, some groups are offered the technology more than others.

Varied perspectives increase the chances of reducing data bias—by tearing down silos in healthcare. “It’s often unusual to have an interdisciplinary environment where a hospital informatics department works closely with clinicians, said Ferryman. “It could be very good practice to create structures where a machine learning person says, ‘Hey, I have all this great data,’ and there is a clinician they feel comfortable calling to say, ’Does this kind of question make sense to ask in this way? What other kinds of things should I consider?’”

Writing in The Lancet Digital Health journal, a team of data researchers recommended organizational action to address the low diversity in health data science. Their suggestions include updating hiring processes, ensuring representation on executive leadership teams, developing leadership pathways to support emerging leaders from historically underrepresented backgrounds, creating inclusive working environments, and actively listening to and learning from the experiences of data scientists from ethnic minority groups, such as Black in AI, the Shuri Network, One HealthTech.

Digging Deeper to Build “Data Empathy”

While doing research on bias and precision medicine, Ferryman recalls a conversation with a software engineer working with health data to build models. He shared with her that he felt he often lacked data empathy. She was surprised to hear this and asked what he meant. The engineer explained that he did not have a deep context or sense of where the data came from, how it was collected or the intentions of who collected it. He found that when he was closer to the data and its origins, the models he built turned out better.

Ferryman advises that after engineers identify the clinical problem being addressed, their very next step should be asking if there are health disparities in the health condition of interest. “That's one way of getting closer to that kind of data empathy… to know about the data that you're collecting, to know what this data is actually measuring.”

Even when starting with a very strong clinical question, Dr. Obermeyer has seen settings where subtle but important problems with the data results in the wrong algorithm. “One example that might be very topical is allocating ventilators and [intensive care unit] beds. Many places that are doing the following exercise: ‘Let's take people who were put on a ventilator because of COVID-19 or for whatever reason, and let's predict who's going to do poorly… so we can give them to people who are going to do better.’ Now, there's one really important problem with that way of thinking, which is that the people who do poorly with a ventilator might also do poorly without a ventilator. What's important is the difference in how they do on the ventilator versus not on the ventilator. Young people do great on ventilators, but they also do great not on ventilators. So we're predicting the wrong thing. We're treating this as a machine learning prediction problem when, in fact, it's a deep question about who do ventilators benefit?”

Imagining a More Equitable Healthcare AI Strategy

To ensure that algorithms of tomorrow are powerful but also fair, we must build infrastructure to deliver the large and diverse data required to train these algorithms.

Dr. Obermeyer believes data can be a “force for evil and reinforce disparities,” but it can also show us where they exist so we can fix them. “I think there's reason for optimism,” he said. “We have a lot of better data, both from nationally representative surveys, from routine financial and electronic health record systems that we can use to more equitably redistribute resources to more equitably train machine learning algorithms. Let's look critically at which data we're using because the difference between variable one and variable two can make a huge difference between a biased algorithm and a biased policy and one that's fundamentally more just.”

Ferryman sees hope in not only working with hospitals and tech companies to address the problem of data bias, but also with regulators and policymakers, giving guidance to those working closely with the data in different settings. “Let's also think about how we bake in an attention to health equity and attention to health disparities,” she said. “With technical guidance in that policy, we can make sure there are good practices in place around machine learning and that the decisions that are made around performance of the algorithms are clear. You’ve got to put social justice glasses on as you are developing machine learning algorithms.”

Researchers recently issued this call to action on tackling data bias: “Data scientists have a responsibility to tackle the different forms of racism that manifest themselves in our sector. Inaction perpetuates existing inequalities and racism; as practitioners, we all need to take more action to address racism and ensure that the benefits from the use of health data are shared equitably.”

The views and opinions expressed in this content or by commenters are those of the author and do not necessarily reflect the official policy or position of HIMSS or its affiliates.