AI Recognises Race in Medical Images

Summarised paper information

Reading Race: AI Recognises Patient’s Racial Identity In Medical Images

Jul 21, 2021

arXiv

Imon Banerjee, Ananth Reddy Bhimireddy, John L. Burns, Leo Anthony Celi, Li-Ching Chen, Ramon Correa, Natalie Dullerud, Marzyeh Ghassemi, Shih-Cheng Huang, Po-Chih Kuo, Matthew P Lungren, Lyle Palmer, Brandon J Price, Saptarshi Purkayastha, Ayis Pyrros, Luke Oakden-Rayner, Chima Okechukwu, Laleh Seyyed-Kalantari, Hari Trivedi, Ryan Wang, Zachary Zaiman, Haoran Zhang, Judy W Gichoya

Previous studies have shown that AI can predict your sex and age from looking at an eye scan, or your race from a chest X-ray.

This is strange — because even the most expert doctors can’t do this. What’s more: They don’t even understand how the AI is doing this…

The fact that you can give an AI model an anonymous X-ray, and it can work out the patient’s race could either be important to aid diagnosis & treatment — or it could enable a terrible amount of bias. (This topic is hotly debated).

What did they do?

The authors picked lots of large scale imaging datasets including chest X-rays, limb X-rays, chest CT scans, mammograms etc.

They trained Convolutional Neural Networks (CNNs) which could identify a patient’s race from looking at radiological imaging.

Convolutional neural networks (CNNs) are deep neural networks with many layers that pick up features. The features get more complicated as you go deeper into the network. In the early layers, the network may recognise lines and colours. These are added together to make shapes and textures. In the final layers, the full image is analysed.

They challenged these CNNs with various experiments to see how they worked, and how were they able to identify race.

This is a big paper with lots of experiments. I’ve picked three of the most interesting ones here.

B4 Can AI predict race using bone density?

This is a chest X-ray. The black parts of the image are gas (less dense) and the white areas are bone (more dense). Thicker bone is whiter. Thinner bone is more grey/translucent.

Reading Race: AI Recognizes Patient’s Racial Identity In Medical Images Clipped Chest X Ray Figure

We know that bone density (how white the bone appears) differs between races, for example, black people generally have higher bone mineral density.

The authors thought that AI models could use these colour differences to work out the bone density of a patient, and therefore predict their race.

So they ‘clipped’ the pictures. Essentially, they put a filter on the images which made everything appear more grey, so the AI couldn’t detect these subtle differences in colour.

Result: the model still performed really well on the clipped images (AUC 0.94–0.96). So bone density is probably not important in its decision making process.

C2 Is AI picking up something we can’t see in high resolution images?

To test this, they presented the AI with high quality images (512×512 pixels) and some really low quality ones (8×8 pixels).

Remarkably, the AI maintained a pretty strong race-predicting performance — even when the images presented to it were extremely low quality.

Reading Race: AI Recognizes Patient’s Racial Identity In Medical Images Pixelating Performance

C3 Is AI picking up on differences in anatomy on imaging to predict race?

Was the AI picking up subtle differences in anatomy to detect race? Different races might have different heart positions, lung sizes etc.

The methodology they used to test this is interesting:

1️⃣ They created saliency maps using Grad-CAM methodology.

Grad-CAM is a technique which provides visual explanations for what an AI (CNN) is doing.

Simply, it creates a heatmap showing areas which were important for the AI’s decision making process. You can read more about it here.


On the left image, you can see the saliency map. This heatmap shows areas in which the AI was paying particular attention when determining a patient’s race (red=more attention).

2️⃣ In this example, it looks like the AI is paying particular ‘attention’ to the heart borders. So they place a black box over the heart border to hide it (right image).

Result: The AI is worse at detecting race, but still performs pretty well (~AUC 0.94 original, ~AUC 0.82 with parts of the image hidden).

This isn’t surprising, since you’re giving less data for the algorithm to work with, it’s inevitable that it will perform worse at any task. But it indicates that the heart borders are just one of many important factors

So what?

There’s some debate about what this means. One of the paper’s authors: Luke Oakden-Rayner believes AI’s ability to detect race so easily is very bad and could lead to bias. Other researchers don’t find this ability as alarming:

(9/9) Agree 100% w/ conclusions (need more transparency, monitoring, etc). Just don’t agree with the alarmist framing. The alarm for systemic racism in healthcare is COVID-19 and how it’s devastating communities of color. That alarm is ringing loud and clear.

— Mark Sendak (@MarkSendak) August 7, 2021

Read More

Written by 

Leave a Reply

Your email address will not be published. Required fields are marked *