A step toward clinically useful machine learning models of brain behavior

Relating brain activity to behavior is an ongoing goal of neuroimaging research because it would help scientists understand how the brain gives rise to behavior—and may open new opportunities for personalized treatment of mental health and neurological conditions. In some cases, scientists use brain images and behavioral data to train machine learning models to predict an individual’s symptoms or disease based on brain function. But these models are only useful if they can generalize to different settings and populations.

In a new study, Yale researchers show that predictive models can perform well on data sets quite different from those on which the model was trained. In fact, they argue that testing models in this way, on diverse data, will be essential to developing clinically useful predictive models.

“It is common for predictive models to perform well when tested on data similar to what they were trained on,” said Brendan Adkinsonthe lead author of the study recently published in the journal Developmental Cognitive Neuroscience. “But when you test them on a dataset with different characteristics, they often fail, making them virtually useless for most real-world applications.”

The problem lies in the differences between the data sets, which include variations in age, sex, race and ethnicity, geography, and clinical symptom presentation among the individuals included in the data sets. But rather than seeing these differences as an obstacle to model development, researchers should see them as a key component, Adkinson says.

““Predictive models will only be clinically valuable if they can effectively predict beyond these dataset-specific idiosyncrasies,” said Adkinson, who is an M.D.-Ph.D. candidate in the lead author’s laboratory Dustin Scheinostassociate professor of radiology and biomedical imaging at Yale School of Medicine.

To test how well the models could perform in diverse data sets, the researchers trained models to predict two traits—language skills and executive function—from three large data sets that were substantially different from each other. Three models were trained—one on each data set—and then each model was tested on the other two data sets.

“We found that even though these datasets were significantly different from each other, the models still performed well by neuroimaging standards during testing,” said Adkinson. “This tells us that generalizable models are achievable and testing on various features of data sets can help.”

Next, Adkinson is interested in exploring the idea of ​​generalization in relation to a specific population.

Large-scale data collection efforts used to generate predictive neuroimaging models are based in metropolitan areas where researchers have access to more people. But building models solely on data collected from people living in urban and suburban areas risks creating patterns that don’t generalize to people living in rural areas, the researchers say.

“If we reach a point where predictive models are robust enough to be used in clinical assessment and treatment but do not generalize to certain populations, such as rural residents, then those populations will not be as well served. as good as others,” Adkinson said. , who himself comes from a rural area. “So we’re looking to generalize the patterns to rural populations.”