³Ô¹ÏÍøÕ¾

“It is important to evaluate the diagnostic accuracy of the AI algorithms that might be used clinically,” says project leader , researcher at the , Karolinska Institutet, and radiologist at Karolinska University Hospital. “Even if they meet the regulatory requirements, it doesn’t mean they work in all contexts.”

In a paper published in the Journal of Medical Imaging, the researchers describe how they have designed the platform in the hope that more teams will be able to develop similar platforms.

Enables objective evaluation of AI systems

“By making our code freely accessible, we hope that it will be of use in implementing similar platforms for algorithm evaluation for other kinds of cancer than breast cancer,” says Fredrik Strand, adding that objective evaluations of this kind have been keenly awaited. Several of the country’s regions are already considering to start using AI for mammography screening.

“There’s an imminent need for a system that can evaluate algorithms under locally relevant conditions,” he says. “It’s important that each hospital chooses the right system to suit its requirements and particular circumstances to avoid missing a breast cancer or recalling an excessive number of healthy women.”

There is currently no established evaluation procedure on designated datasets to assess how accurate AI algorithms are in medical diagnostics. Manufacturers of AI systems have trained and tested their algorithms on disparate datasets, and those results are used for regulatory approval. In the platform, all algorithms process the same images and the results are compared with data on actual cancer diagnoses extracted from the ³Ô¹ÏÍøÕ¾ Quality Register for Breast Cancer (NKBC), revealing differences in the algorithms s. Since today’s AI algorithms have been trained on the specific population groups to which the developers have had access, they may contain biases that should be identified.

“The platform will be able to show an algorithm’s bias, particularly when it comes to age, geographical origin and socioeconomic status of the screened individual,” says Fredrik Strand.

May also appeal to manufacturers

It is hoped that this means of validating algorithms will also appeal to the manufacturers, who will have an independent assessment that can help them improve their products.

“Hospitals will have to do their bit by demanding that the manufactures take part in independent tests, maybe before procurement,” he says.

The development of the validation platform took place under a research project due to conclude in 2024. Before this, the project group is to have presented a proposal on how the platform can be permanently implemented for national use. Meanwhile, more regions are being invited to use the platform.

“We want more regions to take this opportunity, also for evaluating other algorithms than the three already installed,” says Fredrik Strand.

Also involved in the project, alongside Karolinska Institutet and Karolinska University Hospital, are Lund University, Region Skåne, Linköping University, Region Östergötland, KTH Royal Institute of Technology, the Swedish Breast Cancer Association and Medtech4Health AB.

The research was financed by Regional Cancer Centres in Sweden and Vinnova. Some of the researchers have reported potential conflicts of interest, including employment by medical image processing companies and patents on mammography technology.

More details on the national validation project for the use of AI in mammography screening can be found .

Publication

“.” Fernando Cossío, Haiko Schurz, Mathias Engström, Carl Barck-Holst, Apostolia Tsirikoglou, Claes Lundström, Håkan Gustafsson, Kevin Smith, Sophia Zackrisson, Fredrik Strand, Journal of Medical Imaging, online March 20, 2023, doi: 10.1117/1.JMI.10.6.061404