Project CANAIRI takes flight; asks what does it take to successfully integrate AI tools with clinical practice?

AI’s role in the health and medical fields has never been more prolific. Around the globe, medical professionals are utilising AI and machine learning to diagnose disease, treat patients, and maintain treatment plans in ways that were difficult to imagine just a few decades ago.

But how can patients trust the decisions health professionals are making with these tools?

Enter Project CANAIRI, or the Collaboration for trANslational AI tRIals. The program, established and implemented by AIML Deputy Director Dr Melissa McCradden, tests that AI tools function effectively in real-world healthcare settings and meet the needs of those who use them. It also tests that an AI tool that works at one facility or hospital produces the same results at other hospitals. 

The project asks these critical questions: how do we make sure that AI tools perform in the right place, for the right groups, and across the needed time frame?

“CANAIRI emerged in 2019 from my experience at The Hospital for Sick Children (SickKids) [in Toronto] with the AI team there and [realising] how important the silent evaluation stage became over the course of many years of training and testing AI tools in clinical environments,” said Dr McCradden. ‘Silent’ evaluations refer to the AI tool’s predictions being tested in real-time while remaining invisible to the clinical team, meaning that the model is tested without affecting patient care. This allows teams to approximate the clinical conditions of integration before finding out if the model works.

“It surprised me to know that silent evaluations are not common practice; but for those who are more experienced in AI deployments, it is an absolutely essential step,” she said. “By bringing in a wider set of considerations right at the beginning, we can actually streamline translation and minimise research waste.”

Project CANAIRI has three main goals:

• Scoping existing literature and conducting interviews to identify current practices around silent testing and lessons learned
• Using a survey to gather expert opinions in order to create practical guidance for diverse healthcare environments
• Designing knowledge-translation resources to empower patients, clinicians, developers, and decision-makers alike.

“Our goal is to focus on the silent stage, which we call the translational trial,” said Dr McCradden. “For health institutions, the main value of the translational trial is that it offers a way to test an AI tool on your own system, with your own data, without having to affect operations at scale before you know that it works.”

“We focus on this stage because it is the one place where there is [a] total lack of guidance around how the community should be doing this sort of evaluation.”

CANAIRI’s team brings together a wealth of expertise from data science, bioethics, clinical research, and health economics, alongside valuable insights from youth, caregivers, and Aboriginal community representatives. By embedding diversity as a core value, the initiative is designed to address the multifaceted challenges of integrating AI tools responsibly in healthcare.

“We will work with experts and non-experts alike to identify what practices should be part of translational trials and when. We will also develop 'how-to' resources for health settings to use.” 

“I'm really grateful for funding support from the Centre for Augmented Reasoning (CAR) and for the ongoing support from AIML,” said Dr McCradden. “We currently have 30 individuals on the steering group, including University of Adelaide Professor Carolyn Semmler, AIML Professor Lyle Palmer, and AIML Senior Research Fellow Dr. Lauren Oakden-Rayner who is also a Working Group Lead.”

“As a cognitive scientist, I am excited to be involved with a project where human cognitive processes and decision making is considered from the outset, not as an afterthought,” said Professor Semmler, a professor with the University’s School of Psychology. “I was also drawn to the strong female leadership in this project and the shared values of inclusivity and respect that permeate the work.”

“Because AI is a complex and sometimes poorly understood technology, there is a lot of complexity in gathering evidence to assist a diversity of decision makers. Too narrow a focus on model performance without consideration of the entire system will mean that translation fails,” added Professor Semmler. “We have a chance to prevent this with CANAIRI.”

AIML PhD student Lana Tikhomirov is taking a leading role in this project, bringing her unique interdisciplinary approach to AI integration.

“I became involved in 2024 when [Dr McCradden] became my third supervisor in my PhD,” said Tikhomirov. “I was drawn to this project because of the respect for human factors and cognitive science in CANAIRI, which can be rare in a field where technical validation has historically been viewed as more important than clinical validation.”

“I am really passionate about CANAIRI’s role in the future of AI in medicine,” she continued. “I would love to see standardised practices for AI trials in hospitals and clinical settings so we can be smart and safe about how we deploy AI in critical areas such as medicine.”

CANAIRI also aims to inform the public of AI’s growing role in the medical field.

“We are working closely with consumers and will be developing consumer-facing materials to support health AI literacy,” said Dr McCradden. “We're aiming to involve folks from every major geographical region as we move forward with CANAIRI and are always looking for collaborators.” The team is also partnering with the Aboriginal Health Unit at Adelaide's Women’s and Children’s Hospital and connecting with Aboriginal scholars across Australia to ensure that their work respects key issues of rights, sovereignty, and equity.

Dr. McCradden hopes that CANAIRI ushers in a new paradigm of translational trials and changes the way that AI tools are tested and utilised in medicine.  

“My own interests have always been around how we combine medical knowledge, evidence from research, and ethical values to determine what should be done, how, and why,” she said.

“This is critical in a time where AI tools are everywhere, and everyone wants a piece of the action.”

For more information on Project CANAIRI, please visit https://www.adelaide.edu.au/aiml/our-key-initiatives/project-canairi