1/10 2022

Validating our AI: How We Make Safe and Reliable Health Tech

The process behind getting technology ready for a clinical setting can seem daunting. In this interview, Martin Axelsen, Chief Scientific Officer at Radiobotics, explains how we get our AI ready to market and how he sees the future of AI in health tech.

Thoroughness is a key component in the process of developing Radiobotics products. Not only because of the standards you must follow when developing medical devices, but also because of Radiobotics’ dedication to making meticulously validated and safe products.Martin Axelsen’s job is to lead the clinical and regulatory teams in Radiobotics, which means that he’s responsible for conducting clinical trials and subsequently getting the market approvals done for Radiobotics’ products. One of Martin Axelsen’s visions for the clinical team is to push the boundaries for what can be done with AI in medical imaging and create technology that can revolutionize the field of medical imaging.
“It's about proving to the world that it is possible to build the technology and that it can bring value and perform as well as clinical experts, and that we can conduct the studies to show this improvement and this excellent performance from our products is an essential part of it.”
Martin explains that there’s generally a need in health tech for better external validation of the claims made by new technologies. During Covid-19, the extreme pressure on healthcare systems has meant that hospitals have been eager to implement new technology that hasn’t been tested properly. That is risky. A recent publication in Nature Machine Intelligence describes these failings.“It's important that there’s scientifically sound studies behind these technologies in order to ensure the patient's safety. One thing is that the diagnostic accuracy is high, but it should also generalise to the data from a new hospital and that the risks associated with implementing the technology should be mitigated to an acceptable level. Conducting comprehensive studies obviously takes time and requires resources. But it is of the highest importance, and we are happy to have prioritised the resources for this early, so that we now have the framework and excellent collaborators to do it.”

Getting the standards right

“Patient safety is obviously of highest importance when developing medical devices. That is why the risk analysis and risk-based approaches are central in everything we do during development of our products.“ says Martin Axelsen. Medical devices are not like any other app, they must be built according to standards and published guidelines that have been created by international expert committees. The standards help a company develop recipes for how they can build their products in an as safe and effective manner as possible. This recipe book is what's called the quality management system or QMS. So called Notified Bodies are designated by the public authorities to thoroughly review an organization’s QMS and give an important stamp of approval if it complies with the rigorous standards. Subsequently, a product can be assessed by the notified body, and the recieves a stamp if it has been built according to the recipe book (the QMS).In the case of Radiobotics, we are proud to have received these stamps of approval from both European and US authorities. We are certified according to the new European Medical Device Regulations and ISO 13485, and versions of our first product, RBknee, has been both CE-marked and FDA-cleared. These are major milestones for a medical device company and a testament to the hard work of the people at Radiobotics “I think the compliance aspect of building medical devices is what keeps many entrepreneurs from venturing into this field, as it requires a lot of insight and resources to implement. I have had to learn a lot, but we would never have made it without Astrid and John, who run our quality assurance and regulatory efforts ensuring that we comply with both EU and US regulations in an efficient manner” Martin says.

Validating claims

When developing medical devices it’s crucial that the products actually work according to the plan. This means that Radiobotics has to prove the claims behind the product selling points. In the clinical team, Michael and Liv work with all the clinical and research partners of Radiobotics to define protocols for studies that can help prove the claims of the products of Radiobotics.However, before the clinical team engages the clinical partners, they verify the technology on data they already have available and which has been annotated, together with the development team. This means that doctors have already gone through the images and annotated what they are looking for in each image and how they see the disease or abnormality in the imaged anatomy. It’s essential that this verification doesn’t happen on images that the AI has been trained on. There must never be an overlap in what is used for training the AI and what is used for testing or validating it. Read more about how we ensure we have the best data to build the best AI. When this validation is done, the team is ready to bring the technology to the hospitals that they collaborate with.At the hospitals, the image data is used as part of the diagnosing of patients, and is hence analysed and potentially annotated by the medical professionals accordingly. When a product is evaluated at a hospital, the team will implement the technology and run a study to validate the performance in the hospital setting. The purpose of these studies is not only to prove that Radiobotics products work, but also to contribute to the groundbreaking body of research on AI-driven medical devices. We always aim to publish or present the results of the studies with our clinical partners, like this work which was presented at the 2021 OARSI World Congress. Another important aspect of the hospital evaluations is to ensure that the technology fits well into the clinical workflow. Once the technology has been validated in a clinical setting in one of the collaborating hospitals, the next step can be to set up a more comprehensive study. In 2020, Radiobotics was granted €1.3M from the Horizon 2020 EIC Accelerator programme, an EU grant that is funding a number of large multicenter studies in a European consortium between Radiobotics and Charité Hospital in Berlin, Erasmus Medical Center in Rotterdam, and Bispebjerg Hospital in Copenhagen. This will be the largest setup that the team has done so far and will not only allow them to answer the research questions, but will do so with a higher certainty and robustness.“When you are working with AI, you try to include as broad a training set as possible in teaching the model. But subsequently to be able to know how well you actually manage to do, it's important to validate the final product on a diverse dataset. This validation can be done by validating it on a number of different sites - a so-called multicenter study, and that's exactly what we're doing with Charité, Erasmus MC and Bispebjerg in the studies funded by the EIC grant.”

The future of medical imaging

Martin Axelsen and the rest of the team are working on setting up several studies for the future. He anticipates a future where AI is a common tool in the decision support and automation of things within the analysis of medical images. “It's obvious that even if there wasn't a huge lack of radiologists, the fact that the technology is so good and there are so many people, companies and universities investing in developing these tools, proves that there is a possibility to create value in assisting in reading these images. And then it's up to the radiologists and doctors to adopt the technology,” he says. His hopes for the future of healthtech are that the different actors and competitors find a way to collaborate in order to better integrate the different tools into an easy to use package for doctors. A unified way of integrating would avoid the hassle of navigating between different applications and systems. The current systems in place in many health settings are rigid and hard to replace. In this way it’s very unlike the smooth and intuitive user interfaces of our everyday technology.“If we at Radiobotics succeed, that's good. But we're only solving one vertical, one speciality for the radiologist. I hope that the other companies that are in this space, essentially some of our competitors, also succeed. This way we manage to solve more tasks for the radiologists and other doctors, so the new tools can be of greater value to the hospitals, society and most importantly the patients.”