Building AI has its own challenges, building AI within a medical field presents a whole set of other issues. In this post our R&D team shares some of the key learnings and challenges that we’ve found while building the Radiobotics technology.
01: Health tech requires clinical insight
To be able to create great health tech AI products, it is very important to have a deep clinical understanding about the field of radiology and the workflow in the hospital. Since the end user is a radiologist, orthopedic surgeon or another clinical expert, it is very important to involve these people early in the development phase. When we have to determine which features to develop for the next product releases, we are very dependent on the inputs from these clinical experts. They are the only individuals who can tell us which features will contribute the most to our line of products.
It can be very difficult to find the right clinical experts who are willing to spend time on your company and advise on which products to develop. In Radiobotics, we have been lucky to have Michael Boesen and Anders Troelsen (radiology and orthopedic professors) on board from the beginning who have provided all sorts of inputs. They have advised us on which clinical challenges to solve and products and features to develop, and are able to give us valuable insight into the clinical understanding of the different tasks are trying to automate using AI. Our process has proven to us that it’s very important to identify clinical champions who believe in the vision. This can not only open doors but also can provide constructive feedback.
It can be very difficult to find the right clinical experts who are willing to spend time on your company and advise on which products to develop. In Radiobotics, we have been lucky to have Michael Boesen and Anders Troelsen (radiology and orthopedic professors) on board from the beginning who have provided all sorts of inputs. They have advised us on which clinical challenges to solve and products and features to develop, and are able to give us valuable insight into the clinical understanding of the different tasks are trying to automate using AI. Our process has proven to us that it’s very important to identify clinical champions who believe in the vision. This can not only open doors but also can provide constructive feedback.
02: ML requires access to the right kind of data
When developing AI/ML radiology products, one of the most important and biggest challenges is the access to data - the right data. Training deep learning systems requires large quantities of data, and often you see AI systems trained on millions of images, which is just not possible within AI healthcare and radiology. This can be attributed to a limit on the number of images available, these images have to be acquired by X-ray, MRI or CT scans of patients - not random images of cats and dogs. Besides that, the data also needs to represent the problem you are trying to solve, so that your AI system is able to generalize to other hospitals or countries.
Another limiting factor, is the labelling / annotation of the data, and the time it takes to do so. To be able to train AI systems (within supervised learning), you need both the data and the labels (in our case it can be an image and a report describing the image). This labelling can be very time consuming and costly as the labels themselves have to be done by expert clinicians (e.g. radiologists).
In Radiobotics, we started out with a big public data set of knee x-rays from the Osteoarthritis Initiative, where several radiologists have been annotating, specifically knee OA, in a very structured way - perfect for deep learning and machine learning. However, as time went on, we found out that multiple other features had to be added to the product, which meant that we had to set up contracts with expert radiologists who were willing to come into our office and manually label the images. It was both hard to find these experts and annotate enough data for the systems - especially nowadays, physically labelling of the data in our office is just not an option. Therefore we developed an internal web-based annotation tool, which makes it easy for our expert annotators to label data from their home offices. Moreover, have spent a lot of time working on a data agreements to the danish hospitals, which in the end gave us access to data we can use for development of our products.
Now that we are moving on to other new products, the development is still very limited without access to data of that specific problem. No matter what, extracting and annotating medical images is still a big bottleneck.
Another limiting factor, is the labelling / annotation of the data, and the time it takes to do so. To be able to train AI systems (within supervised learning), you need both the data and the labels (in our case it can be an image and a report describing the image). This labelling can be very time consuming and costly as the labels themselves have to be done by expert clinicians (e.g. radiologists).
In Radiobotics, we started out with a big public data set of knee x-rays from the Osteoarthritis Initiative, where several radiologists have been annotating, specifically knee OA, in a very structured way - perfect for deep learning and machine learning. However, as time went on, we found out that multiple other features had to be added to the product, which meant that we had to set up contracts with expert radiologists who were willing to come into our office and manually label the images. It was both hard to find these experts and annotate enough data for the systems - especially nowadays, physically labelling of the data in our office is just not an option. Therefore we developed an internal web-based annotation tool, which makes it easy for our expert annotators to label data from their home offices. Moreover, have spent a lot of time working on a data agreements to the danish hospitals, which in the end gave us access to data we can use for development of our products.
Now that we are moving on to other new products, the development is still very limited without access to data of that specific problem. No matter what, extracting and annotating medical images is still a big bottleneck.
03: Seamless integration is key
In Radiobotics, we are developing products within musculoskeletal radiology - a field where burnout and continuous overload of tasks is part of the daily work. We found that over the past years, many companies, including the healthcare system providers, have developed a lot of different applications and decision support tools with poor user experience and which require a ton of clicks with the mouse. Therefore it has been key to us to make sure our technology processes information in a smooth and swift way without adding any radical changes to the workflow of the user.
In Radiobotics, we are developing products which both generate visual overlays and reports - these outputs have to be integrated into the existing systems at the hospital. Often, the hospitals have different systems for viewing the images (PACS viewers) and the reporting module (RIS system). This increases the complexity of the integration task since our software has to talk to several systems to be seamlessly integrated. We also rely on a hospital's IT department to help us integrate our technology into the existing system.
This means getting real life feedback from the clinicians is a lengthy process.
In Radiobotics, we are developing products which both generate visual overlays and reports - these outputs have to be integrated into the existing systems at the hospital. Often, the hospitals have different systems for viewing the images (PACS viewers) and the reporting module (RIS system). This increases the complexity of the integration task since our software has to talk to several systems to be seamlessly integrated. We also rely on a hospital's IT department to help us integrate our technology into the existing system.
This means getting real life feedback from the clinicians is a lengthy process.
04: Keeping up with the clinical and regulatory standards
We are working within the highly regulated field of medical equipment. This requires us to follow certain standards and obtain the regulatory approval before we can sell our product. To meet the approval, we must have a certified and working quality management system in the company – the QMS. This system has to be certified and comply with different ISO standards. The certification is carried out by an external authority, and this certification process can be a time consuming process.
Though these processes and documentation requirements are there to secure the quality of the end product and make sure our system is working as intended, they also mean that the developing and sales process become somewhat rigid.
This makes working in an agile way very difficult since every single change in the product requirements has to be re-evaluated, tested and documented before a new product release.
Another challenge in this is that regulations change from time to time and some areas lack clear rules or definitions on the regulation. And it is not even safe to say that these regulations and policies are the same from hospital to hospital and again from country to country.
Though these processes and documentation requirements are there to secure the quality of the end product and make sure our system is working as intended, they also mean that the developing and sales process become somewhat rigid.
This makes working in an agile way very difficult since every single change in the product requirements has to be re-evaluated, tested and documented before a new product release.
Another challenge in this is that regulations change from time to time and some areas lack clear rules or definitions on the regulation. And it is not even safe to say that these regulations and policies are the same from hospital to hospital and again from country to country.
05: Are we even solving the right problem?
Probably hardest challenge of all is getting the right “product-market fit“ – actually creating a product that customers will pay for. In Radiobotics, we have developed several products and for us it’s key to distinguish between the customer (the hospital) and the user (the radiologist or the orthopaedic). You can develop fancy algorithms that can automatically detect things in x-rays, however, this doesn't mean that we can actually save time or reduce the radiologists’ workload. Without solving an actual problem or optimizing the clinical workflow it can be really hard to convince a hospital to buy your product.
For us, the technicality of building our product is not the main thing, it is more knowing what to build.
The process can easily look like this: you have a data set, you develop a system based on the data set, get regulatory approval of that system, deploy and test the system and in the end find out that the value generated might not be that big or it might not be the right product. In consumer technology, there are ways of testing products fast (and relatively cheap) with users, and get early validation of initial assumptions. On the contrary, when building medical devices you need access to doctors (sometimes from many specialties and countries) to encompass all different needs, you'll need to understand complex purchasing cycles that varies from country to country (and hospital to hospital), you'll need to understand how to do clinical validation/trials, you’ll need to know how to get it regulatory approved, and you’ll need to make assumptions on pricing, that you can't really validate before you actually have a product with clinical evidence ready for implementation.
All this makes it crucial to think about what problems we are solving before developing the system instead of thinking which features we should implement.
For us, the technicality of building our product is not the main thing, it is more knowing what to build.
The process can easily look like this: you have a data set, you develop a system based on the data set, get regulatory approval of that system, deploy and test the system and in the end find out that the value generated might not be that big or it might not be the right product. In consumer technology, there are ways of testing products fast (and relatively cheap) with users, and get early validation of initial assumptions. On the contrary, when building medical devices you need access to doctors (sometimes from many specialties and countries) to encompass all different needs, you'll need to understand complex purchasing cycles that varies from country to country (and hospital to hospital), you'll need to understand how to do clinical validation/trials, you’ll need to know how to get it regulatory approved, and you’ll need to make assumptions on pricing, that you can't really validate before you actually have a product with clinical evidence ready for implementation.
All this makes it crucial to think about what problems we are solving before developing the system instead of thinking which features we should implement.
← Back to all resources
