With the increasing popularity of Artificial Intelligence (AI), foundation models have demonstrated a tremendous ability to handle quite a lot of problems with only a small amount of knowledge provided by labeled instances. The concept of in-context learning has gained attention with its ability to let a model pick up a task from a number of examples given while being prompted without adjusting the model’s parameters. Considering the sphere of healthcare and the medical domain, in-context learning has the potential to enhance current medical AI models exponentially.
Though in-context learning has shown some great capabilities by way of medical data, as a result of the intrinsic complexity and multimodality of medical data, in addition to the range of tasks that have to be completed, implementing in-context learning in a medical setting offers difficulties. Multimodal medical foundation models have been attempted up to now, akin to ChexZero, which makes a speciality of reading chest X-rays, and BiomedCLIP, which was trained on quite a lot of images linked with captions from biological literature. For surgical footage and electronic health record (EHR) data, several models have been devised. None of those models have included contextual learning for the multimodal medical domain.
To deal with the restrictions, a team of researchers has proposed Med-Flamingo, a singular and highly effective foundation model that’s able to performing multimodal in-context learning specialized for the medical domain. This vision-language model relies on Flamingo, which is certainly one of the primary vision-language models that reveal in-context learning and few-shot learning capabilities. By providing pre-training in multimodal knowledge sources from multiple medical fields, Med-Flamingo expands these capabilities to the medical arena.
The primary phase entails creating an original, interleaved image-text dataset from over 4K medical textbooks, assuring correctness by choosing the dataset from reputable and reliable sources of medical knowledge. With a purpose to evaluate Med-Flamingo, the researchers have focussed on generative medical visual question-answering (VQA) tasks, where the model directly creates open-ended responses somewhat than assessing pre-defined possibilities. A brand new and realistic evaluation process has been developed that yields a human evaluation rating as the important thing parameter. A visible USMLE dataset has also been developed, which is a difficult generative VQA dataset comprising difficult USMLE-style tasks across specialties, enhanced with images, case vignettes, and lab results.
In three generative medical VQA datasets, Med-Flamingo has been shown to outperform earlier models in clinical evaluation scores, suggesting that doctors favor the model’s predictions. It has exhibited medical reasoning skills, something multimodal medical foundation models haven’t previously done, by responding to complicated medical queries and offering justifications. The model’s effectiveness can, though, be constrained by the range and accessibility of the training data in addition to the problem of some medical tasks.
The team has summarized their contributions as follows.
- Med-Flamingo is the primary multimodal few-shot learner designed for the medical domain, offering latest clinical applications like rationale generation and context conditioning.
- The researchers have built a singular dataset for pre-training the model, specifically fitted to multimodal few-shot learning within the medical domain.
- They’ve also introduced an evaluation dataset with USMLE-style problems, incorporating complex medical reasoning in visual query answering.
- Existing evaluation strategies are critiqued, and an in-depth clinical evaluation study has been conducted using a dedicated app involving medical raters to evaluate the model’s open-ended VQA generations.
Try the Paper, Model, and Github. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 27k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more.
Tanya Malhotra is a final 12 months undergrad from the University of Petroleum & Energy Studies, Dehradun, pursuing BTech in Computer Science Engineering with a specialization in Artificial Intelligence and Machine Learning.
She is a Data Science enthusiast with good analytical and significant considering, together with an ardent interest in acquiring latest skills, leading groups, and managing work in an organized manner.