How can an LLM be applied effectively for biomedical entity linking?
Biomedical text is a catch-all term that broadly encompasses documents comparable to research articles, clinical trial reports, and patient records, serving as wealthy repositories of data about various biological, medical, and scientific concepts. Research papers within the biomedical field present novel breakthroughs in areas like drug discovery, drug unwanted side effects, and recent disease treatments. Clinical trial reports offer in-depth details on the protection, efficacy, and unwanted side effects of recent medications or treatments. Meanwhile, patient records contain comprehensive medical histories, diagnoses, treatment plans, and outcomes recorded by physicians and healthcare professionals.
Mining these texts allows practitioners to extract beneficial insights, which may be useful for various downstream tasks. You would mine text to discover hostile drug response extractions, construct automated medical coding algorithms or construct information retrieval or question-answering systems that may help extract information from vast research corpora. Nevertheless, one issue affecting biomedical document processing is the usually unstructured nature of the text. For instance, researchers might use different terms to discuss with the identical concept. What one researcher calls a “heart attack” may be known as a “myocardial infarction” by one other. Similarly, in drug-related documentation, technical and customary names could also be used interchangeably. For example, “Acetaminophen” is the technical name of a drug, while “Paracetamol” is its more common counterpart. The prevalence of abbreviations also adds one other layer of complexity; for example, “Nitric Oxide” may be known as “NO” in one other context. Despite these various terms referring to the identical concept, these variations make it difficult for a layman or a text-processing algorithm to find out whether or not they discuss with the identical concept. Thus, Entity Linking becomes crucial in this case.
- What’s Entity Linking?
- Where do LLMs are available here?
- Experimental Setup
- Processing the Dataset
- Zero-Shot Entity Linking using the LLM
- LLM with Retrieval Augmented Generation for Entity Linking
- Zero-Shot Entity Extraction with the LLM and an External KB Linker
- High quality-tuned Entity Extraction with the LLM and an External KB Linker
- Benchmarking Scispacy
- Takeaways
- Limitations
- References
When text is unstructured, accurately identifying and standardizing medical concepts becomes crucial. To realize this, medical terminology systems comparable to Unified Medical Language System (UMLS) [1], Systematized Medical Nomenclature for Medicine–Clinical Terminology (SNOMED-CT) [2], and Medical Subject Headings (MeSH) [3] play a necessary role. These systems provide a comprehensive and standardized set of medical concepts, each uniquely identified by an alphanumeric code.
Entity linking involves recognizing and extracting entities inside the text and mapping them to standardized concepts in a big terminology. On this context, a Knowledge Base (KB) refers to an in depth database containing standardized information and ideas related to the terminology, comparable to medical terms, diseases, and medicines. Typically, a KB is expert-curated and designed, containing detailed information concerning the concepts, including variations of the terms that may very well be used to discuss with the concept, or the way it is said to other concepts.
Entity recognition entails extracting words or phrases which might be significant within the context of our task. On this context, it often refers to extraction of biomedical terms comparable to drugs, diseases etc. Typically, lookup-based methods or machine learning/deep learning-based systems are sometimes used for entity recognition. Linking the entities to a KB often involves a retriever system that indexes the KB. This technique takes each extracted entity from the previous step and retrieves likely identifiers from the KB. The retriever here can be an abstraction, which could also be sparse (BM-25), dense (embedding-based), or perhaps a generative system (like a Large Language Model, (LLM)) that has encoded the KB in its parameters.
I’ve been curious for some time about one of the best ways to integrate LLMs into biomedical and clinical text-processing pipelines. On condition that Entity Linking is a very important a part of such pipelines, I made a decision to explore how best LLMs may be utilized for this task. Specifically I investigated the next setups:
- Zero-Shot Entity Linking with an LLM: Leveraging an LLM to directly discover all entities and concept IDs from input biomedical texts with none fine-tuning
- LLM with Retrieval Augmented Generation (RAG): Utilizing the LLM inside a RAG framework by injecting details about relevant concept IDs within the prompt to discover the relevant concept IDs.
- Zero-Shot Entity Extraction with LLM with an External KB Linker: Employing the LLM for zero-shot entity extraction from biomedical texts, with an external linker/retriever for mapping the entities to concept IDs.
- High quality-tuned Entity Extraction with an External KB Linker: Finetuning the LLM first on the entity extraction task, and using it as an entity extractor with an external linker/retriever for mapping the entities to concept IDs.
- Comparison with an existing pipeline: How do these methods fare comparted to Scispacy, a commonly used library for biomedical text processing?
All code and resources related to this text are made available at this Github repository, under the entity_linking folder. Be happy to drag the repository and run the notebooks on to run these experiments. Please let me know if you have got any feedback or observations or if you happen to notice any mistakes!
To conduct these experiments, we utilize the Mistral-7B Instruct model [9] as our Large Language Model (LLM). For the medical terminology to link entities against, we utilize the MeSH terminology. To cite the National Library of Medicine website:
“The Medical Subject Headings (MeSH) thesaurus is a controlled and hierarchically-organized vocabulary produced by the National Library of Medicine. It’s used for indexing, cataloging, and searching of biomedical and health-related information.”
We utilize the BioCreative-V-CDR-Corpus [4,5,6,7,8] for evaluation. This dataset incorporates annotations of disease and chemical entities, together with their corresponding MeSH IDs. For evaluation purposes, we randomly sample 100 data points from the test set. We used a version of the MeSH KB provided by Scispacy [10,11], which incorporates information concerning the MeSH identifiers, comparable to definitions and entities corresponding to every ID.
For performance evaluation, we calculate two metrics. The primary metric pertains to the entity extraction performance. The unique dataset incorporates all mentions of entities within the text, annotated on the substring level. A strict evaluation would check if the algorithm has outputted all occurrences of all entities. Nevertheless, we simplify this process for easier evaluation; we lower-case and de-duplicate the entities in the bottom truth. We then calculated the Precision, Recall and F1 rating for every instance and calculate the macro-average for every metric.
Suppose you have got a set of actual entities, ground_truth, and a set of entities predicted by a model, pred for every input text. The true positives TP may be determined by identifying the common elements between pred and ground_truth, essentially by calculating the intersection of those two sets.
For every input, we will then calculate:
precision = len(TP)/ len(pred) ,
recall = len(TP) / len(ground_truth) and
f1 = 2 * precision * recall / (precision + recall)
and at last calculate the macro-average for every metric by summing all of them up and dividing by the variety of datapoints in our test set.
For evaluating the general entity linking performance, we again calculate the identical metrics. On this case, for every input datapoint, we now have a set of tuples, where each tuple is a (entity, mesh_id) pair. The metrics are otherwise calculated the identical way.
Right, let’s kick off things by first defining some helper functions for processing our dataset.
def parse_dataset(file_path):
"""
Parse the BioCreative Dataset.Args:
- file_path (str): Path to the file containing the documents.
Returns:
- list of dict: An inventory where each element is a dictionary representing a document.
"""
documents = []
current_doc = None
with open(file_path, 'r', encoding='utf-8') as file:
for line in file:
line = line.strip()
if not line:
proceed
if "|t|" in line:
if current_doc:
documents.append(current_doc)
id_, title = line.split("|t|", 1)
current_doc = {'id': id_, 'title': title, 'abstract': '', 'annotations': []}
elif "|a|" in line:
_, abstract = line.split("|a|", 1)
current_doc['abstract'] = abstract
else:
parts = line.split("t")
if parts[1] == "CID":
proceed
annotation = {
'text': parts[3],
'type': parts[4],
'identifier': parts[5]
}
current_doc['annotations'].append(annotation)
if current_doc:
documents.append(current_doc)
return documents
def deduplicate_annotations(documents):
"""
Filter documents to make sure annotation consistency.
Args:
- documents (list of dict): The list of documents to be checked.
"""
for doc in documents:
doc["annotations"] = remove_duplicates(doc["annotations"])
def remove_duplicates(dict_list):
"""
Remove duplicate dictionaries from a listing of dictionaries.
Args:
- dict_list (list of dict): An inventory of dictionaries from which duplicates are to be removed.
Returns:
- list of dict: An inventory of dictionaries after removing duplicates.
"""
unique_dicts = []
seen = set()
for d in dict_list:
dict_tuple = tuple(sorted(d.items()))
if dict_tuple not in seen:
seen.add(dict_tuple)
unique_dicts.append(d)
return unique_dicts
We first parse the dataset from the text files provided in the unique dataset. The unique dataset includes the title, abstract, and all entities annotated with their entity type (Disease or Chemical), their substring indices indicating their exact location within the text, together with their MeSH IDs. While processing our dataset, we make just a few simplifications. We disregard the substring indices and the entity type. Furthermore, we de-duplicate annotations that share the identical entity name and MeSH ID. At this stage, we only de-duplicate in a case-sensitive manner, meaning if the identical entity appears in each lower and upper case across the document, we retain each instances in our processing to date.
First, we aim to find out whether the LLM already possesses an understanding of MeSH terminology attributable to its pre-training, and if it may function as a zero-shot entity linker. By zero-shot, we mean the LLM’s capability to directly link entities to their MeSH IDs from biomedical text based on its intrinsic knowledge, without depending on an external KB linker. This hypothesis shouldn’t be entirely unrealistic, considering the supply of data about MeSH online, which makes it possible that the model may need encountered MeSH-related information during its pre-training phase. Nevertheless, even when the LLM was trained with such information, it’s unlikely that this alone would enable the model to perform zero-shot entity linking effectively, attributable to the complexity of biomedical terminology and the precision required for accurate entity linking.
To judge this, we offer the input text to the LLM and directly prompt it to predict the entities and corresponding MeSH IDs. Moreover, we create a few-shot prompt by sampling three data points from the training dataset. It’s important to make clear the excellence in using “zero-shot” and “few-shot” here: “zero-shot” refers back to the LLM as a complete performing entity linking without prior specific training on this task, while “few-shot” refers back to the prompting strategy employed on this context.
To calculate our metrics, we define functions for evaluating the performance:
def calculate_entity_metrics(gt, pred):
"""
Calculate precision, recall, and F1-score for entity recognition.Args:
- gt (list of dict): An inventory of dictionaries representing the bottom truth entities.
Each dictionary must have a key "text" with the entity text.
- pred (list of dict): An inventory of dictionaries representing the anticipated entities.
Much like `gt`, each dictionary must have a key "text".
Returns:
tuple: A tuple containing precision, recall, and F1-score (in that order).
"""
ground_truth_set = set([x["text"].lower() for x in gt])
predicted_set = set([x["text"].lower() for x in pred])
# True positives are predicted items which might be in the bottom truth
true_positives = len(predicted_set.intersection(ground_truth_set))
# Precision calculation
if len(predicted_set) == 0:
precision = 0
else:
precision = true_positives / len(predicted_set)
# Recall calculation
if len(ground_truth_set) == 0:
recall = 0
else:
recall = true_positives / len(ground_truth_set)
# F1-score calculation
if precision + recall == 0:
f1_score = 0
else:
f1_score = 2 * (precision * recall) / (precision + recall)
return precision, recall, f1_score
def calculate_mesh_metrics(gt, pred):
"""
Calculate precision, recall, and F1-score for matching MeSH (Medical Subject Headings) codes.
Args:
- gt (list of dict): Ground truth data
- pred (list of dict): Predicted data
Returns:
tuple: A tuple containing precision, recall, and F1-score (in that order).
"""
ground_truth = []
for item in gt:
mesh_codes = item["identifier"]
if mesh_codes == "-1":
mesh_codes = "None"
mesh_codes_split = mesh_codes.split("|")
for elem in mesh_codes_split:
combined_elem = {"entity": item["text"].lower(), "identifier": elem}
if combined_elem not in ground_truth:
ground_truth.append(combined_elem)
predicted = []
for item in pred:
mesh_codes = item["identifier"]
mesh_codes_split = mesh_codes.strip().split("|")
for elem in mesh_codes_split:
combined_elem = {"entity": item["text"].lower(), "identifier": elem}
if combined_elem not in predicted:
predicted.append(combined_elem)
# True positives are predicted items which might be in the bottom truth
true_positives = len([x for x in predicted if x in ground_truth])
# Precision calculation
if len(predicted) == 0:
precision = 0
else:
precision = true_positives / len(predicted)
# Recall calculation
if len(ground_truth) == 0:
recall = 0
else:
recall = true_positives / len(ground_truth)
# F1-score calculation
if precision + recall == 0:
f1_score = 0
else:
f1_score = 2 * (precision * recall) / (precision + recall)
return precision, recall, f1_score
Let’s now run the model and get our predictions:
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2", torch_dtype=torch.bfloat16).cuda()
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
model.eval()mistral_few_shot_answers = []
for item in tqdm(test_set_subsample):
few_shot_prompt_messages = build_few_shot_prompt(SYSTEM_PROMPT, item, few_shot_example)
input_ids = tokenizer.apply_chat_template(few_shot_prompt_messages, tokenize=True, return_tensors = "pt").cuda()
outputs = model.generate(input_ids = input_ids, max_new_tokens=200, do_sample=False)
# https://github.com/huggingface/transformers/issues/17117#issuecomment-1124497554
gen_text = tokenizer.batch_decode(outputs.detach().cpu().numpy()[:, input_ids.shape[1]:], skip_special_tokens=True)[0]
mistral_few_shot_answers.append(parse_answer(gen_text.strip()))
On the entity extraction level, the LLM performs quite well, considering it has not been explicitly fine-tuned for this task. Nevertheless, its performance as a zero-shot linker is kind of poor, with an overall performance of lower than 1%. This final result is intuitive, though, since the output space for MeSH labels is vast, and it’s a hard task to precisely map entities to a particular MeSH ID.
Retrieval Augmented Generation (RAG) [12] refers to a framework that mixes LLMs with an external KB equipped with a querying function, comparable to a retriever/linker. For every incoming query, the system first retrieves knowledge relevant to the query from the KB using the querying function. It then combines the retrieved knowledge and the query, providing this combined prompt to the LLM to perform the duty. This approach is predicated on the understanding that LLMs may not have all of the crucial knowledge or information to reply an incoming query effectively. Thus, knowledge is injected into the model by querying an external knowledge source.
Using a RAG framework can offer several benefits:
- An existing LLM may be utilized for a brand new domain or task without the necessity for domain-specific fine-tuning, because the relevant information may be queried and provided to the model through a prompt.
- LLMs can sometimes provide incorrect answers (hallucinate) when responding to queries. Employing RAG with LLMs can significantly reduce such hallucinations, because the answers provided by the LLM usually tend to be grounded in facts attributable to the knowledge supplied to it.
Considering that the LLM lacks specific knowledge of MeSH terminologies, we investigate whether a RAG setup could enhance performance. On this approach, for every input paragraph, we utilize a BM-25 retriever to question the KB. For every MeSH ID, we now have access to a general description of the ID and the entity names related to it. After retrieval, we inject this information to the model through the prompt for entity linking.
To research the effect of the variety of retrieved IDs provided as context to the model on the entity linking process, we run this setup by providing top 10, 30 and 50 documents to the model and quantify its performance on entity extraction and MeSH concept identification.
Let’s first define our BM-25 Retriever:
from rank_bm25 import BM25Okapi
from typing import List, Tuple, Dict
from nltk.tokenize import word_tokenize
from tqdm import tqdmclass BM25Retriever:
"""
A category for retrieving documents using the BM25 algorithm.
Attributes:
index (List[int, str]): A dictionary with document IDs as keys and document texts as values.
tokenized_docs (List[List[str]]): Tokenized version of the documents in `processed_index`.
bm25 (BM25Okapi): An instance of the BM25Okapi model from the rank_bm25 package.
"""
def __init__(self, docs_with_ids: Dict[int, str]):
"""
Initializes the BM25Retriever with a dictionary of documents.
Args:
docs_with_ids (List[List[str, str]]): A dictionary with document IDs as keys and document texts as values.
"""
self.index = docs_with_ids
self.tokenized_docs = self._tokenize_docs([x[1] for x in self.index])
self.bm25 = BM25Okapi(self.tokenized_docs)
def _tokenize_docs(self, docs: List[str]) -> List[List[str]]:
"""
Tokenizes the documents using NLTK's word_tokenize.
Args:
docs (List[str]): An inventory of documents to be tokenized.
Returns:
List[List[str]]: An inventory of tokenized documents.
"""
return [word_tokenize(doc.lower()) for doc in docs]
def query(self, query: str, top_n: int = 10) -> List[Tuple[int, float]]:
"""
Queries the BM25 model and retrieves the highest N documents with their scores.
Args:
query (str): The query string.
top_n (int): The variety of top documents to retrieve.
Returns:
List[Tuple[int, float]]: An inventory of tuples, each containing a document ID and its BM25 rating.
"""
tokenized_query = word_tokenize(query.lower())
scores = self.bm25.get_scores(tokenized_query)
doc_scores_with_ids = [(doc_id, scores[i]) for i, (doc_id, _) in enumerate(self.index)]
top_doc_ids_and_scores = sorted(doc_scores_with_ids, key=lambda x: x[1], reverse=True)[:top_n]
return [x[0] for x in top_doc_ids_and_scores]
We now process our KB file and create a BM-25 retriever instance that indexes it. While indexing the KB, we index each ID using a concatenation of their description, aliases and canonical name.
def process_index(index):
"""
Processes the initial document index to mix aliases, canonical names, and definitions right into a single text index.Args:
- index (Dict): The MeSH knowledge base
Returns:
List[List[int, str]]: A dictionary with document IDs as keys and combined text indices as values.
"""
processed_index = []
for key, value in tqdm(index.items()):
assert(type(value["aliases"]) != list)
aliases_text = " ".join(value["aliases"].split(","))
text_index = (aliases_text + " " + value.get("canonical_name", "")).strip()
if "definition" in value:
text_index += " " + value["definition"]
processed_index.append([value["concept_id"], text_index])
return processed_index
mesh_data = read_jsonl_file("mesh_2020.jsonl")
process_mesh_kb(mesh_data)
mesh_data_kb = {x["concept_id"]:x for x in mesh_data}
mesh_data_dict = process_index({x["concept_id"]:x for x in mesh_data})
retriever = BM25Retriever(mesh_data_dict)
mistral_rag_answers = {10:[], 30:[], 50:[]}for k in [10,30,50]:
for item in tqdm(test_set_subsample):
relevant_mesh_ids = retriever.query(item["title"] + " " + item["abstract"], top_n = k)
relevant_contexts = [mesh_data_kb[x] for x in relevant_mesh_ids]
rag_prompt = build_rag_prompt(SYSTEM_RAG_PROMPT, item, relevant_contexts)
input_ids = tokenizer.apply_chat_template(rag_prompt, tokenize=True, return_tensors = "pt").cuda()
outputs = model.generate(input_ids = input_ids, max_new_tokens=200, do_sample=False)
gen_text = tokenizer.batch_decode(outputs.detach().cpu().numpy()[:, input_ids.shape[1]:], skip_special_tokens=True)[0]
mistral_rag_answers[k].append(parse_answer(gen_text.strip()))
entity_scores_at_k = {}
mesh_scores_at_k = {}for key, value in mistral_rag_answers.items():
entity_scores = [calculate_entity_metrics(gt["annotations"],pred) for gt, pred in zip(test_set_subsample, value)]
macro_precision_entity = sum([x[0] for x in entity_scores]) / len(entity_scores)
macro_recall_entity = sum([x[1] for x in entity_scores]) / len(entity_scores)
macro_f1_entity = sum([x[2] for x in entity_scores]) / len(entity_scores)
entity_scores_at_k[key] = {"macro-precision": macro_precision_entity, "macro-recall": macro_recall_entity, "macro-f1": macro_f1_entity}
mesh_scores = [calculate_mesh_metrics(gt["annotations"],pred) for gt, pred in zip(test_set_subsample, value)]
macro_precision_mesh = sum([x[0] for x in mesh_scores]) / len(mesh_scores)
macro_recall_mesh = sum([x[1] for x in mesh_scores]) / len(mesh_scores)
macro_f1_mesh = sum([x[2] for x in mesh_scores]) / len(mesh_scores)
mesh_scores_at_k[key] = {"macro-precision": macro_precision_mesh, "macro-recall": macro_recall_mesh, "macro-f1": macro_f1_mesh}
Usually, the RAG setup improves the general MeSH Identification process, in comparison with the unique zero-shot setup. But what’s the impact of the variety of documents provided as information to the model? We plot the scores as a function of the variety of retrieved IDs provided to the model as context.