Equall.ai an AI company has very recently introduced SaulLM-7B, the primary open-source large language model tailored explicitly for the legal domain.
The sphere of law presents a novel challenge for language models as a consequence of its intricate syntax, specialized vocabulary, and domain-specific nuances. Legal texts, akin to contracts, court decisions, and statutes, are characterised by a definite linguistic complexity that requires a deep understanding of the legal context and terminology.
SaulLM-7B is a 7 billion parameter language model crafted to beat the legal language barrier. The model’s development process involves two critical stages: legal continued pretraining and legal instruction fine-tuning.
- Legal Continued Pretraining: The muse of SaulLM-7B is built upon the Mistral 7B architecture, a robust open-source language model. Nevertheless, the team at Equall.ai recognized the necessity for specialised training to reinforce the model’s legal capabilities. To attain this, they curated an intensive corpus of legal texts spanning over 30 billion tokens from diverse jurisdictions, including america, Canada, the UK, Europe, and Australia.
By exposing the model to this vast and diverse legal dataset in the course of the pretraining phase, SaulLM-7B developed a deep understanding of the nuances and complexities of legal language. This approach allowed the model to capture the unique linguistic patterns, terminologies, and contexts prevalent within the legal domain, setting the stage for its exceptional performance in legal tasks.
- Legal Instruction Fantastic-tuning: While pretraining on legal data is crucial, it is usually not sufficient to enable seamless interaction and task completion for language models. To deal with this challenge, the team at Equall.ai employed a novel instructional fine-tuning method that leverages legal datasets to further refine SaulLM-7B’s capabilities.
The instruction fine-tuning process involved two key components: generic instructions and legal instructions.
When evaluated on the LegalBench-Instruct benchmark, a comprehensive suite of legal tasks, SaulLM-7B-Instruct (the instruction-tuned variant) established a brand new state-of-the-art, outperforming the very best open-source instruct model by a big 11% relative improvement.
Furthermore, a granular evaluation of SaulLM-7B-Instruct’s performance revealed its superior capabilities across 4 core legal abilities: issue spotting, rule recall, interpretation, and rhetoric understanding. These areas demand a deep comprehension of legal expertise, and SaulLM-7B-Instruct’s dominance in these domains is a testament to the ability of its specialized training.
The implications of SaulLM-7B’s success extend far beyond academic benchmarks. By bridging the gap between natural language processing and the legal domain, this pioneering model has the potential to revolutionize the way in which legal professionals navigate and interpret complex legal material.
Biomedical and Healthcare
GatorTron, Codex-Med, Galactica, and Med-PaLM LLM