The highest open source Large Language Models available for industrial use are as follows.
- Llama – 2
Meta released Llama 2, a set of pretrained and refined LLMs, together with Llama 2-Chat, a version of Llama 2. These models are scalable as much as 70 billion parameters. It was discovered after extensive testing on safety and helpfulness-focused benchmarks that Llama 2-Chat models perform higher than current open-source models typically. Human evaluations have shown that they align well with several closed-source models.
The researchers have even taken just a few steps to ensure the safety of those models. This includes annotating data, especially for safety, conducting red-teaming exercises, fine-tuning models with an emphasis on questions of safety, and iteratively and repeatedly reviewing the models.
Variants of Llama 2 with 7 billion, 13 billion, and 70 billion parameters have also been released. Llama 2-Chat, optimized for dialogue scenarios, has also been released in variants with the identical parameter scales.
Project: https://huggingface.co/meta-llama
Paper: https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/
- Falcon
Researchers from Technology Innovation Institute, Abu Dhabi introduced the Falcon series, which incorporates models with 7 billion, 40 billion, and 180 billion parameters. These models, that are intended to be causal decoder-only models, were trained on a high-quality, varied corpus that was mostly obtained from online data. Falcon-180B, the most important model within the series, is the one publicly available pretraining run ever, having been trained on a dataset of greater than 3.5 trillion text tokens.
The researchers discovered that Falcon-180B shows great advancements over other models, including PaLM or Chinchilla. It outperforms models which might be being developed concurrently, comparable to LLaMA 2 or Inflection-1. Falcon-180B achieves performance near PaLM-2-Large, which is noteworthy given its lower pretraining and inference costs. With this rating, Falcon-180B joins GPT-4 and PaLM-2-Large because the leading language models on this planet.
Project: https://huggingface.co/tiiuae/falcon-180B
Project: https://arxiv.org/pdf/2311.16867.pdf
- Dolly 2.0
Researchers from Databricks created the LLM Dolly-v2-12b, which has been designed for industrial use and was created on the Databricks Machine Learning platform. Based on pythia-12b as a base, it’s trained using roughly 15,000 instruction/response pairs (named databricks-dolly-15k) that were produced by Databricks personnel. The several capability areas covered by these instruction/response pairings are brainstorming, classification, closed question-answering, generation, information extraction, open question-answering, and summarising, as stated within the InstructGPT document.
Dolly-v2 can also be available in smaller model sizes for various use cases. Dolly-v2-7b has 6.9 billion parameters and is predicated on pythia-6.9b.
Dolly-v2-3b has 2.8 billion parameters and is predicated on pythia-2.8b.
HF Project: https://huggingface.co/databricks/dolly-v2-12b
Github: https://github.com/databrickslabs/dolly#getting-started-with-response-generation
- MPT
Transformer-based language models have made great progress with the discharge of MosaicML’s MPT-7B. MPT-7B was trained from the start and has been exposed to a large corpus of 1 trillion tokens, which incorporates each text and code.
The efficiency with which MPT-7B was trained is amazing. In only 9.5 days, the total training process, which was carried out with none human involvement, was finished. MPT -7 B was trained at an exceptionally low price, given the scale and difficulty of the project. The training procedure, which made use of MosaicML’s cutting-edge infrastructure, cost about $200,000.
HF Project: https://huggingface.co/mosaicml/mpt-7b
Github: https://github.com/mosaicml/llm-foundry/
- FLAN – T5
Google introduced FLAN – T5, an enhanced version of T5 that has been finetuned in a mix of tasks. Flan-T5 checkpoints reveal robust few-shot performance even compared to significantly larger models like PaLM 62B. With FLAN – T5, The team discussed instruction fine-tuning as a flexible approach for improving language model performance across various tasks and evaluation metrics.
HF Project: https://huggingface.co/google/flan-t5-base
Paper: https://arxiv.org/pdf/2210.11416.pdf
- GPT-NeoX-20B
EleutherAI presented GPT-NeoX-20B, an enormous autoregressive language model with 20 billion parameters. GPT-NeoX-20B’s performance is assessed on a wide range of tasks that include knowledge-based skills, mathematical reasoning, and language comprehension.
The evaluation’s key conclusion is that GPT-NeoX-20B performs admirably as a few-shot reasoner, even when given little or no information. GPT-NeoX-20B performs noticeably higher than comparable-sized devices like GPT-3 and FairSeq, especially in five-shot evaluations.
HF Project: https://huggingface.co/EleutherAI/gpt-neox-20b
Paper: https://arxiv.org/pdf/2204.06745.pdf
- Open Pre-trained Transformers (OPT)
Since LLM models are steadily trained over a whole bunch of 1000’s of computing days, they sometimes need substantial computing resources. This makes replication extremely difficult for researchers that lack substantial funding. Complete access to the model weights is steadily restricted, stopping in-depth research and evaluation, even in cases where these models are made available through APIs.
To deal with these issues, Meta researchers presented Open Pre-trained Transformers (OPT), a set of pre-trained transformers which might be limited to decoders and canopy a broad range of parameter values, from 125 million to 175 billion. OPT’s principal goal is to ddemocratizeaccess to cutting-edge language models by making these models fully and ethically available to academics.
OPT-175B, the flagship model within the OPT suite, is shown by the researchers to perform similarly to GPT-3. But what really distinguishes OPT-175B is that, compared to standard large-scale language model training techniques, it requires only one/seventh of the environmental effect during development.
HF Project: https://huggingface.co/facebook/opt-350m
Paper: https://arxiv.org/pdf/2205.01068.pdf
- BLOOM
Researchers from BigScience developed BLOOM, a major 176 billion-parameter open-access language model. Since BLOOM is a decoder-only Transformer language model, it is especially good at producing text sequences in response to input cues. The ROOTS corpus, an intensive dataset with content from a whole bunch of sources covering 46 natural languages and 13 programming languages for a complete of 59 languages, served as its training ground. Due to great amount of coaching data, BLOOM is ready to understand and produce text in a wide range of linguistic contexts.
Paper: https://arxiv.org/pdf/2211.05100.pdf
HF Project: https://huggingface.co/bigscience/bloom
- Baichuan
Probably the most recent version of the extensive open-source language models created by Baichuan Intelligence Inc. known as Baichuan 2. With 2.6 trillion tokens in its fastidiously chosen corpus, this sophisticated model is taught to capture a big selection of linguistic nuances and patterns. Notably, Baichuan 2 has established latest norms for models of comparable size by exhibiting exceptional performance across credible benchmarks in each Chinese and English.
Baichuan 2 has been released in various versions, each designed for a selected use case. Options are offered in parameter mixtures of seven billion and 13 billion for the Base model. Baichuan 2 provides Chat models in matching variants with 7 billion and 13 billion parameters, that are tailored for dialogue settings. Furthermore, a 4-bit quantized version of the Chat model is obtainable for increased efficiency, which lowers processing needs without sacrificing performance.
HF Project:https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat#Introduction
- BERT
Google introduced BERT (Bidirectional Encoder Representations from Transformers). BERT is specially developed to pre-train deep bidirectional representations from unlabeled text, unlike earlier language models. Which means that BERT can capture a more thorough grasp of linguistic nuances since it concurrently takes into consideration the left and right context in every layer of its architecture.
BERT’s conceptual simplicity and exceptional empirical power are two of its principal advantages. It acquires wealthy contextual embeddings by extensive pretraining on text data, which could also be refined with little effort to supply highly efficient models for a big selection of natural language processing applications. Adding only one extra output layer is often all that’s required for this fine-tuning process, which leaves BERT extremely flexible and adaptable to a big selection of applications without requiring significant task-specific architecture changes.
BERT performs well on eleven distinct natural language processing tasks. It shows notable gains in SQuAD question-answering performance, MultiNLI accuracy, and GLUE rating. For instance, BERT increases the GLUE rating to 80.5%, which is a major 7.7% absolute improvement.
Github: https://github.com/google-research/bert
Paper: https://arxiv.org/pdf/1810.04805.pdf
HF Project: https://huggingface.co/google-bert/bert-base-cased
- Vicuna
LMSYS presented Vicuna-13B, an open-source chatbot that was created through the use of user-shared conversations gathered from ShareGPT to fine-tune the LLaMA model. Vicuna-13B offers consumers superior conversational capabilities and is an enormous leap in chatbot technology.
Within the initial assessment, Vicuna-13B’s performance was judged using the GPT-4. The evaluation results showed that Vicuna-13B outperforms other well-known chatbot models like OpenAI ChatGPT and Google Bard, with a high quality level that surpasses 90%. Vicuna-13B performs higher and is more efficient in producing high-quality responses than other models, comparable to LLaMA and Stanford Alpaca, in greater than 90% of the cases. Vicuna-13B is an amazing device by way of cost-effectiveness. Vicuna-13B might be developed for about $300 in training, which makes it an economical solution.
HF Project: https://huggingface.co/lmsys/vicuna-13b-delta-v1.1
- Mistral
Mistral 7B v0.1 is a cutting-edge 7-billion-parameter language model that has been developed for remarkable effectiveness and performance. Mistral 7B breaks all previous records, outperforming Llama 2 13B in every benchmark and even Llama 1 34B in crucial domains like logic, math, and coding.
State-of-the-art methods like grouped-query attention (GQA) have been used to speed up inference and sliding window attention (SWA) to efficiently handle sequences with different lengths while reducing computing overhead. A customized version, Mistral 7B — Instruct, has also been provided and optimized to perform exceptionally well in activities requiring following instructions.
HF Project: https://huggingface.co/mistralai/Mistral-7B-v0.1
Paper: https://arxiv.org/pdf/2310.06825.pdf
- Gemma
Gemma is a series of state-of-the-art open models that Google has built using the identical technology and research because the Gemini models. These English-language, decoder-only large language models, dubbed Gemma, are intended for text-to-text applications. They’ve three variations: instruction-tuned, pre-trained, and open-weighted. Gemma models do exceptionally well in a wide range of text creation tasks, comparable to summarising, reasoning, and answering questions.
Gemma is exclusive in that it’s lightweight, which makes it ideal for deployment in contexts with limited resources, like desktops, laptops, or personal cloud infrastructure.
HF Project: https://huggingface.co/google/gemma-2b-it
- Phi-2
Microsoft introduced Phi-2, which is a Transformer model with 2.7 billion parameters. It was trained using a mixture of information sources just like Phi-1.5. It also integrates a brand new data source, which consists of NLP synthetic texts and filtered web sites which might be considered instructional and protected. Examining Phi-2 against benchmarks measuring logical pondering, language comprehension, and customary sense showed that it performed almost on the state-of-the-art level amongst models with lower than 13 billion parameters.
HF Project: https://huggingface.co/microsoft/phi-2
- StarCoder2
StarCoder2 was introduced by the BigCode project; a cooperative endeavor centered on the conscientious creation of Large Language Models for Code (Code LLMs). The Stack v2 is predicated on the digital commons of Software Heritage’s (SWH) source code archive, which covers 619 computer languages. A fastidiously chosen set of additional high-quality data sources, comparable to code documentation, Kaggle notebooks, and GitHub pull requests, makes the training set 4 times larger than the initial StarCoder dataset.
StarCoder2 models with 3B, 7B, and 15B parameters are extensively tested on an intensive collection of Code LLM benchmarks after being trained on 3.3 to 4.3 trillion tokens. The outcomes show that StarCoder2-3B performs higher on most benchmarks than similar-sized Code LLMs and even beats StarCoderBase-15B. StarCoder2-15B performs on par with or higher than CodeLlama-34B, a model twice its size, and greatly beats devices of comparable size.
Paper: https://arxiv.org/abs/2402.19173
HF Project: https://huggingface.co/bigcode
- Mixtral
Mistral AI released Mixtral 8x7B, a sparse mixture of expert models (SMoE) with open weights and an Apache 2.0 license. Mixtral sets itself apart by delivering six times faster inference speeds and outperforming Llama 2 70B on most benchmarks. It offers the best cost/performance trade-offs within the industry and is the highest open-weight model with a permissive license. Mixtral outperforms GPT3.5 on a wide range of common benchmarks, reaffirming its standing as the highest model in the sphere.
Mixtral supports English, French, Italian, German, and Spanish, and it handles contexts of as much as 32k tokens with ease. Its usefulness is further increased by the proven fact that it demonstrates excellent proficiency in code-generating jobs. Mixtral can be optimized to change into an instruction-following model, as demonstrated by its high 8.3 MT-Bench evaluation rating.
HF Project: https://huggingface.co/mistralai/Mixtral-8x7B-v0.1
Blog: https://mistral.ai/news/mixtral-of-experts/
Tanya Malhotra is a final yr undergrad from the University of Petroleum & Energy Studies, Dehradun, pursuing BTech in Computer Science Engineering with a specialization in Artificial Intelligence and Machine Learning.
She is a Data Science enthusiast with good analytical and significant pondering, together with an ardent interest in acquiring latest skills, leading groups, and managing work in an organized manner.