Prior to now few years, the AI and ML industry has witnessed a meteoric rise in the event & application of the NLP systems as researchers have been capable of implement NLP practices in highly flexible and task-agnostic ways for downstream transferring tasks. 

Initially, it was the single-layer representations that used word vectors, and were then fed to the task-specific architecture. Next, it was the RNN architecture that used multi-layer representations & contextual state to form higher representations. And most recently, we’ve the transfer language models or pre-trained recurrent models which have entirely removed the necessity for task-specific architectures by fine-tuning these networks. 

The transfer language models have proved to be a serious turning point within the NLP industry as they’ve resulted in tremendous progress on difficult tasks like answering questions, reading comprehensions or blocks of text, textual entailment, and way more. 

Nevertheless, despite their benefits, transfer language models have a serious limitation as they require task-specific finetuning or task-specific dataset to realize the specified performance on a task. Moreover, transfer language models also require developers to finetune the datasets to lots of of hundreds of examples specific to a selected task. 

It goes without saying that removing the requirement for task-specific dataset, and task-specific finetuning will likely be highly desirable, and useful for the NLP industry for various reasons. 

Issues with Existing Pre-Trained Transfer Language Models or Recurrent Models

• Limiting the Practicality & Applicability

Firstly, the requirement of a big dataset with labeled data for every task limits the applicability & practicality of the language models. Language models find their applications in a wide selection of tasks starting from generating a brief story, to correcting grammatical errors, to generating examples on an idea. At times, it’s a difficult task to gather a big supervised dataset with labeled data, especially when the method must be repeated for each individual task. 

• Exploiting Spurious Correlations in Training Data

Limitations & narrowness of the training distribution coupled with expressiveness of the model can lead to a fundamental growth in potential to use spurious correlations in training data. The potential to use the training data can lead to problems throughout the fine-tuning and pre-training paradigm since the transfer language models are designed in a approach to absorb a considerable amount of information during pre-training. 

Moreover, work on prior models have indicated that enormous models don’t lead to higher out of distribution each & each time. Moreover, it’s also been indicated that generalization achieved under such a paradigm can lead to poor performance primarily since the model is very specific to the training data, and can’t perform well on situations beyond the scope of the training data. 

• Comparison with Human Learning

Finally when put next to transfer language models, humans don’t require a big training dataset with regards to learning a majority of language tasks. Most frequently, a transient directive in an individual’s natural language or a small demonstration of the language task is adequate for a human to grasp and perform a language task with a certain level of competitiveness. 

Human’s ability to adapt has quite a few practical benefits because it allows them to either switch between different skill sets or mix them together to higher perform during a dialect, something that’s beyond the capabilities of the present NLP systems. 

Tackling the Issues with Meta Learning & GPT-3

A possible solution to the above challenges is the usage of meta learning, an idea in modern ML that enables a model to develop a bigger & broader set of skills & ability to acknowledge patterns while training, after which uses these learned abilities during interference to adapt rapidly, or recognize the required task. 

Meta Learning is being implemented in language model architecture via a method called “in-context learning” that uses text input of a pre-trained language model as a task specification. In the method, the model conditions on a natural language instruction, and might even use just a few demonstrations, and the model is then expected to finish the remaining of the duty by predicting the subsequent steps. 

The one major issue with Meta Learning is that even though it has shown positive potential, it’s still inferior to the fine-tuning approach in natural language architecture, and it needs further improvement to be able to grow to be a practical method for overcoming language tasks. 

Along with meta learning, one other method that’s gaining popularity is increasing the capability of transformer language models. Prior to now few years, transfer models have witnessed a considerable increase of their capability with the RNSS18 model with 100 million parameters, the DCLT18 model with 300 million parameters, the RWC19 model with 1.5 billion parameters, the SSP19 model with 8 billion parameters, the RSR19 model with 11 billion parameters, and the TUR20 model with 17 billion parameters. 

Increasing the capability of the model or increasing the parameters has historically resulted in improvements in text synthesis, and there’s been a sign that log loss, that correlates with downstream tasks also follows a smooth trend of improving with the size. 

That brings us to the GPT-3 model that has over 175 billion parameters, and when it was launched, it was the transfer language model with the best capability. Let’s now talk in regards to the GPT-3 model. 

An Introduction to the GPT-3 Model

The GPT-3 is an autoaggressive language model with over 175 billion parameters that was released by OpenAI in 2020. GPT-3 can be classified as a big language model that similar to its predecessor the GPT-2 model is a decoder-only deep learning transformer model that uses convolution-based architecture to generate textual data. 

The GPT-3 model measures its own context-learning abilities, and the GPT-3 model is evaluated on over two dozen NLP datasets and multiple novel tasks. For each individual task, the GPT-3 model is evaluated under three conditions,

• Few Shot Learning or In-Context Learning: In few shot learning, the GPT-3 model allows as many distributions that may fit well into the model’s context window. 
• One Shot Learning: In a single shot learning, the model allows just one demonstration. 
• Zero Shot Learning: In zero shot learning, there aren’t any demonstrations, and there’s only an instruction in natural language that’s fed to the model. 

Broadly speaking, the GPT-3 model achieves desired performance in zero-shot, and one-shot settings, and within the few-shot setting, it outperforms the state-of-the-art transfer models more often than not. Moreover, the GPT-3 model performs well in one-shot, and zero-shot settings at natural language tasks designed to check on the fly reasoning, or requires rapid attention like using novel words after a sentence, or unscrambling words, or performing arithmetic operations. However, when operated in a few-shot setting, the GPT-3 model generates synthetic news articles that resemble human writing when passed through human evaluators. 

GPT-3 Model: Approach

The GPT-3 model uses a traditional pre-training approach that comprises model, data, and training, and it resembles the pre-training process followed by the RWC-19 transfer language model. The GPT-3 model scales up the model size, the dataset size, diversity of the dataset, and increases the length of the training period. 

The model also uses an in-context learning approach that when again resembles the RWC-19 model’s approach, but tweaks things up a bit by systematically exploring different settings for learning patterns throughout the context of the dataset. 

So, let’s start by exploring these settings, and evaluate how the GTP-3 model performs on different settings. 

Superb Tuning

Superb-tuning the model has been the standard approach in transfer language models, and this approach involves updating the weights of a pre-trained model by training the model on a supervised dataset that’s specific to the specified task, and lots of of hundreds of labeled examples are used throughout the process. 

The fine-tuning approach is useful since it returns strong performance across quite a few benchmarks. However, the fundamental limitation of using the fine-tuning approach is that it requires a brand new & large dataset for each individual task, has the potential to use spurious features of the training dataset, can potentially lead to unfair comparison with human performance, and poor generalization for out-of-distribution. 

The present scope of the GPT-3 model doesn’t implement the fine-tuning approach due to its task-agnostic performance, although fine-tuning will be applied to the GPT-3 model in the longer term. 

Few Shot

Few Shot is a term that refers back to the setting where the GPT-3 model is given just a few demonstrations of the duty during interference as conditioning, however the weights of the model aren’t updated. Within the few shot settings, the dataset typically has an example with a context, and a desired completion (for instance, a French sentence, and its English translation). The few shot setting gives the model K examples of context, and completion, and it then provides the model with one final context, and expects the model to supply the completion. 

The foremost advantage of using the few shot setting is that it significantly reduces the necessity for task-specific data, and in addition reduces the potential to learn a narrow distribution from a big dataset that is fine-tuned narrowly. However, the most important drawback of using few shot learning is that the outcomes delivered within the few shot setting aren’t up to speed, and significantly poor when put next to other cutting-edge models which might be fine-tuned. 

One Shot

Within the one shot setting, the model is provided only with a single demonstration, and the remaining is analogous to the few shot setting. The rationale why one shot setting is relevant in transfer language models is because out of all of the three settings, one shot is the one which resembles the best way through which tasks are communicated to humans one of the best. It’s because in a lot of the tasks, it is common to present one demonstration of the duty otherwise it is likely to be obscure the context of the duty. 

Zero Shot

Within the zero shot setting, there aren’t any demonstrations, and the model is given a natural language instruction that describes the duty. The zero shot method is the one that gives maximum convenience, is strong, and in addition avoids spurious correlations, however it’s also probably the most difficult of all of the three settings. Its because in some cases, it’s difficult even for us humans to determine the context of a task without seeing an illustration first. 

Regardless, for some tasks, zero-shot setting is the one which resembles how humans perform natural language tasks the closest. 

The above figure compares the few shot, the one shot, and the zero shot setting when performing a natural language task of taking an English sentence, and translating it into French. 

GPT-3: Model Architecture

The GPT-3 model uses the identical architecture because the one utilized in the GPT-2 model, and it includes pre-normalization, modified initialization, and reversible tokenization techniques as they were used on the GPT-model excluding using an alternate strategy for locally banded sparse attention patterns, and alternating dense layers within the transformer layers, just like Sparse Transformer. 

To check the dependency of the model’s performance on the model size, the developers have trained 8 different model sizes that range over three different orders of magnitude from 125 million to over 175 billion parameters, the last one in all them being called the GPT-3 model. Prior work related to LLM models have indicated that Scaling of validation loss with a sufficient amount of coaching data needs to be an approximate smooth power law as a function of size. Training models of various sizes allows developers to check the hypothesis for each downstream language tasks in addition to for validation loss. 

The above figure compares the dimensions & architecture of the 8 different models used for development of GPT-3. Here, n(params) defines the overall variety of trainable patterns, n(layers) defines the overall variety of layers within the model, d(model) defines the variety of units in each layer of the bottleneck, and d(head) defines the scale of every attention head. The context window for every model is identical with 2048 tokens. 

Moreover, to reduce the transfer of knowledge between the nodes, the model is partitioned across the GPUs along the depth & the width of the scale. The architectural parameters for every model have been chosen on the premise of computational efficiency, & load-balancing to maximise precision within the layout of models across GPUs. 

Training Datasets

Typically, the massive language models use datasets which have expanded significantly with recent developments, they usually culminate within the Common Crawl dataset that consists of over a trillion different words. The scale of the dataset is adequate enough to coach the GPT-3 model without updating on the identical sequence multiple times. Nevertheless, studies & performance evaluation indicate that frivolously filtered versions or unfiltered versions of the Common Crawl dataset have low quality when put next to more curated dataset. 

To tackle the problem of the common quality of the dataset, developers took 3 steps to spice up the standard of the dataset. 

1. Developers downloaded & filtered a version of the Common Crawl dataset based on a variety just like high-quality reference corpora. 
2. Developers performed fuzzy duplication on the document level across the dataset in an try and preserve the integrity of their held-out validation set as an efficient measurement of overfitting, and in addition to stop redundancy. 
3. Developers also added high-quality reference corpora to the training data to enhance the Common Crawl dataset, and to further increase the variety of the dataset. 

The next figure shows the ultimate proportion or mixture of the datasets used for training the GPT-3 model. The Common Crawl data consisted of over 45 TB of plaintext before filtering that was reduced to 570 GB of knowledge after filtering, a rough such as over 400 billion byte-pair encoded tokens. It’s value noting that datasets within the training which might be viewed as higher-quality are sampled with more frequency as a substitute of sampling the dataset proportion to their size. Consequently, datasets like Books2 & Common Crawl are sampled lower than one time during training, whereas the opposite datasets are sampled multiple times. It allows the model to just accept a small amount of overfitting in exchange for training on training data with the next quality. 

A big concern with large language models which might be pre-trained on a considerable amount of web data with the capability to memorize & learn a considerable amount of content is the potential contamination of downstream tasks by having their development or test sets seen throughout the pre-training process. To cut back such potential contamination, the developers looked for any overlaps with the test & development sets of the benchmarks studied for GPT-3, and attempted to remove these overlaps. 

The above image shows the overall compute used throughout the training of the GPT-3 model. The model uses Scaling Laws for Neural Language Models to coach much larger models on fewer tokens than typical. Consequently, each GPT-3 and RoBERTa-Large model, that’s 10x smaller than the GPT-3 model took nearly 50 petaflops/day of compute throughout the pre-training process. 

Evaluation

For the few shot learning, the model evaluates each example present within the evaluation data set by drawing K examples randomly from that task’s training dataset as conditioning, and delimits it by 1 or 2 newlines depending upon the duty. For Storycloze, and LAMBADA, the model draws conditioning examples from the event set & evaluates it on the test set due to unavailability of a supervised training set. For Winograd, there exists just one dataset, and so the conditioning samples are drawn directly from it. 

K will be any value starting from 0 to the utmost amount allowed by the model’s context window which is next = 2048 for all of the models, and it typically suits about 10 to 100 examples. Larger values of K often lead to higher results, but not all the time which is why when the model has a test set, and a separate development set available, the model experiments on just a few values of K on the event set, and based on the outcomes, it runs one of the best value on the test set. 

Moreover, on the tasks that require choosing an accurate completion from multiple options, the developers provide K examples of correction plus context completion, and follow it up by providing one example of context only, and the tasks are then compared on the premise of LM likelihood of every completion. For tasks that require binary classification, the models often give options more semantically, and with more meaningful names, after which treats the duty as multiple alternative, and sometimes also frames the duty just like what is completed by the RSR model & architecture. 

For the tasks that require free-form completion, the model uses beam search with an identical parameters as utilized in the RSR framework, with a beam of length 4, and a penalty of 0.6. The model is then scored using either the F1 similarity rating, exact match, or BLEU, depending on the usual for the dataset. 

Results

The above figure displays the training curves for the 8 models utilized in the GPT-3 model architecture, as described within the previous sections. Much like the outcomes from the KMH language model, the performance of the GPT-3 model follows a correct law when using training compute effectively. There’s a slight difference from the law only when the trend is prolonged by two more orders of magnitude. It’d occur to folks that the improvements in cross-entropy loss is likely to be a results of modeling spurious details of the training corpus. Nevertheless, the improvements within the cross-entropy loss result in consistent gains in the general performance across a broad spectrum of a wide range of NLP tasks. 

Before evaluating the 8 different models on a big selection of coaching data, the datasets are grouped into 8 different categories that represent similar tasks. These categories are

1. Evaluation on traditional language modeling tasks, and tasks that resemble language modeling like Cloze tasks, or sentence/paragraph completion tasks. 
2. Evaluation on “closed-book” query answering tasks. 
3. Evaluating the model’s ability to translate between languages (especially one-shot and few-shot)
4. Evaluating the model’s performance on Winograd Schema-like tasks. 
5. Evaluating on datasets that involve commonsense reasoning or query answering. 
6. Evaluating on reading comprehension tasks. 
7. Evaluating on the SuperGLUE benchmark suite. 
8. Exploring NLI. 

Language Modeling, Completion, and Cloze Tasks

On this section, the GPT-3 model’s performance is evaluated on the normal language modeling tasks in addition to tasks that require the prediction of a single word of interest, or completing a paragraph or a sentence, or completing a chunk of a text. Let’s discuss them in short detail. 

Language Modeling

The GPT-3 model calculates the zero-shot perplexity on the PTB or the Penn Tree Bank dataset. The model omits Wikipedia-related tasks since it’s already included within the model’s training data, and the one billion word benchmark can be omitted since it causes a big amount of friction of the dataset being throughout the training data. Nevertheless, the PTB dataset tackles these issues because it might probably predate the fashionable web. The most important model within the GPT-3 model architecture ets latest SOTA on the PTB dataset by a noteworthy margin of 15 points, and achieves a perplexity of 20.50. 

LAMBADA

The LAMBADA dataset is used to check the modeling of the model on long-range dependencies in paragraphs or texts. It implies that the model is asked to predict the last word of a sentence after reading the paragraph for the context. Moreover, the continual scaling of the language models yields diminishing returns on the benchmark. 

The GPT-3 model achieves 76% accuracy on LAMBADA, and has a gain of over 8% over previous best models. Moreover, the LAMBADA model demonstrates the pliability of few-shot learning because it addressed the issue in a way that happens classically with the dataset. The completion of a sentence in LAMBADA is generally the last word of the sentence, but as a language model cannot know that, it assigns a probability not only to the proper ending, but in addition to other continuations within the paragraph. 

Moreover, when the examples fed to the GPT-3 model are modified in a certain way, the model returns an accuracy of over 86%, a rise of over 18% over previous models. Moreover, the outcomes also indicated that the performance of the model in a few-shot setting increases proportionally with the rise in model size. Although this strategy reduces the smallest model within the GPT-3 architecture by 20%, it enhances the accuracy of the first GPT-3 model with 175 billion parameters by 10%. 

Closed Book Query Answering

Closed Book Query Answering is an try and measure the GPT-3 model’s ability to reply questions based on broad factual knowledge. Because such questions often have a high amount of possible queries, the duty is often achieved using an information retrieval system that enables the model to search out relevant text together with the model that learns to generate a response to a solution given the retrieved text, and the query. 

The above image compares the result for the GPT-3 model compared with different models, and running on different datasets. On the TriviaQA dataset, the model achieves an accuracy rating of 64.3% within the zero-shot setting, while it achieves an accuracy rating of 68%, and 71.2% in one-shot, and few-shot settings respectively. 

It could evidently be seen that the GPT-3 model in zero-shot setting outperforms the fine-tuned T5-11B model by over 14%. 

The above figure shows the performance of the GPT-3 model grows easily with a rise within the model size. The performance suggests that the language models proceed to learn from the dataset as their capability increases. 

Final Thoughts

It might be secure to say that GPT-3 was a revolutionizing phase within the LLM industry as GPT-3 helped in pushing the bounds of what a language model could do. It was the developments made, and obstacles overcome by GPT-3 that paved the best way for probably the most advanced, and accurate large language model till date, the GPT-4.