In large language models (LLMs), the landscape of pretraining data is a wealthy mix of diverse sources. It spans from common English to less common languages, including casual conversations and scholarly texts, and even extends to modalities like images and speeches. Inside this mix, the info interact in complex ways, sometimes aligning well, diverging, and infrequently conflicting. The challenge lies in fine-tuning the proportions of this mix, leveraging the strengths of every domain while minimizing potential conflicts through which the resulting models gain enhanced capabilities, a testament to the useful insights gained from extensive real-world use.

Despite being elusive in determining an excellent training data mixture, most existing practices tune the mixture through heuristics to upsample a proportion of high-quality or underrepresented data without disclosing the concrete criteria intimately. Predicting whether these data strategies are effective before ending the training run is tough. Inspired by advancements in scaling laws that show model losses on a given set of evaluation data are quantitatively predictable for a big selection of variables, there’s an exciting prospect. If this principle also applies to mixture proportions, they might estimate the performance of the resulting model before even commencing training.

Researchers from Fudan University and Shanghai AI Laboratory introduced data mixing law and prediction pipeline, which solves the issue of accurately predicting the validation loss for a combination of coaching domains under a hard and fast model size and amount of coaching data. Researchers carried out a Pilot Study on Domain Losses under Two-domain Mixtures to predict model losses regarding data mixtures. That is achieved by training 70M and 160M language models on the combination of Github and Pile-CC subsets from the Pile dataset with five different mixture proportions for Github. All of the models are trained with a batch size of 1M tokens for 30k steps, which is 30B tokens.

This paper addresses various challenges in optimizing data mixtures. A few of them are (a) Discovery of quantitative predictability of model performance regarding data mixture, summarizing this right into a functional relationship, namely the info mixing laws. (b) Proposed a pipeline to predict the model performance of large-scale training on different mixture proportions but only experiments on small models with few training data through nested scaling laws of coaching steps, model sizes, and data mixing laws. (c) Experimental verification of the reliability of information mixing laws and prediction pipeline, showing its effectiveness in optimizing model performance, balancing model capabilities, and the prospects of guiding the design of the info schedule.

Developing a pipeline for loss prediction involved training the models on the mixture of RedPajama and validating against the validation set of the Pile. A series of 70M, 160M, 305M, and 410M models for 30B tokens were trained to stick to the scaling laws of coaching steps and model sizes. Remarkably, the model trained on the optimized mixture achieves performance comparable to that of 1 trained on the default mixture, but with just 73% of the steps. It will definitely surpasses the default mixture’s performance, requiring 48% more steps, underscoring the pipeline’s effectiveness in mixture optimization.

In conclusion, this paper introduces data mixing law and prediction pipeline, which solves the issue of accurately predicting the validation loss for a combination of coaching domains under a hard and fast model size and amount of coaching data. The nested use of scaling laws of coaching steps, model sizes, and data mixture makes predictions with only experiments at small scales, enabling the reuse of existing experiments and reducing computation costs. This study will further facilitate quantitative studies and theoretical evaluation with an increasing deal with data engineering.

Take a look at the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

In the event you like our work, you’ll love our newsletter..

Don’t Forget to hitch our 39k+ ML SubReddit