turn your Llama right into a Giraffe

Context length refers back to the maximum variety of tokens the model can remember when generating text. An extended context window allows the model to know long-range dependencies in text higher. Models with longer contexts can construct connections between ideas far apart within the text, generating more globally coherent outputs.

During training, the model processes the text data in chunks or fixed-length windows. Models should be trained on lengthy texts to really leverage long contexts. Training sequences must contain documents, books, articles, etc., with 1000’s of tokens.
The length of coaching data sets a limit on usable context length.

So, why don’t we train models on longer sequences?

Not so fast.

Increasing context length increases the variety of possible token combos the model must learn to predict accurately.
This permits more robust long-range modeling but in addition require more memory and processing power, resulting in higher training costs.

With none optimization, computation scales quadratically with context length — meaning that a 4096 token model will need 64 times more computation than a 512 token model.

You should use sparse or approximate attention methods to cut back the computation cost, but they might also affect the model’s accuracy.

Training and using large context language models presents three primary challenges:

• Fitting long contexts into the model.
• Accelerating inference and training so that they don’t take endlessly.
• Ensuring a high-quality inference that maintains awareness of the total context.

The eye mechanism is the core component of transformer models. It relates different positions of a sequence to compute its representation, allowing models to give attention to relevant parts of the text and understand it higher. Scaling transformers to longer sequences faces challenges as a consequence of the quadratic complexity of full attention.