Machine Learning has iconic applications in programming languages, from code understanding to code representation or completion. Earlier work focused on exploiting the underlying deep semantic structure of programming languages like Code2Vec, Code2Seq, and Graph Representation Learning for Code. The above architectures are tailor-made for the native structures of Abstract Syntax Trees (AST) / Data Flow Graphs (DFG). They’ve a major limitation: they’ll only be applied for tasks that involve completely executable code.

Later research has shown how transformer-based models may be used like natural language for code on the lexical (text) level. Since then, language models have been widely used to model code on various tasks. Such models are executed every few seconds, especially within the case of code completion. Strong models running on consumer devices are preferred to avoid network latency, make a difference, and address discrepancies concerning gated APIs.

The researchers from Stability AI introduced Stable Code, which serves as a general-purpose base code language model targeting code completion, reasoning, math, and other software engineering-based tasks. Also, they introduce an instruction variant named Stable Code Instruct that enables conversing with the model in a natural chat interface for performing question-answering and instruction-based tasks.

Stable Code is built on top of Stable LM, a state-of-the-art LLM for natural language in English at the three billion parameter scale. The model is a causal decoder-only transformer similar in design to the LLaMA architecture. The most important differences with LLaMA are:

• Rotary Position Embeddings are applied to the primary 25% of head embedding dimensions for improved throughput.
• LayerNorm with learned bias terms versus RMSNorm.
• All bias terms were faraway from the feed-forward networks and multi-head self-attention layers, apart from the biases of the important thing, query, and value projections.

Stable Code matches the performance of Llama and StarCoder on average across programming languages, despite the fact that it is comparatively smaller. Also, Stable Code 3B achieves strong performance on the 3B scale, showing remarkable capabilities in code completion tasks. In addition they evaluated instruct-tuned models on the code subset of the difficult Multi-turn benchmark.

In conclusion, the researchers from Stability AI introduced Stable Code and Stable Code Instruct to handle different software development use cases. Each Stable Code and Stable Code Instruct are compact decoder-only language models. Researchers have conducted extensive model evaluations and comparisons with other similarly-sized models, demonstrating Stable Code and Stable Code Instruct’s remarkable performance. In addition they provide an evaluation of the model on typical edge computing architectures.

Take a look at the Paper and Blog. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

Should you like our work, you’ll love our newsletter..

Don’t Forget to affix our 39k+ ML SubReddit