Artificial neural networks, ubiquitous machine-learning models that may be trained to finish many tasks, are so called because their architecture is inspired by the way in which biological neurons process information within the human brain.
About six years ago, scientists discovered a brand new style of more powerful neural network model often known as a transformer. These models can achieve unprecedented performance, akin to by generating text from prompts with near-human-like accuracy. A transformer underlies AI systems akin to ChatGPT and Bard, for instance. While incredibly effective, transformers are also mysterious: Unlike with other brain-inspired neural network models, it hasn’t been clear learn how to construct them using biological components.
Now, researchers from MIT, the MIT-IBM Watson AI Lab, and Harvard Medical School have produced a hypothesis which will explain how a transformer might be built using biological elements within the brain. They suggest that a biological network composed of neurons and other brain cells called astrocytes could perform the identical core computation as a transformer.
Recent research has shown that astrocytes, non-neuronal cells which are abundant within the brain, communicate with neurons and play a job in some physiological processes, like regulating blood flow. But scientists still lack a transparent understanding of what these cells do computationally.
With the brand new study, published this week in open-access format within the, the researchers explored the role astrocytes play within the brain from a computational perspective, and crafted a mathematical model that shows how they might be used, together with neurons, to construct a biologically plausible transformer.
Their hypothesis provides insights that might spark future neuroscience research into how the human brain works. At the identical time, it could help machine-learning researchers explain why transformers are so successful across a various set of complex tasks.
“The brain is much superior to even the very best artificial neural networks that now we have developed, but we don’t really know exactly how the brain works. There’s scientific value in fascinated by connections between biological hardware and large-scale artificial intelligence networks. That is neuroscience for AI and AI for neuroscience,” says Dmitry Krotov, a research staff member on the MIT-IBM Watson AI Lab and senior creator of the research paper.
Joining Krotov on the paper are lead creator Leo Kozachkov, a postdoc within the MIT Department of Brain and Cognitive Sciences; and Ksenia V. Kastanenka, an assistant professor of neurobiology at Harvard Medical School and an assistant investigator on the Massachusetts General Research Institute.
A biological impossibility becomes plausible
Transformers operate in a different way than other neural network models. As an example, a recurrent neural network trained for natural language processing would compare each word in a sentence to an internal state determined by the previous words. A transformer, alternatively, compares all of the words within the sentence directly to generate a prediction, a process called self-attention.
For self-attention to work, the transformer must keep all of the words ready in some type of memory, Krotov explains, but this didn’t seem biologically possible resulting from the way in which neurons communicate.
Nonetheless, just a few years ago scientists studying a rather different style of machine-learning model (often known as a Dense Associated Memory) realized that this self-attention mechanism could occur within the brain, but provided that there have been communication between at the least three neurons.
“The number three really popped out to me since it is thought in neuroscience that these cells called astrocytes, which usually are not neurons, form three-way connections with neurons, what are called tripartite synapses,” Kozachkov says.
When two neurons communicate, a presynaptic neuron sends chemicals called neurotransmitters across the synapse that connects it to a postsynaptic neuron. Sometimes, an astrocyte can also be connected — it wraps a protracted, thin tentacle across the synapse, making a tripartite (three-part) synapse. One astrocyte may form tens of millions of tripartite synapses.
The astrocyte collects some neurotransmitters that flow through the synaptic junction. Sooner or later, the astrocyte can signal back to the neurons. Because astrocytes operate on a for much longer time scale than neurons — they create signals by slowly elevating their calcium response after which decreasing it — these cells can hold and integrate information communicated to them from neurons. In this fashion, astrocytes can form a style of memory buffer, Krotov says.
“If you happen to give it some thought from that perspective, then astrocytes are extremely natural for precisely the computation we’d like to perform the eye operation inside transformers,” he adds.
Constructing a neuron-astrocyte network
With this insight, the researchers formed their hypothesis that astrocytes could play a job in how transformers compute. Then they got down to construct a mathematical model of a neuron-astrocyte network that might operate like a transformer.
They took the core mathematics that comprise a transformer and developed easy biophysical models of what astrocytes and neurons do when they impart within the brain, based on a deep dive into the literature and guidance from neuroscientist collaborators.
Then they combined the models in certain ways until they arrived at an equation of a neuron-astrocyte network that describes a transformer’s self-attention.
“Sometimes, we found that certain things we desired to be true couldn’t be plausibly implemented. So, we had to consider workarounds. There are some things within the paper which are very careful approximations of the transformer architecture to give you the option to match it in a biologically plausible way,” Kozachkov says.
Through their evaluation, the researchers showed that their biophysical neuron-astrocyte network theoretically matches a transformer. As well as, they conducted numerical simulations by feeding images and paragraphs of text to transformer models and comparing the responses to those of their simulated neuron-astrocyte network. Each responded to the prompts in similar ways, confirming their theoretical model.
“Having remained electrically silent for over a century of brain recordings, astrocytes are one of the abundant, yet less explored, cells within the brain. The potential of unleashing the computational power of the opposite half of our brain is gigantic,” says Konstantinos Michmizos, associate professor of computer science at Rutgers University, who was not involved with this work. “This study opens up a captivating iterative loop, from understanding how intelligent behavior may truly emerge within the brain, to translating disruptive hypotheses into latest tools that exhibit human-like intelligence.”
The subsequent step for the researchers is to make the leap from theory to practice. They hope to match the model’s predictions to those which were observed in biological experiments, and use this data to refine, or possibly disprove, their hypothesis.
As well as, one implication of their study is that astrocytes could also be involved in long-term memory, because the network must store information to give you the option act on it in the longer term. Additional research could investigate this concept further, Krotov says.
“For loads of reasons, astrocytes are extremely essential for cognition and behavior, they usually operate in fundamentally alternative ways from neurons. My biggest hope for this paper is that it catalyzes a bunch of research in computational neuroscience toward glial cells, and specifically, astrocytes,” adds Kozachkov.
This research was supported, partly, by the BrightFocus Foundation and the National Institute of Health.