We’re seeing a progression of Generative AI applications powered by large language models (LLM) from prompts to retrieval augmented generation (RAG) to agents. Agents are being talked about heavily in industry and research circles, mainly for the ability this technology provides to remodel Enterprise applications and supply superior customer experiences. There are common patterns for constructing agents that enable first steps towards artificial general intelligence (AGI).

In my previous article, we saw a ladder of intelligence of patterns for constructing LLM powered applications. Starting with prompts that capture problem domain and use LLM internal memory to generate output. With RAG, we augment the prompt with external knowledge searched from a vector database to manage the outputs. Next by chaining LLM calls we are able to construct workflows to appreciate complex applications. Agents take this to a next level by auto determining how these LLM chains are to be formed. Let’s look intimately.

Agents – Under the hood

A key pattern with agents is that they use the language understanding power of LLM to make a plan on the way to solve a given problem. The LLM understands the issue and provides us a sequence of steps to resolve the issue. Nonetheless, it doesn’t stop there. Agents usually are not a pure support system that can provide you recommendations on solving the issue after which pass on the baton to you to take the beneficial steps. Agents are empowered with tooling to go ahead and take the motion. Scary right!?

If we ask an agent a basic query like this:

Human: Which company did the inventor of the phone start?

Following is a sample of pondering steps that an agent may take.

Agent (THINKING):

• Thought: I would like to look for the inventor of the phone.
• Motion: Search [inventor of telephone]
• Commentary: Alexander Graham Bell
• Thought: I would like to look for an organization that was founded by Alexander Graham Bell
• Motion: Search [company founded by Alexander Graham Bell]
• Commentary: Alexander Graham Bell co-founded the American Telephone and Telegraph Company (AT&T) in 1885
• Thought: I actually have found the reply. I’ll return.

Agent (RESPONSE): Alexander Graham Bell co-founded AT&T in 1885

You’ll be able to see that the agent follows a methodical way of breaking down the issue into subproblems that may be solved by taking specific Actions. The actions listed below are beneficial by the LLM and we are able to map these to specific tools to implement these actions. We could enable a search tool for the agent such that when it realizes that LLM has provided search as an motion, it can call this tool with the parameters provided by the LLM. The search here is on the web but can as well be redirected to look an internal knowledge base like a vector database. The system now becomes self-sufficient and might work out the way to solve complex problems following a series of steps. Frameworks like LangChain and LLaMAIndex offer you a simple technique to construct these agents and hook up with toolings and API. Amazon recently launched their Bedrock Agents framework that gives a visible interface for designing agents.

Under the hood, agents follow a special type of sending prompts to the LLM which make them generate an motion plan. The above Thought-Motion-Commentary pattern is popular in a sort of agent called ReAct (Reasoning and Acting). Other forms of agents include MRKL and Plan & Execute, which mainly differ of their prompting style.

For more complex agents, the actions could also be tied to tools that cause changes in source systems. For instance, we could connect the agent to a tool that checks for vacation balance and applies for leave in an ERP system for an worker. Now we could construct a pleasant chatbot that might interact with users and via a chat command apply for leave within the system. No more complex screens for applying for leaves, an easy unified chat interface. Sounds exciting!?

Caveats and want for Responsible AI

Now what if we now have a tool that invokes transactions on stock trading using a pre-authorized API. You construct an application where the agent studies stock changes (using tools) and makes decisions for you on buying and selling of stock. What if the agent sells the fallacious stock since it hallucinated and made a fallacious decision? Since LLM are huge models, it’s difficult to pinpoint why they make some decisions, hence hallucinations are common in absence of proper guardrails.

While agents are all fascinating you almost certainly would have guessed how dangerous they may be. In the event that they hallucinate and take a fallacious motion that would cause huge financial losses or major issues in Enterprise systems. Hence Responsible AI is becoming of utmost importance within the age of LLM powered applications. The principles of Responsible AI around reproducibility, transparency, and accountability, attempt to put guardrails on decisions taken by agents and suggest risk evaluation to make your mind up which actions need a human-in-the-loop. As more complex agents are being designed, they need more scrutiny, transparency, and accountability to make certain we all know what they’re doing.

Closing thoughts

Ability of agents to generate a path of logical steps with actions gets them really near human reasoning. Empowering them with more powerful tools may give them superpowers. Patterns like ReAct attempt to emulate how humans solve the issue and we’ll see higher agent patterns that shall be relevant to specific contexts and domains (banking, insurance, healthcare, industrial, etc.). The long run is here and technology behind agents is prepared for us to make use of. At the identical time, we want to maintain close attention to Responsible AI guardrails to make certain we usually are not constructing Skynet!