Home Artificial Intelligence Leverage OpenAI Tool calling: Constructing a reliable AI Agent from Scratch From Concept to Reality: Constructing the Basis for AI Agents The Aim of This Article 1. Hands-On: Testing OpenAI Tool Call Calling OpenAI API 2. Optimize Tool handling Handling Missing Information Designing Tool class Testing Missing Information Handling 3. Constructing the Agent Workflow 4. Running the Agent Key Takeaways Looking Ahead

Leverage OpenAI Tool calling: Constructing a reliable AI Agent from Scratch From Concept to Reality: Constructing the Basis for AI Agents The Aim of This Article 1. Hands-On: Testing OpenAI Tool Call Calling OpenAI API 2. Optimize Tool handling Handling Missing Information Designing Tool class Testing Missing Information Handling 3. Constructing the Agent Workflow 4. Running the Agent Key Takeaways Looking Ahead

0
Leverage OpenAI Tool calling: Constructing a reliable AI Agent from Scratch
From Concept to Reality: Constructing the Basis for AI Agents
The Aim of This Article
1. Hands-On: Testing OpenAI Tool Call
Calling OpenAI API
2. Optimize Tool handling
Handling Missing Information
Designing Tool class
Testing Missing Information Handling
3. Constructing the Agent Workflow
4. Running the Agent
Key Takeaways
Looking Ahead

Created with DALL·E

Step-by-Step Workflow for developing and refining an AI Agent while coping with errors

Towards Data Science

When we predict in regards to the way forward for AI, we envision intuitive on a regular basis helpers seamlessly integrating into our workflows and taking up complex, routinely tasks. All of us have found touchpoints that relieve us from the tedium of mental routine work. Yet, the most important tasks currently tackled involve text creation, correction, and brainstorming, underlined by the numerous role RAG (Retrieval-Augmented Generation) pipelines play in ongoing development. We aim to supply Large Language Models with higher context to generate more priceless content.

Fascinated with the longer term of AI conjures images of Jarvis from Iron Man or Rasputin from Destiny (the sport) for me. In each examples, the AI acts as a voice-controlled interface to a fancy system, offering high-level abstractions. As an example, Tony Stark uses it to administer his research, conduct calculations, and run simulations. Even R2D2 can reply to voice commands to interface with unfamiliar computer systems and extract data or interact with constructing systems.

In these scenarios, AI enables interaction with complex systems without requiring the top user to have a deep understanding of them. This may very well be likened to an ERP system in a big corporation today. It’s rare to search out someone in a big corporation who fully knows and understands every facet of the in-house ERP system. It’s not far-fetched to assume that, within the near future, AI might assist nearly every interaction with an ERP system. From the top user managing customer data or logging orders to the software developer fixing bugs or implementing latest features, these interactions could soon be facilitated by AI assistants acquainted with all points and processes of the ERP system. Such an AI assistant would know which database to enter customer data into and which processes and code is likely to be relevant to a bug.

To attain this, several challenges and innovations lie ahead. We’d like to rethink processes and their documentation. Today’s ERP processes are designed for human use, with specific roles for various users, documentation for humans, input masks for humans, and user interactions designed to be intuitive and error-free. The design of those points will look different for AI interactions. We’d like specific roles for AI interactions and different process designs to enable intuitive and error-free AI interaction. That is already evident in our work with prompts. What we consider to be a transparent task often seems to not be so straightforward.

Nonetheless, let’s first take a step back to the concept of agents. Agents, or AI assistants that may perform tasks using the tools provided and make decisions on tips on how to use these tools, are the constructing blocks that might eventually enable such a system. They’re the method components we’d need to integrate into every facet of a fancy system. But as highlighted in a previous article, they’re difficult to deploy reliably. In this text, I’ll exhibit how we are able to design and optimize an agent able to reliably interacting with a database.

While the grand vision of AI’s future is inspiring, it’s crucial to take practical steps towards realizing this vision. To exhibit how we are able to start constructing the inspiration for such advanced AI systems, let’s give attention to making a prototype agent for a typical task: expense tracking. This prototype will function a tangible example of how AI can assist in managing financial transactions efficiently, showcasing the potential of AI in automating routine tasks and highlighting the challenges and considerations involved in designing an AI system that interacts seamlessly with databases. By starting with a selected and relatable use case, we are able to gain priceless insights that may inform the event of more complex AI agents in the longer term.

This text will lay the groundwork for a series of articles geared toward developing a chatbot that may function a single point of interaction for a small business to support and execute business processes or a chatbot that in your personal life organizes the whole lot you have to keep track of. From data, routines, files, to pictures, we wish to easily chat with our Assistant, allowing it to determine where to store and retrieve your data.

Transitioning from the grand vision of AI’s future to practical applications, let’s zoom in on making a prototype agent. This agent will function a foundational step towards realizing the ambitious goals discussed earlier. We’ll embark on developing an “Expense Tracking” agent, an easy yet essential task, demonstrating how AI can assist in managing financial transactions efficiently.

This “Expense Tracking” prototype is not going to only showcase the potential of AI in automating routine tasks but additionally illuminate the challenges and considerations involved in designing an AI system that interacts seamlessly with databases. By specializing in this instance, we are able to explore the intricacies of agent design, input validation, and the combination of AI with existing systems — laying a solid foundation for more complex applications in the longer term.

To bring our prototype agent to life and discover potential bottlenecks, we’re venturing into testing the tool call functionality of OpenAI. Starting with a basic example of expense tracking, we’re laying down a foundational piece that mimics a real-world application. This stage involves making a base model and remodeling it into the OpenAI tool schema using the langchain library’s convert_to_openai_tool function. Moreover, crafting a report_tool enables our future agent to speak results or highlight missing information or issues:

from pydantic.v1 import BaseModel, validator  
from datetime import datetime
from langchain_core.utils.function_calling import convert_to_openai_tool

class Expense(BaseModel):
description: str
net_amount: float
gross_amount: float
tax_rate: float
date: datetime

class Report(BaseModel):
report: str

add_expense_tool = convert_to_openai_tool(Expense)
report_tool = convert_to_openai_tool(Report)

With the info model and tools arrange, the subsequent step is to make use of the OpenAI client SDK to initiate an easy tool call. On this initial test, we’re intentionally providing insufficient information to the model to see if it will probably appropriately indicate what’s missing. This approach not only tests the functional capability of the agent but additionally its interactive and error-handling capacities.

Now, we’ll use the OpenAI client SDK to initiate an easy tool call. In our first test, we deliberately provide the model with insufficient information to see if it will probably notify us of the missing details.

from openai import OpenAI  
from langchain_core.utils.function_calling import convert_to_openai_tool

SYSTEM_MESSAGE = """You might be tasked with completing specific objectives and
must report the outcomes. At your disposal, you will have a wide range of tools,
each specialized in performing a definite style of task.

For successful task completion:
Thought: Consider the duty at hand and determine which tool is best suited
based on its capabilities and the character of the work.

Use the report_tool with an instruction detailing the outcomes of your work.
When you encounter a problem and can't complete the duty:

Use the report_tool to speak the challenge or reason for the
task's incompletion.
You'll receive feedback based on the outcomes of
each tool's task execution or explanations for any tasks that
couldn't be accomplished. This feedback loop is crucial for addressing
and resolving any issues by strategically deploying the available tools.
"""
user_message = "I actually have spend 5$ on a coffee today please track my expense. The tax rate is 0.2."

client = OpenAI()
model_name = "gpt-3.5-turbo-0125"

messages = [
{"role":"system", "content": SYSTEM_MESSAGE},
{"role":"user", "content": user_message}
]

response = client.chat.completions.create(
model=model_name,
messages=messages,
tools=[
convert_to_openai_tool(Expense),
convert_to_openai_tool(ReportTool)]
)

Next, we’ll need a brand new function to read the arguments of the function call from the response:

def parse_function_args(response):
message = response.selections[0].message
return json.loads(message.tool_calls[0].function.arguments)

print(parse_function_args(response))

{'description': 'Coffee',
'net_amount': 5,
'gross_amount': None,
'tax_rate': 0.2,
'date': '2023-10-06T12:00:00Z'}

As we are able to observe, we have now encountered several issues within the execution:

  1. The gross_amount isn’t calculated.
  2. The date is hallucinated.

With that in mind. Let’s attempt to resolve this issues and optimize our agent workflow.

To optimize the agent workflow, I find it crucial to prioritize workflow over prompt engineering. While it is likely to be tempting to fine-tune the prompt in order that the agent learns to make use of the tools provided perfectly and makes no mistakes, it’s more advisable to first adjust the tools and processes. When a typical error occurs, the initial consideration needs to be tips on how to fix it code-based.

Handling missing information effectively is an important topic for creating robust and reliable agents. Within the previous example, providing the agent with a tool like “get_current_date” is a workaround for specific scenarios. Nonetheless, we must assume that missing information will occur in various contexts, and we cannot rely solely on prompt engineering and adding more tools to stop the model from hallucinating missing information.

An easy workaround for this scenario is to change the tool schema to treat all parameters as optional. This approach ensures that the agent only submits the parameters it knows, stopping unnecessary hallucination.

Subsequently, let’s take a have a look at openai tool schema:

add_expense_tool = convert_to_openai_tool(Expense)
print(add_expense_tool)
{'type': 'function',
'function': {'name': 'Expense',
'description': '',
'parameters': {'type': 'object',
'properties': {'description': {'type': 'string'},
'net_amount': {'type': 'number'},
'gross_amount': {'type': 'number'},
'tax_rate': {'type': 'number'},
'date': {'type': 'string', 'format': 'date-time'}},
'required': ['description',
'net_amount',
'gross_amount',
'tax_rate',
'date']}}}

As we are able to see we have now special key required , which we’d like to remove. Here’s how you’ll be able to adjust the add_expense_tool schema to make parameters optional by removing the required key:

del add_expense_tool["function"]["parameters"]["required"]

Next, we are able to design a Tool class that originally checks the input parameters for missing values. We create the Tool class with two methods: .run(), .validate_input(), and a property openai_tool_schema, where we manipulate the tool schema by removing required parameters. Moreover, we define the ToolResult BaseModel with the fields content and success to serve because the output object for every tool run.

from pydantic import BaseModel
from typing import Type, Callable, Dict, Any, List

class ToolResult(BaseModel):
content: str
success: bool

class Tool(BaseModel):
name: str
model: Type[BaseModel]
function: Callable
validate_missing: bool = False

class Config:
arbitrary_types_allowed = True

def run(self, **kwargs) -> ToolResult:
if self.validate_missing:
missing_values = self.validate_input(**kwargs)
if missing_values:
content = f"Missing values: {', '.join(missing_values)}"
return ToolResult(content=content, success=False)
result = self.function(**kwargs)
return ToolResult(content=str(result), success=True)

def validate_input(self, **kwargs) -> List[str]:
missing_values = []
for key in self.model.__fields__.keys():
if key not in kwargs:
missing_values.append(key)
return missing_values
@property
def openai_tool_schema(self) -> Dict[str, Any]:
schema = convert_to_openai_tool(self.model)
if "required" in schema["function"]["parameters"]:
del schema["function"]["parameters"]["required"]
return schema

The Tool class is a vital component within the AI agent’s workflow, serving as a blueprint for creating and managing various tools that the agent can utilize to perform specific tasks. It’s designed to handle input validation, execute the tool’s function, and return the lead to a standardized format.

The Tool class key components:

  1. name: The name of the tool.
  2. model: The Pydantic BaseModel that defines the input schema for the tool.
  3. function: The callable function that the tool executes.
  4. validate_missing: A boolean flag indicating whether to validate missing input values (default is False).

The Tool class two most important methods:

  1. run(self, **kwargs) -> ToolResult: This method is accountable for executing the tool’s function with the provided input arguments. It first checks if validate_missing is ready to True. If that’s the case, it calls the validate_input() method to ascertain for missing input values. If any missing values are found, it returns a ToolResult object with an error message and success set to False. If all required input values are present, it proceeds to execute the tool’s function with the provided arguments and returns a ToolResult object with the result and success set to True.
  2. validate_input(self, **kwargs) -> List[str]: This method compares the input arguments passed to the tool with the expected input schema defined within the model. It iterates over the fields defined within the model and checks if each field is present within the input arguments. If any field is missing, it appends the sphere name to an inventory of missing values. Finally, it returns the list of missing values.

The Tool class also has a property called openai_tool_schema, which returns the OpenAI tool schema for the tool. It uses the convert_to_openai_tool() function to convert the model to the OpenAI tool schema format. Moreover, it removes the "required" key from the schema, making all input parameters optional. This enables the agent to supply only the available information without the necessity to hallucinate missing values.

By encapsulating the tool’s functionality, input validation, and schema generation, the Tool class provides a clean and reusable interface for creating and managing tools within the AI agent’s workflow. It abstracts away the complexities of handling missing values and ensures that the agent can gracefully handle incomplete information while executing the suitable tools based on the available input.

Next, we’ll extend our OpenAI API call. We would like the client to utilize our tool, and our response object to directly trigger a tool.run(). For this, we’d like to initialize our tools in our newly created Tool class. We define two dummy functions which return successful message string.

def add_expense_func(**kwargs):  
return f"Added expense: {kwargs} to the database."

add_expense_tool = Tool(
name="add_expense_tool",
model=Expense,
function=add_expense_func
)

def report_func(report: str = None):
return f"Reported: {report}"

report_tool = Tool(
name="report_tool",
model=ReportTool,
function=report_func
)

tools = [add_expense_tool, report_tool]

Next we define our helper function, that every take client response as input an help to interact with out tools.

def get_tool_from_response(response, tools=tools):  
tool_name = response.selections[0].message.tool_calls[0].function.name
for t in tools:
if t.name == tool_name:
return t
raise ValueError(f"Tool {tool_name} not present in tools list.")

def parse_function_args(response):
message = response.selections[0].message
return json.loads(message.tool_calls[0].function.arguments)

def run_tool_from_response(response, tools=tools):
tool = get_tool_from_response(response, tools)
tool_kwargs = parse_function_args(response)
return tool.run(**tool_kwargs)

Now, we are able to execute our client with our latest tools and use the run_tool_from_response function.

response = client.chat.completions.create(  
model=model_name,
messages=messages,
tools=[tool.openai_tool_schema for tool in tools]
)

tool_result = run_tool_from_response(response, tools=tools)
print(tool_result)

content='Missing values: gross_amount, date' success=False

Perfectly, we now see our tool indicating that missing values are present. Because of our trick of sending all parameters as optional, we now avoid hallucinated parameters.

Our process, because it stands, doesn’t yet represent a real agent. To date, we’ve only executed a single API tool call. To remodel this into an agent workflow, we’d like to introduce an iterative process that feeds the outcomes of tool execution back to the client. The essential process should like this:

Image by creator

Let’s start by making a latest OpenAIAgent class:

class StepResult(BaseModel):  
event: str
content: str
success: bool

class OpenAIAgent:

def __init__(
self,
tools: list[Tool],
client: OpenAI,
system_message: str = SYSTEM_MESSAGE,
model_name: str = "gpt-3.5-turbo-0125",
max_steps: int = 5,
verbose: bool = True
):
self.tools = tools
self.client = client
self.model_name = model_name
self.system_message = system_message
self.step_history = []
self.max_steps = max_steps
self.verbose = verbose

def to_console(self, tag: str, message: str, color: str = "green"):
if self.verbose:
color_prefix = Fore.__dict__[color.upper()]
print(color_prefix + f"{tag}: {message}{Style.RESET_ALL}")

Like our ToolResultobject, we’ve defined a StepResult as an object for every agent step. We then defined the __init__ approach to the OpenAIAgent class and a to_console() method to print our intermediate steps and power calls to the console, using colorama for colourful printouts. Next, we define the center of the agent, the run() and the run_step() method.

class OpenAIAgent:

# ... __init__...

# ... to_console ...

def run(self, user_input: str):

openai_tools = [tool.openai_tool_schema for tool in self.tools]
self.step_history = [
{"role":"system", "content":self.system_message},
{"role":"user", "content":user_input}
]

step_result = None
i = 0

self.to_console("START", f"Starting Agent with Input: {user_input}")

while i < self.max_steps:
step_result = self.run_step(self.step_history, openai_tools)

if step_result.event == "finish":
break
elif step_result.event == "error":
self.to_console(step_result.event, step_result.content, "red")
else:
self.to_console(step_result.event, step_result.content, "yellow")
i += 1

self.to_console("Final Result", step_result.content, "green")
return step_result.content

Within the run() method, we start by initializing the step_history, which is able to function our message memory, with the predefined system_message and the user_input. Then we start our while loop, where we call run_step during each iteration, which is able to return a StepResult Object. We discover if the agent finished his task or if an error occurred, which will probably be passed to the console as well.

class OpenAIAgent:

# ... __init__...

# ... to_console ...
# ... run ...
def run_step(self, messages: list[dict], tools):

# plan the subsequent step
response = self.client.chat.completions.create(
model=self.model_name,
messages=messages,
tools=tools
)

# add message to history
self.step_history.append(response.selections[0].message)

# check if tool call is present
if not response.selections[0].message.tool_calls:
return StepResult(
event="Error",
content="No tool calls were returned.",
success=False
)

tool_name = response.selections[0].message.tool_calls[0].function.name
tool_kwargs = parse_function_args(response)

# execute the tool call
self.to_console(
"Tool Call", f"Name: {tool_name}nArgs: {tool_kwargs}", "magenta"
)
tool_result = run_tool_from_response(response, tools=self.tools)
tool_result_msg = self.tool_call_message(response, tool_result)
self.step_history.append(tool_result_msg)

if tool_result.success:
step_result = StepResult(
event="tool_result",
content=tool_result.content,
success=True)
else:
step_result = StepResult(
event="error",
content=tool_result.content,
success=False
)

return step_result

def tool_call_message(self, response, tool_result: ToolResult):
tool_call = response.selections[0].message.tool_calls[0]
return {
"tool_call_id": tool_call.id,
"role": "tool",
"name": tool_call.function.name,
"content": tool_result.content,
}

Now we’ve defined the logic for every step. We first obtain a response object by our previously tested client API call with tools. We append the response message object to our step_history. We then confirm if a tool call is included in our response object, otherwise, we return an error in our StepResult. Then we log our tool call to the console and run the chosen tool with our previously defined method run_tool_from_response(). We also must append the tool result to our message history. OpenAI has defined a selected format for this purpose, in order that the Model knows which tool call refers to which output by passing a tool_call_id into our message dict. This is finished by our method tool_call_message(), which takes the response object and the tool_result as input arguments. At the top of every step, we assign the tool result to a StepResult Object, which also indicates if the step was successful or not, and return it to our loop in run().

Now we are able to test our agent with the previous example, directly equipping it with a get_current_date_tool as well. Here, we are able to set our previously defined validate_missing attribute to False, because the tool doesn’t need any input argument.

class DateTool(BaseModel):  
x: str = None

get_date_tool = Tool(
name="get_current_date",
model=DateTool,
function=lambda: datetime.now().strftime("%Y-%m-%d"),
validate_missing=False
)

tools = [
add_expense_tool,
report_tool,
get_date_tool
]

agent = OpenAIAgent(tools, client)
agent.run("I actually have spent 5$ on a coffee today please track my expense. The tax rate is 0.2.")

START: Starting Agent with Input: 
"I actually have spend 5$ on a coffee today please track my expense. The tax rate is 0.2."

Tool Call: get_current_date
Args: {}
tool_result: 2024-03-15

Tool Call: add_expense_tool
Args: {'description': 'Coffee expense', 'net_amount': 5, 'tax_rate': 0.2, 'date': '2024-03-15'}
error: Missing values: gross_amount

Tool Call: add_expense_tool
Args: {'description': 'Coffee expense', 'net_amount': 5, 'tax_rate': 0.2, 'date': '2024-03-15', 'gross_amount': 6}
tool_result: Added expense: {'description': 'Coffee expense', 'net_amount': 5, 'tax_rate': 0.2, 'date': '2024-03-15', 'gross_amount': 6} to the database.
Error: No tool calls were returned.

Tool Call: Name: report_tool
Args: {'report': 'Expense successfully tracked for coffee purchase.'}
tool_result: Reported: Expense successfully tracked for coffee purchase.

Final Result: Reported: Expense successfully tracked for coffee purchase.

Following the successful execution of our prototype agent, it’s noteworthy to emphasise how effectively the agent utilized the designated tools in keeping with plan. Initially, it invoked the get_current_date_tool, establishing a foundational timestamp for the expense entry. Subsequently, when attempting to log the expense via the add_expense_tool, our intelligently designed tool class identified a missing gross_amount—a vital piece of knowledge for accurate financial tracking. Impressively, the agent autonomously resolved this by calculating the gross_amount using the provided tax_rate.

It’s vital to say that in our test run, the character of the input expense — whether the $5 spent on coffee was net or gross — wasn’t explicitly specified. At this juncture, such specificity wasn’t required for the agent to perform its task successfully. Nonetheless, this brings to light a priceless insight for refining our agent’s understanding and interaction capabilities: Incorporating such detailed information into our initial system prompt could significantly enhance the agent’s accuracy and efficiency in processing expense entries. This adjustment would ensure a more comprehensive grasp of monetary data right from the outset.

  1. Iterative Development: The project underscores the critical nature of an iterative development cycle, fostering continuous improvement through feedback. This approach is paramount in AI, where variability is the norm, necessitating an adaptable and responsive development strategy.
  2. Handling Uncertainty: Our journey highlighted the importance of elegantly managing ambiguities and errors. Innovations similar to optional parameters and rigorous input validation have proven instrumental in enhancing each the reliability and user experience of the agent.
  3. Customized Agent Workflows for Specific Tasks: A key insight from this work is the importance of customizing agent workflows to suit particular use cases. Beyond assembling a collection of tools, the strategic design of tool interactions and responses is significant. This customization ensures the agent effectively addresses specific challenges, resulting in a more focused and efficient problem-solving approach.

The journey we have now embarked upon is just the start of a bigger exploration into the world of AI agents and their applications in various domains. As we proceed to push the boundaries of what’s possible with AI, we invite you to hitch us on this exciting adventure. By constructing upon the inspiration laid in this text and staying tuned for the upcoming enhancements, you’ll witness firsthand how AI agents can revolutionize the best way businesses and individuals handle their data and automate complex tasks.

Together, allow us to embrace the facility of AI and unlock its potential to rework the best way we work and interact with technology. The long run of AI is shiny, and we’re on the forefront of shaping it, one reliable agent at a time.

As we proceed our journey in exploring the potential of AI agents, the upcoming articles will give attention to expanding the capabilities of our prototype and integrating it with real-world systems. In the subsequent article, we’ll dive into designing a strong project structure that enables our agent to interact seamlessly with SQL databases. By leveraging the agent developed in this text, we’ll exhibit how AI can efficiently manage and manipulate data stored in databases, opening up a world of possibilities for automating data-related tasks.

Constructing upon this foundation, the third article within the series will introduce advanced query features, enabling our agent to handle more complex data retrieval and manipulation tasks. We may also explore the concept of a routing agent, which is able to act as a central hub for managing multiple subagents, each accountable for interacting with specific database tables. This hierarchical structure will allow users to make requests in natural language, which the routing agent will then interpret and direct to the suitable subagent for execution.

To further enhance the practicality and security of our AI-powered system, we’ll introduce a role-based access control system. This can be sure that users have the suitable permissions to access and modify data based on their assigned roles. By implementing this feature, we are able to exhibit how AI agents could be deployed in real-world scenarios while maintaining data integrity and security.

Through these upcoming enhancements, we aim to showcase the true potential of AI agents in streamlining data management processes and providing a more intuitive and efficient way for users to interact with databases. By combining the facility of natural language processing, database management, and role-based access control, we will probably be laying the groundwork for the event of sophisticated AI assistants that may revolutionize the best way businesses and individuals handle their data.

Stay tuned for these exciting developments as we proceed to push the boundaries of what’s possible with AI agents in data management and beyond.

Source Code

Moreover, the whole source code for the projects covered is on the market on GitHub. You may access it at https://github.com/elokus/AgentDemo.

LEAVE A REPLY

Please enter your comment!
Please enter your name here