I believe each of us has wondered at the very least once over the past yr if (or relatively when) ChatGPT will give you the chance to exchange your role. I’m no exception here.

We’ve got a somewhat consensus that the recent breakthroughs in Generative AI will highly affect our personal lives and work. Nonetheless, there isn’t a clear view yet of how our roles will change over time.

Spending plenty of time enthusiastic about different possible future scenarios and their probabilities is perhaps fascinating, but I suggest a completely different approach — to try to construct your prototype yourself. First, it’s relatively difficult and fun. Second, it can help us to take a look at our work in a more structured way. Third, it can give us a possibility to try in practice some of the cutting-edge approaches — LLM agents.

In this text, we are going to start easy and find out how LLMs can leverage tools and do straightforward tasks. But in the next articles, we are going to dive deeper into different approaches and best practices for LLM agents.

So, let the journey begin.

Before moving on to the LLMs, let’s try defining what analytics is and what tasks we do as analysts.

My motto is that the goal of the analytical team is to assist the product teams make the precise decisions based on data within the available time. It’s a great mission, but to define the scope of the LLM-powered analyst, we should always decompose the analytical work further.

I just like the framework proposed by Gartner. It identifies 4 different Data and Analytics techniques:

• Descriptive analytics answers questions like “What happened?”. For instance, what was the revenue in December? This approach includes reporting tasks and dealing with BI tools.
• Diagnostic analytics goes a bit further and asks questions like “Why did something occur?”. For instance, why revenue decreased by 10% in comparison with the previous yr? This system requires more drill-down and slicing & dicing of your data.
• Predictive analytics allows us to get answers to questions like “What’s going to occur?”. The 2 cornerstones of this approach are forecasting (predicting the longer term for business-as-usual situations) and simulation (modelling different possible outcomes).
• Prescriptive analytics impacts the ultimate decisions. The common questions are “What should we deal with?” or “How could we increase volume by 10%?”.

Normally, corporations undergo all these stages step-by-step. It’s almost inconceivable to start out taking a look at forecasts and different scenario analyses if your organization hasn’t mastered descriptive analytics yet (you don’t have a knowledge warehouse, BI tools, or metrics definitions). So, this framework may also show the corporate’s data maturity.

Similarly, when an analyst grows from junior to senior level, she is going to likely undergo all these stages, ranging from well-defined reporting tasks and progressing to vague strategic questions. So, this framework is relevant on a person level as well.

If we return to our LLM-powered analyst, we should always deal with descriptive analytics and reporting tasks. It’s higher to start out from the fundamentals. So, we are going to deal with learning LLM to know the essential questions on data.

We’ve defined our focus for the primary prototype. So, we’re able to move on to the technical questions and discuss the concept of LLM agents and tools.

After we were using LLMs before (for instance, to do topic modelling here), we described the precise steps ourselves within the code. For instance, let’s take a look at the chain below. Firstly, we asked the model to find out the sentiment for a customer review. Then, depending on the sentiment, extract from the review either the benefits or disadvantages mentioned within the text.

In this instance, we clearly defined the LLM’s behaviour, and the LLM solved this task pretty much. Nonetheless, this approach won’t work if we construct something more high-level and vague, like an LLM-powered analyst.

For those who’ve ever worked as or with an analyst for at the very least sooner or later, you’d know that analysts are getting an unlimited range of various questions and asks, ranging from basic questions (like “How many purchasers did we’ve on our site yesterday?” or “Could you make a graph for our Board meeting tomorrow?”) to very high-level ones (for instance, “What are the important customer pain points?” or “What market should we launch next?”). It goes without saying it’s not feasible to explain all possible scenarios.

Nonetheless, there’s an approach that might help us — agents. The core idea of the agents is to make use of LLMs as a reasoning engine that might select what to do next and when it’s time to return the ultimate answer to the shopper. It sounds pretty near our behaviour: we get a task, define needed tools, use them, after which come back with the ultimate answer when ready.

The essential concept related to agents (that I’ve already mentioned above) is tools. Tools are functions that LLM could invoke to get missing information (for instance, execute SQL, use a calculator or call a search engine). Tools are crucial because they help you bring LLMs to the subsequent level and interact with the world. In this text, we are going to primarily deal with OpenAI functions as tools.

OpenAI has fine-tuned models to give you the chance to work with functions in order that:

• You’ll be able to pass to the model the list of functions with descriptions;
• If it’s relevant to your query, the model will return you a function call — function name and input parameters to call it.

You’ll find more info and the up-to-date list of models that support functions within the documentation.

There are two distinguished use cases to make use of functions with LLMs:

• Tagging & extraction — in these cases, functions are used to make sure the output format of the model. As an alternative of the standard output with content, you’ll get a structured function call.
• Tools & routing — this can be a more exciting use case that permits you to create an agent.

Let’s start with the more straightforward use case of extraction to learn the way to use OpenAI functions.

You would possibly wonder what’s the difference between tagging and extraction. These terms are pretty close. The one difference is whether or not the model extracts info presented within the text or labels the text providing latest information (i.e. defines language or sentiment).

Since we’ve decided to deal with descriptive analytics and reporting tasks, let’s use this approach to structure incoming data requests and pull the next components: metrics, dimensions, filters, period and desired output.

It is going to be an example of extraction since we only need information present within the text.

OpenAI Completion API basic example

First, we’d like to define the function. OpenAI expects a function description as a JSON. This JSON can be passed to LLM, so we’d like to inform it all of the context: what this function does and the way to use it.

Here is an example of a function JSON. We’ve specified:

• name and description for the function itself,
• type and description for every argument,
• the list of required input parameters for the function.

extraction_functions = [
{
"name": "extract_information",
"description": "extracts information",
"parameters": {
"type": "object",
"properties": {
"metric": {
"type": "string",
"description": "main metric we need to calculate, for example, 'number of users' or 'number of sessions'",
},
"filters": {
"type": "string",
"description": "filters to apply to the calculation (do not include filters on dates here)",
},
"dimensions": {
"type": "string",
"description": "parameters to split your metric by",
},
"period_start": {
"type": "string",
"description": "the start day of the period for a report",
},
"period_end": {
"type": "string",
"description": "the end day of the period for a report",
},
"output_type": {
"type": "string",
"description": "the desired output",
"enum": ["number", "visualisation"]
}
},
"required": ["metric"],
},
}
]

There’s no must implement the function itself on this use case because we won’t be using it. We only get LLM responses in a structured way as function calls.

Now, we could use the usual OpenAI Chat Completion API to call the function. We passed to the API call:

• model — I’ve used the most recent ChatGPT 3.5 Turbo that may work with functions,
• list of messages — one system message to establish the context and a user request,
• list of functions we’ve defined earlier.

import openai
messages = [
{
"role": "system",
"content": "Extract the relevant information from the provided request."
},
{
"role": "user",
"content": "How did number of iOS users change over time?"
}
]
response = openai.ChatCompletion.create(
model = "gpt-3.5-turbo-1106", 
messages = messages,
functions = extraction_functions
)
print(response)

Consequently, we got the next JSON.

{
"id": "chatcmpl-8TqGWvGAXZ7L43gYjPyxsWdOTD2n2",
"object": "chat.completion",
"created": 1702123112,
"model": "gpt-3.5-turbo-1106",
"decisions": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"function_call": {
"name": "extract_information",
"arguments": "{"metric":"number of users","filters":"platform='iOS'","dimensions":"date","period_start":"2021-01-01","period_end":"2021-12-31","output_type":"visualisation"}"
}
},
"finish_reason": "function_call"
}
],
"usage": {
"prompt_tokens": 159,
"completion_tokens": 53,
"total_tokens": 212
},
"system_fingerprint": "fp_eeff13170a"
}

Keep in mind that functions and performance calls can be counted into the tokens limits and be billed.

The model returned a function call as an alternative of a standard response: we will see that the content is empty and finish_reason is the same as function_call. Within the response, there are also the input parameters for the function call:

• metric = "variety of users",
• filters = "platform = 'iOS'",
• dimensions = "date",
• period_start = "2021-01-01",
• period_start = "2021-12-31",
• output_type = "visualisation".

The model did a fairly good job. The one problem is that it presumed the period out of nowhere. We will fix it by adding more explicit guidance to the system message, for instance, "Extract the relevant information from the provided request. Extract ONLY the knowledge presented within the initial request; don't add the rest. Return partial information if something is missing."

By default, models resolve whether to make use of functions independently (function_call = 'auto'). We will require it to return a selected function call each time or not to make use of functions in any respect.

# all the time calling extract_information function
response = openai.ChatCompletion.create(
model = "gpt-3.5-turbo-1106",
messages = messages,
functions = extraction_functions,
function_call = {"name": "extract_information"}
)
# no function calls
response = openai.ChatCompletion.create(
model = "gpt-3.5-turbo-1106",
messages = messages,
functions = extraction_functions,
function_call = "none"
)

We’ve got the primary working program that uses LLM functions. That’s awesome. Nonetheless, it’s not very convenient to explain functions in a JSON. Let’s discuss the way to do it easier.

Using Pydantic to define functions

To define functions more conveniently, we will leverage Pydantic. Pydantic is the preferred Python library for data validation.

We’ve already used Pydantic to define LangChain Output Parser.

First, we’d like to create a category inheriting from the BaseModel class and define all of the fields (arguments of our function).

from pydantic import BaseModel, Field
from typing import Optional
class RequestStructure(BaseModel):
"""extracts information"""
metric: str = Field(description = "important metric we'd like to calculate, for instance, 'variety of users' or 'variety of sessions'")
filters: Optional[str] = Field(description = "filters to use to the calculation (don't include filters on dates here)")
dimensions: Optional[str] = Field(description = "parameters to separate your metric by")
period_start: Optional[str] = Field(description = "the beginning day of the period for a report")
period_end: Optional[str] = Field(description = "the tip day of the period for a report")
output_type: Optional[str] = Field(description = "the specified output", enum = ["number", "visualisation"])

Then, we will use LangChain to convert the Pydantic class into the OpenAI function.

from langchain.utils.openai_functions import convert_pydantic_to_openai_function
extract_info_function = convert_pydantic_to_openai_function(RequestStructure, 
name = 'extract_information')

LangChain validates the category we provided. For instance, it ensures that the function description is specified since LLM needs it to give you the chance to make use of this tool.

Consequently, we got the identical JSON to pass to LLM, but now we express it as a Pydantic class.

{'name': 'extract_information',
'description': 'extracts information',
'parameters': {'title': 'RequestStructure',
'description': 'extracts information',
'type': 'object',
'properties': {'metric': {'title': 'Metric',
'description': "important metric we'd like to calculate, for instance, 'variety of users' or 'variety of sessions'",
'type': 'string'},
'filters': {'title': 'Filters',
'description': 'filters to use to the calculation (don't include filters on dates here)',
'type': 'string'},
'dimensions': {'title': 'Dimensions',
'description': 'parameters to separate your metric by',
'type': 'string'},
'period_start': {'title': 'Period Start',
'description': 'the beginning day of the period for a report',
'type': 'string'},
'period_end': {'title': 'Period End',
'description': 'the tip day of the period for a report',
'type': 'string'},
'output_type': {'title': 'Output Type',
'description': 'the specified output',
'enum': ['number', 'visualisation'],
'type': 'string'}},
'required': ['metric']}}

Now, we could use it in our call to OpenAI. Let’s switch from OpenAI API to LangChain to make our API calls more modular.

Defining LangChain chain

Let’s define a sequence to extract needed information from the requests. We are going to use LangChain because it’s the preferred framework for LLMs. For those who haven’t worked with it before, I like to recommend you learn some basics in considered one of my previous articles.

Our chain is straightforward. It consists of an Open AI model and prompt with one variable request (a user message).

We’ve also used the bind function to pass functions argument to the model. The bind function allows us to specify constant arguments for our models that should not a part of the input (for instance, functions or temperature).

from langchain.prompts import ChatPromptTemplate
from langchain.chat_models import ChatOpenAI
model = ChatOpenAI(temperature=0.1, model = 'gpt-3.5-turbo-1106')
.bind(functions = [extract_info_function])
prompt = ChatPromptTemplate.from_messages([
("system", "Extract the relevant information from the provided request. 
Extract ONLY the information presented in the initial request. 
Don't add anything else. 
Return partial information if something is missing."),
("human", "{request}")
])
extraction_chain = prompt | model

Now it’s time to try our function. We’d like to make use of the invoke method and pass a request.

extraction_chain.invoke({'request': "How many purchasers visited our site on iOS in April 2023 from different countries?"})

Within the output, we got AIMessage with none content but with a function call.

AIMessage(
content='', 
additional_kwargs={
'function_call': {
'name': 'extract_information', 
'arguments': '''{
"metric":"number of consumers", "filters":"device = 'iOS'",
"dimensions":"country", "period_start":"2023-04-01",
"period_end":"2023-04-30", "output_type":"number"}
'''}
}
)

So, we’ve learned the way to use OpenAI functions in LangChain to get structured output. Now, let’s move on to the more interesting use case — tools and routing.

It’s time to make use of tools and empower our model with external capabilities. Models on this approach are reasoning engines, and so they can resolve what tools to make use of and when (it’s called routing).

LangChain has an idea of tools — interfaces that agents can use to interact with the world. Tools could be functions, LangChain chains and even other agents.

We will easily convert tools into OpenAI functions using format_tool_to_openai_function and keep passing the functions argument to LLMs.

Defining a custom tool

Let’s teach our LLM-powered analyst to calculate the difference between two metrics. We all know that LLMs might make mistakes in math, so we would really like to ask a model to make use of a calculator as an alternative of counting by itself.

To define a tool, we’d like to create a function and use a @tool decorator.

from langchain.agents import tool
@tool
def percentage_difference(metric1: float, metric2: float) -> float:
"""Calculates the proportion difference between metrics"""
return (metric2 - metric1)/metric1*100

Now, this function has name and description parameters that can be passed to LLMs.

print(percentage_difference.name)
# percentage_difference.name
print(percentage_difference.args)
# {'metric1': {'title': 'Metric1', 'type': 'number'},
# 'metric2': {'title': 'Metric2', 'type': 'number'}}
print(percentage_difference.description)
# 'percentage_difference(metric1: float, metric2: float) -> float - Calculates the proportion difference between metrics'

These parameters can be used to create an OpenAI function specification. Let’s convert our tool to an OpenAI function.

from langchain.tools.render import format_tool_to_openai_function
print(format_tool_to_openai_function(percentage_difference))

We got the next JSON because the result. It outlines the structure, but field descriptions are missing.

{'name': 'percentage_difference',
'description': 'percentage_difference(metric1: float, metric2: float) -> float - Calculates the proportion difference between metrics',
'parameters': {'title': 'percentage_differenceSchemaSchema',
'type': 'object',
'properties': {'metric1': {'title': 'Metric1', 'type': 'number'},
'metric2': {'title': 'Metric2', 'type': 'number'}},
'required': ['metric1', 'metric2']}
}

We will use Pydantic to specify a schema for the arguments.

class Metrics(BaseModel):
metric1: float = Field(description="Base metric value to calculate the difference")
metric2: float = Field(description="Recent metric value that we compare with the baseline")
@tool(args_schema=Metrics)
def percentage_difference(metric1: float, metric2: float) -> float:
"""Calculates the proportion difference between metrics"""
return (metric2 - metric1)/metric1*100

Now, if we convert a new edition to the OpenAI function specification, it can include argument descriptions. It’s significantly better since we could share all of the needed context with the model.

{'name': 'percentage_difference',
'description': 'percentage_difference(metric1: float, metric2: float) -> float - Calculates the proportion difference between metrics',
'parameters': {'title': 'Metrics',
'type': 'object',
'properties': {'metric1': {'title': 'Metric1',
'description': 'Base metric value to calculate the difference',
'type': 'number'},
'metric2': {'title': 'Metric2',
'description': 'Recent metric value that we compare with the baseline',
'type': 'number'}},
'required': ['metric1', 'metric2']}}

So, we’ve defined the tool that LLM will give you the chance to make use of. Let’s try it in practice.

Using a tool in practice

Let’s define a sequence and pass our tool to the function. Then, we could test it on a user request.

model = ChatOpenAI(temperature=0.1, model = 'gpt-3.5-turbo-1106')
.bind(functions = [format_tool_to_openai_function(percentage_difference)])
prompt = ChatPromptTemplate.from_messages([
("system", "You are a product analyst willing to help your product team. You are very strict to the point and accurate. You use only facts, not inventing information."),
("user", "{request}")
])
analyst_chain = prompt | model
analyst_chain.invoke({'request': "In April we had 100 users and in May only 95. What's difference in percent?"})

We got a function call with the proper arguments, so it’s working.

AIMessage(content='', additional_kwargs={
'function_call': {
'name': 'percentage_difference', 
'arguments': '{"metric1":100,"metric2":95}'}
}
)

To have a more convenient technique to work with the output, we will useOpenAIFunctionsAgentOutputParser. Let’s add it to our chain.

from langchain.agents.output_parsers import OpenAIFunctionsAgentOutputParser
analyst_chain = prompt | model | OpenAIFunctionsAgentOutputParser()
result = analyst_chain.invoke({'request': "There have been 100 users in April and 110 users in May. How did the variety of users modified?"})

Now, we got output in a more structured way, and we could easily retrieve arguments for our tool as result.tool_input .

AgentActionMessageLog(
tool='percentage_difference', 
tool_input={'metric1': 100, 'metric2': 110}, 
log="nInvoking: `percentage_difference` with `{'metric1': 100, 'metric2': 110}`nnn", 
message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'percentage_difference', 'arguments': '{"metric1":100,"metric2":110}'}})]
)

So, we could execute the function because the LLM requested like this.

statement = percentage_difference(result.tool_input)
print(statement)
# 10

If we wish to get the ultimate answer from the model, we’d like to pass the function execution result back. To do it, we’d like to define a message list to pass to the model observations.

from langchain.prompts import MessagesPlaceholder
model = ChatOpenAI(temperature=0.1, model = 'gpt-3.5-turbo-1106')
.bind(functions = [format_tool_to_openai_function(percentage_difference)])
prompt = ChatPromptTemplate.from_messages([
("system", "You are a product analyst willing to help your product team. You are very strict to the point and accurate. You use only facts, not inventing information."),
("user", "{request}"),
MessagesPlaceholder(variable_name="observations")
])
analyst_chain = prompt | model | OpenAIFunctionsAgentOutputParser()
result1 = analyst_chain.invoke({
'request': "There have been 100 users in April and 110 users in May. How did the variety of users modified?",
"observations": []
})
statement = percentage_difference(result1.tool_input)
print(statement)
# 10

Then, we’d like so as to add the statement to our observations variable. We could use format_to_openai_functions function to format our leads to an expected way for the model.

from langchain.agents.format_scratchpad import format_to_openai_functions
format_to_openai_functions([(result1, observation), ])

Consequently, we got such a message that the LLM can understand.

[AIMessage(content='', additional_kwargs={'function_call': {'name': 'percentage_difference', 
'arguments': '{"metric1":100,"metric2":110}'}}),
FunctionMessage(content='10.0', name='percentage_difference')]

Let’s invoke our chain another time, passing the function execution result as an statement.

result2 = analyst_chain.invoke({
'request': "There have been 100 users in April and 110 users in May. How did the variety of users modified?",
"observations": format_to_openai_functions([(result1, observation)])
})

Now, we got the outcome from the model, which sounds reasonable.

AgentFinish(
return_values={'output': 'The variety of users increased by 10%.'}, 
log='The variety of users increased by 10%.'
)

If we were working with vanilla OpenAI Chat Completion API, we could just add one other message with role = tool . You’ll find an in depth example here.

If we turn on debug, we will see the precise prompt that was passed to OpenAI API.

System: You might be a product analyst willing to assist your product team. You might be very strict to the purpose and accurate. You employ only facts, not inventing information.
Human: There have been 100 users in April and 110 users in May. How did the variety of users modified?
AI: {'name': 'percentage_difference', 'arguments': '{"metric1":100,"metric2":110}'}
Function: 10.0

To change on LangChain debug, execute the next code and invoke your chain to see what is happening under the hood.

import langchain
langchain.debug = True

We’ve tried to work with one tool, but let’s extend our toolkit and see how LLM could handle it.

Routing: using multiple tools

Let’s add a pair more tools to our analyst’s toolkit:

• get monthly energetic users
• using Wikipedia.

First, let’s define a dummy function to calculate the audience with filters by month and city. We are going to again use Pydantic to specify the input arguments for our function.

import datetime
import random
class Filters(BaseModel):
month: str = Field(description="Month of customer's activity within the format %Y-%m-%d")
city: Optional[str] = Field(description="City of residence for purchasers (by default no filter)", 
enum = ["London", "Berlin", "Amsterdam", "Paris"])
@tool(args_schema=Filters)
def get_monthly_active_users(month: str, city: str = None) -> int:
"""Returns variety of energetic customers for the required month"""
dt = datetime.datetime.strptime(month, '%Y-%m-%d')
total = dt.yr + 10*dt.month
if city is None:
return total
else:
return int(total*random.random())

Then, let’s use the wikipedia Python package to permit model query Wikipedia.

import wikipedia
class Wikipedia(BaseModel):
term: str = Field(description="Term to go looking for")
@tool(args_schema=Wikipedia)
def get_summary(term: str) -> str:
"""Returns basic knowledge in regards to the given term provided by Wikipedia"""
return wikipedia.summary(term)

Let’s define a dictionary with all of the functions our model knows now. This dictionary will help us to do routing later.

toolkit = {
'percentage_difference': percentage_difference,
'get_monthly_active_users': get_monthly_active_users,
'get_summary': get_summary
}
analyst_functions = [format_tool_to_openai_function(f) 
for f in toolkit.values()]

I’ve made a few changes to our previous setup:

• I tweaked the system prompt a bit to force LLM to seek the advice of with Wikipedia if it needs some basic knowledge.
• I’ve modified the model to GPT 4 since it’s higher for handling tasks requiring reasoning.

from langchain.prompts import MessagesPlaceholder
model = ChatOpenAI(temperature=0.1, model = 'gpt-4-1106-preview')
.bind(functions = analyst_functions)
prompt = ChatPromptTemplate.from_messages([
("system", "You are a product analyst willing to help your product team. You are very strict to the point and accurate. 
You use only information provided in the initial request. 
If you need to determine some information i.e. what is the name of the capital, you can use Wikipedia."),
("user", "{request}"),
MessagesPlaceholder(variable_name="observations")
])
analyst_chain = prompt | model | OpenAIFunctionsAgentOutputParser()

We will invoke our chain with all of the functions. Let’s start with a fairly straightforward query.

result1 = analyst_chain.invoke({
'request': "What number of users were in April 2023 from Berlin?",
"observations": []
})
print(result1)

We got within the result function call for get_monthly_active_users with input parameters — {'month': '2023–04–01', 'city': 'Berlin'} , which looks correct. The model was capable of find the precise tool and solve the duty.

Let’s attempt to make task a bit more complex.

result1 = analyst_chain.invoke({
'request': "How did the variety of users from the capital of Germany
change between April and May 2023?",
"observations": []
})

Let’s pause for a minute and think how we would really like the model to reason. It’s evident that there’s not enough information for the model to reply right away, so it must make a bunch of function calls:

• call Wikipedia to get the capital of Germany
• call the get_monthly_active_users function twice to get MAU for April and May
• call percentage_difference to calculate the difference between metrics.

It looks pretty complex. Let’s see whether ChatGPT would give you the chance to handle this query.

For the primary call, LLM returned back a function call to Wikipedia with the next params — {'term': 'capital of Germany'}. Thus far, it’s following our plan.

Let’s provide the statement and see what the subsequent steps can be.

observation1 = toolkit[result1.tool](result1.tool_input)
print(observation1)
# The capital of Germany is the  city state of Berlin. It's the seat of 
# the President of Germany, whose official residence is Schloss Bellevue. 
# The Bundesrat ("federal council") is the representation of the Federal States 
# (Bundesländer) of Germany and has its seat at the previous Prussian Herrenhaus 
# (House of Lords). Though a lot of the ministries are seated in Berlin, 
# a few of them, in addition to some minor departments, are seated in Bonn, 
# the previous capital of West Germany.
# Although Berlin is officially the capital of the Federal Republic of Germany,
# 8,000 out of the 18,000 total officials employed on the federal bureaucracy 
# still work in Bonn, about 600 km (370 mi) away from Berlin.
# source: https://en.wikipedia.org/wiki/Capital_of_Germany 
result2 = analyst_chain.invoke({
'request': "How did the variety of users from the capital of Germany change between April and May 2023?",
"observations": format_to_openai_functions([(result1, observation1)])
})

The model desires to execute get_monthly_active_users with arguments {'month': '2023–04–01', 'city': 'Berlin'}. Let’s do it and return the knowledge to the model once more.

observation2 = toolkit[result2.tool](result2.tool_input)
print(observation2)
# 168
result3 = analyst_chain.invoke({
'request': "How did the variety of users from the capital of Germany change between April and May 2023?",
"observations": format_to_openai_functions([(result1, observation1), (result2, observation2)])
})

Then, the model requests to call get_monthly_active_users again with arguments {'month': '2023–05–01', 'city': 'Berlin'}. Thus far, it’s doing a superb job. Let’s follow its logic.

observation3 = toolkit[result3.tool](result3.tool_input)
print(observation3)
# 1046
result4 = analyst_chain.invoke({
'request': "How did the variety of users from the capital of Germany change between April and May 2023?",
"observations": format_to_openai_functions(
[(result1, observation1), (result2, observation2), 
(result3, observation3)])
})

The following result’s a function call for percentage_difference with the next arguments {'metric1': 168, 'metric2': 1046}. Let’s calculate statement and invoke our chain another time. Hopefully, it can be the last step.

observation4 = toolkit[result4.tool](result4.tool_input)
print(observation4)
# 523.27
result5 = analyst_chain.invoke({
'request': "How did the variety of users from the capital of Germany change between April and May 2023?",
"observations": format_to_openai_functions(
[(result1, observation1), (result2, observation2), 
(result3, observation3), (result4, observation4)])
})

Ultimately, we got the next response from the model: The variety of users from Berlin, the capital of Germany, increased by roughly 523.27% between April and May 2023.

Here’s the whole scheme of the LLM calls for this query.

Within the above example, we triggered subsequent calls one after the other manually, but it may be easily automated.

It’s a incredible result, and we were capable of see how LLMs can do reasoning and utilize multiple tools. It took model 5 steps to attain the result, but it surely followed the plan we outlined initially, so it was a fairly logical path. Nonetheless, if you happen to plan to make use of LLMs in production, have in mind that it’d make mistakes and introduce evaluation and quality assurance processes.

You’ll find the total code on GitHub.

This text taught us the way to empower LLMs with external tools using OpenAI functions. We’ve examined two use cases: extraction to get structured output and routing to make use of external information for questions. The outcome inspires me since LLM could answer pretty complex questions using three different tools.

Let’s return to the initial query of whether LLMs can replace data analysts. Our current prototype is basic and removed from the junior analysts’ capabilities, but it surely’s only the start. Stay tuned! We are going to dive deeper into the several approaches to LLM agents. Next time, we are going to attempt to create an agent that may access the database and answer basic questions.

This text is inspired by the “Functions, Tools and Agents with LangChain” course from DeepLearning.AI