No more unstructured data outputs; turn ChatGPT’s completions into structured JSON!

A number of months ago, OpenAI released their API to most people, which excited many developers who wanted to utilize ChatGPT’s outputs in a scientific way. As exciting has this has been, it’s equally been a little bit of a nightmare since we programmers are inclined to work within the realm of structured data types. We like integers, booleans, and lists. The unstructured string may be unwieldy to take care of, and in an effort to get consistent results, a programmer is required to face their worst nightmare: developing an everyday expression () for correct parsing. 🤢

After all, prompt engineering can actually help quite a bit here, but it surely’s still not perfect. For instance, if you desire to have ChatGPT analyze the sentiment of a movie review for positivity or negativity, you would possibly structure a prompt that appears like this:

prompt = f'''
Please perform a sentiment evaluation on the next movie review:
{MOVIE_REVIEW_TEXT}
Please output your response as a single word: either "Positive" or "Negative".
'''

This prompt truthfully does pretty decently, but the outcomes aren’t precisely consistent. For instance, I even have seen ChatGPT produce outputs that appear like the next when it comes to the movie sentiment example:

• Positive
• positive
• Positive.

This won’t look like a giant deal, but on the earth of programming, those are NOT equal. Again, you’ll be able to get around a less complicated example like this with a little bit of , but beyond the incontrovertible fact that most individuals (including myself) are terrible at writing regular expressions, there are simply some instances where even can’t parse the data appropriately.

As you’ll be able to tell, programmers have been hoping that OpenAI would add functionality to support structured JSON outputs, and OpenAI has delivered in the shape of function calling. Function calling is precisely because it sounds: it allows ChatGPT to supply arguments that may interact with a custom function in a fashion that uses structured data types. Yup, no more fancy prompt engineering and to cross your fingers and hope you get the proper end result. On this post, we’ll cover the right way to make use of this latest functionality, but first, let’s start with an example of how we used to try and produce structured data outputs with prompt engineering and .

Before we jump into the majority of our post, please allow me to share a link to this Jupyter notebook in my GitHub. This notebook incorporates all of the code I can be running (and more) as a part of this blog post. Moreover, I’d encourage you to examine out OpenAI’s official function calling documentation for anything that I’ll not cover here.

To reveal what we used to do within the “pre-function calling days”, I wrote a small little bit of text about myself, and we’ll be using the OpenAPI to extract bits of data from this text. Here is the “About Me” text we’ll be working with:

Hello! My name is David Hundley. I’m a principal machine learning engineer at State Farm. I enjoy learning about AI and teaching what I learn back to others. I even have two daughters. I drive a Tesla Model 3, and my favorite video game series is The Legend of Zelda.

Let’s say I would like to extract the next bits of data from that text:

• Name
• Job title
• Company
• Number of youngsters as an integer (This is very important!)
• Automobile make
• Automobile model
• Favorite video game series

Here’s how I’d engineer a few-shot prompt in an effort to produce a structured JSON output:

# Engineering a prompt to extract as much information from "About Me" as a JSON object
about_me_prompt = f'''
Please extract information as a JSON object. Please search for the next pieces of data.
Name
Job title
Company
Number of youngsters as a single integer
Automobile make
Automobile model
Favorite video game series
That is the body of text to extract the data from:
{about_me}
'''
# Getting the response back from ChatGPT (gpt-3.5-turbo)
openai_response = openai.ChatCompletion.create(
model = 'gpt-3.5-turbo',
messages = [{'role': 'user', 'content': about_me_prompt}]
)
# Loading the response as a JSON object
json_response = json.loads(openai_response['choices'][0]['message']['content'])
json_response

Let’s try how ChatGPT returned this completion to me:

As you’ll be able to see, this actually isn’t bad. But it surely’s not ideal and will prove to be dangerous for the next reasons:

• We aren’t guaranteed that OpenAI’s response will provide a clean JSON output. It could have produced something like “Here is your JSON:” followed by the JSON output, meaning that in an effort to use json.loads() to parse the string right into a JSON object, we’d first need to strip out that little little bit of text that opens the response.
• We aren’t guaranteed that the keys within the key-value pairs of the JSON object can be consistent from API call to API call. Recall the instance from above of the three instances of the word Positive. That is precisely the identical risk you run attempting to have ChatGPT parse out keys through few-shot prompt engineering. The one way you may perhaps lock this down is with , which comes with its own baggage as we already discussed.
• We aren’t guaranteed to receive our responses in the correct data type format. While our prompt engineering to extract number of youngsters did parse right into a proper integer, we’re on the mercy of crossing our fingers and hoping we get that consistent result for each API call.

We could summarize these issues right into a single statement: Without function calling, we aren’t guaranteed to get consistent results which are vital for the precision required for systematic implementation. It’s a nontrivial issue that may be very difficult to treatment through prompt engineering and regular expressions.

Now that we’ve built an intuition around why getting structured outputs from ChatGPT was formerly problematic, let’s move into the brand new function calling capability introduced by OpenAI.

Function calling is definitely a little bit of a misnomer. OpenAI is just not actually running your code in a real function call. Relatively, it’s simply organising the structured arguments you’d have to execute your personal custom functions, and I’d argue that is preferred behavior. Whilst you is likely to be considering that it doesn’t make sense that the OpenAI API isn’t executing your custom function, consider that in an effort to try this, you’d need to pass that function code into ChatGPT. This function code probably incorporates proprietary information that you simply would NOT want to reveal to anybody, hence why it’s good that you simply don’t actually need to pass this code to utilize OpenAI’s function calling.

Let’s jump into an example of the right way to enable function calling with a single custom function. Using our “About Me” sample text from the previous section, let’s create a custom function called extract_person_info. This function needs just three bits of data: person name, job title, and number of youngsters. (We’ll revisit extracting the remaining of the data in the following section; I just want to start out simpler for now.) This tradition function is intentionally quite simple and can simply take our arguments and print them together in a single string. Here’s the code for this:

def extract_person_info(name, job_title, num_children):
'''
Prints basic "About Me" information
Inputs:
- name (str): Name of the person
- job_title (str): Job title of the person
- num_chilren (int): The number of youngsters the parent has.
'''
print(f'This person's name is {name}. Their job title is {job_title}, and so they have {num_children} children.')

With a purpose to make use of function calling, we want to establish a JSON object in a particular way that notes the name of our custom function and what data elements we hope ChatGPT will extract from the body of the text. Due to specificity on how this JSON object should look, I’d encourage you reference OpenAI’s developer documentation if you desire to know any details that I don’t cover here.

(Note: Within the OpenAI documentation, I noticed one element within the JSON object called required that seemingly indicates that a parameter have to be present for ChatGPT to properly recognize the function. I attempted testing this out, and either this isn’t how this functionality works or I did something mistaken. Either way, I transparently do not know what this required parameter indicates. 😅)

Here is how we want to structure our JSON object to utilize our custom function:

my_custom_functions = [
{
'name': 'extract_person_info',
'description': 'Get "About Me" information from the body of the input text',
'parameters': {
'type': 'object',
'properties': {
'name': {
'type': 'string',
'description': 'Name of the person'
},
'job_title': {
'type': 'string',
'description': 'Job title of the person'
},
'num_children': {
'type': 'integer',
'description': 'Number of children the person is a parent to'
}
}
}
}
]

You’re probably already accustomed to JSON syntax, although let me draw attention for a moment to the information type associated to every property. In case you are a Python developer like myself, remember that the information typing for this JSON structure is NOT directly corresponding to how we define data structures in Python. Generally speaking, we will find equivalencies that work out alright, but when you desire to know more concerning the specific data types related to this JSON structure, try this documentation.

Now we’re able to make our API call to get the outcomes! Using the Python client, you’ll notice the syntax may be very much like how we obtain completions on the whole. We’re just going so as to add some additional arguments into this call that represent our function calling:

# Getting the response back from ChatGPT (gpt-3.5-turbo)
openai_response = openai.ChatCompletion.create(
model = 'gpt-3.5-turbo',
messages = [{'role': 'user', 'content': about_me}],
functions = my_custom_functions,
function_call = 'auto'
)
print(openai_response)

As you’ll be able to see, we simply pass in our list of custom functions (or in our case for now, our singular custom function) because the functions parameter, and also you’ll also notice a further parameter called function_call that we’ve set to auto. Don’t worry about this for now as we’ll revisit what this auto piece is doing in the following section.

Let’s run this code and try the complete API response from ChatGPT

For essentially the most part, this response looks the identical as a non-function call response, but now there’s a further field within the response called function_call, and nested under this dictionary are two additional items: name and arguments. name indicates the name of our custom function that we can be calling with ChatGPT’s output, and arguments incorporates a string that we will load using json.loads() to load our custom function arguments as a JSON object.

Notice now that we’re getting rather more consistency than we were in our pre-function calling methodology. Now we may be guaranteed that the keys of the key-value pairs WILL be consistent, and the information types WILL be consistent. No need for fancy prompt engineering or regular expressions!

That’s the core of OpenAI’s function calling! After all, this was a really simplistic example to get you going, but you most likely have additional questions. Let’s cover those on this next section.

The previous section covered a quite simple example of the right way to enable function calling, but for those who’re like me, you most likely have some additional questions beyond this point. Naturally, I can’t cover all these questions, but I do need to cover two big ones which are barely more advanced than what we covered within the previous section.

What if the prompt I submit doesn’t contain the data I would like to extract per my custom function?

In our original example, our custom function sought to extract three very specific bits of data, and we demonstrated that this worked successfully by passing in my custom “About Me” text as a prompt. But you is likely to be wondering, what happens for those who pass in another prompt that doesn’t contain that information?

Recall that we set a parameter in our API client call called function_call that we set to auto. We’ll explore this even deeper in the following subsection, but what this parameter is actually doing is telling ChatGPT to make use of its best judgment in determining when to structure the output for one in all our custom functions.

So what happens once we submit a prompt that doesn’t match any of our custom functions? Simply put, it defaults to typical behavior as if function calling doesn’t exist. Let’s test this out with an arbitrary prompt: “How tall is the Eiffel Tower?”

As you’ll be able to see, we’re getting a typical “Completions” output though we passed in our custom function. Naturally, this is sensible since this arbitrary Eiffel Towel prompt incorporates none of the particular information we’re searching for.

What if I would like to pass multiple custom functions and a few of them have overlapping parameters?

Briefly, ChatGPT intelligently handles this with no problem. Where we previously passed in a single custom function as essentially a listing of Python dictionaries, we just have to keep adding additional Python dictionaries to this same list, each representing its own distinct function. Let’s add two latest functions: one called extract_vehicle_info and one other called extract_all_info. Here’s what our adjusted syntax looks like:

# Defining a function to extract only vehicle information
def extract_vehicle_info(vehicle_make, vehicle_model):
'''
Prints basic vehicle information
Inputs:
- vehicle_make (str): Make of the vehicle
- vehicle_model (str): Model of the vehicle
'''
print(f'Vehicle make: {vehicle_make}nVehicle model: {vehicle_model}')
# Defining a function to extract all information provided in the unique "About Me" prompt
def extract_vehicle_info(name, job_title, num_children, vehicle_make, vehicle_model, company_name, favorite_vg_series):
'''
Prints the complete "About Me" information
Inputs:
- name (str): Name of the person
- job_title (str): Job title of the person
- num_chilren (int): The number of youngsters the parent has
- vehicle_make (str): Make of the vehicle
- vehicle_model (str): Model of the vehicle
- company_name (str): Name of the corporate the person works for
- favorite_vg_series (str): Person's favorite video game series.
'''
print(f'''
This person's name is {name}. Their job title is {job_title}, and so they have {num_children} children.
They drive a {vehicle_make} {vehicle_model}.
They work for {company_name}.
Their favorite video game series is {favorite_vg_series}.
''')
# Defining how we would like ChatGPT to call our custom functions
my_custom_functions = [
{
'name': 'extract_person_info',
'description': 'Get "About Me" information from the body of the input text',
'parameters': {
'type': 'object',
'properties': {
'name': {
'type': 'string',
'description': 'Name of the person'
},
'job_title': {
'type': 'string',
'description': 'Job title of the person'
},
'num_children': {
'type': 'integer',
'description': 'Number of children the person is a parent to'
}
}
}
},
{
'name': 'extract_car_info',
'description': 'Extract the make and model of the person's car',
'parameters': {
'type': 'object',
'properties': {
'vehicle_make': {
'type': 'string',
'description': 'Make of the person's vehicle'
},
'vehicle_model': {
'type': 'string',
'description': 'Model of the person's vehicle'
}
}
}
},
{
'name': 'extract_all_info',
'description': 'Extract all information about a person including their vehicle make and model',
'parameters': {
'type': 'object',
'properties': {
'name': {
'type': 'string',
'description': 'Name of the person'
},
'job_title': {
'type': 'string',
'description': 'Job title of the person'
},
'num_children': {
'type': 'integer',
'description': 'Number of children the person is a parent to'
},
'vehicle_make': {
'type': 'string',
'description': 'Make of the person's vehicle'
},
'vehicle_model': {
'type': 'string',
'description': 'Model of the person's vehicle'
},
'company_name': {
'type': 'string',
'description': 'Name of the company the person works for'
},
'favorite_vg_series': {
'type': 'string',
'description': 'Name of the person's favorite video game series'
}
}
}
}
]

Notice specifically how the extract_all_info covers among the same parameters as our original extract_person_info function, so how does ChatGPT know which one to pick out? Simply put, ChatGPT looks for the most effective match. If we pass in a prompt that incorporates all of the arguments needed for the extract_all_info function, that’s the one it’ll select. But when we just pass in a prompt that incorporates either just easy details about me or a prompt about my vehicle, it’ll leverage the respective functions that try this. Let’s execute that in code here with a couple of samples:

• Sample 1: The unique “About Me” text. (See above.)
• Sample 2: “My name is David Hundley. I’m a principal machine learning engineer, and I even have two daughters.”
• Sample 3: “She drives a Kia Sportage.”

With each of the respective prompts, ChatGPT chosen the proper custom function, and we will specifically note that within the name value under function_call within the API’s response object. Along with this being a handy approach to discover which function to make use of the arguments for, we will programmatically map our actual custom Python function to this value to run the proper code appropriately. If that doesn’t make sense, perhaps this in code would make this more clear:

# Iterating over the three samples
for i, sample in enumerate(samples):
print(f'Sample #{i + 1}'s results:')
# Getting the response back from ChatGPT (gpt-3.5-turbo)
openai_response = openai.ChatCompletion.create(
model = 'gpt-3.5-turbo',
messages = [{'role': 'user', 'content': sample}],
functions = my_custom_functions,
function_call = 'auto'
)['choices'][0]['message']
# Checking to see that a function call was invoked
if openai_response.get('function_call'):
# Checking to see which specific function call was invoked
function_called = openai_response['function_call']['name']
# Extracting the arguments of the function call
function_args = json.loads(openai_response['function_call']['arguments'])
# Invoking the correct functions
if function_called == 'extract_person_info':
extract_person_info(*list(function_args.values()))
elif function_called == 'extract_vehicle_info':
extract_vehicle_info(*list(function_args.values()))
elif function_called == 'extract_all_info':
extract_all_info(*list(function_args.values()))

**Beware one thing**: Within the spirit of full transparency, I needed to run that code there multiple times to get it to supply like that. The difficulty is that since the extract_person_info and extract_all_info are more similar in nature, ChatGPT kept confusing those for each other. I assume the lesson to be learned here is that your functions must be extracting distinct information. I also only tested using gpt-3.5-turbo, so it’s possible that a more powerful model like GPT-4 could have handled that higher.

I hope you’ll be able to see now why function calling may be so powerful! On the subject of constructing applications that leverage Generative AI, this sort of function calling is a godsend for programmers. By not having to fret a lot now concerning the output JSON structure, we will now focus our time on constructing out other parts of the appliance. It’s an awesome time to be working on this space 🥳