6.2. Query Answering

Query answering is self-explanatory but let’s make it more interesting. As a substitute of providing a paragraph to the agent and asking questions on the provided information, let’s provide a picture and ask the agent in regards to the content of the image.

I wrote a number of lines in a word document after which saved it as a *.jpg image in my local. Let’s first use PIL to see the image as follows:

from PIL import Image
image = Image.open('/content/jack.jpg')
image.show()

Results:

As you possibly can see, the image has a number of sentences about my imaginary friend in Seattle, named Jack. Next, let’s ask a matter from our agent and see the way it responds. I would really like to ask the agent about Jack’s favorite color, as follows:

task = "in the next 'document', what's Jack's favorite color?"
agent_starcoder.run(task, document=image)

Results:

Once more, let’s take a look at the very last sentence in the underside — That’s pretty good! We are able to see that text extraction shouldn’t be perfect, for instance, it extracted Tesla to ‘tesia’ but still, the agent returned the relevant portion of the image, which answered our query.

Let’s see what tools precisely the agent used:

task = "in the next 'document', what's Jack's favorite color?"
agent_starcoder.run(task, document=image, return_code=True)

Results: