How can we make decisions in each day life? We frequently are biased based on our common sense. What about robots? Can they make decisions based on common sense? Completing human instructions successfully requires embodied agents with common sense. Resulting from the necessity for more details of a practical world, the current LLMs yield infeasible motion sequences. 

Researchers on the Department of Automation and Beijing National Research Centre for Information Science and Technology proposed a TAsk Planning Agent ( TaPA ) in embodied tasks with physical scene constraints. These agents generate executable plans based on the present objects within the scene by aligning LLMs with the visual perception models. 

Researchers claim that TaPA can generate grounded plans without constraining task types and goal objects. They first created a multimodal dataset where each sample is a triplet of visual scenes, instructions, and corresponding plans. From the generated dataset, they finetuned the pre-trained LLaMA network by predicting the motion steps based on the thing list of the scene, which is further assigned as a task planner. 

The embodied agent then effectively visits the standing points to gather RGB images, providing sufficient information in various views to generalize the open-vocabulary detector for multi-view images. This overall process allows TaPA to generate the executable actions step-by-step, considering the scene information and the human instructions. 

How did they generate the multimodal dataset? One in all the ways is to utilize vision-language models and huge multimodal models. Nevertheless, as a result of the shortage of a large-scale multimodel dataset to coach the planning agent, it’s difficult to create and achieve embodied task planning that’s grounded in realistic indoor scenes. They resolved it using GPT-3.5 with the presented scene representation and design prompt to generate the large-scale multimodal dataset for tuning the planning agent. 

Researchers trained the duty planner from the pre-trained LLMs and constructed the multimodal dataset containing 80 indoor scenes with 15 K instructions and motion plans. They designed several image collection strategies to explore the encircling 3D scenes, like location selection criteria for random positions and rotated cameras for obtaining multi-view images for every location selection criteria. Inspired by the clustering methods, they divided all the scene into several sub-regions to enhance the performance of the perception. 

Researchers claim that TaPA agents achieve the next success rate of the generated motion plans than the state-of-the-art LLMs, including LlaMA and GPT-3.5, and huge multimodal models reminiscent of LLaVA. TaPA can higher understand the list of input objects with a 26.7% and 5% decrease in the proportion of hallucination cases in comparison with LLaVA and GPT-3.5, respectively. 

Researchers claim that their statistics of collected multimodal datasets indicate the tasks are rather more complex than the standard benchmarks on instruction following tasks with longer implementation steps and require further latest methods for optimization. 

Try the Paper. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 29k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more.

In case you like our work, please follow us on Twitter