In the television show Star Trek: The Next Generation, Captain Picard and his crew use the holodeck, a room capable of generating 3D environments, to train for missions and for entertainment. Virtual interactive environments are being used today to train robots before deploying them in the real world, a process known as “Sim2Real.” However, there is a shortage of virtual environments, with artists spending a significant amount of time manually creating them. This shortage presents a problem when training robots to navigate complex real-world environments.
To address this issue, a team of researchers at the University of Pennsylvania, in collaboration with researchers from Stanford, the University of Washington, and the Allen Institute for Artificial Intelligence, developed Holodeck. Named after the Star Trek technology, Holodeck is an AI-driven system that can generate a wide range of indoor environments based on user requests. Leveraging the knowledge embedded in large language models, Holodeck interprets user commands and creates interactive 3D environments that can be used to train embodied AI agents.
Holodeck works by engaging in a conversation with a large language model, breaking down user requests into specific parameters to create environments. Users can describe the environments they want using language, and Holodeck will generate them accordingly. The system follows a structured process to create environments step by step, from creating the floor and walls to adding furnishings and objects from a library of digital assets. Researchers tested Holodeck’s ability to generate realistic and accurate scenes against an earlier tool and found that users consistently preferred the environments created by Holodeck in terms of asset selection, layout coherence, and overall preference.
Furthermore, Holodeck was evaluated on its ability to generate scenes that are less typical in robotics research, such as stores, public spaces, and offices. Human evaluators preferred the scenes created by Holodeck over those generated using human-created rules, across a variety of indoor environments. The researchers also used scenes generated by Holodeck to fine-tune an embodied AI agent, helping it to navigate new environments more efficiently and safely. By training the agent in diverse virtual spaces, including offices, daycares, gyms, and arcades, Holodeck improved the agent’s ability to navigate and interact with its surroundings.
In testing the agent’s performance in finding specific objects in various environments, it was found that fine-tuning the agent using scenes generated by Holodeck significantly improved its success rate compared to using a different tool. In some scenarios, the agent’s success rate increased fivefold when trained with scenes created by Holodeck. This ability to efficiently generate diverse environments for training robots represents a significant advancement in the field, opening up new possibilities for research and development in robotics and AI. Overall, Holodeck offers a powerful tool for training embodied AI agents to navigate and interact with complex real-world environments effectively.
