On October 8, NVIDIA published seven examples of how its staff use artificial intelligence agents to build simulations with Omniverse libraries. The projects range from a robot in a warehouse to an application of the International Space Station. The company describes prototypes and internal experiments: their results do not constitute an independent evaluation of the agents’ performance.
The process NVIDIA presents combines natural-language instructions, code generation and human review. Developers set tasks, inspect results and request corrections; Omniverse provides GPU-accelerated physics, rendering and sensor simulation. For application developers, the interest lies in how to connect these components and check their behavior, rather than merely generating a visual scene.
Robots: from warehouses to disassembling parts
Frank DeLise, Omniverse product manager, directed the Astra agent to integrate a SimReady warehouse and a humanoid into a simulator with first- and third-person views. According to NVIDIA, the agent generated animations and code, linking physics libraries, scene updates, rendering and the interface. The goal was to explore how to perform warehouse tasks before automating them.
In Robo Olympics, Tae Kim guided Astra with sports videos and instructions to develop controllers for Unitree G1 humanoids. The project used Newton Physics Engine, NVIDIA Warp and ovrtx rendering. The company reports that a robot cleared a hurdle in 64 of 100 simulated trials; that figure applies to a specific test and does not demonstrate a capability in a physical robot.
Another experiment tackled a disassembly challenge: reaching the bolts on an automotive suspension. Jens Jebens directed Astra to model it in PTC Onshape and configure it in Isaac Sim. NVIDIA explains that the agent measured the available space and designed a suitable wrench; Jebens reported removing a component in the simulation, linking tool design with access testing.
Sensors and driving: compare before correcting
Doyub Kim developed the Zero to Alpamayo prototype with Astra, based on Market Street in San Francisco. According to NVIDIA, he connected the scene elements, traffic, RTX sensors and Alpamayo driving model in stages, reviewing each integration. A separate experiment used Cosmos3-Nano to vary weather and lighting in simulated videos and compare the model’s responses under those conditions.
Ashley Reid started from a different point: the differences between simulated sensors and recorded data. NVIDIA says that, under her direction, Astra and Claude Fable 5 agents created two digital twins and improved two others over about three days. The corrections covered missing objects, geometry and materials; acceptance depended on camera and LiDAR metrics, not just on whether the scenes looked realistic.
A space station and editable rooms
Nic Johns used NASA resources to create an OpenUSD model of the International Space Station with Astra, including telemetry, and bring it to a browser. The workflow described by NVIDIA included Blender and libraries for rendering, scene execution and streaming. After the initial instruction, Johns asked to move the scene to Earth’s daytime side so the planet would be visible.
Chirag Majithia, for his part, turned stereo camera captures into an editable room, with human review of objects and adjustments to collisions involving doors and drawers in Isaac Sim. To get started with a measurable task, NVIDIA recommends rendering an OpenUSD scene using ovrtx’s minimal Python example and defining a sensor metric that can be compared with recorded data.
