{"version":1,"type":"story","url":"https://digestai.news/story/nvidia-releases-cosmos3-h-dreams-real-time-generative-surgical-simulat","json":"https://digestai.news/story/nvidia-releases-cosmos3-h-dreams-real-time-generative-surgical-simulat.json","markdown":"https://digestai.news/story/nvidia-releases-cosmos3-h-dreams-real-time-generative-surgical-simulat.md","slug":"nvidia-releases-cosmos3-h-dreams-real-time-generative-surgical-simulat","headline":"Nvidia releases Cosmos3-H-Dreams real-time generative surgical simulator","summary":"Nvidia announced Cosmos3-H-Dreams, its first real‑time generative surgical simulator trained on clinical cholecystectomy recordings. The system learns visual dynamics directly from synchronized video and robot kinematics, replacing explicit geometry and physics models. It runs interactively on a single NVIDIA RTX PRO 6000 GPU using the FlashDreams accelerated streaming‑inference library. To the authors’ knowledge, this is the first generative simulator that supports live kinematic control for a real surgical procedure.\n\nCosmos3-H-Dreams builds on the 16‑billion‑parameter Cosmos3‑H‑Embodied‑Nano model and specializes a bidirectional teacher on curated cholecystectomy video at 832 × 480 resolution sampled at 10 Hz. Training first extends the horizon to 24 future frames and then to 72 frames (2.4 s and 7.2 s). Distillation creates a causal student that generates video in short blocks as new actions arrive, using four‑step and two‑step checkpoints. The model accepts 28 action dimensions – pose and gripper data for two instruments (a left Cadiere grasper and a right monopolar hook) plus energy and thumb‑stick controls. Developers can explore the trade‑off between visual quality and inference cost, and the repository includes a step‑by‑step guide for adapting the pipeline to new datasets. The authors stress that Cosmos3-H-Dreams is a research simulator, not a validated system for patient care, and that visual realism is only a starting point for further biomechanical validation.","keyPoints":["Cosmos3-H-Dreams runs in real time on a single NVIDIA RTX PRO 6000 GPU using FlashDreams streaming inference.","Model builds on 16‑billion‑parameter Cosmos3‑H‑Embodied‑Nano and trains on 832 × 480 video at 10 Hz from clinical cholecystectomy recordings.","Provides four‑step and two‑step checkpoints, supporting 28‑dimensional instrument actions for two tools (Cadiere grasper, monopolar hook)."],"whyItMatters":"Real‑time generative surgical simulation could speed training and research, showing how large vision‑action models can model complex medical procedures without explicit physics.","category":{"slug":"models","name":"Generative AI & Models","url":"https://digestai.news/category/models"},"entities":{"companies":["NVIDIA","CMR Surgical"],"models":["Cosmos3-H-Dreams","Cosmos3-H-Embodied-Nano"],"people":[]},"firstPublishedAt":"2026-10-20T18:17:47Z","updatedAt":"2026-10-20T18:37:16Z","sourceCount":2,"hasPrimarySource":true,"sources":[{"outlet":"Hugging Face","title":"NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation","url":"https://huggingface.co/blog/nvidia/cosmos3-h-dreams","publishedAt":"2026-10-20T18:17:47Z","type":"primary","primary":true,"lead":true},{"outlet":"Unite.AI","title":"NVIDIA Releases Cosmos3-H-Dreams Real-Time Surgical World Model","url":"https://unite.ai/nvidia-releases-cosmos3-h-dreams-real-time-surgical-world-model","publishedAt":"2026-10-20T18:37:16Z","type":"press","primary":false,"lead":false}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Nvidia releases Cosmos3-H-Dreams real-time generative surgical simulator\", 20 October 2026, https://digestai.news/story/nvidia-releases-cosmos3-h-dreams-real-time-generative-surgical-simulat","publisher":"Digest AI","title":"Nvidia releases Cosmos3-H-Dreams real-time generative surgical simulator","datePublished":"2026-10-20T18:17:47Z","url":"https://digestai.news/story/nvidia-releases-cosmos3-h-dreams-real-time-generative-surgical-simulat"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}