Researchers introduce Mirror, a self‑supervised video‑to‑action model
The paper presents Mirror, a self‑supervised approach that learns to map video plans to actions without any pre‑collected action data or hand‑crafted rewards. Mirror iteratively tries to follow a visual plan, records the resulting video‑action pairs, and uses them to improve its own predictions, gradually closing the gap to the visual planner.
Key points
- Mirror learns video‑to‑action mappings through self‑supervised interaction, needing no prior action data
- Across Atari, OGBench, DeepMind Control and Minecraft, Mirror outperforms leading observation‑learning methods
- In Minecraft survival, Mirror self‑improves to acquire an iron pickaxe without rewards or expert labels
Experiments span Atari games, OGBench benchmarks, DeepMind Control suites, and the Minecraft BASALT challenge. Mirror outperforms leading learning‑from‑observation methods when paired with learned video plans and reaches near‑expert performance with oracle plans. In Minecraft survival, it is the first method to acquire an iron pickaxe solely through self‑improvement, and in the offline setting it sets a new state‑of‑the‑art result for diamond acquisition using expert labels only.
The results suggest that advances in video generative models can be leveraged to build increasingly capable embodied agents without relying on large expert action datasets or manually designed reward functions.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- U.S. Army develops echotribble program to deploy autonomous AI agents for cyber attacks · 1 src
- AMD releases GAIA 0.25 with smaller install and default flagship agent · 1 src
- Amazon Bedrock adds structured JSON outputs for Anthropic Claude models · 1 src
- Amazon SageMaker Studio available via new AWS sign‑in experience for developers · 1 src
- Microsoft introduces Guardian Agents to monitor AI agents as insider threats · 4 src
Comments
via GitHub Discussions