DigestAI news desk

Cut through the AI noise.

Agents & Tools

Researchers introduce Mirror, a self‑supervised video‑to‑action model

The paper presents Mirror, a self‑supervised approach that learns to map video plans to actions without any pre‑collected action data or hand‑crafted rewards. Mirror iteratively tries to follow a visual plan, records the resulting video‑action pairs, and uses them to improve its own predictions, gradually closing the gap to the visual planner.

1 source primary source

Key points

  • Mirror learns video‑to‑action mappings through self‑supervised interaction, needing no prior action data
  • Across Atari, OGBench, DeepMind Control and Minecraft, Mirror outperforms leading observation‑learning methods
  • In Minecraft survival, Mirror self‑improves to acquire an iron pickaxe without rewards or expert labels

Experiments span Atari games, OGBench benchmarks, DeepMind Control suites, and the Minecraft BASALT challenge. Mirror outperforms leading learning‑from‑observation methods when paired with learned video plans and reaches near‑expert performance with oracle plans. In Minecraft survival, it is the first method to acquire an iron pickaxe solely through self‑improvement, and in the offline setting it sets a new state‑of‑the‑art result for diamond acquisition using expert labels only.

The results suggest that advances in video generative models can be leveraged to build increasingly capable embodied agents without relying on large expert action datasets or manually designed reward functions.

Read the original at arXiv cs.AI · by Hyunwoo Ryu, Eric Ming Chen, Max Wilcoxson, Vincent Sitzmann primary sourceOpen source ↗
Topics · follow one to build your own front page

The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.

Comments

via GitHub Discussions

More in Agents & Tools

All →

Related stories