# Mu uses Claude Opus 5.5 to build autonomous AI VTuber room

Digest AI · Agents & Tools · published 2026-09-29T20:22:00Z

Canonical: https://digestai.news/story/mu-uses-claude-opus-5-5-to-build-autonomous-ai-vtuber-room

## Summary

Developer Mu detailed a project using Claude Opus 5.5 via Claude Code to create a 3D streaming room where an AI character can perceive, judge, and move autonomously. The system uses Gemini 3.8 Flash via OpenRouter for the character's decision-making, converting visible objects and internal states into text lists that the model processes to generate actions like walking or looking.

The implementation separates judgment from the body, using a JSON format to define room objects, their properties, and allowed actions. The character has 'eyes' that filter visible objects based on field of view and distance, with memory that blurs over time. Mu also added data-driven gimmicks, such as energy levels, object repair tasks, and event triggers like power outages. The character can interact with viewers and other AI visitors.

Mu noted significant latency, with one instruction taking 12–17 seconds and eye-only movement taking around 30 seconds. The character sometimes describes things it is not looking at or repeats greetings in multi-agent conversations. Mu proposed future improvements, including reducing LLM call frequency by using behavior trees for routine actions and adding object-to-object relationships to the room description.

## Key points

- Claude Opus 5.5 built the 3D room; Gemini 3.8 Flash handles character judgment.
- One instruction takes 12–17 seconds; eye-only movement takes around 30 seconds.
- System separates judgment from body, using JSON for room objects and actions.

## Why it matters

Demonstrates a practical architecture for autonomous AI agents in 3D spaces, highlighting latency challenges and the need for hybrid logic (LLM + behavior trees) for real-time interaction.

## Sources

1. [Making a streaming room where an AI character can judge and move on its own — The 'meaning of objects, eyes, and layers' added with Claude Opus 5.5, and the fundamental](https://note.com/toreshin/n/n17a4d233f799?hl=en) (note.com, 2026-09-29)

## Cite

Digest AI, "Mu uses Claude Opus 5.5 to build autonomous AI VTuber room", 29 September 2026, https://digestai.news/story/mu-uses-claude-opus-5-5-to-build-autonomous-ai-vtuber-room

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/mu-uses-claude-opus-5-5-to-build-autonomous-ai-vtuber-room.json