Mu uses Claude Opus 5.5 to build autonomous AI VTuber room
Developer Mu detailed a project using Claude Opus 5.5 via Claude Code to create a 3D streaming room where an AI character can perceive, judge, and move autonomously. The system uses Gemini 3.8 Flash via OpenRouter for the character's decision-making, converting visible objects and internal states into text lists that the model processes to generate actions like walking or looking.
Key points
- Claude Opus 5.5 built the 3D room; Gemini 3.8 Flash handles character judgment.
- One instruction takes 12–17 seconds; eye-only movement takes around 30 seconds.
- System separates judgment from body, using JSON for room objects and actions.
The implementation separates judgment from the body, using a JSON format to define room objects, their properties, and allowed actions. The character has 'eyes' that filter visible objects based on field of view and distance, with memory that blurs over time. Mu also added data-driven gimmicks, such as energy levels, object repair tasks, and event triggers like power outages. The character can interact with viewers and other AI visitors.
Mu noted significant latency, with one instruction taking 12–17 seconds and eye-only movement taking around 30 seconds. The character sometimes describes things it is not looking at or repeats greetings in multi-agent conversations. Mu proposed future improvements, including reducing LLM call frequency by using behavior trees for routine actions and adding object-to-object relationships to the room description.
Model pages: Claude Opus 5.5 → · Gemini 3.8 Flash →
The story so far
2 episodes →- Mu uses Claude Opus 5.5 to build autonomous AI VTuber roomthis story
Making a streaming room where an AI character can judge and move on its own — The 'meaning of objects, eyes, and layers' added with Claude Opus 5.5, and the fundamental
note.com · 29 September 2026
Loading the full article…
This text was published by note.com and written by むぅ. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- Gemini may delete existing text when asked to append, author reports · 1 src
- Google's Gemini outperforms ChatGPT and Claude as Android assistant · 1 src
- Meta launches Muse AI agent in Mexico, first in Latin America · 11 src
- Meta rolls out real-time voice and expressive avatars for Muse AI in the US · 7 src
- Meta improves Muse and Meta AI with Active Memory and Active Focus · 2 src
Comments
via GitHub Discussions