Together AI releases Together Compiler, agents re-optimize inference up to 2.6x faster
Together AI launched Together Compiler (TKCC), an agent-driven system that continuously rewrites a model's forward pass as production traffic shifts. The system runs on the company's serverless endpoints and has been tested on models including Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash, and Llama 1B across NVIDIA B200, B300, GB300, H100, AMD MI355X, and AWS Trainium2 hardware.
Key points
- Together Compiler uses coding agents to continually re-optimize inference as traffic patterns change
- Self-reported up to 2.6x TPS over best InferenceX configs, tested on five models across six hardware types
- Persistent engine with hot reload delivers 252x more agent experiments than restart baselines
Together AI reports up to 2.6x higher tokens-per-second than the best published InferenceX configurations, with each optimization run completing in under 24 hours on at most two nodes. The highest gain was 1,111 P90 TPS per user on MiniMax M3.1 Flash (B200, batch size 1), a 1.8x improvement over the 633 TPS baseline. The smallest gain was GLM 5.3 on MI355X, moving from 463 to 480 TPS (about 4%). The system uses a persistent engine that avoids cold restarts, achieving 252x higher agent experiment throughput compared to engine-restart baselines. Together AI says naive agents instructed to make the forward pass "as fast as possible" outperformed expert-curated knowledge bases. All figures are self-reported by Together AI.
Model page: GLM 5.3 →
Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster
MarkTechPost · 5 November 2026
Loading the full article…
This text was published by MarkTechPost and written by Asif Razzaq. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- Researchers introduce ReScopeBench to test long-lived AI agents’ experience retention · 1 src
- Researchers propose Agent Harness Awareness to improve agent decision-making with real-time telemetry · 1 src
- OpenAI launches GPT-Live outbound phone calls API at $0.05 per minute · 1 src
- The Story of Turning the Process of Creating AI Manga with Claude Code into a Skill (2-Panel Manga) · 1 src
- Developer reports persistent errors and hallucinations in Cursor Composer 2.5 · 1 src
Comments
via GitHub Discussions