{"version":1,"type":"story","url":"https://digestai.news/story/together-ai-releases-together-compiler-agents-re-optimize-inference-up","json":"https://digestai.news/story/together-ai-releases-together-compiler-agents-re-optimize-inference-up.json","markdown":"https://digestai.news/story/together-ai-releases-together-compiler-agents-re-optimize-inference-up.md","slug":"together-ai-releases-together-compiler-agents-re-optimize-inference-up","headline":"Together AI releases Together Compiler, agents re-optimize inference up to 2.6x faster","summary":"Together AI launched Together Compiler (TKCC), an agent-driven system that continuously rewrites a model's forward pass as production traffic shifts. The system runs on the company's serverless endpoints and has been tested on models including Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash, and Llama 1B across NVIDIA B200, B300, GB300, H100, AMD MI355X, and AWS Trainium2 hardware.\n\nTogether AI reports up to 2.6x higher tokens-per-second than the best published InferenceX configurations, with each optimization run completing in under 24 hours on at most two nodes. The highest gain was 1,111 P90 TPS per user on MiniMax M3.1 Flash (B200, batch size 1), a 1.8x improvement over the 633 TPS baseline. The smallest gain was GLM 5.3 on MI355X, moving from 463 to 480 TPS (about 4%). The system uses a persistent engine that avoids cold restarts, achieving 252x higher agent experiment throughput compared to engine-restart baselines. Together AI says naive agents instructed to make the forward pass \"as fast as possible\" outperformed expert-curated knowledge bases. All figures are self-reported by Together AI.","keyPoints":["Together Compiler uses coding agents to continually re-optimize inference as traffic patterns change","Self-reported up to 2.6x TPS over best InferenceX configs, tested on five models across six hardware types","Persistent engine with hot reload delivers 252x more agent experiments than restart baselines"],"whyItMatters":"Continuous inference optimization could reduce serving costs and latency for production LLM deployments as traffic patterns drift over time.","category":{"slug":"agents","name":"Agents & Tools","url":"https://digestai.news/category/agents"},"entities":{"companies":["Together AI","NVIDIA","AMD","AWS","SemiAnalysis"],"models":["Kimi K3","GLM 5.3","MiniMax M3.1 Flash","DeepSeek v4.1 Flash","Llama 1B"],"people":[]},"firstPublishedAt":"2026-11-05T08:02:58Z","updatedAt":"2026-11-05T08:02:58Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"MarkTechPost","title":"Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster","url":"https://marktechpost.com/2026/11/05/together-ai-together-compiler-continual-compilation-ai-inference","publishedAt":"2026-11-05T08:02:58Z","type":"press","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Together AI releases Together Compiler, agents re-optimize inference up to 2.6x faster\", 5 November 2026, https://digestai.news/story/together-ai-releases-together-compiler-agents-re-optimize-inference-up","publisher":"Digest AI","title":"Together AI releases Together Compiler, agents re-optimize inference up to 2.6x faster","datePublished":"2026-11-05T08:02:58Z","url":"https://digestai.news/story/together-ai-releases-together-compiler-agents-re-optimize-inference-up"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}