Nuance raises $50M to build unified audiovisual AI for natural conversation

A startup that's trying to make AI models better conversationalists with more emotional intelligence has just raised $50 million. Seattle-based Nuance says it wants to build models that can hold face-to-face conversations without the lag and awkwardness of existing chatbots. Fangchang Ma, Nuance's cofounder and CEO, said that existing voicebots or AI avatars are built by duct-taping together voice-to-text generation, a large language model responding to those texts, and text-to-voice generation. This leads to a lot of latency and missing human nuances, such as reactions and microexpressions. "If you use existing AI avatars, when they're listening, they're not reacting, or they're just doing random things," Ma said. Ma, and his two co-founders, all of whom are former researchers at Apple, wanted to build a long-term solution, even though it would be more technically challenging. "We decided to build this on one system, on one model. There's audio video in and audio video out," Ma said. Ma said that the data needed to train its models cannot be scraped from the internet and that it collects data in-house or from data partners. "The perfect data for us will be, imagine the two of us having this conversation, we're having to record it," he said. "I want audiovisual from both sides. Both participants are visible, soundtracks cleanly separated, and then it's also natural, not performative." While the startup is pre-product, Ma said he sees it offering an experience where someone can hop on a video call with an AI to practice interviews, give presentations, or learn a new language. The company plans to sell to businesses, too, and said that use cases can include AI interviewers, sales reps, and customer support. The company plans to open a research preview of its model to the public later this year. Nuance, which was founded last year, has now raised a $50 million Series A round led by Lightspeed Venture Partners, with participation from Accel, Nvidia's NVentures, South Park Commons, and Define Ventures. It follows a $10 million seed round led by Accel in July last year. Ma said that the startup has been using the cash flow to grow its eight-person team. The company is investing in hiring for its research and development team and said its next step is to hire go-to-market experts to nail down its pricing and monetization strategy. Check out the 13-slide pitch deck Nuance used to secure the fresh funding.