# Researchers test how mismatched text and audio affect emotion recognition in speech models

Digest AI · Research · published 2026-11-02T05:00:00Z

Canonical: https://digestai.news/story/researchers-test-how-mismatched-text-and-audio-affect-emotion-recognit

## Summary

A new study on arXiv examines whether mismatches between spoken words and tone hurt speech emotion recognition in audio-capable models. The authors propose an incongruence score to measure the gap between text and audio in a shared valence-arousal-dominance space, then test it on three datasets: eNTERFACE'05, MELD, and CMU-MOSEI. Their findings show that incongruence increases error rates across acted, conversational, and spontaneous speech when audio is present, even after accounting for ground-truth emotions and feature distance.

The research suggests the issue lies in how models integrate raw evidence rather than the signal itself. Supervised classifiers trained on the same features behave differently, and text-only effects vary by dataset. The authors also confirm their core result without using an emotion lexicon, and their incongruence score is the only label-free method that predicts errors across all three datasets. Code, data tables, and prompts are released for further use.

## Key points

- Study introduces an incongruence score to quantify mismatches between text and audio in speech emotion recognition
- Tests three datasets: eNTERFACE'05, MELD, and CMU-MOSEI, finding incongruence raises error odds in all cases
- Authors release code, data tables, and prompts to support reproducibility and further research

## Why it matters

This work could help improve speech models by identifying how mismatched text and audio affect emotion detection, a critical step for applications like accessibility tools, customer service bots, and mental health assistants.

## Sources

1. [Incongruence-Aware Evaluation of Multimodal Speech Emotion Recognition in Audio Large Language Models](https://arxiv.org/abs/2610.40883) (arXiv cs.AI, 2026-11-02, primary source)

## Cite

Digest AI, "Researchers test how mismatched text and audio affect emotion recognition in speech models", 2 November 2026, https://digestai.news/story/researchers-test-how-mismatched-text-and-audio-affect-emotion-recognit

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/researchers-test-how-mismatched-text-and-audio-affect-emotion-recognit.json