Researchers achieve 77.95% F1 on ambivalence detection using audio‑text cross‑attention
Ambivalence and hesitancy are subtle behavioural cues that appear both in spoken words and how they are delivered. A new framework encodes full‑video transcripts with a pre‑trained multilingual BERT model and the accompanying speech waveforms with wav2vec 2.0. Two stacked bidirectional cross‑attention blocks let linguistic tokens attend to acoustic frames and vice‑versa, removing the need for…
Key points
- Audio and text are encoded with multilingual BERT and wav2vec 2.0, then fused via bidirectional cross‑attention.
- Evaluated on the Behavioural Ambivalence/Hesitancy dataset, the model reaches 77.95% A/H F1‑score.
- Sequences are aggregated with masked mean pooling and combined using concatenation, element‑wise products, and absolute differences.
The system was tested on the subject‑independent split of the Behavioural Ambivalence/Hesitancy (BAH) dataset. Reported results include 72.19% accuracy, 75.00% precision, 81.13% recall, a 77.95% A/H F1‑score, and a 70.16% macro‑F1 on the test set. These numbers suggest that jointly modelling linguistic and acoustic information can substantially improve automatic recognition of these nuanced states.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- OpenAI brings back Aleksander Madry to focus on AI safety · 2 src
- OpenAI claims AI-generated proof of quasi‑Riemann hypothesis · 2 src
- Nvidia's Astranomy study reaches 74.8% success in robot control with GPT-6 Astra · 1 src
- PHP patches critical RCE flaws discovered with GPT-5.6 Sol, affecting up to 70% of websites · 8 src
- Ggml-org adds fused top‑k mask operation to llama.cpp · 1 src
Comments
via GitHub Discussions