SoniSpeech: A Large-Scale Open-Vocabulary Tri-Modal Dataset for Wearable Silent Speech Interfaces

AI 中文解读

核心亮点：科学家给智能眼镜装上了"读唇术"，不用出声就能识别你说的话，还能听懂五千多个日常词汇，这在以前是不可能的。通俗解读：以前类似的"无声语音"设备只能识别固定的几句话，比如"开灯""关窗"这类简单指令，因为需要贴电极在脸上，又笨重又局限。这次研究人员推出一个叫SoniSpeech的大型数据集，用一副能发射超声波并捕捉回声的眼镜，同时记录你说话时的嘴唇动作、发声时的声音和脸部画面。重点是，它涵盖五千多个单词和完整发音，覆盖了大量日常对话场景。通过这套数据训练的AI模型，即使你只做口型不出声，也能猜出你想说什么，错误率约为四分之一。实际影响：这项技术最直接的好处是帮到那些声带受损或发音困难的人，让他们通过细微的口型动作就能操控设备、打字交流。普通人也能受益，比如在图书馆、会议室或嘈杂环境里，不用出声就能给手机发消息、下指令，既保护隐私又不打扰别人。未来这类智能眼镜可能变成随身助手，让"无声对话"真正走进生活。

Wearable silent speech interfaces (SSIs) are limited to small, closed vocabularies. Approaches achieving larger vocabularies require obtrusive hardware such as facial electrodes. We present SoniSpeech, the first large-scale, open-vocabulary, trimodal dataset for wearable SSI using acoustic-sensing eyewear. It contains 34 hours across 18,000 utterances with three synchronized modalities: ultrasound echo profiles, voiced audio, and frontal video, in both voiced and silent modes. The corpus draws from the SODA dialogue dataset, providing contemporary conversational English with 5,356 unique words and full phoneme coverage. A CTC-based ResNet-34 baseline achieves 26.3% word error rate (WER) on open-vocabulary silent speech recognition, the first benchmark for this task. Dataset is available at https://doi.org/10.7298/xjjr-9m85