Multimodal Surface EMG Hand Gesture Recognition Using Query-Based Transformers for Prosthetic Control

1. 核心亮点：一项名为EMG-CrossFormer的新模型，通过融合肌肉电信号和运动数据，让假肢手势识别准确率最高提升到93%，突破了传统方法对手势种类增多性能下降的瓶颈。 2. 通俗解读：以前用肌电信号控制假肢，就像只靠听几个关键词猜整句话，手势一多就容易出错。新模型就像一位聪明的翻译官，既能抓住局部肌肉活动的细节，又能看到整个动作的连贯模式，还把手臂运动数据也加进来一起分析。它用一种叫“交叉注意力”的技术，把不同信号像拼图一样完美结合，最后通过“手势查询”精准匹配用户想做的动作。 3. 实际影响：对截肢者来说，这意味着未来假肢能更流畅、更自然地响应各种复杂手势，比如捏东西、握杯子甚至打字。日常使用中，用户只需像正常人一样活动手臂，假肢就能准确执行指令。长远看，这项技术还可能扩展到智能手表、VR手套等设备，让普通人用手指细微动作就能隔空操控手机或电脑，大大提升交互体验。

arXiv:2607.22779v1 Announce Type: new Abstract: Hand gesture recognition via surface electromyography (sEMG) is fundamental to prosthetic control. In this field, deep learning approaches have become the gold standard. However, current architectures struggle to scale; model performance typically decreases as the number of hand movements increases. Performance degradation is tied to the increased statistical complexity of decoding expanded gesture sets and compounded by the limitations of state-of-the-art methods, which primarily rely on low-latency unimodal convolutional architectures. Convolutions operate locally, limiting model's ability to capture long-range sequential patterns. Unimodal setups cannot leverage complementary information from coordinated signals characterizing movement execution, such as inertial and eye-tracking data. These limitations motivate architectures that integrate local and global features across multimodal physiological sequences. To bridge this gap, this study introduces EMG-CrossFormer, an end-to-end hybrid convolutional-transformer for seamless multimodal integration. EMG-CrossFormer combines representations from an arbitrary number of unimodal encoders through cascaded cross-attention fusion layers, and decodes the fused representations using learnable gesture queries. EMG-CrossFormer was evaluated on four NinaPro datasets (DB2, DB3, DB7, and DB10) and benchmarked against six state-of-the-art models using an increasing number of modalities. Using only sEMG, EMG-CrossFormer achieved mean accuracies of 72.33%, 52.48%, 79.16%, and 73.49% on DB2, DB3, DB7, and DB10, respectively. Incorporating inertial signals improved performance to 90.66%, 80.40%, 92.79%, and 92.06%. These results show that joint local-global feature modeling improves sEMG-only decoding and that multimodal fusion substantially amplifies this benefit, underscoring the value of both design principles for complex hand gesture recognition.