跳到正文
Advanced Science· Yüksel·· 8 天前AI 评分34

Experience Filter:将异构信号转化为结构化运动轨迹

Experience Filter: Translating Heterogeneous Signals into Motion

AI 导读

Experience Filter 提出一种无需训练的确定性编码流程,将音乐、图像等异构多媒体信号经模态特异特征提取后映射到共享描述子空间(信号强度、主频谱特征、偏离中性轴距离),再转为结构化运动轨迹并重定向到任意自由度和形态的机器人平台。在真实音乐与图像上的演示生成了连续同步的图像—音频—运动体验,无需人工编排;特征敏感性分析显示各描述子轴相互独立,编码空间拓扑由所选描述子决定而非固定。

AI 深度导读基于原文整理的分节解读
背景
人类经验本身就是多感官的,交织着视觉、听觉与动觉反馈;而数字媒体通常只捕捉视听两个维度,遗漏了机械感觉这一维度。全身(动觉)触觉反馈虽能丰富媒体体验,但从异质信号中导出结构化运动、再与视听内容结合进行渲染,此前尚无通用方法。
做法
Experience Filter 把多媒体信号映射为结构化运动轨迹,采用统一的处理管线:先做模态特定的特征提取,再做模态无关的共享编码。每个信号被归约到共享描述符空间——信号强度、主频特征、到中性轴的距离——再经由固定的、确定性的、无需训练的编码映射为轨迹。由于该编码空间与硬件无关,运动可重定向到任意自由度与形态的机器人平台。工作还做了特征敏感性分析,用以检验描述符轴之间的关系及编码空间拓扑。
结论
特征敏感性分析显示,各描述符轴相互独立,编码空间的拓扑由所选描述符决定,而非固定不变。框架在真实音乐与图像上做了演示,无需人工编写即可生成连续、同步的图像—音频—运动体验。
意义
该框架提供了从多媒体信号到图像—音频—运动联合渲染的解析映射,使跨机器人平台的表达性、用户可调体验成为可能,指向无需手工编排即可为媒体补充动觉维度的路径。

原文摘要

出版方登记 · Crossref

Human experience is inherently rich and multisensory, composed of intertwined visual, auditory, and kinesthetic feedback. Digital media capture only its audiovisual dimensions, omitting its mechanosensory dimensions. While full‐body (kinesthetic) haptic feedback can enrich media, no general method exists for deriving structured motion from heterogeneous signals and combining it with the audiovisual content for rendering. The Experience Filter maps multimedia signals into structured motion trajectories through a common pipeline of modality‐specific feature extraction followed by a modality‐agnostic shared encoding, recombining them into a synchronized multisensory experience: each signal is reduced to a shared descriptor space – signal intensity, dominant spectral feature, and distance from neutral axis – then mapped to a trajectory via a fixed, deterministic, training‐free encoding. Because this encoding space is hardware‐independent, the motion can be retargeted to robotic platforms of arbitrary degrees of freedom and morphology. A feature‐sensitivity analysis shows that the descriptor axes are independent and the encoding‐space topology follows from the chosen descriptors rather than being fixed. The framework is demonstrated on real‐world music and images, producing continuous, synchronized image‐audio‐motion experiences without manual authoring. The Experience Filter thus provides an analytical mapping from multimedia signals to combined image‐audio‐motion rendering, enabling expressive, user‐adjustable experience across robotic platforms.

页面未能读取全文,以上为出版方登记的摘要;方法与图表见原文

本页是 AI 精读,不转载论文原文;完整论文(含方法、图表与参考文献)见原文

来源:Advanced Science · doi.org