Multimodal Fusion for Indoor Sound Source Localization

Jinhui Chen; Ryoichi Takashima; Xingchen Guo; Zhihong Zhang; Xuexin Xu; Tetsuya Takiguchi; Edwin R. Hancock

doi:10.1016/j.patcog.2021.107906

Multimodal Fusion for Indoor Sound Source Localization

Jinhui Chen, Ryoichi Takashima, Xingchen Guo, Zhihong Zhang, Xuexin Xu, Tetsuya Takiguchi, Edwin R. Hancock

Computer Science

Research output: Contribution to journal › Article › peer-review

Abstract

To identify the localization of indoor sound source, especially when attempted using only a single microphone, it is a challenging problem to machine learning. To address these issues, this paper presents a distinct novel solution based on fusing visual and acoustic models. Therefore, we propose two novel approaches. First, to estimate orientation of vocal object in a stable manner, we employ the visual approach as estimation model, where we develop a robust image feature representation method that adopts Fourier analysis to efficiently extract polar descriptors. Second the distance information is estimated by calculating the signal difference between transmit receive ends. To implement these, we use phoneme-level hidden Markov models (HMMs) extracted from clean speech sound, to estimate the acoustic transfer function (ATF), which can capture the speech signal as a network of phoneme HMMs. And using the separated frame sequences of the ATF, we can indicate the signal difference between two positions, which can be used to estimate the distance of sound source. Experimental results show that the proposed method can simultaneously extract the sound source parameters of direction and distance, and thus improves the verification task of sound source localization.

Original language	English
Article number	107906
Number of pages	13
Journal	Pattern recognition
Volume	115
Early online date	23 Feb 2021
DOIs	https://doi.org/10.1016/j.patcog.2021.107906
Publication status	E-pub ahead of print - 23 Feb 2021

Bibliographical note

This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy.

Keywords

Sound source localization
acoustic transfer function
HMM
polar HOG
SVM

Access to Document

10.1016/j.patcog.2021.107906

PR-localization
This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy.
Accepted author manuscript, 2.29 MBLicence: CC BY-NC-ND

https://www.sciencedirect.com/science/article/pii/S0031320321000935

Cite this

@article{befb820f8aa349e7a4d4eb2a7ea7a747,

title = "Multimodal Fusion for Indoor Sound Source Localization",

abstract = "To identify the localization of indoor sound source, especially when attempted using only a single microphone, it is a challenging problem to machine learning. To address these issues, this paper presents a distinct novel solution based on fusing visual and acoustic models. Therefore, we propose two novel approaches. First, to estimate orientation of vocal object in a stable manner, we employ the visual approach as estimation model, where we develop a robust image feature representation method that adopts Fourier analysis to efficiently extract polar descriptors. Second the distance information is estimated by calculating the signal difference between transmit receive ends. To implement these, we use phoneme-level hidden Markov models (HMMs) extracted from clean speech sound, to estimate the acoustic transfer function (ATF), which can capture the speech signal as a network of phoneme HMMs. And using the separated frame sequences of the ATF, we can indicate the signal difference between two positions, which can be used to estimate the distance of sound source. Experimental results show that the proposed method can simultaneously extract the sound source parameters of direction and distance, and thus improves the verification task of sound source localization.",

keywords = "Sound source localization, acoustic transfer function, HMM, polar HOG, SVM",

author = "Jinhui Chen and Ryoichi Takashima and Xingchen Guo and Zhihong Zhang and Xuexin Xu and Tetsuya Takiguchi and Hancock, {Edwin R.}",

note = "This is an author-produced version of the published paper. Uploaded in accordance with the publisher{\textquoteright}s self-archiving policy. ",

year = "2021",

month = feb,

day = "23",

doi = "10.1016/j.patcog.2021.107906",

language = "English",

volume = "115",

journal = "Pattern Recognition",

issn = "0031-3203",

publisher = "Elsevier",

}

TY - JOUR

T1 - Multimodal Fusion for Indoor Sound Source Localization

AU - Chen, Jinhui

AU - Takashima, Ryoichi

AU - Guo, Xingchen

AU - Zhang, Zhihong

AU - Xu, Xuexin

AU - Takiguchi, Tetsuya

AU - Hancock, Edwin R.

N1 - This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy.

PY - 2021/2/23

Y1 - 2021/2/23

N2 - To identify the localization of indoor sound source, especially when attempted using only a single microphone, it is a challenging problem to machine learning. To address these issues, this paper presents a distinct novel solution based on fusing visual and acoustic models. Therefore, we propose two novel approaches. First, to estimate orientation of vocal object in a stable manner, we employ the visual approach as estimation model, where we develop a robust image feature representation method that adopts Fourier analysis to efficiently extract polar descriptors. Second the distance information is estimated by calculating the signal difference between transmit receive ends. To implement these, we use phoneme-level hidden Markov models (HMMs) extracted from clean speech sound, to estimate the acoustic transfer function (ATF), which can capture the speech signal as a network of phoneme HMMs. And using the separated frame sequences of the ATF, we can indicate the signal difference between two positions, which can be used to estimate the distance of sound source. Experimental results show that the proposed method can simultaneously extract the sound source parameters of direction and distance, and thus improves the verification task of sound source localization.

AB - To identify the localization of indoor sound source, especially when attempted using only a single microphone, it is a challenging problem to machine learning. To address these issues, this paper presents a distinct novel solution based on fusing visual and acoustic models. Therefore, we propose two novel approaches. First, to estimate orientation of vocal object in a stable manner, we employ the visual approach as estimation model, where we develop a robust image feature representation method that adopts Fourier analysis to efficiently extract polar descriptors. Second the distance information is estimated by calculating the signal difference between transmit receive ends. To implement these, we use phoneme-level hidden Markov models (HMMs) extracted from clean speech sound, to estimate the acoustic transfer function (ATF), which can capture the speech signal as a network of phoneme HMMs. And using the separated frame sequences of the ATF, we can indicate the signal difference between two positions, which can be used to estimate the distance of sound source. Experimental results show that the proposed method can simultaneously extract the sound source parameters of direction and distance, and thus improves the verification task of sound source localization.

KW - Sound source localization

KW - acoustic transfer function

KW - HMM

KW - polar HOG

KW - SVM

U2 - 10.1016/j.patcog.2021.107906

DO - 10.1016/j.patcog.2021.107906

M3 - Article

VL - 115

JO - Pattern Recognition

JF - Pattern Recognition

SN - 0031-3203

M1 - 107906

ER -