Shape Tracking of Flexible Morphing Matters from Depth Images

Lalithkumar Seenivasan, Fan Bai, Ming Ji, Xiaoyi Gu, Zion Tsz Ho Tse, Hongliang Ren*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


The development and use of soft or flexible structural matters across various research domains have drastically increased in recent decades. Its flexible, compliant nature and interactive safety have made it a preferred candidate compared to its rigid bodied counterparts. However, the lack of robust soft robot detection and localization techniques has constrained its feedback control system, limiting its application. This paper proposes a novel depth sensor-based detection and tracking algorithm adaptive to shape morphing robots. The detection algorithm first employs optimal iterative threshold segmentation on the depth image to remove background and detect occlusions. Blob detection and polygon approximation using Fourier descriptor techniques are then utilized to detect and extract the contours of the shape morphing soft robots. Finally, using the pixel coordinates obtained from the detection algorithm, transformation is applied from the pixel coordinate system to the world coordinate system on the depth image to achieve motion tracking in 3D space. Qualitative and quantitative assessments prove that the detection algorithm is robust and accurate in tracking shape morphing soft robots.

Original languageEnglish
Article number9262941
Pages (from-to)8234-8244
Number of pages11
JournalIEEE Sensors Journal
Issue number6
Early online date18 Nov 2020
Publication statusPublished - 15 Mar 2021

Bibliographical note

Funding Information:
Manuscript received September 21, 2020; accepted November 4, 2020. Date of publication November 18, 2020; date of current version February 17, 2021. This work was supported by the National Key Research and Development Program, Ministry of Science and Technology (MOST) of China under Grant 2018YFB1307703 and the Singapore Academic Research Fund under Grant R-397-000-353-114. The work of Zion Tsz Ho Tse was supported in part by the Royal Society Wolfson Fellowship.The associate editor coordinating the review of this article and approving it for publication was Dr. Qammer H. Abbasi. (Corresponding author: Hongliang Ren.) Lalithkumar Seenivasan, Xiaoyi Gu, and Hongliang Ren are with the NUS (Suzhou) Research Institute (NUSRI), Suzhou 215123, China, and also with the Department of Biomedical Engineering, Faculty of Engineering, National University of Singapore, Singapore 117583.

Publisher Copyright:
© 2001-2012 IEEE.

Copyright 2021 Elsevier B.V., All rights reserved.


  • blob detection
  • optimal threshold segmentation
  • shape detection
  • Soft matters

Cite this