Developing Effective Test Sets and Metrics for Evaluating Automated Media Analysis Systems

TY - CONF

T1 - Developing Effective Test Sets and Metrics for Evaluating Automated Media Analysis Systems

AU - Mateer, John

PY - 2003

Y1 - 2003

N2 - This paper first looks at current methods of evaluating automated content-based media analysis systems. Several key deficiencies are identified, particularly with regard to test set creation and metric design. A new framework is proposed that better reflects real-world conditions and end-user requirements. This is based on the author’s experience as a professional filmmaker and researcher in this domain. Specific approaches for data set selection, including the importance of understanding the physical, production and aesthetic attributes of footage, are presented. A discussion of related evaluation methods and means of effective assessment follow. It is hoped the suggestions proposed will facilitate more effective analysis of these systems.

AB - This paper first looks at current methods of evaluating automated content-based media analysis systems. Several key deficiencies are identified, particularly with regard to test set creation and metric design. A new framework is proposed that better reflects real-world conditions and end-user requirements. This is based on the author’s experience as a professional filmmaker and researcher in this domain. Specific approaches for data set selection, including the importance of understanding the physical, production and aesthetic attributes of footage, are presented. A discussion of related evaluation methods and means of effective assessment follow. It is hoped the suggestions proposed will facilitate more effective analysis of these systems.

M3 - Paper

SP - 201

EP - 204

T2 - IEEE International Conference on Multimedia & Expo

Y2 - 6 July 2003 through 9 July 2003

ER -