Developing Effective Test Sets and Metrics for Evaluating Automated Media Analysis Systems

Research output: Contribution to conferencePaperpeer-review


This paper first looks at current methods of evaluating automated content-based media analysis systems. Several key deficiencies are identified, particularly with regard to test set creation and metric design. A new framework is proposed that better reflects real-world conditions and end-user requirements. This is based on the author’s experience as a professional filmmaker and researcher in this domain. Specific approaches for data set selection, including the importance of understanding the physical, production and aesthetic attributes of footage, are presented. A discussion of related evaluation methods and means of effective assessment follow. It is hoped the suggestions proposed will facilitate more effective analysis of these systems.
Original languageEnglish
Publication statusPublished - 2003
EventIEEE International Conference on Multimedia & Expo - Baltimore, United States
Duration: 6 Jul 20039 Jul 2003


ConferenceIEEE International Conference on Multimedia & Expo
Country/TerritoryUnited States

Cite this