Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

Catherine L. Lawson*, Andriy Kryshtafovych, Paul D. Adams, Pavel V. Afonine, Matthew L. Baker, Benjamin A. Barad, Paul Bond, Tom Burnley, Renzhi Cao, Jianlin Cheng, Grzegorz Chojnowski, Kevin Cowtan, Ken A. Dill, Frank DiMaio, Daniel P. Farrell, James S. Fraser, Mark A. Herzik, Soon Wen Hoh, Jie Hou, Li Wei HungMaxim Igaev, Agnel P. Joseph, Daisuke Kihara, Dilip Kumar, Sumit Mittal, Bohdan Monastyrskyy, Mateusz Olek, Colin M. Palmer, Ardan Patwardhan, Alberto Perez, Jonas Pfab, Grigore D. Pintilie, Jane S. Richardson, Peter B. Rosenthal, Daipayan Sarkar, Luisa U. Schäfer, Michael F. Schmid, Gunnar F. Schröder, Mrinal Shekhar, Dong Si, Abishek Singharoy, Genki Terashi, Thomas C. Terwilliger, Andrea Vaiana, Liguo Wang, Zhe Wang, Stephanie A. Wankowicz, Christopher J. Williams, Martyn Winn, Tianqi Wu, Xiaodi Yu, Kaiming Zhang, Helen M. Berman, Wah Chiu

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 Å). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.

Original languageEnglish
Pages (from-to)156-164
Number of pages9
JournalNature Methods
Issue number2
Publication statusPublished - 4 Feb 2021

Bibliographical note

Funding Information:
EMDataResource (C.L.L., A.K., G.P., H.M.B. and W.C.) is supported by the US National Institutes of Health (NIH)/National Institute of General Medical Science, grant no. R01GM079429. The Singharoy team used the supercomputing resources of the Oak Ridge Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science at the Department of Energy under contract no. DE-AC05-00OR22725. The following additional grants are acknowledged for participant support: grant no. NIH/R35GM131883 to J.S.R. and C.W.; grant no. NIH/P01GM063210 to P.D.A., P.V.A., L.-W.H., J.S.R., T.C.T. and C.W.; National Science Foundation grant no. (NSF)/MCB-1942763 (CAREER) and NIH/R01GM095583 to A.S.; grant nos. NIH/R01GM123055, NIH/R01GM133840, NSF/DMS1614777, NSF/CMMI1825941, NSF/MCB1925643, NSF/DBI2003635 and Purdue Institute of Drug Discovery to D. Kihara; grant no. NIH/R01GM123159 to J.S.F.; Max Planck Society German Research Foundation grant no. IG 109/1-1 to M.I.; Max Planck Society German Research Foundation grant no. FOR-1805 to A.C.V.; grant nos. NIH/R37AI36040 and Welch Foundation/Q1279 to D. Kumar (PI: BVV Prasad); grant no. NSF/DBI2030381 to D. Si.; Medical Research Council grant no. MR/N009614/1 to T.B., C.M.P. and M.W.; Wellcome Trust grant no. 208398/Z/17/Z to A.P.J. and M.W.; Biotechnology and Biological Sciences Research Council grant no. BB/P000517/1 to K.C. and Biotechnology and Biological Sciences Research Council grant no. BB/P000975/1 to M.W.

Publisher Copyright:
© 2021, The Author(s).

Copyright 2021 Elsevier B.V., All rights reserved.
  • 2019 EMDataResource Model Metrics Challenge Dataset

    Lawson, C. L. (Creator), Kryshtafovych, A. (Creator), Adams, P. D. (Creator), Afonine, P. V. (Creator), Baker, M. L. (Creator), Barad, B. A. (Creator), Bond, P. (Creator), Burnley, T. (Creator), Cao, R. (Creator), Cheng, J. (Creator), Chojnowski, G. (Creator), Cowtan, K. D. (Creator), Dill, K. (Creator), DiMaio, F. (Creator), Farrell, D. P. (Creator), Fraser, J. S. (Creator), Herzik, M. A. J. (Creator), Hoh, S. W. (Creator), Hou, J. (Creator), Hung, L. (Creator), Igaev, M. (Creator), Joseph, A. P. (Creator), Kihara, D. (Creator), Kumar, D. (Creator), Mittal, S. (Creator), Monastryrskyy, B. (Creator), Olek, M. (Creator), Palmer, C. M. (Creator), Patwardhan, A. (Creator), Perez, A. (Creator), Pfab, J. (Creator), Pintilie, G. D. (Creator), Richardson, J. S. (Creator), Sarkar, D. (Creator), Schäfer, L. U. (Creator), Schmid, M. F. (Creator), Schröder, G. F. (Creator), Shekhar, M. (Creator), Si, D. (Creator), Singharoy, A. (Creator), Terashi, G. (Creator), Terwilliger, T. C. (Creator), Vaiana, A. (Creator), Wang, L. (Creator), Wankowicz, S. A. (Creator), Williams, C. J. (Creator), Winn, M. (Creator), Wu, T. (Creator), Yu, X. (Creator), Zhang, K. (Creator), Berman, H. M. (Creator) & Chiu, W. (Creator), Zenodo, 28 Oct 2020


Cite this