Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

Catherine L. Lawson; Andriy Kryshtafovych; Paul D. Adams; Pavel V. Afonine; Matthew L. Baker; Benjamin A. Barad; Paul Bond; Tom Burnley; Renzhi Cao; Jianlin Cheng; Grzegorz Chojnowski; Kevin Cowtan; Ken A. Dill; Frank DiMaio; Daniel P. Farrell; James S. Fraser; Mark A. Herzik; Soon Wen Hoh; Jie Hou; Li Wei Hung; Maxim Igaev; Agnel P. Joseph; Daisuke Kihara; Dilip Kumar; Sumit Mittal; Bohdan Monastyrskyy; Mateusz Olek; Colin M. Palmer; Ardan Patwardhan; Alberto Perez; Jonas Pfab; Grigore D. Pintilie; Jane S. Richardson; Peter B. Rosenthal; Daipayan Sarkar; Luisa U. Schäfer; Michael F. Schmid; Gunnar F. Schröder; Mrinal Shekhar; Dong Si; Abishek Singharoy; Genki Terashi; Thomas C. Terwilliger; Andrea Vaiana; Liguo Wang; Zhe Wang; Stephanie A. Wankowicz; Christopher J. Williams; Martyn Winn; Tianqi Wu; Xiaodi Yu; Kaiming Zhang; Helen M. Berman; Wah Chiu

doi:10.1038/s41592-020-01051-w

Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

Catherine L. Lawson^*, Andriy Kryshtafovych, Paul D. Adams, Pavel V. Afonine, Matthew L. Baker, Benjamin A. Barad, Paul Bond, Tom Burnley, Renzhi Cao, Jianlin Cheng, Grzegorz Chojnowski, Kevin Cowtan, Ken A. Dill, Frank DiMaio, Daniel P. Farrell, James S. Fraser, Mark A. Herzik, Soon Wen Hoh, Jie Hou, Li Wei HungMaxim Igaev, Agnel P. Joseph, Daisuke Kihara, Dilip Kumar, Sumit Mittal, Bohdan Monastyrskyy, Mateusz Olek, Colin M. Palmer, Ardan Patwardhan, Alberto Perez, Jonas Pfab, Grigore D. Pintilie, Jane S. Richardson, Peter B. Rosenthal, Daipayan Sarkar, Luisa U. Schäfer, Michael F. Schmid, Gunnar F. Schröder, Mrinal Shekhar, Dong Si, Abishek Singharoy, Genki Terashi, Thomas C. Terwilliger, Andrea Vaiana, Liguo Wang, Zhe Wang, Stephanie A. Wankowicz, Christopher J. Williams, Martyn Winn, Tianqi Wu, Xiaodi Yu, Kaiming Zhang, Helen M. Berman, Wah Chiu

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

Abstract

This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 Å). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.

Original language	English
Pages (from-to)	156-164
Number of pages	9
Journal	Nature Methods
Volume	18
Issue number	2
DOIs	https://doi.org/10.1038/s41592-020-01051-w
Publication status	Published - 4 Feb 2021

Bibliographical note

Funding Information:
EMDataResource (C.L.L., A.K., G.P., H.M.B. and W.C.) is supported by the US National Institutes of Health (NIH)/National Institute of General Medical Science, grant no. R01GM079429. The Singharoy team used the supercomputing resources of the Oak Ridge Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science at the Department of Energy under contract no. DE-AC05-00OR22725. The following additional grants are acknowledged for participant support: grant no. NIH/R35GM131883 to J.S.R. and C.W.; grant no. NIH/P01GM063210 to P.D.A., P.V.A., L.-W.H., J.S.R., T.C.T. and C.W.; National Science Foundation grant no. (NSF)/MCB-1942763 (CAREER) and NIH/R01GM095583 to A.S.; grant nos. NIH/R01GM123055, NIH/R01GM133840, NSF/DMS1614777, NSF/CMMI1825941, NSF/MCB1925643, NSF/DBI2003635 and Purdue Institute of Drug Discovery to D. Kihara; grant no. NIH/R01GM123159 to J.S.F.; Max Planck Society German Research Foundation grant no. IG 109/1-1 to M.I.; Max Planck Society German Research Foundation grant no. FOR-1805 to A.C.V.; grant nos. NIH/R37AI36040 and Welch Foundation/Q1279 to D. Kumar (PI: BVV Prasad); grant no. NSF/DBI2030381 to D. Si.; Medical Research Council grant no. MR/N009614/1 to T.B., C.M.P. and M.W.; Wellcome Trust grant no. 208398/Z/17/Z to A.P.J. and M.W.; Biotechnology and Biological Sciences Research Council grant no. BB/P000517/1 to K.C. and Biotechnology and Biological Sciences Research Council grant no. BB/P000975/1 to M.W.

Publisher Copyright:
© 2021, The Author(s).

Copyright:
Copyright 2021 Elsevier B.V., All rights reserved.

Access to Document

10.1038/s41592-020-01051-wLicence: CC BY

s41592-020-01051-w
© The Author(s) 2021
Final published version, 6.64 MBLicence: CC BY

Cite this

Lawson, C. L., Kryshtafovych, A., Adams, P. D., Afonine, P. V., Baker, M. L., Barad, B. A., Bond, P., Burnley, T., Cao, R., Cheng, J., Chojnowski, G., Cowtan, K., Dill, K. A., DiMaio, F., Farrell, D. P., Fraser, J. S., Herzik, M. A., Hoh, S. W., Hou, J., ... Chiu, W. (2021). Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge. Nature Methods, 18(2), 156-164. https://doi.org/10.1038/s41592-020-01051-w

@article{213d35b33fbf4048a45049f21fa074aa,

title = "Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge",

abstract = "This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 {\AA}). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.",

author = "Lawson, {Catherine L.} and Andriy Kryshtafovych and Adams, {Paul D.} and Afonine, {Pavel V.} and Baker, {Matthew L.} and Barad, {Benjamin A.} and Paul Bond and Tom Burnley and Renzhi Cao and Jianlin Cheng and Grzegorz Chojnowski and Kevin Cowtan and Dill, {Ken A.} and Frank DiMaio and Farrell, {Daniel P.} and Fraser, {James S.} and Herzik, {Mark A.} and Hoh, {Soon Wen} and Jie Hou and Hung, {Li Wei} and Maxim Igaev and Joseph, {Agnel P.} and Daisuke Kihara and Dilip Kumar and Sumit Mittal and Bohdan Monastyrskyy and Mateusz Olek and Palmer, {Colin M.} and Ardan Patwardhan and Alberto Perez and Jonas Pfab and Pintilie, {Grigore D.} and Richardson, {Jane S.} and Rosenthal, {Peter B.} and Daipayan Sarkar and Sch{\"a}fer, {Luisa U.} and Schmid, {Michael F.} and Schr{\"o}der, {Gunnar F.} and Mrinal Shekhar and Dong Si and Abishek Singharoy and Genki Terashi and Terwilliger, {Thomas C.} and Andrea Vaiana and Liguo Wang and Zhe Wang and Wankowicz, {Stephanie A.} and Williams, {Christopher J.} and Martyn Winn and Tianqi Wu and Xiaodi Yu and Kaiming Zhang and Berman, {Helen M.} and Wah Chiu",

note = "Funding Information: EMDataResource (C.L.L., A.K., G.P., H.M.B. and W.C.) is supported by the US National Institutes of Health (NIH)/National Institute of General Medical Science, grant no. R01GM079429. The Singharoy team used the supercomputing resources of the Oak Ridge Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science at the Department of Energy under contract no. DE-AC05-00OR22725. The following additional grants are acknowledged for participant support: grant no. NIH/R35GM131883 to J.S.R. and C.W.; grant no. NIH/P01GM063210 to P.D.A., P.V.A., L.-W.H., J.S.R., T.C.T. and C.W.; National Science Foundation grant no. (NSF)/MCB-1942763 (CAREER) and NIH/R01GM095583 to A.S.; grant nos. NIH/R01GM123055, NIH/R01GM133840, NSF/DMS1614777, NSF/CMMI1825941, NSF/MCB1925643, NSF/DBI2003635 and Purdue Institute of Drug Discovery to D. Kihara; grant no. NIH/R01GM123159 to J.S.F.; Max Planck Society German Research Foundation grant no. IG 109/1-1 to M.I.; Max Planck Society German Research Foundation grant no. FOR-1805 to A.C.V.; grant nos. NIH/R37AI36040 and Welch Foundation/Q1279 to D. Kumar (PI: BVV Prasad); grant no. NSF/DBI2030381 to D. Si.; Medical Research Council grant no. MR/N009614/1 to T.B., C.M.P. and M.W.; Wellcome Trust grant no. 208398/Z/17/Z to A.P.J. and M.W.; Biotechnology and Biological Sciences Research Council grant no. BB/P000517/1 to K.C. and Biotechnology and Biological Sciences Research Council grant no. BB/P000975/1 to M.W. Publisher Copyright: {\textcopyright} 2021, The Author(s). Copyright: Copyright 2021 Elsevier B.V., All rights reserved.",

year = "2021",

month = feb,

day = "4",

doi = "10.1038/s41592-020-01051-w",

language = "English",

volume = "18",

pages = "156--164",

journal = "Nature Methods",

issn = "1548-7091",

publisher = "Nature Publishing Group",

number = "2",

}

Lawson, CL, Kryshtafovych, A, Adams, PD, Afonine, PV, Baker, ML, Barad, BA, Bond, P, Burnley, T, Cao, R, Cheng, J, Chojnowski, G, Cowtan, K, Dill, KA, DiMaio, F, Farrell, DP, Fraser, JS, Herzik, MA, Hoh, SW, Hou, J, Hung, LW, Igaev, M, Joseph, AP, Kihara, D, Kumar, D, Mittal, S, Monastyrskyy, B, Olek, M, Palmer, CM, Patwardhan, A, Perez, A, Pfab, J, Pintilie, GD, Richardson, JS, Rosenthal, PB, Sarkar, D, Schäfer, LU, Schmid, MF, Schröder, GF, Shekhar, M, Si, D, Singharoy, A, Terashi, G, Terwilliger, TC, Vaiana, A, Wang, L, Wang, Z, Wankowicz, SA, Williams, CJ, Winn, M, Wu, T, Yu, X, Zhang, K, Berman, HM & Chiu, W 2021, 'Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge', Nature Methods, vol. 18, no. 2, pp. 156-164. https://doi.org/10.1038/s41592-020-01051-w

TY - JOUR

T1 - Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

AU - Lawson, Catherine L.

AU - Kryshtafovych, Andriy

AU - Adams, Paul D.

AU - Afonine, Pavel V.

AU - Baker, Matthew L.

AU - Barad, Benjamin A.

AU - Bond, Paul

AU - Burnley, Tom

AU - Cao, Renzhi

AU - Cheng, Jianlin

AU - Chojnowski, Grzegorz

AU - Cowtan, Kevin

AU - Dill, Ken A.

AU - DiMaio, Frank

AU - Farrell, Daniel P.

AU - Fraser, James S.

AU - Herzik, Mark A.

AU - Hoh, Soon Wen

AU - Hou, Jie

AU - Hung, Li Wei

AU - Igaev, Maxim

AU - Joseph, Agnel P.

AU - Kihara, Daisuke

AU - Kumar, Dilip

AU - Mittal, Sumit

AU - Monastyrskyy, Bohdan

AU - Olek, Mateusz

AU - Palmer, Colin M.

AU - Patwardhan, Ardan

AU - Perez, Alberto

AU - Pfab, Jonas

AU - Pintilie, Grigore D.

AU - Richardson, Jane S.

AU - Rosenthal, Peter B.

AU - Sarkar, Daipayan

AU - Schäfer, Luisa U.

AU - Schmid, Michael F.

AU - Schröder, Gunnar F.

AU - Shekhar, Mrinal

AU - Si, Dong

AU - Singharoy, Abishek

AU - Terashi, Genki

AU - Terwilliger, Thomas C.

AU - Vaiana, Andrea

AU - Wang, Liguo

AU - Wang, Zhe

AU - Wankowicz, Stephanie A.

AU - Williams, Christopher J.

AU - Winn, Martyn

AU - Wu, Tianqi

AU - Yu, Xiaodi

AU - Zhang, Kaiming

AU - Berman, Helen M.

AU - Chiu, Wah

N1 - Funding Information: EMDataResource (C.L.L., A.K., G.P., H.M.B. and W.C.) is supported by the US National Institutes of Health (NIH)/National Institute of General Medical Science, grant no. R01GM079429. The Singharoy team used the supercomputing resources of the Oak Ridge Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science at the Department of Energy under contract no. DE-AC05-00OR22725. The following additional grants are acknowledged for participant support: grant no. NIH/R35GM131883 to J.S.R. and C.W.; grant no. NIH/P01GM063210 to P.D.A., P.V.A., L.-W.H., J.S.R., T.C.T. and C.W.; National Science Foundation grant no. (NSF)/MCB-1942763 (CAREER) and NIH/R01GM095583 to A.S.; grant nos. NIH/R01GM123055, NIH/R01GM133840, NSF/DMS1614777, NSF/CMMI1825941, NSF/MCB1925643, NSF/DBI2003635 and Purdue Institute of Drug Discovery to D. Kihara; grant no. NIH/R01GM123159 to J.S.F.; Max Planck Society German Research Foundation grant no. IG 109/1-1 to M.I.; Max Planck Society German Research Foundation grant no. FOR-1805 to A.C.V.; grant nos. NIH/R37AI36040 and Welch Foundation/Q1279 to D. Kumar (PI: BVV Prasad); grant no. NSF/DBI2030381 to D. Si.; Medical Research Council grant no. MR/N009614/1 to T.B., C.M.P. and M.W.; Wellcome Trust grant no. 208398/Z/17/Z to A.P.J. and M.W.; Biotechnology and Biological Sciences Research Council grant no. BB/P000517/1 to K.C. and Biotechnology and Biological Sciences Research Council grant no. BB/P000975/1 to M.W. Publisher Copyright: © 2021, The Author(s). Copyright: Copyright 2021 Elsevier B.V., All rights reserved.

PY - 2021/2/4

Y1 - 2021/2/4

N2 - This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 Å). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.

AB - This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 Å). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.

UR - http://www.scopus.com/inward/record.url?scp=85101049839&partnerID=8YFLogxK

U2 - 10.1038/s41592-020-01051-w

DO - 10.1038/s41592-020-01051-w

M3 - Article

C2 - 33542514

AN - SCOPUS:85101049839

SN - 1548-7091

VL - 18

SP - 156

EP - 164

JO - Nature Methods

JF - Nature Methods

IS - 2

ER -

Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

Abstract

Bibliographical note

Access to Document

Other files and links

Flexible-body refinement for Cryogenic Electron Microscopy Applications

CCP4 Advanced integrated approaches to macromolecular structure determination

CCP4 Grant Renewal 2014-2019: Question-driven crystallographic data

2019 EMDataResource Model Metrics Challenge Dataset

Cite this

Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

Abstract

Bibliographical note

Access to Document

Other files and links

Projects

Flexible-body refinement for Cryogenic Electron Microscopy Applications

CCP4 Advanced integrated approaches to macromolecular structure determination

CCP4 Grant Renewal 2014-2019: Question-driven crystallographic data

Datasets

2019 EMDataResource Model Metrics Challenge Dataset

Cite this