MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation

Alexandros Karargyris; Renato Umeton; Micah J. Sheller; Alejandro Aristizabal; Johnu George; Srini Bala; Daniel J. Beutel; Victor Bittorf; Akshay Chaudhari; Alexander Chowdhury; Cody Coleman; Bala Desinghu; Gregory Diamos; Debo Dutta; Diane Feddema; Grigori Fursin; Junyi Guo; Xinyuan Huang; David Kanter; Satyananda Kashyap; Nicholas Lane; Indranil Mallick; Pietro Mascagni; Virendra Mehta; Vivek Natarajan; Nikola Nikolov; Nicolas Padoy; Gennady Pekhimenko; Vijay Janapa Reddi; G Anthony Reina; Pablo Ribalta; Jacob Rosenthal; Abhishek Singh; Jayaraman J. Thiagarajan; Anna Wuest; Maria Xenochristou; Daguang Xu; Poonam Yadav; Michael Rosenthal; Massimo Loda; Jason M. Johnson; Peter Mattson

MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation

Alexandros Karargyris, Renato Umeton, Micah J. Sheller, Alejandro Aristizabal, Johnu George, Srini Bala, Daniel J. Beutel, Victor Bittorf, Akshay Chaudhari, Alexander Chowdhury, Cody Coleman, Bala Desinghu, Gregory Diamos, Debo Dutta, Diane Feddema, Grigori Fursin, Junyi Guo, Xinyuan Huang, David Kanter, Satyananda KashyapNicholas Lane, Indranil Mallick, Pietro Mascagni, Virendra Mehta, Vivek Natarajan, Nikola Nikolov, Nicolas Padoy, Gennady Pekhimenko, Vijay Janapa Reddi, G Anthony Reina, Pablo Ribalta, Jacob Rosenthal, Abhishek Singh, Jayaraman J. Thiagarajan, Anna Wuest, Maria Xenochristou, Daguang Xu, Poonam Yadav, Michael Rosenthal, Massimo Loda, Jason M. Johnson, Peter Mattson

Computer Science

Research output: Working paper › Preprint

Abstract

Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable federated evaluation in which models are securely distributed to different facilities for evaluation, thereby empowering healthcare organizations to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status, and our roadmap. We call for researchers and organizations to join us in creating the MedPerf open benchmarking platform.

Original language	English
Publisher	arXiv
Publication status	Published - 29 Sept 2021

Keywords

cs.LG
cs.DC
cs.PF
cs.SE

Access to Document

2110.01406v3

Cite this

Karargyris, A., Umeton, R., Sheller, M. J., Aristizabal, A., George, J., Bala, S., Beutel, D. J., Bittorf, V., Chaudhari, A., Chowdhury, A., Coleman, C., Desinghu, B., Diamos, G., Dutta, D., Feddema, D., Fursin, G., Guo, J., Huang, X., Kanter, D., ... Mattson, P. (2021). MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation. arXiv.

@techreport{9c8dc39a630a4d5c82240574ca5a64d4,

title = "MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation",

abstract = " Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable federated evaluation in which models are securely distributed to different facilities for evaluation, thereby empowering healthcare organizations to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status, and our roadmap. We call for researchers and organizations to join us in creating the MedPerf open benchmarking platform. ",

keywords = "cs.LG, cs.DC, cs.PF, cs.SE",

author = "Alexandros Karargyris and Renato Umeton and Sheller, {Micah J.} and Alejandro Aristizabal and Johnu George and Srini Bala and Beutel, {Daniel J.} and Victor Bittorf and Akshay Chaudhari and Alexander Chowdhury and Cody Coleman and Bala Desinghu and Gregory Diamos and Debo Dutta and Diane Feddema and Grigori Fursin and Junyi Guo and Xinyuan Huang and David Kanter and Satyananda Kashyap and Nicholas Lane and Indranil Mallick and Pietro Mascagni and Virendra Mehta and Vivek Natarajan and Nikola Nikolov and Nicolas Padoy and Gennady Pekhimenko and Reddi, {Vijay Janapa} and Reina, {G Anthony} and Pablo Ribalta and Jacob Rosenthal and Abhishek Singh and Thiagarajan, {Jayaraman J.} and Anna Wuest and Maria Xenochristou and Daguang Xu and Poonam Yadav and Michael Rosenthal and Massimo Loda and Johnson, {Jason M.} and Peter Mattson",

year = "2021",

month = sep,

day = "29",

language = "English",

publisher = "arXiv",

type = "WorkingPaper",

institution = "arXiv",

}

Karargyris, A, Umeton, R, Sheller, MJ, Aristizabal, A, George, J, Bala, S, Beutel, DJ, Bittorf, V, Chaudhari, A, Chowdhury, A, Coleman, C, Desinghu, B, Diamos, G, Dutta, D, Feddema, D, Fursin, G, Guo, J, Huang, X, Kanter, D, Kashyap, S, Lane, N, Mallick, I, Mascagni, P, Mehta, V, Natarajan, V, Nikolov, N, Padoy, N, Pekhimenko, G, Reddi, VJ, Reina, GA, Ribalta, P, Rosenthal, J, Singh, A, Thiagarajan, JJ, Wuest, A, Xenochristou, M, Xu, D, Yadav, P, Rosenthal, M, Loda, M, Johnson, JM & Mattson, P 2021 'MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation' arXiv.

TY - UNPB

T1 - MedPerf

T2 - Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation

AU - Karargyris, Alexandros

AU - Umeton, Renato

AU - Sheller, Micah J.

AU - Aristizabal, Alejandro

AU - George, Johnu

AU - Bala, Srini

AU - Beutel, Daniel J.

AU - Bittorf, Victor

AU - Chaudhari, Akshay

AU - Chowdhury, Alexander

AU - Coleman, Cody

AU - Desinghu, Bala

AU - Diamos, Gregory

AU - Dutta, Debo

AU - Feddema, Diane

AU - Fursin, Grigori

AU - Guo, Junyi

AU - Huang, Xinyuan

AU - Kanter, David

AU - Kashyap, Satyananda

AU - Lane, Nicholas

AU - Mallick, Indranil

AU - Mascagni, Pietro

AU - Mehta, Virendra

AU - Natarajan, Vivek

AU - Nikolov, Nikola

AU - Padoy, Nicolas

AU - Pekhimenko, Gennady

AU - Reddi, Vijay Janapa

AU - Reina, G Anthony

AU - Ribalta, Pablo

AU - Rosenthal, Jacob

AU - Singh, Abhishek

AU - Thiagarajan, Jayaraman J.

AU - Wuest, Anna

AU - Xenochristou, Maria

AU - Xu, Daguang

AU - Yadav, Poonam

AU - Rosenthal, Michael

AU - Loda, Massimo

AU - Johnson, Jason M.

AU - Mattson, Peter

PY - 2021/9/29

Y1 - 2021/9/29

N2 - Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable federated evaluation in which models are securely distributed to different facilities for evaluation, thereby empowering healthcare organizations to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status, and our roadmap. We call for researchers and organizations to join us in creating the MedPerf open benchmarking platform.

AB - Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable federated evaluation in which models are securely distributed to different facilities for evaluation, thereby empowering healthcare organizations to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status, and our roadmap. We call for researchers and organizations to join us in creating the MedPerf open benchmarking platform.

KW - cs.LG

KW - cs.DC

KW - cs.PF

KW - cs.SE

M3 - Preprint

BT - MedPerf

PB - arXiv

ER -