Mount Sinai AI Algorithm Beats Radiologist in Diagnosing COVID-19 Patients

Mount Sinai AI Algorithm Beats Radiologist in Diagnosing COVID-19 Patients

Mount Sinai researchers are declared the first in the nation to employ artificial intelligence (AI) combined with imaging, and clinical data to analyze patients with SARS-CoV-2, the virus behind COVID-19. The team has developed a unique algorithm capable of rapid detection of COVID-19 based on how lung disease looks in computed tomography (CT scans) of the best, in combination with patient information such as symptoms, age, bloodwork, and possible contact with someone infected by COVID-19. This technology breakthrough could help providers worldwide more rapidly detect the virus, isolate patients and stop the spreading during this pandemic.

The output of this study was published in the May 19 issue of Nature Medicine. This most recent research is actually an expansion on previous Mount Sinai research identifying characteristic patterns of the disease in the lungs of COVID-19 patients. To date, CT scans aren’t widely used for diagnosis of COVID-19 in America but imaging can play an important role and this new tool could accelerate adoption.

The Study

The AI algorithm was the result of a study involving 900 patients that Mount Sinai received from institutional collaborators at hospitals in China. These patients were admitted to 18 medical centers in 13 Chinese provinces between January 17 and March 3, 2020. The scans included 419 confirmed COVID-19 positive cases (most either had recently traveled to Wuhan, China, where the outbreak began, or had contact with an infected COVID-19 patient) and 486 COVID-19-negative scans.

The research team possessed patient clinical information such as blood test results revealing abnormalities in white blood cell counts or lymphocyte counts as well as their age, sex, and symptoms (e.g. fever, cough, etc.). They focused on CT scans and blood tests since doctors in China use both of these to diagnose patients with COVID-19 if they come in with fever or have been in contact with an infected patient.

Integrating the Data Requires an Understanding of Process

First, the Mount Sinai team had to integrate key data from those CT scans with the clinical information to develop an AI algorithm. Intelligently, the team reviewed existing physician workflow and actually designed an approach to mimic existing physician workflow involved when the physical diagnoses COVID-19: in this process, the physician offers a prediction of positive or negative diagnosis.

The AI Model

The actual AI model of course requires all of the data inputs. It then produces separate probabilities of being COVID-19-positive based on CT images, clinical data, and both combined. The researchers had to, of course, train and then fine tune the algorithm on data from 626 out of 905 patients, and then tested the algorithm on the remaining 279 patients in the study group) split between COVID-19 positive and negative cases) to judge the test’s sensitivity; higher sensitivity means better detection performance.

Algorithm Performance beats Radiologists

The actual algorithm performed well—producing results 84% higher when compared to 75% for radiologists evaluating the images and clinical trial data. The AI system also improved detection of COVID-19 positive patients who had negative CT scans. Specifically, it recognized 68% of COVID-19 positive cases, whereas radiologists interpreted all of these cases as negative due to the negative CT appearance.

The Value Proposition

As the AI outperformed the human doctor, these results are especially important to keep patients isolated If scans don’t’ show lung disease when patients first present symptoms (since the previous study showed that lung disease doesn’t always show up on a CT in the first few days) and COVID-19 symptoms are often nonspecific, resembling a flu or common cold, so it can be difficult to diagnose.

Hence Zahi Fayad, PhD, Director of the Biomedical Engineering and Imaging Institute (BMEII) at the Icahn School of Medicine at Mount Sinai and one of the lead authors, reports, “AI has huge potential for an analyzing large amounts of data quickly, an attribute that can have a big impact in a situation such as a pandemic. At Mount Sinai, we recognized this early and were able to mobilize the expertise of our faculty and our international collaborations to work on implementing a novel AI model using CT data from coronavirus patients in Chinese medical centers. We were able to show that the AI model was as accurate as an experienced radiologist in diagnosing the disease, and even better in some cases where there was no clear sign of lung disease on CT.”

Importance & Perspective

Matthew Levin, MD, Director of the Mount Sinai Health System’s Clinical Data Science Team and a member of the Mount Sinai COVID Informatics Center, noted on this “proof-of-concept study” that first and foremost they can include their own patient data in a quest to further develop and refine algorithms tailored more to the populations of their region—e.g. New York City. Moreover, the study reveals that AI can definitely “help with early identification of COVID-19, and this can be used in the clinical setting to triage or prioritize the evaluation of sick patients early in their admission to the emergency room.”

About Mount Sinai Health System

The Mount Sinai Health System is New York City’s largest academic medical system, encompassing eight hospitals, a leading medical school, and a vast network of ambulatory practices throughout the greater New York region. Mount Sinai is a national and international source of unrivaled education, translational research and discovery, and collaborative clinical leadership ensuring that we deliver the highest quality care–from prevention to treatment of the most serious and complex human diseases. The Health System includes more than 7,200 physicians and features a robust and continually expanding network of multispecialty services, including more than 400 ambulatory practice locations throughout the five boroughs of New York City, Westchester, and Long Island. The Mount Sinai Hospital is ranked No. 14 on U.S. News & World Report’s “Honor Roll” of the Top 20 Best Hospitals in the country and the Icahn School of Medicine as one of the Top 20 Best Medical Schools in country. Mount Sinai Health System hospitals are consistently ranked regionally by specialty by U.S. News & World Report.

Lead Research/Investigator

Hence Zahi Fayad, PhD, Director of the BioMedical Engineering and Imaging Institute (BMEII) at the Icahn School of Medicine at Mount Sinai

Call to Action: The team is now developing the ability to use the tool at home and collaborate on findings. Apparently, according to Fayad, the toolkit is deployable easily by other hospitals either online or integrated into their own provider systems. Mount Sinai researchers are now focused on further developing the model to find clues about how well patients will do based on subtleties in their CT data and clinical information. They say this could be important to optimize treatment and improve outcomes.