Evaluation of an artificial intelligence-based medical device for diagnosis of autism spectrum disorder

Jonathan T Megerian; Sangeeta Dey; Raun D Melmed; Daniel L Coury; Marc Lerner; Christopher J Nicholls; Kristin Sohl; Rambod Rouhbakhsh; Anandhi Narasimhan; Jonathan Romain; Sailaja Golla; Safiullah Shareef; Andrey Ostrovsky; Jennifer Shannon; Colleen Kraft; Stuart Liu-Mayo; Halim Abbas; Diana E Gal-Szabo; Dennis P Wall; Sharief Taraman

doi:10.1038/s41746-022-00598-6

Evaluation of an artificial intelligence-based medical device for diagnosis of autism spectrum disorder

NPJ Digit Med. 2022 May 5;5(1):57. doi: 10.1038/s41746-022-00598-6.

Authors

Jonathan T Megerian^{1

2}, Sangeeta Dey^{3

4}, Raun D Melmed⁵, Daniel L Coury^{6

7}, Marc Lerner^{1

2}, Christopher J Nicholls^{8

9}, Kristin Sohl¹⁰, Rambod Rouhbakhsh^{11

12}, Anandhi Narasimhan¹³, Jonathan Romain^{1

2}, Sailaja Golla¹⁴, Safiullah Shareef¹⁵, Andrey Ostrovsky^{16

17}, Jennifer Shannon¹⁸, Colleen Kraft¹⁸, Stuart Liu-Mayo¹⁸, Halim Abbas¹⁸, Diana E Gal-Szabo¹⁸, Dennis P Wall^{18

19}, Sharief Taraman^{20

21

22

23}

Affiliations

¹ CHOC Children's, Orange, CA, USA.
² University of California, Irvine School of Medicine, Department of Pediatrics, Irvine, CA, USA.
³ Bay Area Neuropsychological and Developmental Services, Palo Alto, CA, USA.
⁴ Stanford University, Department of Developmental Behavioral Pediatrics, Lucile Packard Children's Hospital, Stanford, CA, USA.
⁵ Melmed Center, Scottsdale, AZ, USA.
⁶ Nationwide Children's Hospital, Columbus, OH, USA.
⁷ The Ohio State University, College of Medicine, Columbus, OH, USA.
⁸ The Nicholls Group, Scottdale, AZ, USA.
⁹ Arizona State University, Department of Psychology, Tempe, AZ, USA.
¹⁰ University of Missouri, School of Medicine, Columbia, MO, USA.
¹¹ Forrest General Hospital, Family Medicine Residency Program, Hattiesburg, MS, USA.
¹² Hattiesburg Clinic, MediSync Clinical Research, Petal, Hattiesburg, MS, USA.
¹³ Private Practice, Los Angeles, CA, USA.
¹⁴ Texas Institute for Neurological Disorders, Frisco, TX, USA.
¹⁵ Texas Child Neurology, Plano, TX, USA.
¹⁶ Social Innovation Ventures, Washington, DC, USA.
¹⁷ Children's National Health System, Washington, DC, USA.
¹⁸ Cognoa Inc., Palo Alto, CA, USA.
¹⁹ Stanford University, Department of Pediatrics, Department of Biomedical Data Science, and Department of Psychiatry and Behavioral Sciences, Stanford, CA, USA.
²⁰ CHOC Children's, Orange, CA, USA. sharief@cognoa.com.
²¹ University of California, Irvine School of Medicine, Department of Pediatrics, Irvine, CA, USA. sharief@cognoa.com.
²² Cognoa Inc., Palo Alto, CA, USA. sharief@cognoa.com.
²³ Chapman University, Dale E. and Sarah Ann Fowler School of Engineering, Orange, CA, USA. sharief@cognoa.com.

Abstract

Autism spectrum disorder (ASD) can be reliably diagnosed at 18 months, yet significant diagnostic delays persist in the United States. This double-blinded, multi-site, prospective, active comparator cohort study tested the accuracy of an artificial intelligence-based Software as a Medical Device designed to aid primary care healthcare providers (HCPs) in diagnosing ASD. The Device combines behavioral features from three distinct inputs (a caregiver questionnaire, analysis of two short home videos, and an HCP questionnaire) in a gradient boosted decision tree machine learning algorithm to produce either an ASD positive, ASD negative, or indeterminate output. This study compared Device outputs to diagnostic agreement by two or more independent specialists in a cohort of 18-72-month-olds with developmental delay concerns (425 study completers, 36% female, 29% ASD prevalence). Device output PPV for all study completers was 80.8% (95% confidence intervals (CI), 70.3%-88.8%) and NPV was 98.3% (90.6%-100%). For the 31.8% of participants who received a determinate output (ASD positive or negative) Device sensitivity was 98.4% (91.6%-100%) and specificity was 78.9% (67.6%-87.7%). The Device's indeterminate output acts as a risk control measure when inputs are insufficiently granular to make a determinate recommendation with confidence. If this risk control measure were removed, the sensitivity for all study completers would fall to 51.6% (63/122) (95% CI 42.4%, 60.8%), and specificity would fall to 18.5% (56/303) (95% CI 14.3%, 23.3%). Among participants for whom the Device abstained from providing a result, specialists identified that 91% had one or more complex neurodevelopmental disorders. No significant differences in Device performance were found across participants' sex, race/ethnicity, income, or education level. For nearly a third of this primary care sample, the Device enabled timely diagnostic evaluation with a high degree of accuracy. The Device shows promise to significantly increase the number of children able to be diagnosed with ASD in a primary care setting, potentially facilitating earlier intervention and more efficient use of specialist resources.