Abstract:
Background
Despite progress made towards malaria control in endemic countries, key indicators have stalled since 2015. Achieving the World Health Organization's (WHO) Global Technical Strategy for Malaria 2016–2030 targets—aiming for 90% reduction in malaria cases and deaths by 2030—will require innovative approaches. Here, we present the results of a study which explores the deployment and adoption of an artificial intelligence (AI)-driven Connected Diagnostics (ConnDx) system for malaria Rapid Diagnostic Test (mRDT) interpretation, using digital platforms. ConnDx is an integrated system that uses HealthPulse app to capture mRDT images, upload them to the cloud, interpret them using AI, and display aggregated results on a dashboard.
Methods
This cross-sectional study was conducted within five health facilities in Kisumu County, western Kenya. We tested febrile patients (≥ 6 months of age) suspected of malaria using standard mRDTs and uploaded photographs of the tests to a cloud server for AI (ConnDx) interpretation leveraging computer vision models. Two healthcare workers from each of the five facilities received training on performing mRDTs using the ConnDx system. The performance of the AI model and test administrators was evaluated using an Expert Panel – a group of three independent external readers specialized in high quality data annotation–as “ground truth”. Performance was assessed through weighted F1 score for accuracy and Cohen’s Kappa for concordance. The deviations in concordance levels between AI and test administrators, relative to the Expert Panel, were assessed using a two-proportion Z-test.
Results
Between May and December 2023, a total of 3,851 mRDTs were conducted across the facilities, of which 3,620 had complete associated data and were included in the analysis. The AI model demonstrated weighted F1 score of 0.975 (95% CI: 0.970–0.981) and Cohen’s Kappa of 0.92 (95% CI: 0.91–0.94) compared to the Expert Panel. The overall concordance between the AI and the Expert Panel in interpreting mRDT results was 3,472 (96.4%). Agreement between test administrators and Expert Panel was 3,496 (97.1%), with Cohen’s Kappa of 0.92 (95% CI: 0.91–0.93). No statistically significant difference was found between the concordance of AI and that of test administrators when each of the methods was compared to the Expert Panel (P = 0.11). Additionally, the AI model demonstrated a sensitivity of 96.1% (95% CI: 94.6–97.3) and a specificity of 98.0% (95% CI: 97.4–98.5) relative to the Expert Panel. Notably, interpretation accuracy metrics varied considerably across the five facilities, indicating potential differences in testing conditions or user performance.
Conclusion
ConnDx demonstrates strong potential for collecting malaria diagnostic data in real-time with improved quality assurance of mRDT interpretation in resource-limited settings.