Table 2

Abstract

Lancet. 2018 Nov 24;392(10161):22632264. doi: 10.1016/S01406736(18)32819-8. Epub 2018 Nov 6 2018 UK / Australia None Expert opinion / correspondence Symptom checkers have great potential to improve diagnosis, quality of care, and health system performance worldwide. However, systems that are poorly designed or lack rigorous clinical evaluation can put patients at risk and likely increase the load on health systems. Evaluation guidelines specific to symptom checkers have three benefits. First, they would provide system creators with a fixed set of criteria, ahead of time, on which they will be assessed. Second, they would allow external observers to assess the comprehensiveness and quality of evaluation, discouraging system creators from inflating the importance of their results. Finally, they would facilitate policy makers in determining a minimum level of evidence required before wide-scale use of a system. Academic writing using references Evidence considered on level of expert opinion It is not possible to determine how well the Babylon Diagnostic and Triage System would perform on a broader randomized set of cases or with data entered by patients instead of doctors. Babylon’s study does not offer convincing evidence that its Babylon Diagnostic and Triage System can perform better than doctors in any realistic situation, and there is a possibility that it might perform significantly worse. Evaluation of symptom checkers should follow a multistage process of Paper Review: the Babylon Chatbot[Internet]. The Guide to Health Informatics 3rd Edition. 2018 [cited 2019 May 29]. Available from: https://coiera.com/2018/06/29/paperreview-the-babylon-chatbot/ 2018 Australia (from reference list) None Critical review / expert opinion The used vignettes were designed to test known capabilities of the system. Independently created vignettes exploring other diagnoses would likely have resulted in a much poorer performance. This tests Babylon on what it knows not what it might find ‘in the wild. It seems the presentation of information was in the OSCE format, which is artificial and not how patients might present. So there was no real testing of consultation and listening skills that would be needed to manage a real world patient presentation. A better evaluation model would have been to draw a random subset of cases and present them to both GPs and Babylon. Important views from expert Evidence considered on level of expert opinion The reviewed study are considered a very preliminary and artificial test of a Bayesian reasoner on cases for which it has already been trained. In machine learning this would be roughly equivalent to in-sample reporting of performance on the data used to develop the algorithm. Good practice is to report out of sample performance on previously unseen cases. The results are confounded by artificial conditions and use of few and non-independent assessors. There is lack of clarity in the way data are analyzed and there are numerous risks of bias. Razzaki S, Baker A, Perov Y, Middleton K, Baxter J, Mullarkey D, et al. A comparative study of artificial intelligence and human doctors for the purpose of triage and diagnosis. ArXiv180610698 Cs Stat [Internet]. 2018 Jun 27 [cited 2019 May 15]; Available from: 2018 UK/U.S. (From reference list) Roleplay based on 56 vignettes on average between 7 doctors. 100 vignettes for algorithm. Accurate outcome analysis A prospective validation study of the accuracy and safety of an AI powered Triage and Diagnostic System was performed using an experimental paradigm designed to simulate realistic consultations. It was found that the Babylon Triage and Diagnostic System was able to identify the condition modelled by a clinical vignette with accuracy comparable to human doctors (in terms of precision and recall). It was also found that the triage advice recommended by the Babylon Triage and Diagnostic System was safer on average than human doctors, when compared to the ranges provided by independent expert judges, with only minimal reduction in appropriateness. In other words, the AI system was able to safely triage patients without reverting to overly pessimistic fallback decisions validation study using an experimental paradigm designed to simulate realistic consultations. Evaluated only clinical cases that were based on a single underlying condition. Similar cases was used to train the algorithm. Artificial intelligence powered symptom checkers have the potential to provide diagnostic and triage advice with a level of accuracy and safety approaching that of human doctors. Such systems may hold the promise of reduced costs and improved access to healthcare worldwide, but realising this requires greater levels of confidence from the medical community and the wider public. Key to this confidence is a better understanding of the strengths and weaknesses of Quro: Facilitating User Symptom Check Using a Personalised Chatbot-Oriented Dialogue System. Ghosh S, Bhatia S, Bhatia A. Stud Health Technol Inform. 2018;252:51-56 2018 Australia 30 test-case vignettes. Evaluation of chatbot triage accuracy Accurate outcome analysis Symptom extraction from user input involves employing algorithms used to recognize potential medical entity substrings in natural language text. A "medical entity" can refer to an instance of a medical concept such as Sign, Symptom, Disease, Drug and many more. Typically, medical entity recognition consists in: (i) identifying medical entities in the free text, and (ii) determining their categories. This is followed by detecting potential semantic relationships between the extracted medical entities The bot achieved an accurate outcome (in 1out 3 correct criteria) in 25 out of 30 cases (83.3%) and in (2 out 3 correct criteria) in 20 out of 30 cases (66.6%). Interestingly, the chatbot demonstrated a high recall of 100% for emergent care. An interesting aspect of our system was even though we populate our database with red-flag symptoms (for emergent care), our system does not rely on these red flag rules to infer an emergent care condition. There were 25 true positives (TPs) and 3 false positives (FPs) in criteria 1, and 20 TPs and 6 FPs in criteria 2. Accordingly, our chatbot showed an overall average Thorough and apparently transparent description of chatbot function and development. The study evaluation is based on only 30 clinical scenarios vignets, and are not evaluated by real patients in a natural patients language. Should be evaluated through a much larger number real patient cases. Conflict of interest since study done by developers of the program. An automated medical conversational platform powered by learning algorithms that provides personalized assessments based on symptoms is described. The bot’s symptom recognition and condition assessment performance could be greatly improved by adding support for more medical features, such as location, adverse events, and medical entities.