An AI bot may be better at diagnosing patients in emergency care than real doctors, a study shows. ChatGPT performed as well or better as real-life medics when given the same doctor’s notes of a patient, Dutch researchers found.
The chatbot could one day help doctors to reduce waiting times for patients, they suggested. Dr Hidde ten Berg, of Jeroen Bosch Hospital, said: “We found that ChatGPT performed well in generating a list of likely diagnoses and suggesting the most likely option.
“We also found a lot of overlap with the doctors’ lists of likely diagnoses. “Simply put, this indicates that ChatGPT was able to suggest medical diagnoses much like a human doctor would.”
ChatGPT was launched last year by OpenAI and uses one of the most sophisticated language models ever developed. Previous research showed it was able to pass the US Medical Licensing Exam and has a better bedside manner than real doctors.
The latest study, presented at European Emergency Medicine Congress, looked at how well it performed in A&E. Researchers anonymized details on 30 patients who were treated at Jeroen Bosch Hospital’s emergency department in 2022.
They entered physicians’ notes on patients’ signs, symptoms, and physical examinations into two versions of ChatGPT — the free 3.5 version and the subscriber 4.0 version.
The chatbots were asked to produce a list of possible diagnoses for the patients, which were compared to real doctors’ lists. Researchers found a large overlap — around 60 percent — between the shortlists generated by ChatGPT and the doctors.
Doctors had the correct diagnosis within their top five likely diagnoses in 87 percent of the cases, compared to 97 percent for ChatGPT version 3.5 and 87 percent for version 4.0.
Dr Ten Berg said: “We included a case of a patient presenting with joint pain that was alleviated with painkillers, but redness, joint pain, and swelling always recurred.
“In the previous days, the patient had a fever and sore throat. “A few times there was a discoloration of the fingertips. Based on the physical exam and additional tests, the doctors thought the most likely diagnosis was probably rheumatic fever, but ChatGPT was correct with its most likely diagnosis of vasculitis.”
He added: “It’s vital to remember that ChatGPT is not a medical device and there are concerns over privacy when using ChatGPT with medical data.
“However, there is potential here for saving time and reducing waiting times in the emergency department. “The benefit of using artificial intelligence could be in supporting doctors with less experience, or it could help in spotting rare diseases.”
Independent experts said the study adds to research suggesting AI could one day be used to assist doctors in hospitals. Professor Youri Yordanov, of the St Antoine Hospital emergency department, said: “We are a long way from using ChatGPT in the clinic.
