While artificial intelligence tools like ChatGPT are proving useful in the medical field for diagnosing and easing the workload of doctors, they are still far from perfect. A study published in the medical journal Nature Medicine by researchers from Harvard and Stanford universities highlights the limitations of these tools.
They conducted a test using a tool called CRAFT-MD to evaluate the performance of four AI models (GPT-4, GPT-3.5, Mistral, and Meta's Llama 2-7b) in scenarios closely mimicking real patient interactions, yielding disappointing results.
After assessing the capabilities of these models across 12 medical specialties, the researchers noted that while they performed adequately on exam-like questions, their effectiveness declined during more realistic doctor-patient conversations. They pointed out that all AI models exhibited limitations in conducting clinical dialogues and reasoning based on patient information, affecting their ability to gather medical histories and make accurate diagnoses.
