In recent times, chatbots have been explored as a potential tool for responding to patient queries, but questions remain regarding patients' ability to differentiate between human and chatbot responses, as well as their trust in chatbots' abilities. This study aimed to investigate the feasibility of using ChatGPT (Chat Generative Pre-trained Transformer) or a similar AI-based chatbot for patient-provider communication.
The Study
In January 2023, researchers conducted a survey involving ten typical patient-provider interactions extracted from electronic health records. Patient questions were fed into ChatGPT with a request to generate responses of similar length to those provided by human providers. The survey presented participants with both human and ChatGPT-generated responses, and they were asked to identify the source of each response. Additionally, participants were questioned about their level of trust in chatbots' role in patient-provider communication, using a scale from 1 to 5.
The Participants
A total of 430 adults aged 18 and above, representing a diverse group from the United States, were recruited for the study through an academic crowdsourcing platform called Prolific. Of these, 392 participants provided complete survey responses.
Findings
The results showed that the accuracy in distinguishing between human and chatbot responses varied across different questions. Participants correctly identified the source of responses between 49% and 85.7% of the time, depending on the specific question. On average, chatbot responses were correctly identified 65.5% of the time, while human provider responses were correctly identified 65.1% of the time. This suggests that people could not consistently tell the difference between the two sources.
Regarding the level of trust in chatbots for patient-provider communication, the average score on the Likert scale was 3.4 out of 5, indicating a somewhat positive but not overwhelming trust in chatbots. Interestingly, participants showed lower trust in chatbots for more complex health-related questions.
Conclusion
The study's findings suggest that responses generated by ChatGPT were not easily distinguishable from those of human providers. Laypeople generally demonstrated a level of trust in using chatbots for answering lower-risk health questions. However, as chatbots potentially take on more clinical roles in healthcare, further research on patient-chatbot interactions is crucial. Understanding the strengths and limitations of AI-based communication tools will be essential as we explore the integration of chatbots into healthcare settings.
Reference:
Comments