Currently submitted to: JMIR Research Protocols
Date Submitted: Mar 8, 2024
Open Peer Review Period: Mar 11, 2024 - May 6, 2024
(currently open for review)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Mixed Method Feasibility Study Protocol for Socrates 2.0: A Novel Cognitive Behavioral Therapy-Based Generative AI Tool to Facilitate Socratic Dialogue
ABSTRACT
Background:
Digital mental health tools designed to augment traditional mental health treatments are becoming increasingly important due to a wide range of barriers to access, including a growing shortage of clinicians. Most existing tools use rule-based algorithms, often leading to unnatural-feeling interactions compared to human therapists. Large language models (LLMs) offer a solution for the development of more natural, engaging digital tools. In this manuscript, we detail the development of Socrates 2.0, which was designed to engage users in Socratic dialogue surrounding unrealistic or unhelpful beliefs, a core technique in cognitive behavioral therapies (CBTs). The multi-agent LLM-based tool features an AI therapist, 'Socrates', which receives automated feedback from an AI supervisor and an AI rater. The combination of multiple agents appeared to help address common LLM issues like looping and improved the overall dialogue experience. Initial user feedback from individuals with lived experiences of mental health problems as well as cognitive behavioral therapists has been positive. Moreover, tests in ~500 scenarios showed that Socrates 2.0 engaged in harmful responses in under 1% of cases, with the AI supervisor promptly correcting the dialogue each time. However, formal feasibility studies with potential end users are needed.
Objective:
This manuscript details a mixed method feasibility study of Socrates 2.0.
Methods:
Based on the initial data we devised a formal feasibility study of Socrates 2.0 to gather qualitative and quantitative data about users’ and clinicians’ experience of interacting with the tool. Using a mixed method approach, the goal is to gather feasibility and acceptability data from both 100 users and 50 clinicians to inform the eventual implementation of generative AI tools like Socrates 2.0 in mental health treatment. We designed this study to better understand how users and clinicians interact with the tool including the frequency, length and time of interactions, users’ satisfactions with the tool overall, as well as the quality of each dialogue and even individual responses, as well as ways in which the tool should be improved before it is used in efficacy trials. Descriptive and inferential analyses will be performed on data from validated usability measures. Thematic analysis will be performed on the qualitative data.
Results:
Recruitment will begin in February 2024 and is expected to conclude by February 2025.
Conclusions:
The development of Socrates 2.0 and outlined feasibility study are important first steps in applying generative AI to mental health treatment delivery and lays the foundation for formal feasibility studies. Clinical Trial: N/A
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.