PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS

1
1

2 Title: Chatbot-delivered acceptance and commitment therapy with adolescents: A pilot

3 randomized controlled trial

5 Stapleton1†, A., Lavelle1, J., & McHugh1, L.

7
1
8 School of Psychology, University College Dublin, Belfield, Dublin 4, Ireland.

9
†
10 Corresponding author: Alison Stapleton, School of Psychology, Newman Building,

11 University College Dublin, Belfield, Dublin 4, Ireland.

12 Email: [email protected]

13

14 Data are available upon reasonable request.

15 This research did not receive any specific grant from funding agencies in the public,

16 commercial, or not-for-profit sectors.

17

18 Acknowledgments: Thanks to Dr. Abubakr Siddig and Dr. Andrew Hines from the School of

19 Computer Science at University College Dublin for building our chatbot.

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
2
20 Abstract

21 To date, no published randomized controlled trial (RCT) has evaluated chatbot-

22 delivered acceptance and commitment therapy (ACT) for youth mental health. The present

23 pilot study was a three-arm parallel-group RCT that aimed to (i) evaluate universal chatbot-

24 delivered ACT for adolescents and (ii) assess the feasibility of nesting high temporal density

25 measures of problematic rule-following (i.e., daily ecological momentary assessments) in the

26 chatbot. 148 young people aged 15 to 17 years were randomly assigned to receive either

27 chatbot-delivered ACT, psychoeducation via a booklet, or neither of these (non-active control).

28 Participants completed self-report measures of generalized pliance, tracking, affect,

29 psychological inflexibility, and valued living at three time-points (immediately pre-

30 intervention, immediately post-intervention, and at a six-week follow-up), with participants

31 assigned to the chatbot further completing daily single-item assessments of problematic

32 pliance. No statistically significant changes in any dependent variable were observed across

33 conditions or over time. As per intervention satisfaction data, chatbot-delivered ACT

34 demonstrated adequate acceptability. Given both the apparent convergence with a standardized

35 measure and minimal missing data, nesting ecological momentary assessments in the chatbot

36 was deemed highly feasible. Findings are discussed with explicit reference to (i) appropriate

37 session duration and (ii) means of enhancing artificial intelligence and natural language

38 processing technologies underlying the chatbot going forward.

39 Highlights

40 • Three-arm parallel-group RCT: Chatbot ACT, psychoeducation, non-active control

41 • Self-reports of rule-following, affect, and psyflex at pre-, post-, & 6-week follow-up

42 • No change in any dependent variable was observed across conditions or over time

43 • As per intervention satisfaction data, chatbot-delivered ACT was deemed acceptable

44 • Nesting ecological momentary assessments in the chatbot was deemed highly feasible
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
3
45 Introduction

46 Mental health challenges are one of three main contributors to the global burden of

47 disease among young people, accounting for 45% of years lost due to disability (Gore et al.,

48 2011). At adolescence, poor mental health is strongly associated with low educational

49 attainment, substance misuse, risk-taking, aggression, and suicide (Dray et al., 2017; Patel et

50 al., 2007; Rao et al., 1993). The effects of poor youth mental health are lasting, evidenced to

51 impact a range of domains in later life, including well-being and work productivity (Patel et

52 al., 2007; Roza et al., 2003). Moreover, mental health challenges at adolescence are a strong

53 precursor for mental health diagnoses in adulthood, especially when left untreated (e.g.,

54 Keenan-Miller et al., 2007).

55 Given the short- and long-term effects of youth mental health and given that

56 adolescence is a critical developmental period (e.g., a time of apparent increased sensitivity to

57 environmental contingencies due to puberty-related synaptic reorganization; Blakemore &

58 Choudhury, 2006; Crone & Dahl, 2012), promoting and protecting youth mental health is

59 imperative (Eyre & Thapar, 2014; Patton et al., 2016). Departing from deficit- and disease-

60 centered approaches to psychological well-being (Arango et al., 2018), many interventionists

61 working with young people today target both distress and ‘positive’ mental health (i.e., skills

62 for responding effectively to everyday stressors; Fusar‐Poli et al., 2021; O’Connor et al., 2022).

63 Although related to distress, ‘positive’ mental health represents a distinct continuum

64 that can (theoretically) be enhanced across all phases of life/clinical stages (O’Connor et al.,

65 2022). This means interventionists have the potential to enact meaningful change among both

66 clinical cohort and general population members (Arango et al., 2018; O’Connor et al., 2022).

67 Interventions targeting both samples (i.e., delivered to individuals irrespective of

68 symptomatology presence/severity) are termed ‘universal’ and primarily focus on skills

69 building (Burckhardt et al., 2017). This focus is important for two key reasons. First, the mere
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
4
70 absence of distress is not sufficient to guarantee favorable developmental outcomes (Grové,

71 2021; Suldo & Shaffer, 2008). Second, this focus aligns with recent prosocial agendas calling

72 for societies to extend beyond mere symptom management, and instead explicitly increase

73 creativity, vitality, productivity, and connectedness among citizens (e.g., Biglan & Glenn,

74 2013).

75 One approach evidenced to reduce distress and increase ‘positive’ mental health among

76 young people is acceptance and commitment therapy (ACT; Fang & Ding, 2020; Hayes et al.,

77 2012; Swain et al., 2015). ACT blends acceptance, mindfulness, and behavior change

78 techniques to promote engagement in valued action while simultaneously making space for

79 inevitable unwanted internal experiences (Hayes et al., 2012). ACT is underpinned by the

80 theoretical psychological flexibility model (Levin et al., 2012; McCracken & Morley, 2014),

81 which is generally understood in terms of six interrelated components (hexaflex), namely

82 acceptance, defusion, self-as-context, present moment awareness, values, and committed

83 action (Hayes et al., 2012). See Hayes et al. (2006) for a comprehensive overview of the

84 psychological flexibility model via the hexaflex.

85 When discussing the hexaflex, there are two important caveats to address. First,

86 hexaflex components are considered “middle-level” terms; they are top-down, theory-specific,

87 non-technical terms intended to pragmatically orient researchers/practitioners towards a

88 particular domain (Barnes-Holmes et al., 2016; Wilson, 2016). Simply put, hexaflex

89 components do not originate from empirical work and thus lack (relative) precision (Wilson,

90 2016). Second, hexaflex components are often described as both processes (e.g., viewing

91 defusion as the process of broadening stimulus control) and outcomes (e.g., a state of being

92 ‘defused’), further reducing their precision (Assaz et al., 2022). Thus, at best, hexaflex

93 components serve as “short-hand summaries for sets of functional analyses” (Hayes et al.,

94 2021, p.176) that guide the development of interventions due to their associations with
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
5
95 favorable outcomes. Indeed, ACT is evidenced to beneficially impact youth mental health

96 across presentations (as per practitioner-, parent- and child-reported measures of symptoms and

97 functioning) according to recent a systematic review (Swain et al., 2015) and meta-analysis

98 (Fang & Ding, 2020).

99 So far, we have outlined the importance of universal interventions for youth mental

100 health, further presenting ACT as a viable option. We now turn to pliance, a type of rule-

101 following integrated into the relational frame theory (RFT) account of rule-governed behavior

102 (Barnes-Holmes et al., 2001; Hayes et al., 1998). Pliance refers to rule-following controlled by

103 a history of arbitrary speaker-mediated reinforcement for frames of coordination between rules

104 and behavior (Hayes et al., 1998; Ruiz et al., 2019); pliance means rule-following for the sake

105 of rule-following itself and is consequated by the verbal community. If the verbal community

106 fails to appropriately contextualize pliance for an individual, pliance will generalize and

107 dominate their behavioral repertoire (i.e., generalized pliance; Ruiz et al., 2019).

108 Topographically, generalized pliance might look like excessive ‘people-pleasing’ or

109 hypersensitivity to ‘peer pressure’ because individuals displaying generalized pliance are

110 particularly reliant on arbitrary social approval (Törneke et al., 2008).

111 For young people, generalized pliance is likely particularly problematic since young

112 people are often making decisions here and now that will shape the rest of their lives (Ruiz et

113 al., 2019; Salazar et al., 2018a). For example, imagine a young person choosing between school

114 subjects/classes. If they base their decision solely on what they believe will please their

115 parents/teachers/peers (i.e., pliance), when they do enroll in their selected subject/class, they

116 may struggle to contact relevant sources of positive reinforcement, thus entering loops of

117 negative reinforcement (Ruiz et al., 2019). Generalized pliance may also precipitate reduced

118 sensitivity to shifting environmental contingencies (i.e., regardless of experienced utility, plys

119 may be resistant to change), exacerbating unhelpful patterns of behavior in the long run
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
6
120 (Luciano et al., 2012; O’Connor et al., 2019). Thus, decreasing adolescents’ pliant rule-

121 following in favor of accurate tracking is desirable, further aligning with both prosocial and

122 ‘positive’ mental health agendas.

123 Given that (i) RFT is compatible with ACT, (ii) ACT promotes behaving in line with

124 ‘freely chosen’ values (i.e., the antithesis of pliance), and (iii) ACT-RFT provides ways to

125 symbolically alter the supports for pliance (i.e., via language; Villatte et al., 2015), ACT

126 interventions may foster context-sensitive, flexible rule-following that is useful for the

127 individual. In this way, ACT may result in reductions in problematic pliance. Supporting this,

128 research has observed negative correlations between generalized pliance and hexaflex

129 components (Stapleton et al., 2022a; Waldeck et al., 2019), in addition to positive correlations

130 between tracking and hexaflex components (Ruiz et al., 2020a). Although few empirical studies

131 have directly examined the effects of ACT on pliant rule-following, available studies report

132 that ACT was associated with lesser problematic pliance post-intervention (Ruiz et al., 2020b;

133 Salazar et al., 2020; Stapleton et al., 2022b). Notably, to date, no studies have examined the

134 effects of ACT on tracking.

135 Thus far, we have argued that ACT is a viable universal intervention that may reduce

136 distress (i.e., affect), increase ‘positive’ mental health (i.e., psychological flexibility and valued

137 living), and decrease problematic rule-following (i.e., decrease generalized pliance and

138 increase tracking) among adolescents. We now turn our attention to modality/format (i.e., how

139 ACT is delivered to young people). Given that, relative to other age cohorts, young people

140 spend more time online (Edwards‐Hart & Chester, 2010), internet-delivered ACT may be

141 readily accessible and highly acceptable for young people. Automated internet-delivered

142 interventions (e.g., text-based conversational agents, hereafter ‘chatbots’) are particularly

143 desirable because they can reduce demands on systems’ finances, resources, and
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
7
144 practitioner/clinician time, while simultaneously providing some support between sessions/to

145 wait-listed individuals (Lavelle et al., 2022).

146 Zhou et al.’s (2021) recent systematic review of randomized controlled trials (RCTs)

147 examining online interventions for youth mental health found no published RCT testing

148 chatbot-delivered ACT for young people. However, Zhou et al. (2021) did report that online

149 ACT was somewhat effective in beneficially impacting young people’s well-being; of five

150 identified RCTs, ACT beneficially impacted outcomes in three studies (Eustis et al., 2018;

151 Lappalainen et al., 2021; Levin et al., 2020), but not two other studies (Levin et al., 2014;

152 Puolakanaho et al., 2019). Given that (i) chatbots mirror human interactions and thus are more

153 engaging than online self-help/webpages (Grové, 2021; Lavelle et al., 2022) and (ii) chatbots

154 are evidenced to positively impact adolescents’ well-being when delivering other types of

155 therapeutic interventions (Bird et al., 2018; Fitzpatrick et al., 2017; Fulmer et al., 2018),

156 chatbot-delivered ACT may beneficially impact youth mental health.

157 The present pilot RCT evaluated universal chatbot-delivered ACT for youth mental

158 health. Specifically, we examined whether chatbot-delivered ACT would yield greater

159 reductions in problematic rule-following (i.e., decreased generalized pliance and increased

160 tracking), negative affect, and psychological inflexibility, in addition to greater gains in

161 positive affect and valued living, relative to psychoeducation via a booklet. This line of

162 research is important for myriad reasons. First, promoting flexible rule-following and

163 protecting youth mental health is imperative; developing effective automated internet-

164 delivered interventions may increase support accessibility while decreasing the strain on

165 existing systems. Second, relative to the use of ACT with adults, there are fewer empirical

166 studies examining the effectiveness of ACT with young people (Swain et al., 2015), with no

167 published RCTs evaluating chatbot-delivered ACT for youth (Zhou et al., 2021). Third, few

168 studies have explicitly targeted problematic rule-following (Ruiz et al., 2020b; Salazar et al.,
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
8
169 2020; Stapleton et al., 2022b), with none measuring both pliance and tracking. Finally, as per

170 Feeley et al. (2009), conducting a pilot RCT would allow us to assess the feasibility and

171 acceptability of our design, procedure, and the chatbot intervention itself (i.e., both

172 participants’ satisfaction and intervention effectiveness), in addition to facilitating estimates of

173 effect sizes yielded from our intervention (if any) for use in future sample-size calculations.

174 A final aim of the present pilot RCT was to determine the feasibility of nesting

175 (relatively) high temporal density measures of problematic rule-following (i.e., daily ecological

176 momentary assessments) in future RCTs. More specifically, we wanted to know whether there

177 would be many missing data points and whether these assessments trended in expected

178 directions as per visual inspections (e.g., if a participant reported lesser pliance post-

179 intervention relative to pre-intervention, would their ecological momentary assessment data

180 appear to decrease across the intervention period?). This aim represents an important step

181 toward retaining degrees of nuance in individuals’ responses to interventions (McDonald et al.,

182 2017). Simply put, since conclusions we draw from inter-individual data may not apply at the

183 intra-individual level (Hayes et al., 2020, 2021; McDonald et al., 2017), we need to develop

184 and evaluate high temporal density assessments of dependent variables that enable us to

185 meaningfully look within individuals.

186 To summarize, the present pilot RCT aimed to (i) evaluate universal chatbot-delivered

187 ACT for youth mental health and (ii) assess the feasibility of nesting ecological momentary

188 assessments of problematic rule-following in the chatbot. We predicted that the chatbot would

189 yield greater reductions in generalized pliance, negative affect, and psychological inflexibility,

190 in addition to greater gains in tracking, positive affect, and valued living, relative to

191 psychoeducation via a booklet, further hypothesizing that no changes would be observed in

192 these variables in the non-active control condition. The CONSORT guidelines by Schulz et al.

193 (2010) were used to guide the present write-up.

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
9
194

195 Method

196 Participants and Recruitment

197 One of the aims of the present pilot randomized controlled trial was to facilitate the

198 determination of effect sizes associated with our chatbot for use in future sample-size

199 calculations. Thus, guided by previous evaluations of chatbots for youth mental health (e.g.,

200 Fulmer et al.’s (2018) two-arm trial with 75 participants) and previous pilot randomized

201 controlled trials assessing ACT with youth (e.g., Guerrini Usubini et al.’s (2022) two-arm trial

202 with 49 participants; White et al.’s (2022) two-arm trial with 89 participants), we aimed to

203 recruit 150 adolescents.

204 Eligible participants were (a) aged 15 to 17 years, (b) enrolled in secondary school (high

205 school) in [REMOVED FOR MASKED REVIEW], (c) willing to provide written assent, (d)

206 able to obtain written consent from a parent/guardian, (e) users of Facebook Messenger that

207 owned a device capable of running Facebook Messenger (i.e., could be randomized to the

208 chatbot condition), and (f) not diagnosed with/recognized to experience a language difficulty

209 or intellectual/physical disability that may impact their capacity to assent to taking part in the

210 study.

211 Participants were recruited from [REMOVED FOR MASKED REVIEW] in line with

212 [REMOVED FOR MASKED REVIEW] policies. To facilitate recruitment, [REMOVED FOR

213 MASKED REVIEW] first approached school principals to describe the research. School

214 principals then decided whether to present the research participation opportunity to the

215 parents/guardians of adolescents attending their school. If both the young person and their

216 parent/guardian were willing for them to participate, they submitted signed assent and consent

217 forms respectively to the principal for the researcher. Once these signed forms were received

218 by the researcher, participants were enrolled in the trial and randomized to one of the
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
10
219 conditions. Ethical approval for the present pilot randomized trial was obtained from the

220 [REMOVED FOR MASKED REVIEW] Research Ethics Committee and procedures were

221 performed in accordance with the Declaration of Helsinki (World Medical Association, 2013).

222 The final sample that was randomly allocated to one of the conditions was comprised

223 of 148 young people aged 15 to 17 years (Mage = 15.71, SDage = .620) that identified as male

224 (n = 94) and female (n = 54). Participants predominantly identified as White (132; Black = 4;

225 Asian = 4; Hispanic = 1; Latino = 1; Arab = 1; Multiracial = 4; Mixed race = 1). Descriptive

226 statistics at Time 1 (pre-intervention) are presented in Table 1. Participant flow through the

227 trial is shown in Figure 1.

228

229 Design

230 The present pilot was a three-arm parallel-group randomized controlled trial.

231 Participants were randomly assigned (at the individual level) to experience one of three

232 conditions, namely (i) chatbot-delivered ACT, (ii) psychoeducation via a booklet, or (iii)

233 neither chatbot-delivered ACT nor psychoeducation via a booklet (i.e., non-active control);

234 group assignment was the between-subjects variable. Dependent variables included

235 quantitative self-report measures of generalized pliance, tracking, affect, psychological

236 inflexibility (avoidance), and valued living that participants were invited to complete at three

237 time-points, namely immediately pre-intervention, immediately post-intervention, and at a six-

238 week follow-up; time was the within-subjects variable. During the intervention period,

239 participants assigned to the chatbot condition were also prompted to respond to the single-item

240 assessments of problematic rule-following daily.

241

242 Conditions

243 Chatbot-Delivered ACT

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
11
244 After completing the pre-intervention self-report measures, participants in this

245 condition were invited to complete five daily chatbot-delivered ACT sessions. The average

246 duration of these sessions was 13-minutes (i.e., spanning five days, the intervention lasted

247 approximately one hour in total). The chatbot was built into Facebook Messenger (an instant

248 messaging application) and thus did not require a new, additional application to be installed on

249 participants’ devices. In addition, the chatbot could be accessed from a computer, tablet, or

250 mobile phone (i.e., the chatbot was relatively easily accessible). Participants engaged with the

251 chatbot in their own time via response “buttons” and typed responses (see Figure 2). This

252 button-typed response chatbot interface was piloted with five members of the [REMOVED

253 FOR MASKED REVIEW] to optimize user experience and streamline the content. We

254 implemented feedback received from this group before the present randomized trial began (e.g.,

255 based on feedback, we amended the chatbot to allow participants to select a new time for

256 tomorrow’s session at the end of each day, etc.).

257 The chatbot was designed by the lead author, [REMOVED FOR MASKED REVIEW],

258 who has extensive experience delivering ACT interventions and has completed an 18-month

259 advanced training in ACT and mindfulness-based interventions. The chatbot was designed to

260 appear interactive. For example, the chatbot would call participants by their chosen names,

261 restate/refer to previous responses submitted by the participant, etc. The chatbot was also

262 designed to initiate sessions with participants at a time selected by them (i.e., the chatbot invited

263 participants to complete each session every day for five days; “Hey [participant name], is now

264 a good time to chat?”). Finally, the chatbot was designed to reengage participants that did not

265 complete a session. For example, if a participant completed 50% of session two, after one hour

266 of no response from the participant, the chatbot would gently prompt the participant (e.g., “I’m

267 here if you would still like to chat”, presented alongside a response button to continue the

268 conversation).
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
12
269 Session content was partially adapted from Hayes and Strosahl (2005), McHugh et al.

270 (2019), and Villatte et al. (2015). Exercises were initially piloted with a group of adolescents

271 aged 15 to 16 years (n = 7) and subsequently tailored to participants’ preferences and

272 developmental stage. In line with existing chatbots targeting mental health (e.g., Fitzpatrick et

273 al.’s (2017) Woebot; Inkster et al.’s (2018) Wysa; Lavelle et al.’s (2022) Littlebot), content

274 was presented in an informal, conversational style. Based on feedback from our pilots, the

275 chatbot used a mixture of images, text, and emojis to convey session content. Although the

276 chatbot integrated participants’ unique responses into session content (e.g., if participants

277 stated a rule on day two, the chatbot would restate this rule on day three, inviting participants

278 to assess its utility), content was entirely manualized (i.e., the chatbot progressed through

279 predetermined paths of the script based on participants’ responses; see Figure 2). Once the

280 present trial began, no changes were made to the chatbot script.

281 Content targeted each of the six components of the psychological flexibility model.

282 Over the course of five brief sessions, the chatbot guided participants in noticing, naming, and

283 assessing the workability of rules they hold in different domains of their lives (e.g., rules around

284 interpersonal relationships; “if I want good relationships with other people then I should…”).

285 Across sessions, the chatbot defined workability/utility in terms of whether an action brought

286 the participant closer toward or further away from who and where they want to be (i.e., their

287 values). It is important to note that values were never presented as superior rules to live by

288 (Brock et al., 2015). Sessions focused on helping adolescents to notice how their rules served

289 them, when their rules seemed to help them or hurt them, etc. The chatbot was designed to

290 promote context-sensitive rule-following; it was about noticing motivations underlying rule-

291 following and creating space between “self” and “rule” to facilitate flexible responding. Each

292 session included a minimum of one rapport-building prompt (e.g., a joke), one experiential
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
13
293 exercise, and one item for home practice. See Table 2 for a summary of the session content

294 (complete chatbot script available upon reasonable request).

295 At the end of each session, participants were prompted to respond to the single-item

296 assessments of problematic rule-following. After completing all five sessions, participants in

297 this condition immediately completed the post-intervention self-report measures. Participants

298 were invited to complete these measures again at a six-week follow-up.

299 Psychoeducation via a Booklet

300 After completing the pre-intervention self-report measures, participants in this

301 condition received an electronic 26-page adaptation of an informational booklet targeted

302 towards young people aged 15 years and over. Participants were asked to read this booklet in

303 their own time over the next five days. The original booklet was created by the national Health

304 Service Executive (HSE, 2019). For the purposes of the present study, adaptations to this

305 booklet included adding contact details for youth mental health services in participants’

306 regions, updating key statistics, etc. The booklet was designed to help and support young

307 people by providing accessible, clear, and youth-friendly information on mental health while

308 simultaneously normalizing challenges faced by young people across a range of domains (e.g.,

309 relationships, sexuality, grief, substance misuse, etc.).

310 Five days after receiving the booklet, participants in this condition were invited to

311 complete the post-intervention self-report measures. Participants were invited to complete

312 these measures again at a six-week follow-up. Once the present trial ended (i.e., after follow-

313 up data were collected), participants assigned to this condition were offered access to the

314 chatbot.

315 Neither Chatbot-Delivered ACT nor Psychoeducation via a Booklet (i.e., Non-Active

316 Control)
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
14
317 Participants in this condition received neither the chatbot nor booklet. Participants in

318 this condition were only invited to complete the quantitative self-report measures pre-

319 intervention, five days later (i.e., post-intervention), and at a six-week follow-up. Once the

320 present trial ended (i.e., after follow-up data were collected), participants assigned to this

321 condition were offered access to the chatbot and the informational booklet.

322

323 Measures

324 Generalized Pliance

325 Generalized pliance was measured using the Generalized Pliance Questionnaire for

326 Children (GPQ-C; Salazar et al., 2018a). Participants rated each of the eight GPQ-C items on

327 a 5-point scale ranging from 1 (never true) to 5 (always true), with higher scores indicating

328 higher generalized pliance. Sample items include: “Making an effort is only worth it if others

329 recognize it” and “The opinions of others influence my decisions a lot”. Research has supported

330 the factor structure, convergent validity, criterion validity, and reliability of the GPQ-C

331 (Salazar et al., 2018a, 2018b; Stapleton & McHugh, 2020). The GPQ-C displayed good internal

332 consistency in the present study (α = .806).

333 Generalized Tracking

334 Generalized tracking was measured using the Generalized Tracking Questionnaire for

335 Children (GTQ-C; Pereira et al., 2020). Participants rated each of the 12 GTQ-C items on a 5-

336 point scale ranging from 1 (never true) to 5 (always true), with higher scores indicating higher

337 generalized tracking. Sample items include: “I make decisions based on my experience, not on

338 what other people say” and “When I realize something is not working, I try something

339 different”. Research has supported the factor structure, internal consistency, and measurement

340 invariance with regard to gender of the GTQ-C (Pereira et al., 2020). The GTQ-C displayed

341 good internal consistency in the present study (α = .825).

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
15
342 Affect

343 Affect was measured using Ebesutani et al.’s (2012) shortened Positive and Negative

344 Affect Schedule-Children (PANAS-C), a ten-item scale comprised of two five-item subscales

345 targeting positive affect (PA; joyful, cheerful, happy, lively, and proud) and negative affect

346 (NA; miserable, mad, afraid, scared, and sad). Participants rated how often they experienced

347 each mood state “over the past few weeks” on a five-point scale ranging from 1 (very slightly

348 or not at all) to 5 (extremely), with higher subscale scores indicative of higher positive/negative

349 affect. Research has supported the factor structure, internal consistency, criterion-related

350 validity, and discriminant validity of the PANAS-C (Ebesutani et al., 2012). Both the PA and

351 NA subscales displayed adequate internal consistency in the present study (α = .849, α = .717,

352 respectively).

353 Psychological Inflexibility

354 Psychological inflexibility was measured using the Avoidance and Fusion

355 Questionnaire for Youth short form (AFQ-Y8; Greco et al., 2008). Participants rated eight

356 AFQ-Y8 items on a five-point scale ranging from 0 (not at all true) to 4 (very true), with higher

357 scores indicative of higher inflexibility/rigidity. Sample items include: “I stop doing things that

358 are important to me whenever I feel bad” and “I am afraid of my feelings”. Research by Greco

359 et al. (2008) supported the reliability, convergent validity, and discriminant validity of the

360 AFQ-Y8. The AFQ-Y8 displayed good internal consistency in the present study (α = .816).

361 Valued Living

362 Personal values enactment over the past week was assessed using Smout et al.’s (2014)

363 Valuing Questionnaire (VQ), a ten-item scale comprised of two five-item subscales targeting

364 values progress (VQP; e.g., “I made progress in the areas of my life I care most about”) and

365 values obstruction (VQO; e.g., “Difficult thoughts, feelings or memories got in the way of what

366 I really wanted to do”). Participants rated each item on a seven-point scale ranging from 0 (not
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
16
367 at all true) to 6 (completely true), with higher subscale scores indicative of higher values

368 progress/obstruction. Research has supported the factor structure, internal consistency, and

369 convergent validity of the VQ (Ruiz et al., 2022; Smout et al., 2014). Both the VQP and VQO

370 subscales displayed adequate internal consistency in the present study (α = .785, α = .773,

371 respectively).

372 Participants’ Satisfaction with the Interventions (i.e., Chatbot and Booklet; Time 2 Only)

373 At Time 2 (post-intervention), participants in both active conditions were invited to

374 report their satisfaction with the intervention (i.e., the chatbot or booklet). Three items were

375 dichotomous “yes”/”no” questions: “Do you think that you learned something new from the

376 chatbot/booklet?”, “Was the chatbot/booklet relevant to your everyday life?”, and “Would you

377 use the chatbot/booklet again?”. Two additional items (“How happy were you with the

378 chatbot/booklet overall?” and “How happy were you with the information you received?”)

379 were rated on a five-point scale ranging from 1 (not at all happy) to 5 (really happy).

380 Participants were also invited to submit qualitative comments in response to the following

381 prompts: “What was the best thing about the chatbot/booklet?” and “What would you change

382 about chatbot/booklet?”.

383 High Temporal Density Assessments of Problematic Rule-Following

384 To assess the feasibility of nesting (relatively) high temporal density measures of

385 problematic rule-following (i.e., frequent ecological momentary assessments) in future

386 randomized controlled trials, single-item assessments of problematic pliance were drawn from

387 the GPQ-C. Specifically, participants were asked to rate the truthfulness and usefulness of the

388 following item “Making an effort is only worth it if others recognize it” on a five-point scale

389 ranging from 1 (not at all true/useful) to 5 (very true/useful), with higher scores indicating

390 higher dependence on arbitrary social approval and perceived ply workability. This item was

391 selected because it had high face validity and demonstrated good factor loading as per Salazar
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
17
392 et al.’s (2018a) initial validation study. Participants assigned to the chatbot condition were

393 asked to respond to each of these items at the end of each session (i.e., participants submitted

394 daily data for five days).

395

396 Procedure

397 Once [REMOVED FOR MASKED REVIEW] enrolled participants in the trial, they

398 completed Time 1 measures of generalized pliance (GPQ-C), tracking (GTQ-C), affect

399 (PANAS-C), psychological inflexibility (avoidance; AFQ-Y8), and valued living (VQ) on

400 school grounds via Qualtrics accessed via school computers. Next, participants were randomly

401 assigned to one of the three conditions (i.e., chatbot, booklet, or non-active control). Once set

402 up, participants in the active conditions were asked to engage with their assigned intervention

403 in their own time over the next five days. No blinding was involved. During this time,

404 participants assigned to the chatbot condition were prompted to respond to the single-item

405 assessments of problematic rule-following daily. Once the intervention period elapsed, all

406 participants were invited to complete Time 2 measures via a Qualtrics link they received from

407 their teachers. Time 2 measures were the same as Time 1 measures with additional intervention

408 satisfaction items added for participants in active conditions. Finally, all participants were

409 invited to complete Time 3 measures at a six-week follow-up, again via a Qualtrics link they

410 received from their teachers.

411

412 Data Analysis Strategy

413 Preliminary Analyses

414 First, we ran a series of independent t-tests to compare completers’ (i.e., participants

415 that submitted data at each of the three time-points) Time 1 data with non-completers’ (i.e.,

416 participants that did not submit data for at least one time-point) Time 1 data. In addition, we
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
18
417 used Pearson’s Chi Square test to screen for differences between completers and non-

418 completers based on gender and experimental group assignment. We also ran a series of

419 ANOVAs to test whether there were any differences in any of the dependent variables observed

420 between groups (i.e., chatbot, booklet, and non-active control) at Time 1.

421 Intervention Effectiveness

422 To examine whether group and/or time were associated with changes in any of the

423 dependent variables, we ran a series of 3x3 mixed ANOVAs. Group assignment was the

424 between-subjects variable (three levels: chatbot, booklet, and non-active control) and time was

425 the within-subjects variable (three levels: pre-intervention, post-intervention, and six-week

426 follow-up). To reduce family wise error alpha was decreased via Bonferroni correction to

427 0.0071 (0.05/7). If a statistically significant time*group interaction effect was observed, we

428 planned to use ANOVA & t-tests (wherein statistically significant effects were observed via

429 ANOVA) to examine within-group effects further. Bonferroni was the test for multiple

430 comparisons of choice and explicate between-group differences.

431

432 Results

433 Preliminary Analyses

434 Attrition and Missing Data

435 Of the 148 participants enrolled in the present pilot, 65 submitted data at Times 1, 2,

436 and 3. Of the 45 participants assigned to the chatbot, 38 completed all five sessions and

437 submitted Time 2 data. Seven chatbot participants withdrew from the study immediately

438 following a chatbot malfunction (e.g., the chatbot became stuck in a loop of messages and did

439 not allow the participant to advance). See Figure 1 for a breakdown of attrition across phases

440 of the study.

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
19
441 Due to a technical fault at Time 1, some non-completers’ data were missing at Time 1

442 for Psychological Inflexibility (missing = 4) and Valued Living Progress (missing = 2). This

443 technical fault also impacted participants that submitted data at each of the three time-points:

444 (i) two participants (booklet and non-active conditions) were missing Psychological

445 Inflexibility data but had completed responses for each other measure at each time point, and

446 (ii) one participant (chatbot condition) was missing Valued Living Obstruction data but had

447 completed responses for each other measure at each time point (i.e., 63 and 64 participants

448 were considered AFQ-Y8 and VQO completers respectively). No data were missing for pliance

449 (GPQ-C), tracking (GTQ-C), affect (PANAS-C), or valued living progress (VQP) among the

450 65 participants that submitted data at each of the three time-points (i.e., all 65 participants were

451 considered completers). Missing data were not imputed; only completers were included in

452 analyses assessing intervention effectiveness (see Table 1).

453 Comparing completers’ Time 1 data with non-completers’ Time 1 data, a series of

454 independent t-tests revealed that non-completers (M = 9.49) reported significantly less negative

455 affect (PANAS-C NA) than completers (M = 10.72), t = 1.977, p > .05. In addition, non-

456 completers (M = 12.51) reported significantly less valued living obstruction (VQO) than

457 completers (M = 15.09), t = 2.659, p > .05 (see Table 1).

458 Completion by Experimental Group and Gender

459 Pearson’s Chi-Squared tests were used to examine differences in completion/non-

460 completion across (i) groups and (ii) genders. The former of these tests revealed a statistically

461 significant difference in completion rates across groups, X2 = 17.633, p < .001; completion

462 rates were higher among participants assigned to the chatbot (61%) than the booklet (39%) or

463 non-active control (28%). The latter of these tests revealed a statistically significant difference

464 in completion rates across genders, X2 = 64.177, p < .001; completion rates were higher among

465 females (87%) than males (19%).

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
20
466 Time 1 Differences in Dependent Variables Across Groups

467 No statistically significant differences in any of the dependent variables were observed

468 between groups at Time 1 (see Table 3).

469

470 Intervention Effectiveness

471 See Table 4 and Table 5 for descriptive statistics of dependent variables by condition

472 and time. A series of 3x3 mixed ANOVAs were run to examine whether group and/or time

473 were associated with changes in any of the dependent variables.

474 Generalized Pliance

475 For GPQ-C scores, no significant group*time interaction effect was observed, Λ = .985,

476 F(4, 124) = .233, p = .92, ηp2 = .008. The main effect for time was not significant, Λ = .907,

477 F(2, 61) = 3.136, p = .051, ηp2 = .093; nor was the main effect for group, F(2,62) = 1.537, p =

478 .223, ηp2 = .047.

479 Generalized Tracking

480 For GTQ-C scores, no significant group*time interaction effect was observed, Λ = .966,

481 F(4, 122) = .539, p = .707, ηp2 = .017. The main effect for time was not significant, Λ = .974,

482 F(2, 61) = .801, p = .454, ηp2 = .026; nor was the main effect for group, F(2,62) =.386, p =

483 .682, ηp2 = .012.

484 Positive Affect

485 For PANAS-C PA scores, no significant group*time interaction effect was observed,

486 Λ = .971, F(4, 122) = .455, p = .768, ηp2 = .015. The main effect for time was not significant,

487 Λ = .922, F(2, 61) = .259, p = .772, ηp2 = .008; nor was the main effect for group, F(2,62) =

488 2.033, p = .140, ηp2 = .062.

489 Negative Affect

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
21
490 For PANAS-C NA scores, no significant group*time interaction effect was observed,

491 Λ = .980, F(4, 122) = .314, p = .868, ηp2 = .010. The main effect for time was not significant,

492 Λ = .973, F(2, 61) = .833, p = .440, ηp2 = .027; nor was the main effect for group, F(2,62) =

493 .308, p = .736, ηp2 = .010.

494 Psychological Inflexibility

495 For AFQ-Y8 scores, no significant group*time interaction effect was observed, Λ =

496 .923, F(4, 118) = 1.208, p = .311, ηp2 = .039. The main effect for time was not significant, Λ

497 = .920, F(2, 59) = 2.569, p = .085, ηp2 = .08; nor was the main effect for group, F(2,60) =

498 1.079, p = .346, ηp2 = .035.

499 Valued Living Progress

500 For VQP scores, no significant group*time interaction effect was observed, Λ = .944,

501 F(4, 122) = .895, p = .469, ηp2 = .029. The main effect for time was not significant, Λ = .964,

502 F(2, 61) = 1.132, p = .329, ηp2 = .036; nor was the main effect for group, F(2,62) =.740, p =

503 .481, ηp2 = .023.

504 Valued Living Obstruction

505 For VQO scores, no significant group*time interaction effect was observed, Λ = .906,

506 F(4, 120) = 1.514, p = .92, ηp2 = .202. The main effect for time was not significant, Λ = .989,

507 F(2, 60) = .346, p = .709, ηp2 = .011; nor was the main effect for group, F(2,61) = 1.011, p =

508 .370, ηp2 = .032.

509

510 Participants’ Satisfaction with the Interventions

511 One chatbot participant only submitted satisfaction data at Time 2 (i.e., there are 39,

512 not 38 participant responses for chatbot satisfaction). Most participants thought that they had

513 learned something new from the chatbot (~ 80%), agreed that the chatbot was relevant to their

514 everyday life (~ 70%), and would use the chatbot again (~ 75%). Of participants that agreed
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
22
515 they learned something new, most seemed to have achieved the intervention learning objectives

516 (e.g., “I’ve realised that rules don’t always have to be followed and its only situational”, “It

517 makes me realise that rules aren't so black and white”, and “Even though it’s important to abide

518 by your rules most of the time, sticking to them absolutely can be problematic. There is no one

519 rule for life, and you have to look at each situation differently”). Most participants were happy

520 (22; really happy = 8; neutral = 8; a little unhappy = 1) with the chatbot overall. Most

521 participants were happy (22; really happy = 5; neutral = 11; a little unhappy = 1) with the

522 information they received from the chatbot.

523 In terms of the qualitative satisfaction data, participants liked that the chatbot initiated

524 conversations, responded promptly, told jokes, asked their opinion, and prompted reflection

525 (e.g., “It was nice to have time in a busy day to sit down and talk about my feelings”).

526 Participants also commented on the chatbot’s relevance, comprehensibility, and ease of use.

527 Participants requested longer sessions, more response button options, and additional prompts

528 to clarify rules. Participants also commented on technical issues evident with the chatbot (e.g.,

529 the chatbot becoming stuck in a loop; the chatbot having a limited understanding of slang).

530 Notably, some participants responded to the chatbot in ways that suggest a degree of

531 anthropomorphism (e.g., “Hi! Sorry I'm late :((” and “he was like a friend”).

532 No participants that received the booklet responded to the quantitative or qualitative

533 satisfaction questions.

534

535 High Temporal Density Assessments of Problematic Rule-Following

536 Due to a chatbot malfunction, 12 single-item data points were missing (see Table 6 for

537 the raw data). Of GPQ-C completers assigned to the chatbot (n = 31), relative to Time 1, 18

538 reported descriptively lower GPQ-C scores at Time 2 (i.e., lesser pliance; though not overall

539 statistically significantly lower as per our intervention effectiveness analyses). From a visual
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
23
540 inspection, looking at the raw ecological momentary assessments for participants reporting

541 lesser pliance at Time 2, truthfulness and usefulness ratings appear to trend downward (i.e.,

542 seem to decrease across the five days of the intervention).

543 These data must be interpreted with extreme caution because (i) we did not include a

544 baseline (i.e., ecological momentary assessments were only completed over the course of the

545 chatbot intervention), and (ii) we are relying on visual inspection alone to interpret data for

546 participants that reported decreases in GPQ-C scores at Time 2. With these caveats in mind,

547 while there were no overall significant changes in GPQ-C scores, these single-item data

548 tentatively suggest that there may be minute changes happening at the individual level that

549 single-items are more sensitive to.

550

551 Discussion

552 The present three-arm parallel-group pilot RCT aimed to (i) evaluate universal chatbot-

553 delivered ACT for youth mental health (i.e., intervention effectiveness and participant

554 satisfaction) and (ii) assess the feasibility of nesting ecological momentary assessments of

555 problematic rule-following in the chatbot (i.e., degree of missing data and visual inspections

556 of apparent trends). To these ends, participants were randomly assigned to receive either

557 chatbot-delivered ACT, psychoeducation via a booklet, or neither of these (non-active control).

558 Participants completed self-report measures of generalized pliance, tracking, affect,

559 psychological inflexibility, and valued living at three time-points (immediately pre-

560 intervention, immediately post-intervention, and at a six-week follow-up), with participants

561 assigned to the chatbot further completing daily single-item assessments of problematic

562 pliance.

563 Addressing study aim one, we observed no statistically significant changes in any

564 dependent variable either across conditions or over time (i.e., we failed to support our
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
24
565 hypothesis that participants receiving chatbot-delivered ACT would report greater beneficial

566 shifts in problematic rule-following, affect, psychological inflexibility, and valued living,

567 relative to those receiving psychoeducation). Thus, according to self-report standardized

568 measures, the intervention was not effective. Turning to participant satisfaction, of participants

569 assigned to the chatbot, most agreed that the chatbot taught them something new, that the

570 content was relevant to their everyday life, and that they would use the chatbot again. Looking

571 at the qualitative data, participant responses suggested that they both achieved the intervention

572 learning objectives and enjoyed various features of the chatbot. Several participant responses

573 indicated high-level engagement with the chatbot (i.e., that the chatbot could effectively

574 simulate human-to-human interaction, prompt reflection, etc.), in addition to a degree of

575 anthropomorphism. Thus, we found chatbot-delivered ACT had adequate acceptability,

576 although our chatbot’s content may need to be further revised to occasion significant shifts in

577 the dependent variables.

578 Addressing study aim two, looking at the raw high temporal density assessments of

579 problematic rule-following, there does seem to be a degree of convergence between

580 participants’ responses to the standardized measure (GPQ-C) and ecological momentary

581 assessments. More specifically, for participants reporting lesser pliance at Time 2, across the

582 intervention period, ply truthfulness and usefulness ratings appeared to trend downward as per

583 visual inspections of the data. It is important to note that these data must be interpreted with

584 extreme caution due to (i) a lack of baseline and (ii) interpretation based on visual inspection

585 alone. However, with these caveats in mind, although there were no overall significant changes

586 in GPQ-C scores, these single-item data tentatively suggest that there may be minute changes

587 happening at the individual level that single-items are more sensitive to. Simply put, while we

588 did not detect overall significant group-level decreases in problematic pliance, our ecological

589 momentary assessments suggest that changes may have been happening at the individual level.
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
25
590 Given both the apparent convergence with the GPQ-C data and minimal missing data, nesting

591 ecological momentary assessments within the chatbot in a larger RCT was deemed highly

592 feasible.

593 Our findings somewhat align with those of Puolakanaho et al.’s (2019) three-arm RCT

594 wherein intention-to-treat analyses revealed no significant between-group (two types of

595 internet-delivered ACT and non-active control) differences in students’ (aged 16 to 17 years)

596 stress (i.e., affect) and academic buoyancy (i.e., ‘positive’ mental health). Our findings also

597 somewhat align with those of Levin et al.’s (2014) RCT wherein universal internet-delivered

598 ACT did not yield significant changes in depression or anxiety (i.e., affect) among non-

599 distressed undergraduate students. However, our findings also deviate from those of Levin et

600 al. (2014), since unlike them, we observed no significant improvements in psychological

601 flexibility post-intervention. Finally, our findings regarding the effects of ACT on pliance and

602 tracking (as per the standardized measures) depart from extant research evidencing that ACT

603 reduces problematic rule-following (Ruiz et al., 2020b; Salazar et al., 2020; Stapleton et al.,

604 2022b).

605 Recognizing that there may have been minute intervention-related shifts in our

606 dependent variables that were lost to group-level analyses (as suggested by the ecological

607 momentary assessment data), we will now explore limitations of the present pilot, in addition

608 to possible reasons chatbot-delivered ACT did not occasion predicted changes. First, the

609 intervention period was relatively short (i.e., spanning five days, the present intervention lasted

610 approximately one hour in total). Although brief interventions are often preferred because they

611 can maximize the efficiency of services (Kroska et al., 2020), our findings suggest

612 approximately one hour of chatbot-delivered ACT is insufficient. Future iterations of the

613 present chatbot should extend the intervention duration, perhaps to 90-minutes since Kroska et

614 al. (2020) found that 90-minutes of group-based ACT was as effective as 3-hour and 6-hour
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
26
615 group sessions in reducing adults’ depressive symptomatology. Future iterations may also

616 increase the number of days included in the intervention period to allow chatbot users more

617 time to engage in reflective practice between sessions.

618 Second, there were some technical issues with the chatbot that may have undermined

619 its speaker credibility (Stapleton, 2020). For example, some participants experienced the

620 chatbot becoming stuck in a loop of messages, rigidity in the chatbot’s ability to

621 understand/respond appropriately (e.g., “Hi, You can call me Orla. I’m you’re new chatbot

622 buddy!” as opposed to “Hi, Orla”), or the chatbot having a limited understanding of

623 colloquialisms/slang. Each of these factors was identified by Gaffney et al. (2019) in a

624 systematic review of factors impacting the effectiveness of chatbot interventions. Chatbot

625 flexibility/intelligence is an important usability issue (Lavelle et al., 2022) that may have

626 inhibited participants’ rapport and working alliance with the present chatbot (Bendig et al.,

627 2019; Morris et al., 2018; Prochaska et al., 2021). Future iterations of the present chatbot could

628 further develop the underlying artificial intelligence and natural language processing

629 technologies (e.g., training using speech samples from young people, with slang and typing

630 mannerisms included) to reduce the likelihood of these issues occurring.

631 Notwithstanding these limitations, there are a number of strengths of the present study;

632 the present study makes an important contribution to the literature. First, the present pilot is the

633 first RCT testing chatbot-delivered ACT for young people (as per Zhou et al.’s 2021 review).

634 Second, the present study is one of few to explicitly target problematic rule-following (Ruiz et

635 al., 2020b; Salazar et al., 2020; Stapleton et al., 2022b), and is the only study to measure both

636 pliance and tracking. Third, we collected both quantitative and qualitative intervention

637 satisfaction data, the latter of which is often overlooked in acceptability trials (Lavelle et al.,

638 2022). Fourth, we assessed the feasibility of relatively high temporal density assessments of

639 problematic rule-following (i.e., daily ecological momentary assessments), representing an

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
27
640 important step towards retaining degrees of nuance in individuals’ responses to interventions

641 even within RCTs (McDonald et al., 2017).

642 Our findings may be used to inform future chatbot research in myriad ways. First, we

643 recommend future chatbot-delivered ACT interventions last longer than one hour in duration,

644 perhaps also spanning longer than five days. Second, we recommend training any artificial

645 intelligence and natural language processing technologies underlying the chatbot with speech

646 samples from the target population (including slang and typing mannerisms). Third, we

647 recommend nesting high temporal density measures of dependent variables in the chatbot, since

648 we found this was both feasible and potentially sensitive to intervention-related shifts that may

649 have been lost to group-level analyses. Future research is needed to further explore the present

650 ecological momentary assessments of pliance (e.g., are responses to this item stable at baseline,

651 does this item converge with standardized measures in other samples, etc.). Finally, although

652 determining whether interventions are effective is worthwhile, future research must work

653 towards identifying the key mechanisms/processes of change underlying ACT and chatbot-

654 delivered interventions to better facilitate intervention optimization and effectively

655 promote/protect youth mental health.

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
28
656 References

657 Arango, C., Díaz-Caneja, C. M., McGorry, P. D., Rapoport, J., Sommer, I. E., Vorstman, J.

658 A., McDaid, D., Marín, O., Serrano-Drozdowskyj, E., Freedman, R., & Carpenter, W.

659 (2018). Preventive strategies for mental health. The Lancet Psychiatry, 5(7), 591-604.

660 https://doi.org/10.1016/s2215-0366(18)30057-9

661 Assaz, D. A., Tyndall, I., Oshiro, C. K., & Roche, B. (2022). A process-based analysis of

662 cognitive defusion in acceptance and commitment therapy. Behavior Therapy.

663 https://doi.org/10.1016/j.beth.2022.06.003

664 Barnes-Holmes, D., O'Hora, D., Roche, B., Hayes, S. C., Bisset, R. T., & Lyddy, F. (2001).

665 Understanding and verbal regulation. In S.C. Hayes, D. Barnes-Holmes, & B.T.

666 Roche (Eds.) Relational frame theory: A post-Skinnerian account of human language

667 and cognition (pp. 103-119). Plenum.

668 Barnes-Holmes, Y., Hussey, I., McEnteggart, C., Barnes-Holmes, D., & Foody, M. (2016).

669 Scientific ambition: The relationship between relational frame theory and middle-

670 level terms in acceptance and commitment therapy. In R. D. Zettle, S. C. Hayes, D.

671 Barnes-Holmes, & A. Biglan (Eds.), The wiley handbook of contextual behavioral

672 science (pp. 452-473). John Wiley & Sons.

673 Bendig, E., Erb, B., Schulze-Thuesing, L., & Baumeister, H. (2019). The next generation:

674 Chatbots in clinical psychology and psychotherapy to foster mental health: A scoping

675 review. Verhaltenstherapie, 1–13. https://doi.org/10.1159/000501812

676 Biglan, A., & Glenn, S. S. (2013). Toward prosocial behavior and environments: Behavioral

677 and cultural contingencies in a public health framework. In G. J. Madden, W. V.

678 Dube, T. D. Hackenberg, G. P. Hanley, & K. A. Lattal (Eds.), APA handbook of

679 behavior analysis, Vol. 2. Translating principles into practice (pp. 255–275).

680 American Psychological Association. https://doi.org/10.1037/13938-010

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
29
681 Bird, T., Mansell, W., Wright, J., Gaffney, H., & Tai, S. (2018). Manage your life online: A

682 web-based randomized controlled trial evaluating the effectiveness of a problem-

683 solving intervention in a student sample. Behavioural and Cognitive Psychotherapy,

684 46(5), 570-582. https://doi.org/10.1017/S1352465817000820

685 Blakemore, S. J., & Choudhury, S. (2006). Development of the adolescent brain: Implications

686 for executive function and social cognition. Journal of Child Psychology and

687 Psychiatry 47(3/4), 296–312. https://doi.org/10.1111/j.1469-7610.2006.01611.x

688 Brock, M. J., Batten, S. V., Walser, R. D., & Robb, H. B. (2015). Recognizing common

689 clinical mistakes in ACT: A quick analysis and call to awareness. Journal of

690 Contextual Behavioral Science, 4(3), 139-143.

691 https://doi.org/10.1016/j.jcbs.2014.11.003

692 Burckhardt, R., Manicavasagar, V., Batterham, P. J., Hadzi-Pavlovic, D., & Shand, F. (2017).

693 Acceptance and commitment therapy universal prevention program for adolescents: A

694 feasibility study. Child and Adolescent Psychiatry and Mental Health, 11(1), 27.

695 https://doi.org/10.1186/s13034-017-0164-5

696 Crone, E. A., & Dahl, R. E. (2012). Understanding adolescence as a period of social-affective

697 engagement and goal flexibility. Nature Review Neuroscience, 13(1), 636–650.

698 https://doi.org/10.1038/nrn3313

699 Dray, J., Bowman, J., Campbell, E., Freund, M., Wolfenden, L., Hodder, R. K., McElwaine,

700 K., Tremain, D., Bartlem, K., Bailey, J., Small, T., Palazzi, K., Oldmeadow, C., &

701 Wiggers, J. (2017). Systematic review of universal resiliencefocused interventions

702 targeting child and adolescent mental health in the school setting. Journal of the

703 American Academy of Child & Adolescent Psychiatry, 56(10), 813-824.

704 https://doi.org/10.1016/j.jaac.2017.07.780
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
30
705 Ebesutani, C., Regan, J., Smith, A., Reise, S., Higa-McMillan, C., & Chorpita, B. F. (2012).

706 The 10-item positive and negative affect schedule for children, child and parent

707 shortened versions: Application of item response theory for more efficient

708 assessment. Journal of Psychopathology and Behavioral Assessment, 34(2), 191–203.

709 https://doi.org/10.1007/s10862-011-9273-2

710 Edwards‐Hart, T., & Chester, A. (2010). Online mental health resources for adolescents:

711 Overview of research and theory. Australian Psychologist, 45(3), 223-230.

712 https://doi.org/10.1080/00050060903584954

713 Eustis, E. H., Hayes-Skelton, S. A., Orsillo, S. M., & Roemer, L. (2018). Surviving and

714 thriving during stress: A randomized clinical trial comparing a brief web-based

715 therapist-assisted acceptance-based behavioral intervention versus waitlist control for

716 college students. Behavior Therapy, 49(6), 889-903.

717 https://doi.org/10.1016/j.beth.2018.05.009

718 Eyre, O., & Thapar, A. (2014). Common adolescent mental disorders: transition to adulthood.

719 The Lancet, 383(9926), 1366-1368. https://doi.org/10.1016/S0140-6736(13)62633-1

720 Fang, S., & Ding, D. (2020). A meta-analysis of the efficacy of acceptance and commitment

721 therapy for children. Journal of Contextual Behavioral Science, 15(1), 225-234.

722 https://doi.org/10.1016/j.jcbs.2020.01.007

723 Feeley, N., Cossette, S., Côté, J., Héon, M., Stremler, R., Martorella, G., & Purden, M.

724 (2009). The importance of piloting an RCT intervention. Canadian Journal of

725 Nursing Research Archive, 41(2), 85-99.

726 Fitzpatrick, K., Darcy, A., & Vierhile, M. (2017). Delivering cognitive behavior therapy to

727 young adults with symptoms of depression and anxiety using a fully automated

728 conversational agent (Woebot): A randomized controlled trial. Journal of Medical

729 Internet Research Mental Health, 4(2), e19. https://doi.org/10.2196/mental.7785

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
31
730 Fulmer, R., Joerin, A., Gentile, B., Lakerink, L., & Rauws, M. (2018). Using psychological

731 artificial intelligence (Tess) to relieve symptoms of depression and anxiety:

732 Randomized controlled trial. JMIR Mental Health, 5(4), e9782.

733 https://doi.org/10.2196/mental.9782

734 Fusar‐Poli, P., Correll, C. U., Arango, C., Berk, M., Patel, V., & Ioannidis, J. P. (2021).

735 Preventive psychiatry: A blueprint for improving the mental health of young people.

736 World Psychiatry, 20(2), 200-221. https://doi.org/10.1002/wps.20869

737 Gaffney, H., Mansell, W., & Tai, S. (2019). Conversational agents in the treatment of mental

738 health problems: Mixed-method systematic review. Journal of Medical Internet

739 Research Mental Health, 6(10), e14166. https://doi.org/10.2196/14166

740 Gore, F. M., Bloem, P. J., Patton, G. C., Ferguson, J., Joseph, V., Coffey, C., Sawyer, S. M.,

741 & Mathers, C. D. (2011). Global burden of disease in young people aged 10–24 years:

742 A systematic analysis. The Lancet, 377(9783), 2093-2102.

743 https://doi.org/10.1016/s0140-6736(11)60512-6

744 Greco, L. A., Lambert, W., & Baer, R. A. (2008). Psychological inflexibility in childhood

745 and adolescence: Development and evaluation of the avoidance and fusion

746 questionnaire for youth. Psychological Assessment, 20(2), 93-102.

747 https://doi.org/10.1037/1040-3590.20.2.93

748 Grové, C. (2021). Co-developing a mental health and wellbeing chatbot with and for young

749 people. Frontiers in Psychiatry, 11(1), 606041.

750 https://doi.org/10.3389/fpsyt.2020.606041

751 Guerrini Usubini, A., Cattivelli, R., Radaelli, A., Bottacchi, M., Landi, G., Tossani, E.,

752 Grandi, S., Castelnuovo, G., & Sartorio, A. (2022). Preliminary results from the

753 ACTyourCHANGE in teens protocol: A randomized controlled trial evaluating

754 acceptance and commitment therapy for adolescents with obesity. International
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
32
755 Journal of Environmental Research and Public Health, 19(9), 5635.

756 https://doi.org/10.3390/ijerph19095635

757 Hayes, S. C., Gifford, E. V., & Hayes, G. J. (1998). Moral behavior and the development of

758 verbal regulation. The Behavior Analyst, 21(2), 253-279.

759 https://doi.org/10.1007/BF03391967

760 Hayes, S. C., Hofmann, S. G., & Ciarrochi, J. (2020). A process-based approach to

761 psychological diagnosis and treatment: The conceptual and treatment utility of an

762 extended evolutionary meta model. Clinical Psychology Review, 82(1), 101908.

763 https://doi.org/10.1016/j.cpr.2020.101908

764 Hayes, S. C., Luoma, J. B., Bond, F. W., Masuda, A., & Lillis, J. (2006). Acceptance and

765 commitment therapy: Model, processes and outcomes. Behaviour Research and

766 Therapy, 44(1), 1-25. https://doi.org/10.1016/j.brat.2005.06.006

767 Hayes, S. C., Merwin, R. M., McHugh, L., Sandoz, E. K., A-Tjak, J. G., Ruiz, F. J., Barnes-

768 Holmes, D., Bricker, J. B., Ciarrochi, J., Dixon, M. R., Fung, K., Gloster, A. T.,

769 Gobin, R. L., Gould, E. R., Hofmann, S. G., Kasujja, R., Karekla, M., Luciano, C., &

770 McCracken, L. M. (2021). Report of the ACBS Task Force on the strategies and

771 tactics of contextual behavioral science research. Journal of Contextual Behavioral

772 Science, 20(1), 172-183. https://doi.org/10.1016/j.jcbs.2021.03.007

773 Hayes, S. C., & Strosahl, K. D. (2005). A practical guide to acceptance and commitment

774 therapy. Springer Science+Business Media.

775 Hayes, S. C., Strosahl, K. D., & Wilson, K. G. (2012). Acceptance and commitment therapy:

776 The process and practice of mindful change (2nd ed.). Guildford Press.

777 Health Service Executive. (2019). Youth mental health resource booklet. The Cork Life

778 Centre. https://www.hse.ie/eng/services/list/4/mental-health-services/connecting-for-

779 life/news/cork-youth-mental-health-resource-booklet.pdf
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
33
780 Inkster, B., Sarda, S., & Subramanian, V. (2018). An empathy-driven, conversational

781 artificial intelligence agent (Wysa) for digital mental well-being: Real-world data

782 evaluation mixed-methods study. Journal of Medical Internet Research mHealth

783 uHealth, 6(11), e12106. https://doi.org/10.2196/12106

784 Keenan-Miller, D., Hammen, C. L., & Brennan, P. A. (2007). Health outcomes related to

785 early adolescent depression. Journal of Adolescent Health, 41(3), 256–262.

786 https://doi.org/10.1016%2Fj.jadohealth.2007.03.015

787 Kroska, E. B., Roche, A. I., & O'Hara, M. W. (2020). How much is enough in brief

788 Acceptance and Commitment Therapy? A randomized trial. Journal of Contextual

789 Behavioral Science, 15(1), 235-244. https://doi.org/10.1016/j.jcbs.2020.01.009

790 Lappalainen, R., Lappalainen, P., Puolakanaho, A., Hirvonen, R., Eklund, K., Ahonen, T.,

791 Muotka, J., & Kiuru, N. (2021). The Youth Compass-the effectiveness of an online

792 acceptance and commitment therapy program to promote adolescent mental health: A

793 randomized controlled trial. Journal of Contextual Behavioral Science, 20(1), 1-12.

794 https://doi.org/10.1016/j.jcbs.2021.01.007

795 Lavelle, J., Dunne, N., Mulcahy, H. E., & McHugh, L. (2022). Chatbot-delivered cognitive

796 defusion versus cognitive restructuring for negative self-referential thoughts: A pilot

797 study. The Psychological Record, 72(2), 247-261. https://doi.org/10.1007/s40732-

798 021-00478-7

799 Levin, M. E., Hildebrandt, M. J., Lillis, J., & Hayes, S. C. (2012). The impact of treatment

800 components suggested by the psychological flexibility model: A meta-analysis of

801 laboratory-based component studies. Behavior Therapy, 43(4), 741-756.

802 https://doi.org/10.1016/j.beth.2012.05.003

803 Levin, M. E., Krafft, J., Hicks, E. T., Pierce, B., & Twohig, M. P. (2020). A randomized

804 dismantling trial of the open and engaged components of acceptance and commitment
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
34
805 therapy in an online intervention for distressed college students. Behaviour Research

806 and Therapy, 126(1), 103557. https://doi.org/10.1016/j.brat.2020.103557

807 Levin, M. E., Pistorello, J., Seeley, J. R., & Hayes, S. C. (2014). Feasibility of a prototype

808 web-based acceptance and commitment therapy prevention program for college

809 students. Journal of American College Health, 62(1), 20-30.

810 https://doi.org/10.1080/07448481.2013.843533

811 Luciano, C., Valdivia-Salas, S., & Ruiz, F. J. (2012). The self as the context for rule-

812 governed behavior. In L. McHugh, & I. Stewart (Eds.). The self and perspective

813 taking: Research and applications (pp. 143–160). Context Press.

814 McCracken, L. M., & Morley, S. (2014). The psychological flexibility model: A basis for

815 integration and progress in psychological approaches to chronic pain management.

816 The Journal of Pain, 15(3), 221-234. https://doi.org/10.1016/j.jpain.2013.10.014

817 McDonald, S., Quinn, F., Vieira, R., O’Brien, N., White, M., Johnston, D. W., & Sniehotta,

818 F. F. (2017). The state of the art and future opportunities for using longitudinal n-of-1

819 methods in health behaviour research: a systematic literature overview. Health

820 Psychology Review, 11(4), 307-323. https://doi.org/10.1080/17437199.2017.1316672

821 McHugh, L., Stewart, I., & Almada, P. (2019). A contextual behavioral guide to the self:

822 Theory and practice. New Harbinger Publications.

823 Morris, R. R., Kouddous, K., Kshirsagar, R., & Schueller, S. M. (2018). Towards an

824 artificially empathic conversational agent for mental health applications: System

825 design and user perceptions. Journal of Medical Internet Research, 20(6), e10148.

826 https://doi.org/10.2196/10148

827 O’Connor, M., Byrne, P., Ruiz, F. J., & McHugh, L. (2019). Generalized pliance in relation

828 to contingency insensitivity and mindfulness. Mindfulness, 10(5), 833–840.

829 https://doi.org/10.1007/s12671-018-1046-5
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
35
830 O’Connor, M., O'Reilly, G., Murphy, E., Connaughton, L., Hoctor, E., & McHugh, L. (2022).

831 Universal process-based CBT for positive mental health in early adolescence: A

832 cluster randomized controlled trial. Behaviour Research and Therapy, 154(1),

833 104120. https://doi.org/10.1016/j.brat.2022.104120

834 Patel, V., Flisher, A. J., Hetrick, S., & McGorry, P. (2007). Mental health of young people: A

835 global public-health challenge. The Lancet, 369(9569), 1302-1313.

836 https://doi.org/10.1016/s0140-6736(07)60368-7

837 Patton, G. C., Sawyer, S. M., Santelli, J. S., Ross, D. A., Afifi, R., Allen, N. B., Arora, M.,

838 Azzopardi, P., Baldwin, W., Bonell, C., Kakuma, R., Kennedy, E., Mahon, J.,

839 McGovern, T., Mokdad, A., Patel, V., Petroni, S., Reavley, N., Taiwo, K., ... Viner,

840 R. M. (2016). Our future: A Lancet commission on adolescent health and wellbeing.

841 The Lancet, 387(10036), 2423-2478. https://doi.org/10.1016/S0140-6736(16)00579-1

842 Pereira, C., Ayala Ávila, J. M., García Martín, M. B., Suárez Falcón, J. C., & Ruiz, F. J.

843 (2020). Psychometric properties and measurement invariance of the generalized

844 tracking questionnaire-children. International Journal of Psychology and

845 Psychological Therapy, 20(2), 177-188.

846 Prochaska, J. J., Vogel, E. A., Chieng, A., Kendra, M., Baiocchi, M., Pajarito, S., &

847 Robinson, A. A. (2021). Therapeutic relational agent for reducing problematic

848 substance use (woebot): Development and usability study. Journal of Medical

849 Internet Research, 23(3), e24850. https://doi.org/10.2196/24850

850 Puolakanaho, A., Lappalainen, R., Lappalainen, P., Muotka, J. S., Hirvonen, R., Eklund, K.

851 M., Ahonen, T. P. S., & Kiuru, N. (2019). Reducing stress and enhancing academic

852 buoyancy among adolescents using a brief web-based program based on acceptance

853 and commitment therapy: A randomized controlled trial. Journal of Youth and

854 Adolescence, 48(2), 287-305. https://doi.org/10.1007/s10964-018-0973-8

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
36
855 Rao, U., Weissman, M. M., Martin, J. A., & Hammond, R. W. (1993). Childhood depression

856 and risk of suicide: A preliminary report of a longitudinal study. Journal of the

857 American Academy of Child & Adolescent Psychiatry, 32(1), 21–27.

858 https://doi.org/10.1097/00004583-199301000-00004

859 Roza, S. J., Hofstra, M. B., Van Der Ende, J., & Verhulst, F. C. (2003). Stable prediction of

860 mood and anxiety disorders based on behavioral and emotional problems in

861 childhood: A 14-year follow-up during childhood, adolescence, and young adulthood.

862 American Journal of Psychiatry, 160(12), 2116-2121.

863 https://doi.org/10.1176/appi.ajp.160.12.2116

864 Ruiz, F. J., García-Martín, M. B., Suárez-Falcón, J. C., Bedoya-Valderrama, L., Segura-

865 Vargas, M. A., Peña-Vargas, A., Henao, Á. M., & Ávila-Campos, J. E. (2020a).

866 Development and initial validation of the Generalized Tracking Questionnaire. PloS

867 One, 15(6), e0234393. https://doi.org/10.1371/journal.pone.0234393

868 Ruiz, F. J., Odriozola-González, P., Suárez-Falcón, J. C., & Segura-Vargas, M. A. (2022).

869 Psychometric properties of the valuing questionnaire in a Spaniard sample and

870 factorial equivalence with a Colombian sample. PeerJ, 10(1), e12670.

871 https://doi.org/10.7717%2Fpeerj.12670

872 Ruiz, F. J., Peña-Vargas, A., Ramírez, E. S., Suárez-Falcón, J. C., García-Martín, M. B.,

873 García-Beltrán, D. M., Henao, Á. M., Monroy-Cifuentes, A., & Sánchez, P. D.

874 (2020b). Efficacy of a two-session repetitive negative thinking-focused acceptance

875 and commitment therapy (ACT) protocol for depression and generalized anxiety

876 disorder: A randomized waitlist control trial. Psychotherapy, 57(3), 444–456.

877 https://doi.org/10.1037/pst0000273
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
37
878 Ruiz, F. J., Suárez-Falcón, J. C., Barbero-Rubio, A., & Flórez, C. L. (2019). Development

879 and initial validation of the generalized pliance questionnaire. Journal of Contextual

880 Behavioral Science, 12(1), 189-198. https://doi.org/10.1016/j.jcbs.2018.03.003

881 Salazar, D. M., Ruiz, F. J., Flórez, C. L., & Suárez Falcón, J. C. (2018a). Psychometric

882 properties of the generalized pliance questionnaire-children. International Journal of

883 Psychology and Psychological Therapy, 18(3), 273-287

884 Salazar, D. M., Ruiz, F. J., & García Martín, M. B. (2018b). Generalized pliance and

885 insensitivity to contingencies in children with learning difficulties [Unpublished

886 manuscript]. Fundación Universitaria Konrad Lorenz.

887 Salazar, D. M., Ruiz, F. J., Ramírez, E. S., & Cardona-Betancourt, V. (2020). Acceptance and

888 commitment therapy focused on repetitive negative thinking for child depression: A

889 randomized multiple-baseline evaluation. The Psychological Record, 70(3), 373-386.

890 https://doi.org/10.1007/s40732-019-00362-5

891 Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 statement: Updated

892 guidelines for reporting parallel group randomised trials. BMJ, 340(1), c332.

893 https://doi.org/10.1136/bmj.c332

894 Smout, M., Davies, M., Burns, N., & Christie, A. (2014). Development of the valuing

895 questionnaire (VQ). Journal of Contextual Behavioral Science, 3(3), 164-172.

896 https://doi.org/10.1016/j.jcbs.2014.06.001

897 Stapleton, A. (2020). Choosing not to follow rules that will reduce the spread of COVID-19.

898 Journal of Contextual Behavioral Science, 17(1), 73-78.

899 https://doi.org/10.1016/j.jcbs.2020.07.002

900 Stapleton, A., & McHugh, L. (2020). Adolescent correlates of the generalized pliance

901 questionnaire – children. Journal of Contextual Behavioral Science, 15(1), 131-134.

902 https://doi.org/10.1016/j.jcbs.2019.12.006
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
38
903 Stapleton, A., Ruiz, F. J., & McHugh, L. (2022a). Comparative investigation of adolescents’

904 generalized pliance and psychological inflexibility across cultural contexts. The

905 Psychological Record, 72(1), 105-109. https://doi.org/10.1007/s40732-020-00412-3

906 Stapleton, A., Stynes, G., Cassidy, S., & McHugh, L. (2022b). Assessing acceptance and

907 commitment therapy for adolescent mental health: Single-case A-B design with high

908 temporal density assessments. PsyArXiv. https://doi.org/10.31234/osf.io/pb9hn

909 Suldo, S. M., & Shaffer, E. J. (2008). Looking beyond psychopathology: The dual-factor

910 model of mental health in youth. School Psychology Review, 37(1), 52-68.

911 https://doi.org/10.1080/02796015.2008.12087908

912 Swain, J., Hancock, K., Dixon, A., & Bowman, J. (2015). Acceptance and commitment

913 therapy for children: A systematic review of intervention studies. Journal of

914 Contextual Behavioral Science, 4(2), 73-85.

915 https://doi.org/10.1016/j.jcbs.2015.02.001

916 Törneke, N., Luciano, C., & Valdivia-Salas, S. (2008). Rule-governed behavior and

917 psychological problems. International Journal of Psychology and Psychological

918 Therapy, 8(2), 141-156.

919 Villatte, M., Villatte, J. L., & Hayes, S. C. (2015). Mastering the clinical conversation:

920 Language as intervention. Guilford Publications

921 Waldeck, D., Pancani, L., & Tyndall, I. (2019). An examination of the construct validity of

922 the generalized pliance questionnaire. Journal of Contextual Behavioral Science,

923 11(1), 50-54. https://doi.org/10.1016/j.jcbs.2018.12.003

924 White, K., Lubans, D. R., & Eather, N. (2022). Feasibility and preliminary efficacy of a

925 school-based health and well-being program for adolescent girls. Pilot and Feasibility

926 Studies, 8(1), 1-16. https://doi.org/10.1186/s40814-021-00964-3

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
39
927 Wilson, K. (2016). Contextual behavioral science: Holding terms lightly. In R. D. Zettle, S.

928 C. Hayes, D. Barnes-Holmes, & A. Biglan (Eds.), The wiley handbook of contextual

929 behavioral science (pp. 85-106). John Wiley & Sons.

930 World Medical Association. (2013). Declaration of Helsinki: Ethical principles for medical

931 research involving human subjects. JAMA, 310(20), 2191–2194.

932 Zhou, X., Edirippulige, S., Bai, X., & Bambling, M. (2021). Are online mental health

933 interventions for youth effective? A systematic review. Journal of Telemedicine and

934 Telecare, 27(10), 638-666. https://doi.org/10.1177%2F1357633X211047285

PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
40
Figure 1.

Flow Diagram of Participants’ Progress Through the Phases of the Pilot Randomized Controlled Trial.

Randomized (n = 148)

Allocated to chatbot-delivered
Allocated to psychoeducation Allocated to
Allocation acceptance and commitment
via booklet non-active control
therapy
(n = 46) (n = 57)
(n = 45)

Participants completed pre- Participants completed pre- Participants completed pre-

Pre-Intervention intervention measures at Time 1 intervention measures at Time 1 intervention measures at Time 1
(n = 45) (n = 46) (n = 57)

Post-Intervention Participants completed post- Participants completed post- Participants completed post-
(Five days after intervention measures at Time 2 intervention measures at Time 2 intervention measures at Time 2
pre-intervention) (n = 38) (n = 28) (n = 31)

Follow-Up Participants completed follow- Participants completed follow- Participants completed follow-
(Six weeks after up measures at Time 3 up measures at Time 3 up measures at Time 3
post-intervention) (n = 34) (n = 19) (n = 20)

Participants that completed all Participants that completed all Participants that completed all
Analysis components of the study and components of the study and components of the study and
were included in analyses were included in analyses were included in analyses
(n = 31) (n = 18) (n = 16)
PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
41
Figure 2.
Excerpts of Participant 10’s (Left) and 18’s (Right) Interactions with the Chatbot on Day 2 to Demonstrate
Predetermined Paths of Script Experienced by Participants via Facebook Messenger. Visual Representation
of Occasional Button Options (Below). Participants Used Both Typed Responses and Button Options to
Interact with the Chatbot.

Chatbot: Chatbot:
And finally, finish the next sentence And finally, finish the next sentence
by typing below 👇 … 💭 IF I feel by typing below 👇 … 💭 IF I feel
bad inside THEN I should... bad inside THEN I should...
Participant 10: Participant 18:
If I feel bad inside then I should Help
talk about it to someone,
Chatbot:
unless it will hurt them
Remember, there are no right or wrong
Chatbot:
answers! 😊 This is about your own
So I follow my rules because I’m personal rules, you can phrase them
programmed that way 👀 …Tell me by however you want! Just try your best to
help me understand what you mean.
typing below 👇 … why do you follow
your rules? Participant 18:
Participant 10: If I feel bad inside then I should
I’m not sure try to think positive thoughts

Chatbot: Chatbot:

That’s okay! 😊 It can be hard to notice So I follow my rules because I’m

why we follow rules, especially if we programmed that way 👀 …Tell me by
have been following them for a long typing below 👇 … why do you follow
time. It might be helpful to remember your rules?
why you started following these rules in
the first place? Participant 18:
Participant 10: I don’t know

Because I was told it’s Chatbot:

the right thing to do
That’s okay! 😊 It can be hard to notice
Chatbot: why we follow rules, especially if we
have been following them for a long
And how useful are these rules for you time. It might be helpful to remember
on a scale of 1 (not useful) to 5 (very why you started following these rules in
useful)? the first place?

Participant 10: Participant 18:

3 Because I have to

Chatbot:

Even though I’m getting better and better at talking to my human friends, the
buttons are the best way to chat to me so I can understand. Okay?

Sample participant response buttons that displayed above the text

entry box in Facebook Messenger:

Alright Cool So…

 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
42
Table 1.

Demographic Data and Descriptive Statistics at Time 1 (Pre-Intervention) with T-Tests Comparing Completers and Non-Completers.

Measure Total Sample Completers Non-Completers Comparison

Female Male Range M SD Skew Female Male Range M SD Skew Female Male Range M SD Skew t

Pliance 54 94 11-40 24.52 5.554 0.232 47 18 14-40 25.37 5.915 0.403 7 76 11-36 23.86 5.194 -0.063 1.655

Tracking 54 94 16-58 43.01 5.988 -0.557 47 18 16-58 43.22 6.328 -1.031 7 76 28-56 42.86 5.741 -0.093 0.362

Positive Affect 54 94 6-25 16.37 4.391 -0.304 47 18 6-25 15.91 4.464 -0.058 7 76 6-25 16.73 4.325 -0.512 -1.139

Negative Affect 54 94 5-22 10.03 3.791 0.8 47 18 5-19 10.72 3.655 0.619 7 76 5-22 9.49 3.83 1.027 1.977*

Psychological Inflexibility 53 89 0-32 11.68 6.542 0.593 46 17 0-32 12.86 7.034 0.645 7 72 0-28 10.75 6.003 0.404 1.928

Valued Living Progress 54 92 5-30 18.47 5.525 -0.108 47 18 7-30 18.08 5.693 0.102 7 74 5-30 18.79 5.401 -0.287 -0.774

Valued Living Obstruction 54 94 1-30 13.63 5.971 0.011 46 18 2-30 15.09 6.036 0.166 8 76 1-25 12.51 5.705 -0.202 2.659*

Notes. * < .05; Completers = Participants that submitted data at each time point (i.e., Time 1, 2, and 3); Non-Completers = Participants that did not submit data for at

least one time point; Comparison = Co-efficient from independent t-test comparing Completers’ Time 1 data with Non-Completers’ Time 1 data.
PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
43
Table 2.

Summary of Chatbot-Delivered Acceptance and Commitment Therapy Sessions.

Day Key Aims Key Exercises

1 Familiarize participants with the • Reflect on goals for engaging with

chatbot’s functionalities (e.g., the chatbot

explain that, unless otherwise

specified, the buttons provided

should be used to chat); provide

context for the chatbot (i.e., outline

the purpose of the intervention);

rapport building (e.g., participants

introduce themselves to the chatbot);

prompt selection of a time for

session two

2 Space to reflect on self-rules and the • Exercise: Rules of the game

rules others may project onto us (Hayes & Strosahl, 2005)

• Assess the usefulness of their rules

(five-point rating scale)

• Invitation to notice their

motivations underlying engaging

in particular behaviors (e.g.,

people pleasing versus valued

action)

3 Focus on participants’ rules about • Exercise: Assess the usefulness of

“having good relationships with their rules (describe a time their

PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
44
other people”; undermine rule was useful; describe a time

problematic functions of self-rules their rule was not useful)

• Orient toward the present moment:

Prompts to notice that often we

“can get stuck following rules,

even when they aren’t useful!

Although it can be tricky, we

should check in with our rules and

see if they are still working for us

or not”

• Invitation to notice more rules

they have/ others around them

have before session four

4 Introduction to values; opportunity • Introduction to values and

to experiment with what values align committed action in accordance

most with who and where we want with acceptance and commitment

to be therapy, explicit reference to

values as free from aversive

control (e.g., pliance)

• Reminder to notice their

motivations underlying engaging

in particular behaviors

• Exercise: Values clarification

(selecting their top three most

PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
45
important values from a menu of

options)

• Invitation to notice each time they

are aware of each of their top three

values

5 Review and good goodbye • Discussing the relationships

between valuing and self-as-

distinct from unhelpful self-rules

in the context of flexible rule-

following

• Exercise: Determine whether

actions align with or diverge from

one’s values

• Invitation to notice when rules

may prevent them from being who

and where they want to be

PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
46
Table 3.

Results of ANOVA Testing for Time 1 Differences in Dependent Variables Across Groups

(No Statistically Significant Differences Were Observed).

Dependent Variable df F p

Pliance (GPQ-C) 2, 145 0.417 0.660

Tracking (GTQ-C) 2, 145 1.307 0.274

Positive Affect (PANAS-C PA) 2, 145 1.027 0.361

Negative Affect (PANAS-C NA) 2, 145 0.317 0.729

Psychological Inflexibility (AFQ-Y8) 2, 139 0.732 0.483

Valued Living Progress (VQP) 2, 143 1.867 0.158

Valued Living Obstruction (VQO) 2, 144 1.17 0.313

PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
47
Table 4.
Descriptive Statistics for Pliance, Tracking, and Affect by Condition and Time Among Participants that Submitted Data at Each Time
Point.
Condition Pliance Tracking Positive Affect Negative Affect
n M SD n M SD n M SD n M SD
Time 1
Chatbot 31 24.68 6.139 31 42.81 4.483 31 15.9 4.534 31 10.61 4.088
Booklet 18 24.94 4.065 18 44.78 6.32 18 17.06 4.929 18 10.56 3.535
Non-Active 16 27.19 7.101 16 42.25 9.007 16 14.62 3.612 16 11.13 3.03
Time 2
Chatbot 31 23.9 7.295 31 42.42 4.863 31 16.84 4.1 31 10.06 4.033
Booklet 18 24.11 4.788 18 43.39 7.138 18 16.94 5.23 18 10.11 4.639
Non-Active 16 26.56 5.91 16 41.25 8.598 16 14.81 3.544 16 10.56 3.502
Time 3
Chatbot 31 24.97 5.964 31 43.06 5.138 31 15.84 4.22 31 10.84 4.337
Booklet 18 24.67 3.911 18 43.17 7.098 18 17.67 4.511 18 10.06 3.857
Non-Active 16 28.13 5.737 16 43.44 5.366 16 14.75 4.025 16 11.75 4.465
PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
48
Table 5.
Descriptive Statistics for Psychological Inflexibility, Valued Living Progress, and Valued Living Obstruction by
Condition and Time Among Participants that Submitted Data at Each Time Point.
Condition Psychological Inflexibility Valued Living Progress Valued Living Obstruction
n M SD n M SD n M SD
Time 1
Chatbot 31 12.35 7.543 31 17.1 5.504 30 14.4 6.662
Booklet 17 12.35 6.585 18 19.61 5.942 18 15.06 4.832
Non-Active 15 14.47 6.632 16 18.25 5.733 16 16.44 6.153
Time 2
Chatbot 31 12 6.583 31 18 5.842 30 15.3 4.801
Booklet 17 13.88 6.744 18 18.89 5.368 18 14 5.258
Non-Active 15 15.6 7.288 16 16.63 3.462 16 15.38 4.588
Time 3
Chatbot 31 13.1 7.236 31 17.39 6.07 30 14.9 4.915
Booklet 17 12.88 6.518 18 18.44 6.205 18 13.61 5.031
Non-Active 15 16.47 7.643 16 16.5 4.017 16 17.44 4.953
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
49
Table 6.
Raw High Temporal Density Assessment Data for Participants that Submitted Generalized Pliance Questionnaire Data at Times 1, 2, and 3.
ID Gender T1 T2 T3 T2-T1 T3-T1 EMA1T EMA2T EMA3T EMA4T EMA5T EMA1U EMA2U EMA3U EMA4U EMA5U
2 Female 23 26 25 +3 +2 2 2 2 2 3 2 2 3 4 4
3 Female 28 30 26 +2 -2 3 2 3 2 1 1 1 3 2 1
4 Female 19 29 28 +10 +9 5 1 1 Missing 1 5 1 1 Missing 1
5 Female 28 24 23 -4 -5 4 4 3 2 2 5 5 2 2 2
6 Male 19 18 20 -1 +1 3 3 1 1 1 3 3 1 1 1
7 Male 20 20 27 0 +7 2 2 2 2 2 2 2 2 2 2
10 Female 35 33 28 -2 -7 5 5 3 2 2 4 4 1 1 1
11 Female 22 23 23 +1 +1 3 3 3 3 3 1 3 3 3 3
12 Male 16 14 17 -2 +1 1 1 Missing Missing Missing 2 1 Missing Missing Missing
13 Male 30 34 31 +4 +1 3 3 5 2 1 2 2 2 1 1
14 Female 31 29 31 -2 0 4 4 3 3 3 4 3 3 3 3
15 Female 21 19 24 -2 +3 2 2 1 1 1 1 1 1 1 1
17 Female 40 33 35 -7 -5 5 5 2 2 2 3 3 1 1 1
19 Female 25 15 25 -10 0 4 4 1 1 1 4 4 1 1 1
21 Male 25 28 29 +3 +4 1 1 1 1 1 1 1 1 1 1
22 Male 16 12 16 -4 0 2 2 1 1 1 2 2 1 1 1
23 Female 26 25 25 -1 -1 3 3 1 1 1 2 2 2 2 2
24 Female 14 10 12 -4 -2 1 1 1 1 1 2 1 1 2 2
26 Male 20 21 29 +1 +9 1 1 1 1 1 1 1 1 1 1
27 Male 22 28 22 +6 0 3 4 2 4 2 4 3 1 2 1
28 Female 27 26 24 -1 -3 4 4 4 2 2 3 3 3 3 3
29 Female 22 17 18 -5 -4 4 2 2 2 2 2 2 2 1 1
30 Male 16 15 20 -1 +4 1 1 1 1 1 3 2 1 1 1
31 Female 27 22 16 -5 -11 4 4 2 Missing 1 4 4 1 Missing 1
34 Male 33 29 30 -4 -3 5 5 5 2 2 5 3 3 3 3
 PILOT RCT: CHATBOT-DELIVERED ACT WITH ADOLESCENTS
50
36 Male 28 33 29 +5 +1 4 Missing 4 4 3 3 Missing 4 5 3
37 Female 24 18 24 -6 0 2 2 2 1 1 2 2 2 1 1
38 Female 34 38 39 +4 +5 4 4 3 4 4 3 4 3 4 4
40 Female 21 16 19 -5 -2 2 2 1 1 1 1 1 2 2 2
42 Female 29 32 32 +3 +3 1 1 3 1 1 3 1 3 1 1
43 Female 24 24 27 0 +3 1 1 1 1 1 5 5 5 1 1
Notes. ID = Participant identification number; T1 = GPQ-C scores at Time 1; T2 = GPQ-C scores at Time 2; T3 = GPQ-C scores at Time 3; T2-T1 = Time 2 GPQ-
C scores minus Time 1 GPQ-C scores (i.e., a score such as -3 indicates a decrease from pre- to post-intervention); T3-T1 = Time 3 GPQ-C scores minus Time 1
GPQ-C scores; EMA1T-EMA5T = Daily truthfulness ratings of “Making an effort is only worth it if others recognize it” on a five-point scale ranging from 1 (not
at all true) to 5 (very true); EMA1U-EMA5U = Daily usefulness of thinking “Making an effort is only worth it if others recognize it” on a five-point scale ranging
from 1 (not at all useful) to 5 (very useful); Missing = Data lost to chatbot malfunction; Shaded cells = Participants that reported descriptively lower GPQ-C scores
at Time 2 (i.e., lesser pliance; though not overall statistically significantly lower as per our intervention effectiveness analyses).