Innovative Methods to Assess Psychotherapy Practices (imAPP)

Cognitive Behavioral Therapy (CBT) has been demonstrated to be effective for numerous
presenting problems, including depression, anxiety, and post-traumatic stress disorder
(PTSD). Several large mental health systems have invested heavily in programs to train their
clinicians in CBTs, but relatively little attention has been devoted to the monitoring or
promotion of CBT quality after training is complete. Identifying strategies to do so can
facilitate research and training, and is critical to ensuring consumer access to high
quality, evidence-based treatments. The lack of a scalable, effective, and efficient method
of monitoring quality is a key barrier to efforts to promote high-quality implementation.
Self-report fidelity assessments increase clinician and consumer burden and may not
accurately reflect clinician skill or the intensity with which CBT interventions are
delivered. Observation and expert ratings are time and resource intensive and unlikely to be
feasible or affordable in large systems. To maximize the likelihood of broad implementation
once effective strategies to monitor quality are established, it is essential that these
strategies are feasible and acceptable in routine care contexts, leveraging information
collected during routine care. To date, few monitoring strategies that do not involve
observation, client/caregiver reports, or clinician self-reports have been tested. To address
this critical implementation challenge, we propose to refine and evaluate a method of
monitoring quality that is based on an evaluation of CBT worksheets that are completed in
session. Because the worksheets were developed to implement core cognitive and behavioral
elements and are embedded in CBTs across diagnostic categories, they may be used to elucidate
the clinician's ability to guide the client through CBT interventions in session. Preliminary
research with this measure demonstrated high correlations between the measure and observer
ratings of clinician competence, associations with subsequent symptom change, and high
agreement between raters with differing levels of familiarity with CBT. Completion of the
ratings based on worksheets requires only a small fraction of time required for session
observation and ratings. This project will compare this novel strategy to observer ratings
and adherence checklists that are embedded in clinical notes. Furthermore, it will compare
the accuracy of worksheet data collected by mobile app to paper-form worksheets, and assess
the feasibility and acceptability of these strategies. Because the core elements of CBT and
its worksheets are common across many CBTs, this research has broad implications for
monitoring fidelity to CBTs in a variety of mental health and healthcare systems and
settings. This research will be conducted by a team of investigators with expertise in CBT,
training, implementation, psychotherapy process and outcome research, psychometrics,
longitudinal data analysis, mobile technologies and healthcare economics, with input from
community partners and end-users. The resulting products have the potential to significantly
improve efforts to monitor and ensure ongoing high quality implementation of CBT in routine
care settings.

Inclusion Criteria:

A. Clinician eligibility:

- Private practice or employment at an agency at which the administration agrees to
allow the recruitment and participation of their providers in research related
activities

- English Speaking

- No anticipated plans to leave their current agency for at least 18 months

- Willingness to allow their CBT sessions, worksheets, surveys, and interview data, and
clinical notes to be used for research purposes

- Carry a caseload that typically includes patients who experience depression, anxiety,
or PTSD, with whom they regularly conduct individual therapy sessions and/or capacity
to increase the proportion of such patients (e.g., clinic sees a substantial number of
individuals with PTSD or depression)

- Must be trained (worksheet or web-based training and consultation) or in training for
Cognitive Processing Therapy (CPT) for PTSD or CBT for depression or anxiety that uses
worksheets Does not include CBT-i Can include aspects (i.e., worksheets) of CBT for
substance use as long as depression or anxiety is primary diagnoses

- Must anticipate at least 3 eligible patients

- Must be willing to record sessions and provide worksheets and symptom measures to the
study

- Must have computer and internet access

- Must be willing to use a mobile app on a tablet or mobile device

B. Patient eligibility:

- Must be 18 yrs. of age or older

- Experience one or more of the following (both a diagnosis and cut off score):

Clinician diagnosis of primary PTSD (PTSD-Checklist-5 score of 33 or more) Depressive
disorder (e.g., major depressive disorder, dysthymia; PHQ of 10 or above) Or an Anxiety
Disorder (Beck Anxiety Disorder score of 22 or above)

- Note that if a patient has a score close to the cut-off score, it's up to the
therapist digression

- Must be willing to allow the team to collect session recordings, measures, notes
and worksheets

- Must be able to read and write at a sixth-grade level or above

- Able to participate in sessions conducted in English, or Spanish (if working with
bilingual clinicians in community or private practice)

- Must be willing to engage in CBT/CPT

- Therapist considers the treatment with the individual patient to be "mostly" CBT
or CPT

Cannot have*:

- Imminent risk of suicide or homicide (requiring hospitalization) that require immediate
treatment In need of detoxification (can be enrolled when substance abuse treatment is not
the primary treatment target) Active psychosis or manic episode unless well controlled by
medication and not the primary focus of treatment Cognitive impairments that preclude any
participation in therapy

C. Administrator eligibility:

- Must have a support or managerial/supervisory position at the clinic(s) from which A
and B are being recruited from

- Must also be willing to complete a packet of study measures/partake in an interview