Evaluation of the Research to Policy Collaboration Model

Objective: The aim of this study is to examine the effectiveness of a model for supporting
policymakers' use of research evidence within a randomized controlled trial (RCT) employing
both qualitative and quantitative analyses of survey, interview, and observational data to
understand the use of research in policy and collaboration. Legislators' research use will be
assessed with multiple methods, including (1) self-report via a survey protocol, (2)
legislative activity (i.e., introduced bills and official statements), and (3) qualitative
interviews. Additionally, interviews and surveys will assess researchers' reported efficacy,
skill and engagement in public policy, and the extent to which their work is informed by the
policy process.

Intervention: The Research-to-Policy Collaboration (RPC) builds upon theoretical and
empirical literature that suggests the need to (a) translate research relevant to current
policy priorities, and (b) facilitate productive interactions between policymakers and
researchers. The RPC is a nonpartisan, manualized model implemented by an intermediary
organization that cultivates relationships between researchers and legislative offices.
Implementation occurs in two phases involving a series of seven interrelated activities.
During the capacity-building phase, the RPC simultaneously aims to support researcher's
development of honest knowledge broker skills with training and coaching, while also
conducting iterative needs assessments with congressional offices regarding their current
priorities and desire for research evidence. In the collaboration phase, researchers with
relevant expertise are coalesced into rapid response teams that are matched with legislative
offices. Through a scaffolded series of interactions, the RPC model is used to cultivate
productive researcher-policymaker relationships and support responses to legislative
requests. Evaluating the effectiveness of the RPC is the focus of the current study.

Study Design:

This study employs both qualitative and quantitative analyses of survey, interview, and
observational data within the context of an RCT. Three overarching research questions guide
this work and align with the study's three different components - self-report via a
quantitative survey, observation of research use via coded legislative activity, and
ethnography via in-depth interviews and participant observations.

1. How does the RPC impact researchers and legislative staff? (Self-Report)

1. Do researchers report improved policy competencies and motivation for conducting
policy-relevant research?

2. Will legislative staff report increases in positive attitudes toward, knowledge of,
intended application, and actual use of research evidence?

3. Might the impact on legislative and research participants will vary as a function
of aspects of their collaborative relationship (e.g., enhanced by mutual trust)?

2. How does the RPC impact legislative activity? (Observation)

a) Will congressional offices increase their use of research evidence in legislative
activities (i.e., bills, official statements)?

3. How might perceptions and experiences of collaboration through the RPC relate to
different forms of evidence use among researchers and policymakers? (Ethnography)

1. What are the barriers and facilitators of participation in the RPC?

2. What types of interactions between researchers and policymakers does the RPC
influence?

3. How does the RPC affect participants' assumptions about each other, interactions
with each other, and their intentions?

4. What types of evidence use are supported by RPC participation?

Sampling Framework All study components draw from an initial sampling framework through which
researchers and congressional staff who opt into participating in the RPC will be asked to
participate in the study; those who agree to participate in the study will be randomized for
either the full RPC intervention or a light-touch control group that receives little direct
engagement. This study will oversample to combat anticipated attrition. The final sample is
expected to include 60 congressional offices (30 in RPC, 30 in control group) and 60
researchers (30 in RPC, 30 in control group). These samples will be derived from an
identified population of eligible intervention participants.

- A population of relevant legislative offices (Senate and House) will be identified based
on the number of child/family bills legislator have (co)sponsored. The random selection
process will be stratified to ensure proportional partisan representation and
representation of legislators who range in activity related to child welfare. Prior to
randomization, the population of relevant legislators will be assigned an Activity
Ranking based on the number of times a legislator has sponsored a child/family policy
bill. Activity Rankings will range from 1-10 such that 10% of legislators who
(co)sponsored the fewest bills will be ranked 1, and 10% of legislators who co-sponsored
the most bills will be ranked 10. Recruitment and stratified sampling across a continuum
of activity on child/family policy will improve bipartisan representation of the sample.
Also aiding with bipartisan representation are targeted outreach efforts that intensify
along party lines if participation becomes skewed.

- Researchers who work on relevant child/family research will be identified via research
institutions, professional networks and listservs, literature searches, and referrals
from other researchers. Researchers include research-oriented practitioners and program
evaluators from both academic and non-academic settings who are interested in
translating their expertise with policymakers. Subsequent to opting-in to participate in
a policy engagement intervention, researchers are asked to participate in the study.
Study participants will be randomized to receive the RPC or the control condition.

Study Component 1: Quantitative Evaluation of Self-Reported RPC Impact A structured survey
will be used to assess researchers' capacity for engaging in public policy processes. A
corresponding survey protocol will assess legislative staff's attitudes, knowledge, intent,
and reported behavior regarding the use of research evidence. Both researchers and
legislative staff will be asked about the nature of their collaborative interactions with one
another. Surveys will be conducted approximately every three months. Survey constructs are
described in outcome measures. Missing data will be handled with multiple imputation because
this is best practice when data are not missing at random. Multilevel modelling will be used
to examine individual trajectories in outcome variables. Multiple regression will examine
changes between select time points; for instance, whether researchers' reported policy skills
improve between baseline and subsequent to trainings. These analyses will allow us to examine
whether RPC participation is associated with increases in (1) researchers' reported efficacy
to engage with policymakers, (2) legislative staff's interest and willingness to use research
evidence, and (3) both researchers' and policymakers' positive sentiment toward working
together. Longitudinal analyses will further allow us to explore the trajectory of outcome
variables over time (e.g., improved researcher competencies, legislators' use of
research)—specifically modelling change in evidence use and other outcomes of interest.
Analyses of researcher and legislative staff outcomes are all powered at least at a 90% level
to detect effect sizes of 0.2 or greater (power analysis conducted with Optimal Design plus
Empirical Evidence).

Study Component 2: Quantitative Evaluation of Observed RPC Impact on Legislative Activity To
complement the self-reported quantitative assessment of change, use of research evidence in
legislative activities will be observed before and following RPC participation. Data will be
selected, extracted, and coded prior to quantitative analyses with indicators derived from
deductive codes. Legislative data will be selected based upon the use of keywords indicating
relevance to child/family. Since each bill will comprise more text than is feasible to code
with traditional qualitative methods, relevant provisions will be extracted prior to coding
(those containing keywords). Further, all subsection titles for each sampled bill will be
reviewed for potential relevance and extracted if research evidence is used but not captured
otherwise by keywords. Data on legislative activity will be coded based on a detailed
codebook informed by prior findings of how research has been used by policymakers and in
legislation. This coding scheme will be applied by two coders and the codebook will be
revised and further specified until the intercoder reliability reaches at least .70 and 30%
of the data for the first time period has been coded. Subsequently, trained coders will code
individually and a lead investigator will randomly sample 10% of these to ensure consistency
in coding quality. These codes will be converted into quantitative indicators of research
use, which are described in outcome measures. Legislative activity will be assessed annually.
Multivariate Poisson regression analyses will assess change in research use in thematic codes
of (1) bills and (2) official statements between the RPC intervention and control conditions.
Moderation analyses will further explore the extent to which changes in legislative activity
depend on the nature of collaboration experiences, other survey data (e.g., policymakers'
perceptions of the value of research), and legislator characteristics (e.g., party
affiliation, years of age).

Study Component 3: Ethnographic Evaluation of the RPC model The qualitative component of this
study is closely linked with the quantitative work and seeks to better understand the
experiences of RPC participants and processes underlying the use of research evidence in the
RPC context. Using participatory and ethnographic methods, data will be collected from two
sources: (1) semi-structured interviews of trial participants and RPC staff, and (2)
observations of researcher trainings and researcher-staff meetings. Interview participants
will be sampled from the pool of study participants randomly assigned to receive the RPC,
including 11 RPC researchers and 11 RPC congressional office staff that will be interviewed
prior to their involvement in the RPC, and 11 RPC researchers and 11 congressional office
staff that will be interviewed following collaboration activities via the RPC. Additionally,
3 RPC staff will be interviewed prior to and following the RPC. To gain additional insight
into the RPC operations and impact, observations will be conducted of trainings for
researchers, formal meetings and informal interactions between researchers and congressional
staff, and meetings that occur as part of any RPC collaboration. Following ethnographic best
practices, "open-observations" of key meetings and training events will be conducted to
enrich the interviews with more information about how attendees interact and how they respond
to the RPC. Field notes and transcripts will be analysed with thematic analysis and an
interpretive policy analysis that explores different rationalities and rhetoric when
discussing evidence used in the collaboration.

Inclusion Criteria:

- Participants who voluntarily enlist in the RPC will be asked to participate in the
trial.

Exclusion Criteria:

- Participants who choose to stop participating in the study or the RPC itself. All
study participants can choose to opt-out of the study at any time.