Efficacy of a Two-Year Intensive Reading Intervention for Middle School English Learners With Reading Difficulties

Primary Hypothesis: Students receiving the RISE intervention will outperform those receiving
Business as Usual (BAU) instruction across reading-related elements, including word reading,
fluency, and comprehension at the end of year two of treatment.

Secondary Hypothesis: Students receiving the RISE intervention will outperform those
receiving Business as Usual (BAU) instruction on measures of word reading and reading fluency
following one year of treatment.

Research Design Overview

Participants in Cohort 1 will be recruited in fall of 2018 (6th and 7th grade) and Cohort 2
participants will be recruited in fall 2020 (6th and 7th grade). Participants who meet
eligibility criteria will be randomly assigned to one of two conditions: (1) the RISE
intervention (one 45 to 50-minute class, 5 days per week) or (2) a BAU comparison condition.
Students identified as having disabilities and receiving special education who otherwise meet
all criteria will be allowed to participate in the study and additional interventions will be
documented. Treatment teachers are hired and trained by the investigators. Participants in
the control condition typically participate in an elective during that class period (e.g.,
cooking, music), and sometimes in a preparation class for the state high stakes test.
Teachers will be assigned to one school and responsible for providing instruction to students
in groups ranging from 3 to 12 students.

Participants

The Project 3 sample will be composed of 6th and 7th grade students from six moderate to high
poverty middle schools in Austin/San Antonio and in Houston with large numbers of students
who are learning English as a second language. To evaluate the primary hypothesis, one
randomized control trial integrating two nonoverlapping cohorts of participants (two-year
intervention) will be conducted. The investigators will identify sample participants using
extant school records and as described in the inclusion/exclusion criteria.

Cohort 1 will consist of 410 Els with significant reading difficulties, 205 assigned to
Intervention (RISE) and 205 to the no intervention BAU comparison condition beginning in Fall
2018 (Year 1) and continuing in their assigned condition in year 2 (students 7th or 8th grade
year). Cohort 2 will be a nonoverlapping sample of 410 students identified using the same
inclusion/exclusion criteria and also randomly assigned to Intervention (RISE) or the no
intervention BAU comparison condition and remaining in that condition through year 2.
Students assigned to control may receive BAU services (researcher documented) from their
schools, but will not receive researcher-provided reading intervention. These two cohorts (N
= 820 students) will provide a sample to fully power data analysis.

Measures and Assessment Procedures

The data for Project 3 is collected at 4 time points (beginning and end of year for year 1
and year 2) for each cohort, permitting an analysis of treatment effects following one and
two years of treatment (see intervention description). The research team that is responsible
for hiring, training, and supervising data collection in schools, and for recruiting schools
and participants, is experienced and has worked together for the past 10 years. Data
collectors are blind to participants' randomly assigned condition.

Primary and secondary outcome measures are described in the section 9 (outcome measures).

Intervention Procedures

The intervention is described in detail in section 8 (Arms, Groups, and Interventions).

Fidelity of Implementation. Fidelity of implementation will be evaluated a minimum of three
times per year per intervention teacher. All intervention teachers will audio-record all
intervention lessons. A random sample of recordings will be selected and key indicators of
intervention implementation adherence and quality of implementation will be evaluated by a
coder. All coders will be trained and reliability will be established prior to independent
coding.

Intervention Teachers. At each site, highly trained personnel hired and supervised by the
investigators will deliver all treatments. These teachers will be provided 20 hours of
training prior to implementation and then ongoing training and on-site support for at least
10 hours per month. All treatments will be provided daily in group sizes of about 8-11
students for 50 minutes a day.

Business as Usual (BAU) Comparison Condition. Students assigned to the BAU or comparison
condition will participate in an elective class that includes such options as music, cooking,
film, study time, or high stakes test preparation. These students will participate in the
full normative educational program at their schools, but will not receive any instruction
from the research team.

Data Management

Data Acquisition. The data management team will construct data acquisition forms and
guidelines for completing those forms. When modifications to procedures or forms are
necessitated, data management staff will communicate these changes to staff at each site and
ensure that procedures and forms manuals are appropriately updated. Data management staff
will also work to ensure consistency in forms layout, forms revision, and forms numbering.
Data management and statistical staff also work with data managers to conduct data audits to
ensure that error rates are negligible for all data fields.

Data Management. Data management personnel conduct all database design, management, and
collection-related activities in a manner that results in all project data being written to
the Texas Institute for Measurement, Evaluation, and Statistics (TIMES) data warehouse. The
warehouse is designed for maximum data integrity and standardization within and across
projects, while allowing for necessary flexibility across TIMES projects in their designs and
specific measures and methods. The use of the warehouse as the primary project management
data structure increases standardization across projects. The warehouse also automatically
constructs an electronic audit trail of all data management activities that result in the
modification of even a single data element, which together with standardization is essential
to quality control in electronic databases.

Data Analysis

Overall Strategy, Methodology and Analyses. Descriptive features of the data will be examined
prior to analysis. Non-normal dependent variables will be transformed (logarithmic, square
root, inverse, etc.) as necessary and appropriate. Outliers will be identified using modified
z-score analysis, and handled on a case-by-case basis according to their leverage and
influence in specific models. Assumptions that errors are normally distributed,
homoscedastic, and independent across sampling units and levels of the model will be
evaluated by analyzing residuals. Residuals at higher levels are typically assumed to be
multivariate normal and independent of lower level errors. The investigators will augment
residual analyses with influence diagnostics. Heterogeneity will be addressed according to
its apparent source(s), using nonlinear transformations of predictor and/or dependent
variables as appropriate and necessary. The investigators will address the primary research
hypothesis in the context of multiple group (RISE intervention v. BAU), multilevel regression
and structural equation models (SEM).

Analytic Plan. Multiple group, multilevel regressions and SEMs will be fit to estimate the
main effect of treatment on 1) word reading and fluency outcomes after 1 year of intervention
and word reading, fluency, and comprehension after 2 years of intervention For each model,
pretest scores will be centered as appropriate and used as level-1 covariates. School-level
means at pretest will be included as "contextual effects" to minimize school-level
variability, improving statistical power. Treatment main effects (assuming an intent-to-treat
model) will be estimated at the student level by comparing conditional posttest means for the
RISE group and the BAU group in the context of nested models. Fit indices for the
group-specific model (i.e., unique posttest means estimated for each group) and the
constrained model (i.e., posttest means fixed as equal across treatment and groups) will be
compared and the difference (Δχ2) will be tested against the critical value that corresponds
to the difference in degrees of freedom across models (Δdf). Effect sizes will be calculated
as Hedges g.

Power and Effect Size. Data simulation was utilized to estimate statistical power for the
main effect on student outcomes. These are intended as examples that can be applied across
all primary and secondary outcomes. For estimating power, nesting at the site and cohort
levels is ignored, as Intra Class Correlations in previous studies in similar schools have
been negligible when controlling for pretest. Additionally, partial nesting was ignored based
on previous findings for multiple-cohort, multi-year interventions and partially nested
designs implemented in these two sites. When blocking on schools and when controlling for
covariates at the school-level, clustering effects of site, cohort, and small instructional
groups were trivial for comprehension-related outcomes (.00-.01 for the 2-year intervention
with partial nesting). Variability in effect size did not differ statistically from 0 (σδ2 <
.0001), suggesting fixed treatment effects.

Power analyses assumed a time 1 sample of 820 students across 6 schools (n=205/school). In
the population model, the investigators randomly assigned cases to treatment or BAU within
schools. The investigators specified student attrition as .10 annually for both conditions
across sites and across cohorts with no differential attrition by condition or by
interactions involving condition, suggesting a sample of approximately 740 students at
assessment time 2 (spring year 1) and 670 at assessment time 4 (spring year 2). An annual
attrition rate of 10% is consistent with previous studies conducted by the research team at
similar schools. The investigators modeled a school-level covariate for the pretest to
maximize power. In previous work, pretest measures of student reading outcomes (i.e., word
reading, comprehension, etc.) accounted for about 75% of the outcome variance. Posttest
reading scores were modeled as multivariate normal with a mean of 0 and variance of 1 in the
untreated population. For the subpopulation assigned to treatment, the posttest mean was set
at .20, which represents a population-level standardized mean difference between treatment
and control). A literacy-related treatment effect of .20 represents a meaningful impact in
this population, on both word-level reading measures and on reading comprehension, spelling
and writing outcomes.

In the sample model, the investigators estimated posttest means as free parameters in the
treatment and comparison conditions, using population values as starting values. The
probability of rejecting the null hypothesis when it is false (i.e., statistical power) was
.99 when modeling effects as fixed (σδ2 = 0) for samples of 740 and 670. Bias for the
estimator72 was less than 1%. Under the same assumptions, a standardized mean difference of
.10 was associated with power of at least .80. Power estimates were calculated for
independent tests of statistical significance. However, many of the above contrasts are not
independent. To control for inflated Type 1 error associated with multiple comparisons, the
investigators propose the Step M approach for multilevel and mixed effects models and
hierarchical data, which takes advantage of the dependence structure of individual test
statistics and offers a more powerful approach than alternatives for handling family-wise
error in a nested data.

Inclusion Criteria:

- Participants are enrolled in 6th or 7th grade at a participating school.

- Participants are currently identified as English Learners (designated by school as
limited English proficient) or were previously identified as limited English
proficient and were redesignated as English proficient within the last 3 years by
their school.

- A parent reported that Spanish was spoken at home at initial school entry.

- The participant failed the state level reading comprehension test the previous school
year

Exclusion Criteria:

- The potential participant has a sensory disorder that precludes participation int he
assessment and intervention protocols.

- The potential participant participates in an alternative curriculum (i.e., life skills
course) due to moderate to severe learning disorders.

- The potential participant received a score of Beginner (Level 1) on the Texas English
Language Proficiency Assessment System (TELPAS) listening and speaking subtest,
indicating limited opportunity to learn English.