paper.Rnw

 \documentclass[notitlepage,12pt]{jedm}

\input{preamble.tex}

<<include=FALSE>>=
library(knitr)
library(lubridate)
library(scales)
library(reshape2)
library(dplyr)
library(ggplot2)
library(xtable)

#library(tikzDevice)
opts_chunk$set(
echo=FALSE, results='asis',cache=TRUE,warning=FALSE,error=FALSE,message=FALSE,autodep = FALSE
    )

#options(tikzDefaultEngine = "pdftex")
@

<<setup,include=FALSE>>=

library(lubridate)
library(scales)
library(reshape2)
library(dplyr)
library(ggplot2)
library(xtable)

addOverallState <- function(dat){
    levels(dat$state) <- c(levels(dat$state),'Overall')
    dat2 <- dat
    dat2$state <- 'Overall'
    dat2 <- rbind(dat,dat2)
    dat2$state <- factor(dat2$state, levels= levels(dat$state))
    dat2
}

@


<<data,include=FALSE,cache=TRUE>>=
if('cpPaper.RData'%in%list.files() & file.info('cpPaper.RData')$mtime>file.info('dataMerge.r')$mtime){
    load('cpPaper.RData')
} else source('dataMerge.r')
@



\begin{document}
\maketitle

\input{intro.tex}

\section{The RAND Effectiveness Trial}\label{sec:RANDtrial}
The study to measure the effectiveness of CTAI included 7 states, 73
high schools, and 74 middle schools with nearly 18,700 high school
students and 6,800 middle school students participating. Schools were enrolled in a
total of 52 school districts that were distributed among urban,
suburban, and rural areas. Schools were matched on a set of
covariates, and then randomly assigned to the treatment or control
group. Schools in the control group continued with their current
algebra curriculum, and schools in the treatment group used Carnegie
Learning's curriculum which includes CTAI textbook materials and sofware. Each school participated for two years, with a different cohort
of students participating the second year (with a small fraction of
students present in the study both years because they repeated
algebra). It should be noted that this study did not include statewide
implementations; the study results cannot be generalized to all
schools within the state. In some states, one large school district
participated, while in other states, a set of smaller school districts
participated. The states included Alabama (AL), Connecticut (CT),
Kentucky (KY), Louisiana (LA), Michigan (MI), New Jersey (NJ), and
Texas (TX). Each state participated in both the middle school and high
school arms of the study, except AL, which participated only in the
middle school arm. The current study focuses on high school students
only.

There are some limitations to the available data for this study.
Log data from some schools, and some students within schools, were
missing either because the log files were not retrievable, or because
of an imperfect ability to link log data to other study data files.
For this reason, this study uses only data from the
\Sexpr{n_distinct(data[['schoolid2']])} treatment schools for which at least
80\%
of students in both study years appear in the log data file.
This sample includes \Sexpr{n_distinct(data[['field_id']])} students, around
\Sexpr{round(n_distinct(data[['field_id']])/n_distinct(stud[['field_id']])*100)}\%
of the treated high-school sample.
Table \ref{tab:nByState} gives the number of students in the sample by
state and year.
The states in the table are ordered by the total number of students
they represent in the sample; they will appear in this order in
all of the forthcoming tables and figures.
Some figures will only show data from a subset of states; since so few
students were in New Jersey, it will be excluded from almost all
state-by-state comparisons (but included analyses that pool across states).
<<nByState,dependson='data'>>=
sampleSizeTable <- data%>%group_by(state,year)%>%summarize(n=n_distinct(field_id))%>%dcast(year~state)
sampleSizeTable <- sampleSizeTable[,-1]
rownames(sampleSizeTable) <- c('Year 1','Year 2')

print(xtable(sampleSizeTable,caption='Numbers of students in the sample by state and study year',label='tab:nByState'),include.rownames=TRUE)
@

In this sample from \Sexpr{n_distinct(data[['schoolid2']])} schools, \Sexpr{sum(data[['obsUsage']]==FALSE)}
students who participated in the RAND study do not appear in the log data; they may have not used the CTAI
software at all, or may have been excluded from the log data for other
reasons.
Since we don't know which is true, we exclude these students from most
analyses.

It is likely that some usage data were missing, even for students who
appear in the usage dataset.
However, it is impossible to know in which cases these data were
missing or why; for the most part, we ignore this problem, but it
should be kept in mind nonetheless.

\section{Standard and Customized Curricula}\label{sec:curricula}

\begin{figure}
  \centering
<<curricula,dependson=c('setup','data'),fig.width=6.4,fig.height=3>>=

bbb <- addOverallState(data)%>%filter(!is.na(Curriculum) & State!='NJ')%>%
    group_by(state,Curriculum,overall,Yr)%>%
    summarize(n=n())
staten <- addOverallState(data)%>%filter(!is.na(Curriculum) & State!='NJ')%>%group_by(state,Yr)%>%summarize(n=n())

for(i in 1:nrow(bbb)) bbb$n[i] <- bbb$n[i]/staten$n[staten$state==bbb$state[i] & staten$Yr==bbb$Yr[i]]

COLS <- c('#1b9e77','#d95f02','#7570b3')

ggplot(bbb,aes(Yr,n,fill=Curriculum,alpha=overall))+geom_col()+facet_grid(~state)+scale_alpha_manual(values=c(1,0.5))+scale_fill_manual(values=COLS)+scale_y_continuous(labels=percent)+labs(x='',y='% of Problems Worked',alpha='')


@
\caption{Percentage of worked problems coming from various courses
  (denoted by color, with Algebra II and Geometry bundled as ``$>$Algebra I''), from
  standard and customized variants, denoted by shading.}
\label{fig:curricula}
\end{figure}

Students' automatic progress through the Cognitive Tutor (CT) software
is normally governed by
the sequences of sections and units embedded in the software.
Without external meddling, the curriculum a student works on
determines the sequence, and thus what section he or she will be directed towards next after
mastering (or exhausting the problems) from a previous section.
In the CTAI effectiveness trial, the most common curriculum was,
naturally, Algebra I.
This came with three closely related variants, due to new software releases.
Students requiring more remediation were able to work on a less
advanced curriculum, called ``Bridge to Algebra,'' and more advanced
students could work on Algebra II or Geometry.

In the second year of the study, some high schools, primarily in
Texas, Michigan, and Kentucky, requested customized variants of the
curricula.
This was typically due to state standards, testing schedules, or local
scope and sequence guidelines.
These ``customized curricula'' altered the order of some sections and
units, and were usually particular to schools.

Figure \ref{fig:curricula} shows the percentage of worked problems
from each curriculum, from standard and customized varieties, by state
and year.
First, note that the vast majority of worked problems were from the
Algebra I sequence.
A small but notable number of less advanced problems were worked in
Kentucky in year 2, and some more advanced problems were worked in
Michigan and Louisiana.
Secondly, note the rise in ``customized curricula'' in year 2 in
Texas, Kentucky, and Michigan, the three states with the most students
in our dataset.
In particular, Texas shifted almost entirely to customized curricula
from years 1 to 2.




Throughout the school year, teachers could have a class of students
working on multiple curricula either sequentially, where the students
changed curricula in lock step, or simultaneously, where students
worked on different curricula at the same time. As an example, two
teachers located in Kentucky had their students working on Algebra I
throughout most of the year and then reassigned them to Algebra II in
the last month of school. In contrast, a different teacher in Kentucky had students variously
enrolled in three different curricula throughout the entire year
(Bridge-to-Algebra, Algebra I, and a customized Geometry curriculum),
while a year 2 teacher in Michigan enrolled students in
three curricula sequentially throughout the year: Algebra I
until November, followed by a customized curriculum until February, and
ending with a different customized curriculum until June. While there are numerous instances of these
uses of multiple curricula in year 2, there are also many occurrences
of teachers who had their students enrolled in the standard Algebra I
throughout the entire year, including all Connecticut teachers.
There were also teachers, mostly in Texas, who used customized
curricula exclusively throughout the
second year.

\section{Student Usage Across States and Years}\label{sec:usage}

<<usageMedians,dependson=c('data','setup'),results='asis'>>=
secByStud <- data%>%group_by(state,Yr,field_id) %>% summarize(numSec=n_distinct(section,na.rm=TRUE),
                                                              numUnit=n_distinct(unit,na.rm=TRUE),
                                                              time=sum(total_t1,na.rm=TRUE),
                                                              mastered=n_distinct(section[status=='graduated'],na.rm=TRUE),
                                                              nprob=n_distinct(unit,section,Prob1,na.rm=TRUE))

secByStud$time <- secByStud$time/3600000

secByStud <- within(secByStud,{
    numUnit[numUnit==0] <- NA
    numSec[numSec==0] <- NA
    time[time==0] <- NA
    mastered[mastered==0] <- NA
})

secByStud2 <- addOverallState(secByStud)

secByStud2 <-
  secByStud2 %>%
  group_by(state,Yr) %>%
  mutate(outlier.time = time > quantile(time,0.75,na.rm=TRUE) + IQR(time,na.rm=TRUE) * 1.5,
         outlier.sec = numSec > quantile(numSec,0.75,na.rm=TRUE) + IQR(numSec,na.rm=TRUE) * 1.5,
         outlier.prob = nprob > quantile(nprob,0.75,na.rm=TRUE) + IQR(nprob,na.rm=TRUE) * 1.5,
         outlier.unit = numUnit > quantile(numUnit,0.75,na.rm=TRUE) + IQR(numUnit,na.rm=TRUE) * 1.5
         ) %>%
  ungroup


tab <- ungroup(secByStud)%>%group_by(Yr)%>%dplyr::select(time,nprob,numSec,numUnit)%>%summarize_all(median,na.rm=TRUE)
tab$Yr <- NULL
tab <- as.data.frame(tab)
rownames(tab) <- c('Year 1','Year 2')
names(tab) <- c('Hours','Problems','Sections','Units')

xtable(tab,caption='Median numbers of hours, problems, sections, and units worked by each student in the dataset in the two years of the study. Students with no usage data were excluded.',label='tab:medUsage',digits=c(1,2,0,1,0))
@


Table \ref{tab:medUsage} shows the median numbers of hours,
problems, sections, and units worked on by each student in the dataset
in the two years of the study.
Apparently, usage decreased markedly in the second year: the median
of hours worked decreased by
\Sexpr{round(tab['Year 1','Hours']-tab['Year 2','Hours'])}, the median number of problems decreased by
\Sexpr{tab['Year 1','Problems']-tab['Year 2','Problems']}, and
the median number of sections decreased by
\Sexpr{tab['Year 1','Sections']-tab['Year 2','Sections']} from
years 1 to 2.
Yet, as discussed below, the median number of units worked increased
by \Sexpr{tab['Year 2','Units']-tab['Year 1','Units']}.


\begin{figure}
\centering
<<usageTime, fig.height=3,fig.width=6,dependson=c('data','setup','usageMedians')>>=


print(timeStateYear <- ggplot(filter(secByStud2,state!='NJ'),aes(Yr,time))+geom_boxplot(outlier.shape=NA)+
          geom_jitter(data = function(x) dplyr::filter_(x, ~ outlier.time), width=0.2)+
          facet_grid(~state)+coord_cartesian(ylim=c(0,110))+labs(x='',y='Hours on CT Software per Student'))
                                        #ggsave('timeStatYear.jpg',width=6,height=3)





@
\caption{Boxplots of hours each student spent on Cognitive Tutor
  software over by year and state. Students  with no timestamp data
  ($n=$\Sexpr{sum(is.na(secByStud[['time']]))}), with anomalous negative time
  ($n=$\Sexpr{sum(secByStud[['time']]<0,na.rm=TRUE)}) or with more than 110 hours ($n=$\Sexpr{sum(secByStud[['time']]>110,na.rm=TRUE)})
  were excluded.}
\label{fig:timeByStud}
\end{figure}

<<supplementalFigs1,include=FALSE,dependson=c('data','setup','usageMedians')>>=
secStateYear <- ggplot(filter(secByStud2,state!='NJ'),aes(Yr,numSec))+geom_boxplot(outlier.shape=NA)+
          geom_jitter(data = function(x) dplyr::filter_(x, ~ outlier.sec), width=0.2)+
          facet_grid(~state)+labs(x='',y='# of Sections Worked per Student')+coord_cartesian(ylim=c(0,200))
ggsave('secStateYear.jpg',width=6,height=3)

probStateYear <- ggplot(filter(secByStud2,state!='NJ'),aes(Yr,nprob))+geom_boxplot(outlier.shape=NA)+
          geom_jitter(data = function(x) dplyr::filter_(x, ~ outlier.prob), width=0.2)+
          facet_grid(~state)+labs(x='',y='# of Problems Worked per Student')+coord_cartesian(ylim=c(0,2000))
ggsave('probStateYear.jpg',width=6,height=3)

@



Figure \ref{fig:timeByStud} shows that the number of hours students
spent working on the CT software in some more detail, via
state-by-year boxplots.
Analogous figures for the numbers of problems and sections students
worked,
showed similar patterns.
Usage time varied substantially between students and across states and
years.
Students in Texas,
Connecticut, and New Jersey worked far fewer hours than students in
Kentucky, Louisiana, and Michigan.
Not every state reduced its usage from years 1 to 2---while students
in Texas, Kentucky and New Jersey used the software less in the second
year than in the first, students in Michigan, Louisiana, and
Connecticut increased their usage.

Overall, usage varied a bit more in year 2 than in year
1---the median absolute deviation of time spent was \Sexpr{round(mad(secByStud[['time']][secByStud[['Yr']]=='Yr 1'],na.rm=TRUE,constant=1),1)} hours in the first year, compared to \Sexpr{round(mad(secByStud[['time']][secByStud[['Yr']]=='Yr 2'],na.rm=TRUE,constant=1),1)} in  the second year.
The increase in variation seems to be driven both by increasing
between-state variation, and a between-student increase in Louisiana.
One intriguing possibility is that the amount of CT usage may have been better tailored to
teachers and students in the second year than in the first. Perhaps
usage increased for
students who stood to gain more from the software and decreased for
students who stood to gain less.

In contrast to the decreasing numbers of hours, problems, and sections students
worked in year 2, Table \ref{tab:medUsage} shows that the median number of units
students worked increased by
\Sexpr{round(tab['Year 2','Units']-tab['Year 1','Units'])}
in year 2.
This suggests students in year 2 were exposed, on average, to a
slightly wider range of topics.
Figure \ref{fig:unitsByStud} shows boxplots of the numbers of units
worked by state and year.
The geographic variation in units worked mirrors the pattern in Figure
\ref{fig:timeByStud}, with more usage in Kentucky, Michigan, and
Louisiana but less in Texas, Connecticut, and New Jersey.
However, in every state the median year 2 student worked at least as
many different units as the median year 1 student.
Variation in the number of units worked also increased slightly
from years 1 to 2---the interquartile range (IQR) increased in every state
except for Kentucky, where a decrease in IQR was accompanied by an
increase in the number of outliers.



\begin{figure}
\centering
<<unitsWorked,dependson=c('usageMedians','data','setup'),fig.width=6,fig.height=3>>=
secByStud2 <-
  secByStud2 %>%
  group_by(state,Yr) %>%
  mutate(outlier = numUnit > quantile(numUnit,0.75,na.rm=TRUE) + IQR(numUnit,na.rm=TRUE) * 1.5) %>%
  ungroup

print(unitsStateYear <- ggplot(filter(secByStud2,state!='NJ'),aes(Yr,numUnit))+geom_boxplot(outlier.shape=NA)+geom_jitter(data = function(x) dplyr::filter_(x, ~ outlier), width=0.2)+facet_grid(~state)+labs(x='',y='# Units Worked Per Student')+coord_cartesian(ylim=c(0,55)))
@
\caption{Boxplots of the number of units of Cognitive Tutor
  software each student worked, by year and state. Students
  working more than 55 units (
  \Sexpr{sum(secByStud[['numUnit']]>55,na.rm=TRUE)} of
  \Sexpr{nrow(secByStud)}) and students
  with no usage data (\Sexpr{sum(is.na(secByStud[['numUnit']]))})
  were excluded.}
\label{fig:unitsByStud}
\end{figure}

All in all, students used CT software less in year 2 than in year
1.
On the other hand, students in the second year tended to see a
slightly wider range of topics, and varied somewhat more in their usage.



\section{Working Units in Order---Or Not}\label{sec:order}

Overall, students used CT less in the second year than in the
first.
How was this difference distributed across CTAI units?

\begin{figure}
  \centering
<<whichUnits,fig.height=4,fig.width=6,dependson=c('data','setup')>>=

curricula <- read.csv('~/Box Sync/CT/data/sectionLevelUsageData/RAND_study_curricula.csv',stringsAsFactors=FALSE)
curricula <- subset(curricula,curriculum_name=='algebra i')
curricula$unit <- tolower(curricula$unit)
curricula <- subset(curricula,unit%in%intersect(curricula$unit[curricula$ct=='2007'],curricula$unit[curricula$ct=='2008r1']))
units <- curricula$unit[curricula$ct=='2007']
sectionStats <- read.csv('~/Box Sync/CT/data/sectionLevelUsageData/section_stats_withAbb.csv',stringsAsFactors=FALSE)
UnitName <- sectionStats$unit_name_abb[match(units,sectionStats$unit_id)]
UnitName[units=='inequality-systems-solving'] <- 'Systems of Lin. Ineq.'
UnitName[units=='intro-pythag-theorem'] <- 'Pythagorean Theorem'
UnitName[units=='linear-inequality-graphing'] <- 'Graphs of Lin. Ineq.'
UnitName[units=='linear-systems-solving'] <- 'Systems of Lin. Eq. Solving'
UnitName[units=='probability'] <- 'Probability'
UnitName[units=='unit-conversions'] <- 'Unit Conversions'

nstud <- data%>%filter(!is.na(unit))%>%group_by(Year)%>%summarize(nstud=n_distinct(field_id))
data$Unit <- data$unit
data$Unit[grep('unit-conversions',data$Unit)] <- 'unit-conversions'

unitLevel <- data%>%filter(Unit%in%units)%>%group_by(Unit,Year)%>%summarize(numWorked= n_distinct(field_id,na.rm=TRUE),numCP=sum(status=='changed placement',na.rm=TRUE),meanCP=mean(status=='changed placement',na.rm=TRUE))

unitLevel$perWorked <- unitLevel$numWorked/nstud$nstud[match(unitLevel$Year,nstud$Year)]

unitLevel$Unit <- factor(unitLevel$Unit,levels=units)
levels(unitLevel$Unit) <- UnitName

unitLevel$year <- factor(ifelse(unitLevel$Year=='Year 1',1,2))
print(unitsWorked <- ggplot(unitLevel,aes(x=Unit,y=perWorked,color=year,group=year))+geom_point()+geom_line()+theme(axis.text.x = element_text(angle = 90, hjust = 1,vjust=0.5,size=10),legend.text=element_text(size=10),legend.position=c(.96,.82))+labs(x='',y='% Worked',color='Year')+scale_y_continuous(labels=percent))

@
\caption{The percentages of students with usage data who worked at
  least one problem from each unit in the Algebra I curriculum. The
  units are arranged in order for the standard curriculum. In the 2008
version of the software, the ``Unit Conversions'' unit was broken up
into two smaller units; for the sake of between-year comparisons, we
re-combined them.}
\label{fig:unitsWorked}
\end{figure}


Figure \ref{fig:unitsWorked} shows the units of algebra along the
horizontal axis, according to their order in the standard CTAI
curriculum.
The vertical axis shows the percentage of students with usage data who
worked each unit.

In year 1, the curve is almost monotonically decreasing, as one would
expect if students adhered to the curriculum.
Students varied in the number of units they worked---with the variation due
to both student ability and the amount of time allocated to CTAI
within a classroom---but they mostly followed the standard curriculum.
Students who worked fewer units stopped earlier in the sequence, and those who worked more units progressed farther.
Hence, earlier sections were worked by higher proportions of students
than later units.

In contrast, in year 2 students were much more likely to depart from the
standard unit order.
For instance, Figure \ref{fig:unitsWorked} suggests that some students
skipped ``Unit Conversions'' to work on  ``1st
Quadrant Linear Graphs'' or skipped ``1 step Linear Equations'' to work on ``Independent Variables in Linear Models''
In both these cases, the subsequent unit was worked on by a greater
proportion of students than the immediately prior unit.

Most strikingly, ``Linear Equations with Variables on Both Sides'' was worked by a
greater proportion of students in year 2 than in year 1, and by a
greater proportion of students than any of the previous six sections.
Presumably teachers and administrators wanted students to focus on
that unit, perhaps because they found it to be particularly effective,
because students tend to struggle with its main topic, or because its
topic may figure prominently in an upcoming standardized test.

\begin{figure}
  \centering
<<unitsWorkedCust,dependson=c('data','setup','whichUnits'),fig.height=4,fig.width=6>>=
### by customized curriculum (at school level)
cust <- data%>%filter(Year=='Year 2')%>%group_by(schoolid2,state)%>%summarize(cust=mean(overall=='Customized',na.rm=T))%>%arrange(cust)
data$cust <- ifelse(data$schoolid2%in%cust$schoolid2[cust$cust>0.8],'Customized','Standard')

nstudCust <- data%>%filter(!is.na(unit) & Year=='Year 2')%>%group_by(cust)%>%summarize(nstud=n_distinct(field_id))
unitLevelCust <- data%>%filter(Unit%in%units & Year=='Year 2' )%>%group_by(Unit,cust)%>%summarize(numWorked= n_distinct(field_id,na.rm=TRUE),numCP=sum(status=='changed placement',na.rm=TRUE),meanCP=mean(status=='changed placement',na.rm=TRUE))

unitLevelCust$perWorked <- unitLevelCust$numWorked/nstudCust$nstud[match(unitLevelCust$cust,nstudCust$cust)]

unitLevelCust$Unit <- factor(unitLevelCust$Unit,levels=units)
levels(unitLevelCust$Unit) <- UnitName
ggplot(unitLevelCust,aes(x=Unit,y=perWorked,color=cust,group=cust))+geom_point()+geom_line()+theme(axis.text.x = element_text(angle = 90, hjust = 1,vjust=0.5,size=10),legend.text=element_text(size=10),legend.position=c(.90,.82))+labs(x='',y='% Worked',color='Curriculum',title='Year 2')+scale_y_continuous(labels=percent)
@
\caption{The percentages of year-2 students with usage data who worked at
  least one problem from each unit in the Algebra I curriculum. The
  units are arranged in order for the standard curriculum. Students
  are divided between those attending schools using primarily a customized
  curriculum and those using primarily the standard Algebra I curriculum.}
\label{fig:unitsWorkedCust}
\end{figure}

Most of the variation in unit order was driven by the rise, in year 2,
of customized curricula.
Figure \ref{fig:unitsWorkedCust} divides year-2 students into those
attending schools using primarily a customized curriculum, and those
attending schools using primarily a standardized
curriculum.\footnote{At least
  \Sexpr{trunc(100*min(cust[['cust']][cust[['cust']]>0.8]))}\% of
  problems worked by students at ``Customized'' schools were from a customized curriculum,
  and at most
  \Sexpr{ceiling(100*max(cust[['cust']][cust[['cust']]<0.8]))}\% of
  problems at ``Standard'' schools were from a customized curriculum.}
Students using a standardized curriculum followed the standard
sequence---more or less---while students using customized curricula
did not.
That said, there were some order violations in the standard group:
specifically, more students worked problems from units ``2-step Linear
Equations'' and ``Exponents'' than worked the preceding sections;
this suggests that some teachers used the reassignment tool to
prioritize particular topics.
Of course, teacher reassignment may have occurred in schools with customized
curricula as well---a possibility we will discuss in the next
section.

\begin{figure}
  \centering
<<unitsBySchool,dependson=c('data','setup','whichUnits'),fig.height=7,fig.width=6>>=
### by school
nstudSch <- data%>%filter(!is.na(unit) & Year=='Year 2')%>%group_by(schoolid2,state)%>%summarize(nstud=n_distinct(field_id))
unitLevelSch <- with(filter(data,Unit%in%units & Year=='Year 2'),expand.grid(Unit=unique(Unit),schoolid2=unique(schoolid2)))
unitLevelSch$state <- data$state[match(unitLevelSch$schoolid2,data$schoolid2)]
unitLevelSch$cust <- data$cust[match(unitLevelSch$schoolid2,data$schoolid2)]
unitLevelSch$numWorked <- with(filter(data,Year=='Year 2'),vapply(1:nrow(unitLevelSch),function(i)
    n_distinct(field_id[schoolid2==unitLevelSch$schoolid2[i] & Unit==unitLevelSch$Unit[i]]),1))

#unitLevelSch <- data%>%filter(Unit%in%units & Year=='Year 2')%>%group_by(Unit,schoolid2,cust,state)%>%summarize(numWorked= n_distinct(field_id,na.rm=TRUE),numCP=sum(status=='changed placement',na.rm=TRUE),meanCP=mean(status=='changed placement',na.rm=TRUE))

unitLevelSch$perWorked <- unitLevelSch$numWorked/nstudSch$nstud[match(unitLevelSch$schoolid2,nstudSch$schoolid2)]

unitLevelSch$Unit <- factor(unitLevelSch$Unit,levels=units)
levels(unitLevelSch$Unit) <- UnitName

ggplot(filter(unitLevelSch,state!='NJ'),aes(x=Unit,y=perWorked,color=schoolid2,group=schoolid2,linetype=cust))+geom_point(size=.5)+geom_line()+theme(axis.text.x = element_text(angle = 90, hjust = 1,vjust=0.5,size=10),legend.text=element_text(size=10))+
                                        #,legend.position=c(.96,.82))+
    facet_grid(state~.)+scale_y_continuous(labels=percent)+scale_color_discrete(guide=FALSE)+
    labs(x='',y='% Worked',linetype='Curriculum',title='Year 2')
@
\caption{The percentages of year-2 students with usage data in each
  school who worked at least one problem from each unit in the Algebra I curriculum. The
  units are arranged in order for the standard curriculum. Schools are
  classified as either using primarily customized curricula (solid
  line) or using primarily the standard Algebra I curriculum (dotted).}
\label{fig:unitsBySchool}
\end{figure}

Figure \ref{fig:unitsBySchool} further decomposes the year-2 results
by school and state, showing a large amount of variation between
states, as well as variation between schools within states.
In Texas, every school used customized curricula, most of which seem to
prioritize some of the same units, for instance, ``Linear Patterns,''
``Independent Variables in Linear Models,'' and
``Linear Equations with Variables on Both Sides''
On the other hand, there was also variance between schools.
For instance, one school prioritized units ``2 Step Linear Equations''
and ``4-Quadrant Linear Graphs'' while nearly eliminating ``Linear Patterns.''

Between-school variation is evident in the other states, as well.
In four of the five Kentucky schools, nearly every student worked on
the first nine units; in the one Kentucky school that used a
customized curriculum, nearly every student worked on the first 13
units, omitted the 15th (``Lin. Mod. in General Form''), and worked on
the 16th and 17th (``Literal Equations'' and ``Linear Equations with Variables on Both Sides'').
In the remaining school, nearly every student worked on the first
section, but usage decreased rapidly from there.
In one Michigan school which used the standard curriculum,
no students seem to have worked on the ``Linear Models \& Ratios'' section.

If unit order and topic scaffolding are important to CT's mastery
learning mechanism, the wide variation in students' realized curricula
would seem to pose a problem.
The fact that the prescribed order was followed less in the second
year of the study, when CTAI was effective, than in the first year,
when it wasn't, suggests that the standard curriculum may play a
smaller role than one might otherwise imagine.

\section{Mastering the Material---Or Not}\label{sec:mastery}
The central idea behind mastery learning is that students progress
through the curriculum as they master skills.
In the context of CT, skills are clustered within sections, which are
in turn clustered within units.
Students progress from the current section to the next section after
mastering all of the current section's skills.
Ideally, students would master all of the skills in all of the
sections they work.

By default, the software operates by automatically moving students
from section to section based on the sequence of topics defined by the
curriculum they were currently enrolled in. In this
software-controlled sequencing, students ideally spend the time
necessary to learn the material of a section, are judged by the
software to have mastered the material, and then ``graduate'' to the
next section.
However, the software will also ``promote'' a student to the next
section if the student exhausts a section's material without mastering
its skills.
Additionally, teachers are able to modify a student's path within the curriculum.
They can ``reassign'' students from their current sections to other
sections earlier or later in the intended sequence, including sections
they worked on previously.
Finally, if the semester ends, or a student stops using CT for some
other reason, while in the middle of working through a
section, the section is designated ``final.''
All in all, each CT section a student encounters ends in one of four possible ways:
mastery, promotion, reassignment, or as the student's final section.

\begin{figure}
  \centering
<<overallStatus,dependson=c('data','setup'),fig.height=3,fig.width=6>>=
statusPerSec <- data%>%
    filter(!is.na(Curriculum) &!(state=='MI'&Curriculum=='Customized'&year==1))%>%
    group_by(field_id,Yr,state,section,Curriculum,status)%>%
        summarize(cp=any(status=='changed placement'))%>%group_by(state,Curriculum,Yr,field_id)%>%summarise(pcp=mean(cp,na.rm=TRUE),ncp=sum(cp,na.rm=TRUE))

statusOverall <-  data%>%filter(!is.na(status))%>%
    group_by(field_id,Yr,state,unit,section,Curriculum,overall)%>%summarize(status=max(status))

statusStateYr <- addOverallState(statusOverall) %>% group_by(state,Yr)%>%
    summarize(pgrad=mean(status=='graduated'),pfoi=mean(status=='final_or_incomplete'),
              pcp=mean(status=='changed placement'),pprom=mean(status=='promoted'))%>%
    melt(measure.vars=c('pgrad','pfoi','pcp','pprom'))

levels(statusStateYr$variable) <- list(Final='pfoi',Reassigned='pcp',Promoted='pprom',Mastered='pgrad')


 ggplot(filter(statusStateYr,state!='NJ'),aes(Yr,value,fill=variable))+geom_col()+facet_grid(~state)+labs(y='% of Sections Worked',x='',fill='Exit Status')+scale_y_continuous(labels=percent)

@
\caption{The distributions of outcomes of worked sections, by state and across the
  entire sample, in the two study years.}
\label{fig:overallStatus}
\end{figure}

\begin{table}
  \centering
 \begin{tabular}{rllllll}%|llllll}

&   \multicolumn{6}{c}{Year 1}\\%&\multicolumn{6}{c}{Year 2}\\
<<statusTab,dependson='overallStatus',results='asis'>>=
statusStateYr2 <- addOverallState(statusOverall)
statusStateYr2a <- statusStateYr2%>%group_by(state,Yr)%>%
    summarize(pgrad=sum(status=='graduated'),pfoi=sum(status=='final_or_incomplete'),
              pcp=sum(status=='changed placement'),pprom=sum(status=='promoted'))%>%
    melt(measure.vars=c('pgrad','pfoi','pcp','pprom'))
statusStateYr2b <- statusStateYr2%>%group_by(state,Yr)%>%
    summarize(pgrad=mean(status=='graduated'),pfoi=mean(status=='final_or_incomplete'),
              pcp=mean(status=='changed placement'),pprom=mean(status=='promoted'))%>%
    melt(measure.vars=c('pgrad','pfoi','pcp','pprom'))


levels(statusStateYr2a$variable) <- list(Final='pfoi',Reassigned='pcp',Promoted='pprom',Mastered='pgrad')
levels(statusStateYr2b$variable) <- list(Final='pfoi',Reassigned='pcp',Promoted='pprom',Mastered='pgrad')

tab1 <- dcast(subset(statusStateYr2a,Yr=='Yr 1'&state!='NJ'),variable~state)
tab2 <- dcast(subset(statusStateYr2a,Yr=='Yr 2'&state!='NJ'),variable~state)
#tab <- cbind(tab1,tab2[-1])

tab1b <- dcast(subset(statusStateYr2b,Yr=='Yr 1'&state!='NJ'),variable~state)
tab2b <- dcast(subset(statusStateYr2b,Yr=='Yr 2'&state!='NJ'),variable~state)


tab1$variable <- as.character(tab1$variable)
tab2$variable <- as.character(tab2$variable)
cat('&')
cat(names(tab1)[-1],sep='&')
cat('\\\\')
for(i in 1:nrow(tab1)){
    cat(tab1[i,1],round(unlist(tab1[i,-1])),sep='&')
    cat('\\\\')
}
cat('Total',round(colSums(tab1[,-1])),sep='&')
cat('\\\\')
cat('&\\multicolumn{6}{c}{Year 2}\\\\ \n')
for(i in 1:nrow(tab2)){
    cat(tab2[i,1],round(unlist(tab2[i,-1])),sep='&')
    cat('\\\\')
}
cat('Total',round(colSums(tab2[,-1])),sep='&')
cat('\\\\')

rownames(tab1) <- rownames(tab2) <- tab1$variable
tab1 <- as.matrix(tab1[,-1])
tab2 <- as.matrix(tab2[,-1])

rownames(tab1b) <- rownames(tab2b) <- tab1b$variable
tab1b <- as.matrix(tab1b[,-1])
tab2b <- as.matrix(tab2b[,-1])



@

\end{tabular}
\caption{Numbers of worked sections that ended in each of the
  four possible outcomes, across states and study years.}
\label{tab:overallStatus}
\end{table}

Figure \ref{fig:overallStatus} and Table \ref{tab:overallStatus} show
the proportions of worked sections in each state and study year that
ended with mastery, promotion, or reassignment, or as the
student's final section.
In the first year, about
\Sexpr{round(mean(tab1b['Mastered',c('TX','KY','MI','LA')])*100)}\% of
worked sections are mastered, except in Connecticut.
Other than in Texas, about
\Sexpr{paste(round(range(tab1b['Promoted',c('CT','KY','MI','LA')])*100),collapse='--')}\%
of sections end in promotion.
About \Sexpr{round(tab1b['Reassigned','TX']*100)}\% of sections in Texas
and \Sexpr{round(tab1b['Reassigned','CT']*100)}\%
in Connecticut end in reassignment, which is even rarer in the other states.

With the exception of Texas, sections tended to be completed similarly
in both years.
In Texas, however, the percentage of sections ending in reassignment increased by a factor
of about about
\Sexpr{round(tab2b['Reassigned','TX']/tab1b['Reassigned','TX'])},
to about \Sexpr{round(tab2b['Reassigned','TX']*100)}\%.
The proportion of Texas sections labeled ``Final'' increased as
well---the expected result of decreasing the overall number of worked
sections and holding fixed the likelihood of ending
usage while in the middle of a section.

Across states, sections ended in reassignment at a rate of about
\Sexpr{round(tab1b['Reassigned','Overall']*100)}\% in year 1 and
\Sexpr{round(tab2b['Reassigned','Overall']*100)}\% in year 2.

\subsection{Section Mastery and Curriculum}
\begin{figure}
  \centering
<<statusCur,dependson='overallStatus',fig.height=3,fig.width=6>>=
statusOverall$Curr2 <- with(statusOverall,
                        ifelse(Curriculum=='Algebra I',
                                     ifelse(overall=='Standard','Algebra I','Algebra I (Cust.)'),
                                     as.character(Curriculum)))
statusCurr <- statusOverall%>%filter(!is.na(Curriculum))%>% group_by(Curr2,Yr)%>%
    summarize(pgrad=mean(status=='graduated'),pfoi=mean(status=='final_or_incomplete'),
              pcp=mean(status=='changed placement'),pprom=mean(status=='promoted'))%>%
    melt(measure.vars=c('pgrad','pfoi','pcp','pprom'))
levels(statusCurr$variable) <- list(Final='pfoi',Reassigned='pcp',Promoted='pprom',Graduated='pgrad')
statusCurr$Curr2 <-
    factor(statusCurr$Curr2,levels=c('Bridge-to-Algebra','Algebra I','Algebra I (Cust.)','>Algebra I'))
levels(statusCurr$Curr2)[1] <- 'Bridge-to-Alg.'
print(ggplot(statusCurr,aes(Yr,value,fill=variable))+geom_col()+facet_grid(~Curr2)+
      labs(y='% of Sections Worked',x='',fill='Exit Status')+scale_y_continuous(labels=percent))

@
\caption{The distributions of outcomes of worked sections, by
  curriculum, in the two study years. (There were no Bridge-to-Algebra
sections in customized curricula in our dataset.)}
\label{fig:statusCur}
\end{figure}

A well-designed curriculum, can, in theory, play an important role in
students' attainment of mastery.
Students who work on appropriate problems that build on their current
set of skills should be more likely to master new skills than students
working on problems above their level.
What role did variations in the CT curriculum play in mastery during the effectiveness trial?

Figure \ref{fig:statusCur} shows the proportions of worked sections
that were mastered or ended in promotion, reassignment, or finality, in
standard and customized versions of each CT curriculum.
Mastery proportions do, indeed, depend on curriculum.
Specifically, students mastered sections from more advanced curricula
less frequently.
Sections from the most basic curriculum, Bridge to Algebra, were
mastered
\Sexpr{round(
  mean(statusOverall$status[statusOverall$Curriculum=='Bridge-to-Algebra']=='graduated',na.rm=T)*100)}\%
of the time;
those from Algebra I were mastered
\Sexpr{round(
  mean(statusOverall$status[statusOverall$Curriculum=='Algebra I']=='graduated',na.rm=T)*100)}\%
of the time, and those from more advanced curricula were mastered
at a rate of
\Sexpr{round(
  mean(statusOverall$status[statusOverall$Curriculum=='>Algebra I']=='graduated',na.rm=T)*100)}\%.
This is unsurprising, since more advanced curricula may be expected
to be more challenging.
However, it may suggest that some students studying advanced topics
would fare better in more standard curricula.

Algebra I sections from customized curricula tended to end in
reassignment more often than sections from the standard Algebra I
curriculum (\Sexpr{round(100*subset(statusCurr,Curr2=='Algebra I (Cust.)' &
  variable=='Reassigned')[['value']])}\% vs.
\Sexpr{round(100*subset(statusCurr,Curr2=='Algebra I' &
  variable=='Reassigned' & Yr=='Yr 2')[['value']],1)}\%, in year 2).
This may indicate an overall skepticism towards the Carnegie Learning standards
among certain schools and teachers, manifested in both adoption of
alternative curricula and reassignment.%\marginpar{does that make more sense?}


\section{Digging Deeper into Section Reassignment}\label{sec:cp}
The proportion of worked sections in our dataset ending in
reassignment was small.
Nevertheless, since reassignment represents the only mechanism by
which individual teachers can affect their students' progress through
the Cognitive Tutor, exploring patterns of reassignment can provide
insight into how CT was used.

<<cpDat,dependson=c('data','setup'),include=FALSE>>=
secLev <- data%>%filter(is.finite(status) & is.finite(timestamp) &is.finite(date))%>%
    group_by(field_id,unit,section,Year,Yr,classid2,schoolid2)%>%
    summarize(startDate=min(date),endDate=max(date),startTime=min(timestamp),endTime=max(timestamp),state=state[1],
              status=max(status),Curriculum=Curriculum[1],overall=overall[1],version=version[1])%>%
    arrange(endDate)

### cp over time
secLev <- within(secLev,endMonth <- factor(month(secLev$endDate,TRUE,TRUE),levels=c('Aug','Sep','Oct','Nov','Dec','Jan','Feb','Mar','Apr','May','Jun','Jul')))

@

\subsection{How Do Reassignment Patterns Vary?}

\begin{figure}
  \centering
<<vcs,fig.height=3,fig.width=6.5,dependson=c('data','setup')>>=
#load('vcMods.RData')
library(lme4)

vcDat <- data%>%filter(unit%in%unique(data$unit[data$curriculum=='Algebra I']))%>%group_by(field_id,unit,section,classid2,state)%>%summarize(status=max(status,na.rm=TRUE),startDate=min(date,na.rm=TRUE))
vcDat <- merge(vcDat,stud,all.x=TRUE,all.y=FALSE)

vcDat$month <- months(vcDat$startDate,TRUE)

vcDat$cp <- vcDat$status=='changed placement'
vcDat$mast <- vcDat$status=='graduated'

vcModYr <- list(yr1=glmer(cp~(1|field_id)+(1|classid2)+(1|schoolid2)+(1|state)+(1|unit),family=binomial,data=subset(vcDat,year==1)),yr2=glmer(cp~(1|field_id)+(1|classid2)+(1|schoolid2)+(1|state)+(1|unit),family=binomial,data=subset(vcDat,year==2)))

vcMods <- list()
for(st in c('TX','KY','MI'))
 for(yr in c(1,2))
     vcMods[[paste0(st,'_',yr)]] <- glmer(cp~(1|field_id)+(1|classid2)+(1|schoolid2)+(1|unit),family=binomial,data=subset(vcDat,state==st & year==yr))


vcFun <- function(nm){
    mod <- vcMods[[nm]]
    out <- unlist(summary(mod)$varcor)
    out <- data.frame(sig2=out,comp=names(out),stringsAsFactors=FALSE)
    out <- rbind(out,data.frame(sig2=pi^2/3,comp='resid'))
    out$state <- strsplit(nm,'_')[[1]][1]
    out$year <- strsplit(nm,'_')[[1]][2]
    out$sig2 <- out$sig2/sum(out$sig2)
    out
}

vcDat <- do.call('rbind',lapply(names(vcMods),vcFun))

yr1 <- data.frame(sig2=c(unlist(summary(vcModYr[[1]])$varcor),pi^2/3),
                          comp=c(names(summary(vcModYr[[1]])$varcor),'resid'),
                          state='Overall',
                  year='1')
yr1$sig2 <- yr1$sig2/sum(yr1$sig2)

yr2 <- data.frame(sig2=c(unlist(summary(vcModYr[[2]])$varcor),pi^2/3),
                          comp=c(names(summary(vcModYr[[2]])$varcor),'resid'),
                          state='Overall',
                  year='2')
yr2$sig2 <- yr2$sig2/sum(yr2$sig2)

vcDat <- rbind(vcDat,yr1,yr2)

vcDat$comp <- factor(vcDat$comp)
levels(vcDat$comp)=list(State='state',School='schoolid2',Class='classid2',Student='field_id',Unit='unit',Residual='resid')


vcDat$state <- factor(vcDat$state,levels=c('TX','KY','MI','Overall'))

vcs <- list()
for(s in unique(vcDat$state)) for(y in 1:2){
 ddd <- round(subset(vcDat,state==s&year==y,select=c(sig2))*100)
 rownames(ddd) <- gsub('\\d','',rownames(ddd))
 rownames(ddd)[nrow(ddd)] <- 'resid'
 vcs[[paste0(s,'_',y)]] <- ddd
}


ggplot(vcDat,aes(year,sig2,fill=comp))+geom_col()+facet_grid(~state)+scale_fill_manual(values=rev(c('white','grey','#e41a1c','#377eb8','#4daf4a','#984ea3')))+labs(x='',y='% Variance Explained',fill='',title='Model: Multilevel Logistic Unconditional')+scale_y_continuous(labels=percent)



@
\caption{Results from a set of eight multilevel logistic regressions
  predicting section reassignment. For each year, in the entire sample (``Overall'')
  and in the three states with the largest numbers of reassignments (Texas,
  Kentucky, and Michigan), we regressed a binary variable indicating
  whether a section ended in reassignment on random intercepts for
  school, class, student, and unit, and in the overall case, for state
  as well, and recorded their variance. The residual variance was set
  as the variance of the standard logistic distribution,
  $\pi/3$. These bar charts give the proportion of the total variance
  attributable to each random effect.}
\label{fig:vc}
\end{figure}
Teachers alone control reassignment.
Nevertheless, the factors influencing student reassignment vary at a
number of levels.
For instance, state and district standards may prod teachers into
reassigning students to particular units.
Some principals may encourage teachers to adhere to the official
curriculum and avoid reassignment.
Some students may be more prone to reassignment than others.
Certain units in the CTAI curriculum may be harder than others,
causing students to tarry and teachers to reassign.
Finally, a host of other factors, at these levels and others, may spur reassignment.

To better understand the source of the variation in
reassignment---what drives some, but not other, sections worked by
students to end in
reassignment---we fit a set of multilevel models.
We fit separate models to data from each the three states with the highest numbers of reassignments, Texas,
Kentucky, and Michigan, and in the sample as a whole, in each of the
two study years, yielding a total of eight models.
Each model was a logistic regression: a binary indicator for section
reassignment was regressed on a random intercept for unit, as well as
nested random intercepts for student, classroom, and school.
Models fit to data from all six states included an additional random
intercept for state.

Logistic regression can be represented in terms of an underlying
latent variable $Z^*$: student $i$ working section $sec$ is reassigned
when $Z_{sec,i}^*>0$.
The model for $Z^*$ is:
\begin{equation*}
 Z^*_{sec,i}=\alpha_0+\beta_{u[sec]}+\gamma_i+\delta_{c[i]}+\epsilon_{s[i]}+e_{sec,i}
\end{equation*}
Where $\alpha_0$ is an overall intercept,
and $\beta_{u[sec]}$, $\gamma_i$, $\delta_{c[i]}$, and $\epsilon_{s[i]}$
are random intercepts for the unit in which $sec$ appears, for student
$i$, for $i$'s classroom, and for $i$'s school, respectively.
Again, the model fit to all six states includes an additional random
intercept for state.
The random intercepts are modeled as independent and normally
distributed, each with its own variance.
The regression error $e_{sec,i}$ is given the standard logistic
distribution, with ``residual'' variance $\pi/3$.
It is convenient to represent variance in reassignment probabilities
in terms of the variance of $Z^*$.

Figure \ref{fig:vc} gives the variance components estimated from these
logistic regressions: variances of the random intercept terms, as a
percentage of the total variance of $Z^*$.
Overall, in both years of the study, the largest determinant of
reassignment was school, accounting for
\Sexpr{vcs[['Overall_1']]['schoolid','sig2']}\% of the variation in year 1,
and \Sexpr{vcs[['Overall_2']]['schoolid',1]}\% in year 2.
After school, state was the most important, accounting for
\Sexpr{vcs[['Overall_1']]['state',1]}\% and
\Sexpr{vcs[['Overall_2']]['state',1]}\% in the two years, and unit,
accounting for \Sexpr{vcs[['Overall_1']]['unit',1]}\% and \Sexpr{vcs[['Overall_2']]['unit',1]}\%.
Surprisingly, classroom and student-level factors only accounted for
\Sexpr{vcs[['Overall_1']]['classid',1]}\% and
\Sexpr{vcs[['Overall_1']]['field_id',1]}\% in year 1, respectively,
and \Sexpr{vcs[['Overall_2']]['classid',1]}\% and
\Sexpr{vcs[['Overall_2']]['field_id',1]}\% in year 2.
The pattern was similar in Texas---where school accounted for over half
the variation in reassignment in both years---and in Michigan to a
lesser extent.\footnote{Percentages in state specific models, in which
  there is no between-state variance, cannot be
  directly compared to those from the overall model.}
In Kentucky, unit played the largest role
(\Sexpr{vcs[['KY_1']]['unit',1]}\%) in year 1, and classroom played
the largest role in year 2 (\Sexpr{vcs[['KY_2']]['classid',1]}\%).
Across states and years, student level factors never accounted for
more than \Sexpr{max(sapply(vcs,function(x) x['field_id',1]))}\% of
the variation in reassignment.
Other than in Kentucky in year 2, classroom never accounted for more
than \Sexpr{sort(sapply(vcs,function(x) x['classid',1]),dec=T)[2]}\%  of
the variation.

\textbf{Summary.} Although teachers control reassignment, their decisions appear to be largely
determined by broader policies, occurring at the state or school level.


\subsection{When are Students Reassigned?}

The timing of reassignments can also provide a window into what drives
teachers' decisions to reassign students.
Figure \ref{fig:byMonth} shows the proportion of worked sections in
each month that end in reassignment.
In both years, reassignments were much more common in the second half
of the school year than in the first.
This may be the result of teachers learning how to use the software as
the year progresses, or responding the pressure of upcoming
standardized tests by accelerating students' progress and reassigning
students to relevant sections.

As we've seen, reassignment was more common in year 2 than in year 1.
In fact, reassignment increases fairly steadily over the entire length
of the study.
Through December of the first year, reassignment was rare. From
January through May of year 1, between one and two percent of sections
ended in reassignment.
Year 2 begin where year 1 left off, with one to two percent of
sections reassigned.
Finally, from February through May of the second year, the rate of
reassignment increased again.



\begin{figure}
  \centering
<<byMonth,dependson='cpDat',fig.height=3,fig.width=6>>=
secLevMonth <- secLev%>%group_by(endMonth,Year)%>%
 summarize(nsec=n(),CPper=mean(status=='changed placement',na.rm=TRUE))
secLev$cp <- as.numeric(secLev$status=='changed placement')

ggplot(filter(secLevMonth,nsec>100),aes(endMonth,CPper,group=Year,color=Year,size=nsec))+geom_point()+geom_line(size=1)+
    ## geom_smooth(aes(as.numeric(endMonth)+day(endDate)/31-0.5,cp,group=Year,
    ##                 color=Year,size=1),
    ##             method = "glm", formula = y ~ splines::bs(x, 4),data=secLev,method.args=list(family='binomial'),
                                        #            show.legend=FALSE)+
scale_y_continuous(breaks=seq(0,0.05,.01),labels=percent)+
    coord_cartesian(ylim=c(0,0.05))+
    labs(x='Month',y='% of Sections Ending in Reassignment',size='# Worked\nSections',color='')


@
\caption{The proportion of worked sections ending in reassignment, by
  month and year.}
\label{fig:byMonth}
\end{figure}


\begin{table}
  \centering
  \begin{tabular}{rll|ll}
    &\multicolumn{2}{c}{Year 1}&\multicolumn{2}{c}{Year 2}\\
    &Aug--Dec&Jan--Jun&Aug--Dec&Jan--Jun\\
<<byMonthTab,results='asis'>>=

tab <- NULL
for(st in c('TX','KY','MI','LA','CT','NJ')){
 tab <- rbind(tab,
  c(sum(secLev$status[secLev$state==st & secLev$Yr=='Yr 1'&
     secLev$endMonth%in%c('Aug', 'Sep', 'Oct', 'Nov', 'Dec')]=='changed placement'),
    sum(secLev$status[secLev$state==st & secLev$Yr=='Yr 1'&
     secLev$endMonth%in%c('Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun')]=='changed placement'),
    sum(secLev$status[secLev$state==st & secLev$Yr=='Yr 2'&
     secLev$endMonth%in%c('Aug', 'Sep', 'Oct', 'Nov', 'Dec')]=='changed placement'),
    sum(secLev$status[secLev$state==st & secLev$Yr=='Yr 2'&
     secLev$endMonth%in%c('Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun')]=='changed placement')))
}
tab <- rbind(tab,
    c(sum(secLev$status[secLev$Yr=='Yr 1'&
     secLev$endMonth%in%c('Aug', 'Sep', 'Oct', 'Nov', 'Dec')]=='changed placement'),
    sum(secLev$status[ secLev$Yr=='Yr 1'&
     secLev$endMonth%in%c('Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun')]=='changed placement'),
    sum(secLev$status[ secLev$Yr=='Yr 2'&
     secLev$endMonth%in%c('Aug', 'Sep', 'Oct', 'Nov', 'Dec')]=='changed placement'),
    sum(secLev$status[ secLev$Yr=='Yr 2'&
     secLev$endMonth%in%c('Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun')]=='changed placement')))
rownames(tab) <- c('TX','KY','MI','LA','CT','NJ','Overall')

print(xtable(tab),only.contents=TRUE,include.colnames=FALSE,hline.after=6)
@
\hline
\end{tabular}
\caption{The number of reassignments in each state and overall in the
  first and second halves of the year}
\label{tab:byMonth}
\end{table}

Figure \ref{fig:byMonthState} and Table \ref{tab:byMonth} decompose
these trends by state, Figure \ref{fig:byMonthState} includes just
the three states with the highest number of reassignments, Texas,
Kentucky, and Michigan. Table \ref{tab:byMonth} shows data for each of
the six states and overall.
For the two study years, Figure \ref{fig:byMonthState} shows the proportion of all of each
state's reassignments the occurred in each month of the school year.
(Note that while both Figure \ref{fig:byMonth} and \ref{fig:byMonthState}
show proportions for each month, the denominators are not the same:
Figure \ref{fig:byMonth} gives the proportion of each month's worked
sections that ended in reassignment, and Figure \ref{fig:byMonthState}
gives the proportion of all of the state's reassignments over the
course of the year, that occurred in each month.)

Figure \ref{fig:byMonthState} reveals patterns that are not apparent
in Figure 10.
For example, the bulk of Kentucky's reassignments took place in May.
A fifth of Michigan's reassignments in year 1 occurred in October;
for the rest of the year, Michigan roughly followed the same pattern as Texas.
In year 2, the vast majority of Michigan's reassignments also occurred near
the beginning of the year, in September and October.


\begin{figure}
  \centering
<<byMonthState,dependson='cpDat',fig.height=3,fig.width=6>>=

secLevMonthSt <- secLev%>%filter(status=='changed placement'&state%in%c('TX','KY','MI') & endMonth!='Jul')%>%
 group_by(endMonth,Year,state)%>%summarize(cpPer=n())
secLevNum <- secLev%>%filter(status=='changed placement'&state%in%c('TX','KY','MI'))%>%
 group_by(Year,state)%>%summarize(ncp=n())

for(st in c('TX','KY','MI')) for(yr in c('Year 1','Year 2'))
 secLevMonthSt$cpPer[secLevMonthSt$state==st & secLevMonthSt$Year==yr] <-
  secLevMonthSt$cpPer[secLevMonthSt$state==st & secLevMonthSt$Year==yr]/
   secLevNum$ncp[secLevNum$state==st & secLevNum$Year==yr]


## secLevMonthSt <- secLev%>%filter(state%in%c('TX','KY','MI') & endMonth!='Jul')%>%group_by(endMonth,state,Year)%>%summarize(nsec=n(),CPper=mean(status=='changed placement',na.rm=TRUE))
ggplot(secLevMonthSt,aes(endMonth,cpPer,group=state,color=state))+geom_point()+facet_grid(Year~.)+
    geom_line(size=1)+
    scale_y_continuous(labels=percent)+
    labs(x='Month',y='% of State\'s Total Reassignments',color='')


@
\caption{The proportion of a state's reassignments that occurred in
  each month of the year.}
\label{fig:byMonthState}
\end{figure}

\textbf{Summary.} In Texas and overall, reassignments were more common
in the second half of the school year than in the first half. In
Kentucky and Michigan, they were clustered in a few specific months.

\subsection{Does Reassignment Depend on Classmates?}

Student individuality and independence might be the most important
motivating factors behind mastery learning---each student learns at
his or her own pace, and struggles on a unique set of skills.
Students are supposed to move through the CT curriculum independently
of each other.
However, reassignments give teachers the ability to override this
feature, and coordinate students' progress.
Teachers might identify students who are behind their classmates and
reassign them to later sections.
They may also move an entire class together to a particular section or
unit of interest.
To what extent did these and similar considerations drive reassignment
in the CTAI study?


\begin{figure}
  \centering
<<classmates,dependson='cpDat',fig.height=6,fig.width=7>>=
secLev$unitSectionClass <- paste(secLev$unit,secLev$section,secLev$classid2)
cpDat <- subset(secLev,status=='changed placement' & Curriculum=='Algebra I')
cpClasses <- unique(cpDat$unitSectionClass)
secLevCP <- filter(secLev,unitSectionClass%in%cpClasses)
levels(secLevCP$status) <- list(promGrad=c('promoted','graduated','final_or_incomplete'),cp='changed placement')
secLevCPsplit <- split(secLevCP[,c('field_id','endDate','status')],secLevCP$unitSectionClass)


ncp <- nrow(cpDat)
seqDat <- matrix(NA,nrow=ncp,ncol=6)
for(i in 1:ncp){
    #if(i%%100==0) cat(round(i/ncp*100,2),'% ',sep='')
    cls <- secLevCPsplit[[cpDat$unitSectionClass[i]]]
    if(nrow(cls)==1){
        seqDat[i,] <- 0
        next
    }
    cls <- filter(cls,field_id!=cpDat$field_id[i])
    date <- cpDat$endDate[i]
    ord <- factor(ifelse(cls$endDate<date,'bf',
                  ifelse(cls$endDate==date,'same','after')),levels=c('bf','same','after'))
    clsSplit <- split(cls,list(ord,cls$status))
    seqDat[i,] <- vapply(clsSplit,nrow,1)
}
cat('\n')
colnames(seqDat) <- names(clsSplit)

classSize <- data%>%filter(!is.na(section))%>%group_by(classid2)%>%summarize(nstud=n_distinct(field_id))
cpDat$totClassmates <- classSize$nstud[match(cpDat$classid2,classSize$classid2)]

for(nn in names(clsSplit)) cpDat[[nn]] <- seqDat[,nn]/cpDat$totClassmates
cpDat$nw <- (cpDat$totClassmates-rowSums(seqDat))/cpDat$totClassmates
cpDat$total <- rowSums(seqDat)
cpDat$bf.promGrad <- cpDat$bf.promGrad+cpDat$same.promGrad
cpDat$same.promGrad <- NULL

### plot them all

mmm <- with(subset(cpDat,totClassmates>15),max(table(state,Year)))

cpDat2 <- cpDat%>%filter(totClassmates>15)%>%arrange(bf.promGrad,bf.cp,same.cp,after.cp,after.promGrad)%>%group_by(Year,state)%>%mutate(cpid=seq(floor((mmm-n())/2)+1,floor((mmm-n())/2)+n()))%>%ungroup()

aaa <- melt(cpDat2,measure.vars=c('bf.promGrad','bf.cp','same.cp','after.cp','after.promGrad','nw'))
levels(aaa$variable) <- list(
    `Never Worked\n Section`='nw',
    `Mastered/Promoted\n Later`='after.promGrad',
    `Reassigned\n Later`='after.cp',
    `Reassigned\n Same Day`='same.cp',
    `Reassigned\n Before`='bf.cp',
    `Mastered/Promoted\n Before`='bf.promGrad')

behind <- with(cpDat2,mean(bf.promGrad+bf.cp+same.cp>.75))
behind1 <- with(subset(cpDat2,Yr=='Yr 1'),mean(bf.promGrad+bf.cp+same.cp>.75))
behind2 <- with(subset(cpDat2,Yr=='Yr 2'),mean(bf.promGrad+bf.cp+same.cp>.75))

behind1tx <- with(subset(cpDat2,Yr=='Yr 1'&state=='TX'),mean(bf.promGrad+bf.cp+same.cp>.75))
behind1ky <- with(subset(cpDat2,Yr=='Yr 1'&state=='KY'),mean(bf.promGrad+bf.cp+same.cp>.75))
behind1mi <- with(subset(cpDat2,Yr=='Yr 1'&state=='MI'),mean(bf.promGrad+bf.cp+same.cp>.75))

behind2tx <- with(subset(cpDat2,Yr=='Yr 2'&state=='TX'),mean(bf.promGrad+bf.cp+same.cp>.75))
behind2ky <- with(subset(cpDat2,Yr=='Yr 2'&state=='KY'),mean(bf.promGrad+bf.cp+same.cp>.75))
behind2mi <- with(subset(cpDat2,Yr=='Yr 2'&state=='MI'),mean(bf.promGrad+bf.cp+same.cp>.75))

mi2cp <- round(with(subset(cpDat2,Yr=='Yr 2'&state=='MI'),mean(bf.cp+same.cp+after.cp+nw>.75))*100)
ky1cp <- round(with(subset(cpDat2,Yr=='Yr 1'&state=='KY'),mean(bf.cp+same.cp+after.cp+nw>.75))*100)

#aaa$variable <- factor(aaa$variable,levels=rev(c('pGrad','pProm','pCPbf','pCPsame','pLater','pNW')))
ggplot(filter(aaa,state%in%c('TX','KY','MI')),aes(as.factor(cpid),value,fill=variable))+geom_col()+facet_grid(Year~state)+theme(axis.title.x=element_blank(),
        axis.text.x=element_blank(),
        axis.ticks.x=element_blank()) + scale_y_continuous(label=percent)+labs(fill='% of Classmates')+
    scale_fill_manual(values=c('#1b9e77','#d95f02','#7570b3','#e7298a','#66a61e','#e6ab02'))+
    labs(y='')

@
\caption{Distributions of classmates' statuses at each reassignment in
  Texas, Kentucky, and Michigan. Each reassignment that took place in
  these three states is represented by a bar showing the proportions of
  the reassigned student's classmates who had exited the section on
  the same day or earlier via promotion or mastery, who had been
  reassigned from that section on an earlier date, on the same date,
  or on a later date, who mastered the section or were promoted on a
  later date, or who never worked the section at all. The bars are
  rank ordered according to those proportions, in the order listed.}
\label{fig:classmates}
\end{figure}

Figure \ref{fig:classmates} addresses this question by plotting a
students' classmates' statuses at the time he or she is reassigned.
For each reassignment in Texas, Kentucky, and Michigan, Figure \ref{fig:classmates}
plots a vertical bar colored to show the proportions of the reassigned
students' classmates (represented in the usage data) who  had exited the section on
the same day or earlier via promotion or mastery, who had been
reassigned from that section on an earlier date, on the same date,
or on a later date, who mastered the section or were promoted on a
later date, or who never worked the section at all.
The bars are rank ordered according to those same proportions: first by the
proportion of classmates who were promoted or mastered the section on
the same day as the reassignment in question, or earlier, next by the
proportion who were reassigned from the same section on an earlier
date, next by the proportion who were reassigned from the same section
on the same date, and finally by the proportion who were reassigned
from the same section on a later date.

Do teachers reassign students in order to help them catch up with classmates?
According to Figure \ref{fig:classmates}, that might be part of the
story, but isn't all of it.
Across years and states, in about \Sexpr{round(behind*100)}\% of
reassignments, at least 75\% of the rest of the class had exited the
section through graduation or promotion on the same date or earlier.
The figure also reveals that the proportions of students who had graduated or been
promoted from the same section on the same day or earlier was smaller in year 2
than in year 1, especially in Texas and Michigan.

In Texas, year 2 saw a dramatic increase in the proportions of
students reassigned from the same section on the same day, suggesting
that some teachers may have been moving the class together through the
curriculum.
In Michigan in year 2, teachers reassigned almost all students who
worked certain sections.
That is, in \Sexpr{mi2cp}\% of the instances in which year-two Michigan
students were reassigned from a section, at least 75\% of their
classmates were, at some point, reassigned from the same section, or
never worked it.
Across all three states and both years, it is exceedingly rare for
students to master or be promoted from a section after someone in
their class has been reassigned from the same section.

\textbf{Summary.} Three different patterns emerge from Figure
\ref{fig:classmates}: teachers reassigning students who have fallen
behind their classmates, teachers reassigning an entire class from the
same section on the same day, and teachers reassigning almost all
students who begin to work on particular sections.
Each of these patterns mostly takes place in different states and
years.

\subsection{Where To?}

Teachers who reassign students may simply move them to the next
section within the same unit.
Say that a teacher believes that a particular student had already mastered
the skills in one of the CTAI sections or is wheel-spinning---working
problems without learning---or a teacher dislikes one of the sections in a CT unit.
Still, the teacher wants the student to learn as much as possible from
the current unit.
Then moving the student to the next section within the same unit might
make sense.
Alternatively, teachers who believe that some of their students are
not progressing quickly enough may reassign them out of their current
units entirely, and into the next unit in the sequence---whatever that
may be.
Finally, a teacher who wanted his or her students to focus on a
particular topic might reassign them all to the appropriate unit
when the time is appropriate.

Which of these patterns is most prevalent?
More generally, when teachers reassign their students, where in the
curriculum do they send them?



<<secOrder,include=FALSE,dependson=c('data','setup')>>=
#### what's the next section each student works?
secOrder <- data%>%filter(is.finite(status) & is.finite(timestamp))%>%group_by(field_id,section,unit,Curriculum,overall,year,Year,Yr,state,classid2,schoolid2)%>%summarize(time=max(timestamp),status=max(status))%>%arrange(time)%>%group_by(field_id,year,Year,Yr,state,classid2,schoolid2)%>%mutate(prevSec=c(NA,section[-n()]),prevStatus=c(NA,status[-n()]),prevUnit=c(NA,unit[-n()]),nextUnit=c(unit[-1],NA))

secOrder$cp <- secOrder$status=='changed placement'
secOrder$mast <- secOrder$status=='graduated'


secOrderCP <- subset(secOrder,status=='changed placement')


secOrderCP$Unit <- secOrderCP$unit
secOrderCP$Unit[grep('unit-conversions',secOrderCP$Unit)] <- 'unit-conversions'

secOrderCP$unitName <- factor(secOrderCP$Unit,levels=units)
levels(secOrderCP$unitName) <- UnitName

secOrderCP$nextUnit[grep('unit-conversions',secOrderCP$nextUnit)] <- 'unit-conversions'

secOrderCP$nextUnitName <- factor(secOrderCP$nextUnit,levels=units)
levels(secOrderCP$nextUnitName) <- UnitName

@



\begin{figure}
  \centering
  \begin{subfigure}{5in}
<<transition1,fig.height=3,fig.width=5,dependson='secOrder'>>=

trans1 <- with(filter(secOrderCP,year==1),table(unitName,nextUnitName))
trans1o <- trans1

## top senders and receivers
cols <- which(colSums(trans1)>30)
rows <- which(rowSums(trans1)>30)

cols2 <- cols
rows2 <- rows

## top 2 receivers for each sender, vice versa
for(r in rows)
    while(sum(trans1[r,cols2])/sum(trans1[r,])<0.75){
    nextBest <- which.max(trans1[r,seq(ncol(trans1))[-cols2]])
    cols2 <- c(cols2,which(colnames(trans1)==names(nextBest)))
}

for(cc in cols){
    while(sum(trans1[rows2,cc])/sum(trans1[,cc])<0.75){
        nextBest <- which.max(trans1[seq(nrow(trans1))[-rows2],cc])
        rows2 <- c(rows2,which(rownames(trans1)==names(nextBest)))
    }

}

cols <- sort(unique(c(cols2,rows2)))
rows <- sort(unique(rows2))

trans1 <- trans1[rows,cols]

propTotCP1 <- sum(trans1)/sum(secOrderCP$year==1)

uuu <- levels(secOrderCP$unitName)
uuu <- uuu[uuu%in%c(colnames(trans1),rownames(trans1))]
nsec <- length(uuu)

par(mar=c(.1,.1,.1,.1))
plot(1,1,xlim=c(-.5,3.5),ylim=c(0,nsec+2),cex=0,xaxt='n',yaxt='n',xlab='',ylab='')
text(1,nsec+1.5,expression(underline(bold(paste(From)))),adj=c(1,NA))
text(2,nsec+1.5,expression(underline(bold(paste(To)))),adj=c(0,NA))

for(i in 1:nsec) if(uuu[i]%in%rownames(trans1))
  text(1,nsec-i+1,paste0(uuu[i],' (',which(levels(secOrderCP$unitName)==uuu[i]),')'),adj=c(1,NA),cex=0.75)
for(i in 1:nsec) if(uuu[i]%in%colnames(trans1))
  text(2,nsec-i+1,paste0(uuu[i],' (',which(levels(secOrderCP$unitName)==uuu[i]),')'),adj=c(0,NA),cex=0.75)

for(rr in rownames(trans1))
 for(cc in colnames(trans1)){
#     if(trans1[rr,cc]>20){
      arrows(1.1,nsec-which(uuu==rr)+1,1.9,nsec-which(uuu==cc)+1,lwd=5*trans1[rr,cc]/sum(trans1o[rr,]),
       col=rgb(0,0,0,alpha=ifelse(trans1[rr,cc]>50,1,trans1[rr,cc]/50)))
}


@
\end{subfigure}
\begin{subfigure}{1in}
  \centering
<<trans1legend,fig.width=1,fig.height=3,dependson='transition1'>>=
par(mar=c(.1,.1,.1,.1))
plot(1,1,xlim=c(0,2),ylim=c(0,nsec+2),cex=0,xaxt='n',yaxt='n',xlab='',ylab='')
conv <- (nsec+2)/11
text(1,nsec+1.5,expression(underline(bold(paste(Legend)))))
text(1,9*conv,'Thickness:\n Percent',cex=0.75)
text(.5,7.5*conv,'25%')
text(.5,6.5*conv,'50%')
text(.5,5.5*conv,'75%')
arrows(c(1,1,1),c(7.5,6.5,5.5)*conv,c(2,2,2),lwd=5*c(.25,.5,.75))
text(1,4.2*conv,'Darkness:\n Number',cex=0.75)
text(.5,3*conv,'10')
text(.5,2*conv,'25')
text(.5,1*conv,'50+')
arrows(c(1,1,1),c(3,2,1)*conv,c(2,2,2),col=rgb(0,0,0,alpha=c(1/5,.5,1)))

@
\end{subfigure}
\caption{Reassignment transition plot for study year 1. The units on
  the left of the plot are the Algebra I units that ended in reassignment at
  least 30 times in year 1. The units on the right are those that were
  the destination of at least 30 reassignments. Also included on the
  left are the top two sending units for each of the units on the
  right, and also included on the right are all the units on the left,
  along with those units' top two receiving units. The units are numbered
  according their order in the standard Algebra I curriculum. There is an arrow from a
  unit on the left to a unit on the right if a student was assigned
  from the unit on the left to the one on the right. The thickness of
  the arrows is proportional to the percentage of all reassignments from
  the sending unit that ended in that receiving unit. The darkness of
  the arrows is proportional to the number of reassignments from the
  sending unit to the receiving unit.}
\label{fig:trans1}
\end{figure}


\begin{figure}
  \centering
  \begin{subfigure}{5in}
<<transition2,fig.height=5,fig.width=5,dependson='secOrder'>>=

trans2 <- with(filter(secOrderCP,year==2),table(unitName,nextUnitName))
trans2o <- trans2

## top senders and receivers
cols <- which(colSums(trans2)>30)
rows <- which(rowSums(trans2)>30)

cols2 <- cols
rows2 <- rows

## top 2 receivers for each sender, vice versa
for(r in rows)
    while(sum(trans2[r,cols2])/sum(trans2[r,])<0.75){
    nextBest <- which.max(trans2[r,seq(ncol(trans2))[-cols2]])
    cols2 <- c(cols2,which(colnames(trans2)==names(nextBest)))
}

for(cc in cols){
 #   print('new col')
  #  print(cc)
    while(sum(trans2[rows2,cc])/sum(trans2[,cc])<0.75){
        nextBest <- which.max(trans2[seq(nrow(trans2))[-rows2],cc])
        rows2 <- c(rows2,which(rownames(trans2)==names(nextBest)))
    }

}

cols <- sort(unique(c(cols2,rows2)))
rows <- sort(unique(rows2))

trans2 <- trans2[rows,cols]


propTotCP2 <- sum(trans2)/sum(secOrderCP$year==1)

nsec <- nrow(trans2)
uuu <- levels(secOrderCP$unitName)
uuu <- uuu[uuu%in%c(colnames(trans2),rownames(trans2))]
nsec <- length(uuu)

par(mar=c(.1,.1,.1,.1))
plot(1,1,xlim=c(-.5,3.5),ylim=c(0,nsec+2),cex=0,xaxt='n',yaxt='n',xlab='',ylab='')
text(1,nsec+1.5,expression(underline(bold(paste(From)))),adj=c(1,NA))
text(2,nsec+1.5,expression(underline(bold(paste(To)))),adj=c(0,NA))

for(i in 1:nsec) if(uuu[i]%in%rownames(trans2))
  text(1,nsec-i+1,paste0(uuu[i],' (',which(levels(secOrderCP$unitName)==uuu[i]),')'),adj=c(1,NA),cex=0.75)
for(i in 1:nsec) if(uuu[i]%in%colnames(trans2))
  text(2,nsec-i+1,paste0(uuu[i],' (',which(levels(secOrderCP$unitName)==uuu[i]),')'),adj=c(0,NA),cex=0.75)

for(rr in rownames(trans2))
 for(cc in colnames(trans2)){
#     if(trans2[rr,cc]>20){
      arrows(1.1,nsec-which(uuu==rr)+1,1.9,nsec-which(uuu==cc)+1,lwd=5*trans2[rr,cc]/sum(trans2o[rr,]),
       col=rgb(0,0,0,alpha=ifelse(trans2[rr,cc]>50,1,trans2[rr,cc]/50)))
}
@
\end{subfigure}
\begin{subfigure}{1in}
<<trans2legend,fig.width=1,fig.height=3,dependson='transition2'>>=
par(mar=c(.1,.1,.1,.1))
plot(1,1,xlim=c(0,2),ylim=c(0,nsec+2),cex=0,xaxt='n',yaxt='n',xlab='',ylab='')
conv <- (nsec+2)/11
text(1,nsec+1.5,expression(underline(bold(paste(Legend)))))
text(1,9*conv,'Thickness:\n Percent',cex=0.75)
text(.5,7.5*conv,'25%')
text(.5,6.5*conv,'50%')
text(.5,5.5*conv,'75%')
arrows(c(1,1,1),c(7.5,6.5,5.5)*conv,c(2,2,2),lwd=5*c(.25,.5,.75))
text(1,4.2*conv,'Darkness:\n Number',cex=0.75)
text(.5,3*conv,'10')
text(.5,2*conv,'25')
text(.5,1*conv,'50+')
arrows(c(1,1,1),c(3,2,1)*conv,c(2,2,2),col=rgb(0,0,0,alpha=c(1/5,.5,1)),lwd=2.5)

@
\end{subfigure}
\caption{Reassignment transition plot for study year 2. See caption of
Figure \ref{fig:trans1} for details.}
\label{fig:trans2}
\end{figure}

To address these questions, we focus on the units in the standard
Algebra I sequence.
Figures \ref{fig:trans1} and \ref{fig:trans2} give transition plots
for reassignment in the two study years.
On the left of each figure, marked ``From,'' are the top ``sending''
units: the units that students were reassigned \emph{from} at least 30
times.
They are numbered according to the order they appear in the standard Algebra I curriculum.
On the right, under ``To,'' are the top ``receiving'' units: the units
that students were reassigned \emph{to} at least 30 times.
For completeness, each of the top two receivers for each sending unit,
and each of the top two senders for each receiving unit were also included.
Finally, all of the sending units listed on the left were also included in the
receiving column.
All in all, \Sexpr{round(propTotCP1*100)}\% of the first-year reassignments and
\Sexpr{round(propTotCP1*100)}\% of the second-year reassignments are
captured in the figures.

The arrows in the plot represent reassignments.
An arrow from a sending unit to a receiving unit indicates
reassignment from the former to the latter.
The thickness of the arrows represents the proportion of reassignments
originating in the sending unit whose destination was the receiving unit.
The darkness of the arrows represents the number of such reassignments.
For instance, in Figure \ref{fig:trans1}, the arrow from
``4-Quadrant Linear Graphs'' on the left to ``4-Quadrant Linear
Graphs'' on the right is fairly thick, since
\Sexpr{round(trans1o['4-Quadrant Linear Graphs','4-Quadrant Linear Graphs']/
  sum(trans1o['4-Quadrant Linear Graphs',])*100)}\%
of the reassignments from ``4-Quadrant Linear Graphs'' end in the
same unit.
Yet it is also fairly faint, since it only represents
\Sexpr{trans1o['4-Quadrant Linear Graphs','4-Quadrant Linear Graphs']}
reassignments.

Inspecting the figures shows that the most common pattern is for
students to be reassigned to the next unit in the curriculum---this pattern comprises
\Sexpr{round(sum(diag(trans1o[,-1]))/sum(trans1o)*100)}\% of the reassignments in year 1 and
\Sexpr{round(sum(diag(trans2o[,-1]))/sum(trans2o)*100)}\% of those in
year 2.
On the other hand, it is relatively rare for students to be reassigned within the same unit
(these reassignments account for \Sexpr{round(sum(diag(trans1o))/sum(trans1o)*100)}\% and \Sexpr{round(sum(diag(trans2o))/sum(trans2o)*100)}\% in the two years).
It is also rare for students to be reassigned to units earlier in the curriculum (comprising
\Sexpr{round(sum(trans1o[lower.tri(trans1o)])/sum(trans1o)*100)}\% and
\Sexpr{round(sum(trans2o[lower.tri(trans2o)])/sum(trans2o)*100)}\%).

The transition plots also reveal some interesting cases worth highlighting.
In year 1 (Figure \ref{fig:trans1}), students reassigned from the first unit, ``Linear Patterns'', were primarily placed two units ahead, in ``1st Quadrant Linear Graphs,'' skipping ``Unit Conversions,'' perhaps suggesting a disinterest in ``Unit Conversions'' on the part of the reassigning teachers.
Similarly,
\Sexpr{trans1o['Ind. Variables in Lin. Mod.','Systems of Lin. Eq.']}
of the students
(\Sexpr{round(trans1o['Ind. Variables in Lin. Mod.','Systems of Lin. Eq.']/sum(trans1o['Ind. Variables in Lin. Mod.',])*100)}\%)
reassigned from the section ``Independent Variables in Linear Models'' were placed 13 units later in ``Systems of Linear Equations''
and all \Sexpr{trans1o['Quad. Mod. & Area','Exponents']} of the students reassigned from ``Quadratic Models \& Area'' were placed in ``Exponents.''
This suggests that some teachers may have considered the units ``Systems of Linear Equations'' and ``Exponents'' to contain particularly important material.

Since the total number of reassignments was higher in year 2, the corresponding plot (Figure \ref{fig:trans2}) is larger and more complex.
It is also more common in year 2 for students to be reassigned to units other than the next unit in the sequences.
This may be partly due to the proliferation of customized curricula.
Two units, ``4-Quadrant Linear Graphs'' and ``Lin. Equations with
Variables on Both Sides,'' were common destinations from a wide
variety of earlier units, suggesting strong teacher interest in those units.
The majority (\Sexpr{round(trans2o['Probability','Probability']/sum(trans2o['Probability',])*100)}\%)
of the students reassigned from ``Probability'' were placed in another section of the same unit, perhaps suggesting problems with some of that unit's sections; in fact, all of the students reassigned within the ``Probability'' unit were reassigned from one its first three sections (of seven).
Finally, \Sexpr{trans2['Pythagorean Theorem','Product Rule for Exponents']}
(\Sexpr{round(trans2['Pythagorean Theorem','Product Rule for Exponents']/sum(trans2o['Pythagorean Theorem',])*100)}\%) of the reassignments from ``Pythagorean Theorem'' ended in the earlier ``Product Rule for Exponents'' unit.

\textbf{Summary.} The most prevalent pattern was for students to be reassigned
to the following unit, suggesting that teachers may be mostly
interested in advancing students who are behind.
On the other hand, a number of examples of other patterns---students moving within the same unit, or to units out of sequence---appear as well, suggesting that some teachers may be finely manipulating their students' curricula.



\section{Effects of Reassignment}\label{sec:effects}
The goal of CTAI is to help students learn Algebra, so
the most important questions about reassignment are about its effect on learning.
Although the data from this study came from a randomized trial, it was
CTAI as a whole that was randomized, not individual behaviors within CTAI.
Specifically, student reassignment was not randomized.
Therefore, precise estimates of causal effects of reassignment on
learning require strong untestable assumptions that are unlikely to be
true.
As in all observational studies, this includes the assumption that all
confounding variables---variables that predict both reassignment and
learning---have been measured well and modeled correctly.
Further complicating matters, although reassignment itself is a
well-defined process, in practice it can take many forms, as we have
endeavored to show.
There is no reason to expect the effects of reassignment to be
the same regardless of whether the teacher used it to help lagging
students catch up, to allocate time to important topics, or for some
other reason.

All that said, observational estimates of reassignment's average
causal effects can be valuable, if interpreted cautiously, for
instance by assessing their sensitivity to unmeasured confounding, as
we do below.
In the absence of evidence from randomized trials, observational
studies can help guide intuition, future research, and even---when
combined with other relevant information and theory---practice.

<<effects,include=FALSE,dependson=c('data','setup')>>=
cpDat <- data%>%filter(!is.na(status)& status!='final_or_incomplete')%>%
 group_by(field_id,year,state,section,unit,race,sex,grade,spec_speced,xirt,spec_gifted,
  spec_esl,frl,pretest,y_yirt,classid2,schoolid2)%>%
 summarize(status=max(status,na.rm=TRUE),date=max(date),totalTime=sum(total_t1,na.rm=TRUE),
  ttNA=sum(is.na(total_t1)),nprob=n(),probNA=sum(is.na(Prob1)))%>%ungroup()%>%
 group_by(field_id,year,state,race,sex,grade,spec_speced,xirt,spec_gifted,spec_esl,frl,pretest,y_yirt,
  classid2,schoolid2)%>%
 summarize(nsec=n(),nprob=sum(nprob,na.rm=TRUE),totalTime=sum(totalTime,na.rm=TRUE),
  timeNA=sum(ttNA>0),probNA=sum(probNA>0),ncp=sum(status=='changed placement'),
  nprom=sum(status=='promoted'))%>%mutate(gainScore=y_yirt-pretest)%>%droplevels()



## mode imputation:
for(vv in c('race','sex','grade','spec_speced','spec_gifted','spec_esl','frl')){
    num <- is.numeric(cpDat[[vv]])
    cpDat[[paste0(vv,'MIS')]] <- is.na(cpDat[[vv]])
    cpDat[[vv]][is.na(cpDat[[vv]])] <- names(which.max(table(cpDat[[vv]])))
    if(num) cpDat[[vv]] <- as.numeric(cpDat[[vv]])

}

cpDat$grade <- factor(ifelse(cpDat$grade==9,'9','10+'))
levels(cpDat$race) <- list(White=c('WHITE NON-HISPANIC','ASIAN / PACIFIC ISLANDER'),Black=c('BLACK NON-HISPANIC','OTHER RACE / MULTI-RACIAL'),Hispanic=c('HISPANIC','AMERICAN INDIAN / ALASKAN NATIVE'))

cpDat <- mutate(cpDat,totalTime=totalTime/3600000,totalTime=ifelse(totalTime<0,NA,totalTime),
                totalTime=ifelse(totalTime>360,NA,totalTime))

cpDat$ncpCat <- factor(ifelse(cpDat$ncp>=4,'4+',cpDat$ncp))
cpDat$everCP <- cpDat$ncp>0

exirt <- data[match(cpDat$field_id,data$field_id),grep('Exirt2',names(data))]
names(exirt) <- gsub('_0','_',names(exirt))
for(n in names(exirt)) cpDat[[n]] <- exirt[[n]]

@
<<models,include=FALSE,dependson='effects'>>=
#### models!!
## ind var: ncp
addCovs <- .~.+pretest+race+sex+grade+spec_speced+spec_gifted+spec_esl+frl+frlMIS
mod1 <- lm(y_yirt~ncp+classid2,data=cpDat)
mod1.1 <- update(mod1,addCovs)

## ind var: ncpCat
mod2 <- lm(y_yirt~ncpCat+classid2,data=cpDat)
mod2.1 <- update(mod2,addCovs)

## ind var: everCP
mod3 <- lm(y_yirt~everCP+classid2,data=cpDat)
mod3.1 <- update(mod3,addCovs)


miMod <- function(mod){
  if(inherits(mod,'lm')) CCC <- coef
  if(inherits(mod,'lmerMod')) CCC <- fixef
    ests <- NULL
    covs <- list()
    for(i in 1:20){
        formNew <- as.formula(paste0('.~.-pretest+Exirt2_',i))
        modNew <- update(mod,formNew)
        ests <- cbind(ests,CCC(modNew))
        covs[[i]] <- vcov(modNew)
    }
    return(list(ests=ests,covs=covs))
}
miPool <- function(fitMods){
    ests <- fitMods$ests
    covs <- fitMods$covs
    interest <- grep('CP',rownames(ests),ignore.case=TRUE)
    SEs <- sqrt(rowMeans(do.call('cbind',lapply(covs,function(cc) diag(as.matrix(cc[interest,interest])))))+
                1.05*apply(rbind(ests[interest,]),1,var))
    ests <- rowMeans(rbind(ests[interest,]))
    cbind(ests,SEs)
}

mi <- function(mod){
    miPool(miMod(mod))
}

adjustedEsts1 <- list()
for(i in 1:3){
  mm <- paste0('mod',i)
  mm1 <- paste0('mod',i,'.1')
  adjustedEsts1[[mm]] <- mi(get(mm))
  adjustedEsts1[[mm1]] <- mi(get(mm1))
}
@
<<sensitivity,include=FALSE,dependson='models'>>=
source('hhh.r')
### we're only computing sensitivity for omitting pretest
X <- as.data.frame(model.matrix(mod3.1)[,-1])
Ts <- Tz(X = X,treatment = 'everCPTRUE')
X$everCPTRUE <- NULL
R <-  partialCor(cpDat$y_yirt,X,'pretest')^2
multPre <- MEmult(T=Ts['pretest'],R=R,df=mod3.1$df,q=1.96)

Rhos <- vapply(names(X)[-grep('classid2|pretest',names(X))],
               function(nn) partialCor(cpDat$y_yirt,X,nn),1)^2
multOth <- MEmult(T=max(abs(Ts[names(Rhos)])),max(Rhos),df=mod3.1$df,q=1.96)

stopifnot(names(Rhos)[which.max(abs(Ts[names(Rhos)]))]=="raceHispanic")
stopifnot(names(Rhos)[which.max(abs(Rhos))]=="spec_esl")

@

<<heterogeneityMods,include=FALSE,dependson='models'>>=
### heterogeneity
## year
cpDat$Year=as.factor(cpDat$year)
mod4 <- lmer(y_yirt~everCP*Year+state+(1|classid2)+(1|schoolid2),data=cpDat)
mod4.1 <- update(mod4,addCovs)

## classroom
mod5 <- lmer(y_yirt~everCP+year+state+(everCP|classid2)+(1|schoolid2),data=cpDat)
mod5.1 <- update(mod5,addCovs)
@

\begin{figure}
  \centering
<<cpYyear,fig.width=6,fig.height=4,dependson='effects'>>=
levels(cpDat$Year) <- list(`Year 1`='1',`Year 2`='2')
ggplot(cpDat,aes(ncp,gainScore))+geom_jitter(height = 0,width=.3)+geom_boxplot(aes(group=ncp),alpha=0.5)+geom_smooth(method='loess')+facet_grid(Year~.)+labs(x='# of Reassignments',y='Gain Score (posttest - pretest)') + scale_x_continuous(breaks=seq(0,max(cpDat$ncp)))
@
\caption{Students' Gain scores (posttest minus pre-test) versus the number of times the were reassigned over the course of the year (jittered), with a Loess smoother added.}
\label{fig:cpYyear}
\end{figure}

%\subsection{Average Effects}
Figure \ref{fig:cpYyear} shows students' gain scores---the difference between their posttest and pre-test scores---as a function of the number of times they were reassigned.
(The number of reassignments was jittered---random noise was added on
the horizontal dimension---to avoid overplotting.)
Overall, the relationship between the two variables is positive.
Nevertheless, some non-linearity seems to be present, especially in year 2.
Further, the distribution of the number of reassignments is right-skewed---again, especially in year 2.
Care in modeling the number of reassignments, then, is especially important---observations from students reassigned an unusually large number of times can exert undo influence on a regression model and generate misleading results, particularly in the presence of non-monotonic relationships.
We settled on three different strategies: first, the variable $R^{bin}$ dichotomizes reassignment---$R^{bin}=0$ for students who were never reassigned, and $R^{bin}=1$ for students who were.
Next, $R^{cat}$ defines a categorical variable taking the values $R^{cat}=0,1,2,3,4+$ for students who were reassigned 0, 1, 2, 3 or four or more times, respectively.
Finally, $R^{num}$ is the raw number of reassignments, which we include for completeness.

We used linear models to estimate the effect of reassignment, regressing posttest scores on $R^{bin}$, $R^{cat}$, or $R^{num}$ along with fixed effects for classroom, essentially modeling reassignment as randomly assigned within classroom.
Since this is unlikely to be the case (even approximately), we ran a
second set of models including student level covariates as well:
pretest scores,\footnote{These are measured with error, and missing at a relatively high rate (
\Sexpr{round(mean(is.na(cpDat[['xirt']]))*100)}\%). To account for
this, we used regression calibration based on the 20 ``multiple
imputations'' used in the original CTAI study,
\citeN{pane2014effectiveness}.} race, sex, grade, special education
and gifted status, English as a second language (ESL), and free and reduced-price lunch eligibility.


\begin{table}
  \centering
\begin{tabular}{r|c|c|c|}
&\multicolumn{3}{c}{Parametrization}\\
&$R^{bin}$&$R^{cat}$&$R^{num}$\\
&\makecell[c]{Effect of\\ $\ge 1$ Reassignment}&\makecell[l]{Effect of\\  \# Reassignments:}& \makecell[c]{Effect per\\ Reassignment:}\\
\hline
<<effectResults,results='asis',dependson='models'>>=
## model 3
cat('\\makecell[r]{No\\\\Covariates}&')
cat(round(adjustedEsts1[['mod3']][1,'ests'],2),'$\\pm$',round(2*adjustedEsts1[['mod3']][1,'SEs'],2),'&')
## model 2
res <- adjustedEsts1[['mod2']]
cat('\\makecell[l]{')
for(rr in 1:(nrow(res)-1))
  cat(rr,':  ',round(res[rr,1],2),'$\\pm$',round(2*res[rr,'SEs'],2),'\\\\')
cat(nrow(res),'+: ',round(res[nrow(res),1],2),'$\\pm$',round(2*res[nrow(res),'SEs'],2),'}&')
## model 1
    cat(round(adjustedEsts1[['mod1']][1,'ests'],2),'$\\pm$',round(2*adjustedEsts1[['mod1']][1,'SEs'],2),'\\\\')
cat('\n')
cat('\\hline')
###################################
cat('\\makecell[r]{Covariate\\\\ Adjusted}&')
cat(    round(adjustedEsts1[['mod3.1']][1,'ests'],2),'$\\pm$',round(2*adjustedEsts1[['mod3.1']][1,'SEs'],2),'&')
## model 2
cat('\\makecell[l]{')
res <- adjustedEsts1[['mod2.1']]
for(rr in 1:(nrow(res)-1)) cat(rr,': ',round(res[rr,1],2),'$\\pm$',round(2*res[rr,'SEs'],2),'\\\\')
cat(nrow(res),'+: ',round(res[nrow(res),1],2),'$\\pm$',round(2*res[nrow(res),'SEs'],2),'}&')
## model 1
cat(
    round(adjustedEsts1[['mod1.1']][1,'ests'],2),'$\\pm$',round(2*adjustedEsts1[['mod1.1']][1,'SEs'],2),'\\\\')

@
\hline
\end{tabular}
\caption{Estimates of effects of reassignment on posttests, with 95\% margins of error.}
\label{effectResults}
\end{table}

The results are reported in Table \ref{effectResults}.
All the effect estimates are negative, indicating that reassigning students may hurt their algebra learning.
The estimated effects decrease in magnitude for three or four reassignments, but these estimates are very noisy---very few students were reassigned more than two times.
The magnitudes of the effects are rather large: \citeN{pane2014effectiveness} reported an effect, in year 2, of about 0.2; the estimated effect of at least one reassignment is \Sexpr{round(abs(adjustedEsts1[['mod3.1']][1,'ests'])/0.2*100)}\% of that.

But what of unmeasured confounding?
For instance, the negative effect may be due to baseline differences
in ability, beyond what is captured in pretest scores.
\citeN{hhh} suggest a method of estimating the sensitivity of a regression to an omitted confounder based on bench-marking from observed confounders.
In order to confound the causal relationship between reassignment and posttests, a confounder would have to predict both.
Roughly speaking, the idea is to widen the confidence interval from an
ostensibly causal linear model to account for the possibility of a hypothetical unmeasured confounder that predicts reassignment and posttests to the same extent as one of the observed covariates.
These ``sensitivity intervals'' account for uncertainty from two sources: random error, and systematic error due to the omission of a confounder.
As is typical, the pretest is our most important measured covariate, both in terms of its prediction of reassignment and of posttest scores.
The sensitivity interval for the effect of being reassigned at least
once on posttest scores, allowing for the possible omission of a
hypothetical confounder at most as important as pretest, is
\Sexpr{round(adjustedEsts1[['mod3.1']][1,'ests'],2)} $\pm$
\Sexpr{round(adjustedEsts1[['mod3.1']][1,'SEs']*multPre,2)}.
This interval is quite wide, implying that such a covariate could
explain much of the observed relationship between reassignment and
posttest scores (or that the relationship may be much stronger).
On the other hand, the sensitivity interval allowing for the possible
omission of a less important hypothetical confounder---one that
predicts posttests as well as ESL status and reassignment as well as
Hispanic ethnicity, the next best observed
predictors---is
\Sexpr{round(adjustedEsts1[['mod3.1']][1,'ests'],2)} $\pm$
\Sexpr{round(adjustedEsts1[['mod3.1']][1,'SEs']*multOth,2)}.
This interval is substantially tighter.
All in all, unmeasured confounding may play an important role here,
but there is good reason to believe that it does not explain all of
the observed relationship.

The wide variety in the use of reassignment that we have documented
here might suggest that reassignment's treatment effect varies as
well.
Figure \ref{fig:trtHet} shows estimated classroom-specific treatment
effects of $R^{bin}$, being reassigned at least once.
The estimates came from a multilevel model in which posttest scores
were regressed on $R^{bin}$ and student-level covariates, along with
random effects for classroom and random slopes for $R^{bin}$ varying
by classroom.
Unlike fixed effects models, multilevel models ``partially pool'' data
across classrooms to estimate classroom specific effects more precisely \cite{gelmanHill}.
This is especially important given the small sample sizes within
classrooms.
Figure \ref{fig:trtHet} shows a wide variation in the effect of
reassignment across classrooms---the estimated standard deviation of
these effects was
\Sexpr{round(sqrt(summary(mod5.1)$varcor$classid2[2,2]),2)},
larger than the average effect itself.
While the effect was negative in most classrooms, it was positive in some.
This variation could be due to a number of factors, including
differences in the composition of classrooms, but supports the
hypothesis that differences in when and how reassignment is used
lead to differences in its effect.

\begin{figure}
  \centering
<<hetroPlot,fig.width=6,fig.height=3,dependson='heterogeneityMods'>>=
re <- ranef(mod5.1,condVar=TRUE)
reVar <- attr(re$classid2,'postVar')[2,2,]
feVar <- vcov(mod5.1)['everCPTRUE','everCPTRUE']

hetPlotDat <- data.frame(eff=re$classid2$everCPTRUE+fixef(mod5.1)['everCPTRUE'],
                         se=sqrt(reVar+feVar),
                         year=as.factor(data$year[match(rownames(re$classid2),data$classid2)]),
                         state=data$state[match(rownames(re$classid2),data$classid2)])
hetPlotDat$ord <- rank(hetPlotDat$eff)

ggplot(hetPlotDat,aes(ord,eff))+geom_point()+
#    geom_errorbar(aes(ymin=eff-2*se,ymax=eff+2*se),lwd=.2)+
    geom_hline(yintercept=0,lty=2)+
#    geom_hline(yintercept=fixef(mod5.1)['everCPTRUE'])
    scale_x_continuous(labels=NULL,breaks=NULL)+
    labs(y='Effect of Any Reassignment',x='Classroom')
@
\caption{The effect of being reassigned at least once, in each
  classroom, as estimated by a multilevel model. The classrooms are
  sorted by these estimated effects. The dotted line indicated an
  effect of zero.}
\label{fig:trtHet}
\end{figure}



\section{Summary and Discussion}\label{sec:discussion}

The effectiveness of the Cognitive Tutor software presumably depends
on how much, and how, it is used.
This paper exploited available log data from the high-school arm of the CTAI
effectiveness trial---which yielded an impressive result for the
software in year 2---to describe variation and patterns in the software's
usage, paying particular attention to issues of mastery learning.
We found that:
\begin{itemize}
 \item The amount the software was used varied widely between states
   and decreased, overall, from years 1 to 2.
 \item Year 2 saw the proliferation of ``customized'' curricula in
   three states, altering which units students worked, and in what
   order.
 \item Year 2 saw frequent departures from the standard CTAI unit
   sequence, driven mainly, but not entirely, by customized
   curricula.
 \item Examination of section mastery found that:
 \begin{itemize}
  \item About \Sexpr{round(tab1b['Mastered','Overall']*100)}\% of
    worked sections in year 1 and
    \Sexpr{round(tab2b['Mastered','Overall']*100)}\% in  year 2 ended
    with the student having mastered all of the included skills.
  \item There were three ways students worked a section without
    mastering its contents: exhausting its problems and being
    promoted, being reassigned to a new section by the teacher, and
    ending CT use altogether.
  \item Reassignment was rare, though it was more prevalent in Texas
    and Connecticut than in other states, and more common in year 2
    than 1.
  \item Mastery rates were lower for more advanced curricula.
 \end{itemize}
 \item Examination of reassignment patterns found that:
  \begin{itemize}
   \item Reassignment rates were determined more by factors varying by state and school
     than by classroom.
   \item Reassignment was more common in the second half of the year
     than in the first---particularly in Texas.
   \item Depending on state and year, reassignment typically takes
     place in one of three scenarios: teachers reassigning students
     who have fallen behind, teachers reassigning (almost) the entire
     class together, and teachers assigning (almost) all students
     \emph{out} of a particular section.
   \item Students are most commonly reassigned to the next unit in the
     curriculum---suggesting that teachers may be advancing lagging
     students---but in some cases they might be finely manipulating
     their students' curricula.
   \end{itemize}
 \item Reassignment appears to lower students' performance on the
   post-test relative to that of their peers; however, the effect
   appears to vary widely between classrooms (and, presumably, how
   reassignment is used).
 \end{itemize}

In practice, mastery learning and topic scaffolding often unfolded
quite differently from the vision of CT's designers.
In some cases, the departures were apparently a matter of a student's inability
to achieve mastery quickly enough, and in other cases of educators' preferences that ran counter to CTAI's design.

Notably, both the amount of usage and fidelity to CTAI's design
decreased from years 1 to 2---just as the estimated \emph{effect} of
CTAI increased.
In year 2, when the effect was substantial, students spent less time,
and followed the CTAI curriculum and guidelines less closely than in
year 1, when the estimated effect was negative but statistically
insignificant.
This raises questions as to the roles of structure and mastery in
CTAI's effectiveness.
Does flexibility lead to higher effects? Is mastery learning an
important mechanism for CTAI?
Answers to these questions could prove crucial for optimizing the
realized effectiveness of CTAI and other intelligent tutors.

On the other hand, reassignment appears to hurt students' performance
on the posttest, relative to their classmates (though based on data
from a randomized experiment, this analysis was observational, so
confounding can't be ruled out).
Does this, in contrast, suggest that achieving mastery is an important
component of effective intelligent tutoring?

As educational technology spreads, attention to the details of
implementation may yield important insights---or important
questions---about effectiveness, and the science of when
intelligent tutors work, and when they don't.

\bibliographystyle{acmtrans}
\bibliography{ct,cp}

\end{document}