Student Achievement and the Appalachian Math and Science Part.
This study uses school level data for all elementary, middle, and high schools in Kentucky for the academic years 2000-01 to 2005-06. These years are chosen because they include a period of time prior to the introduction of AMSP into the schools and allow a comparison of pre-and post- AMSP. Partnership training began to be introduced in 2002-2003. Although the AMSP project involves 56 school districts in the central Appalachian regions of Kentucky, Tennessee, Virginia and West Virginia, this first study utilizes only those from Kentucky, which constitute the majority.
AMSP teacher participation data are collected as part of the AMSP project. Other data were collected from the Kentucky Department of Education and the Kentucky Education Professional Standards Board. The student outcome data are school performance data that have been used by Kentucky to measure school performance since its major reform (KERA) in 1991. These outcome data are the performance measures used to fulfill the requirements of No Child Left Behind (NCLB). For our purposes, only the math and science components of the student testing will be used.
In addition to current student test scores, data were collected on prior student test scores, school demographics such as percent of students on free and reduced-price lunch, race, and gender. Characteristics of the schools and their teachers are also collected. These include school size, teacher experience, and percent of teachers with master's degrees.
As stated above, the goal of the analysis is to test the effects of AMSP participation on student outcomes. But prior to estimating a model of the participation effects, this study recognizes that participation in AMSP is not random and that we, therefore, do not have a randomized trial for evaluation purposes. Put differently, all schools and all teachers in Kentucky do not have an equal probability of participating in AMSP. If we attempt to measure the effects of AMSP without correcting for the nonrandom selection, our estimated coefficients will be biased. To control for the non-random selection of schools and teachers into AMSP, we construct a first-stage model that estimates the level of participation in AMSP. For example, we know that schools in the Appalachian region of Kentucky are more likely to participate than those schools in the far western part of the state. The designation of Appalachian schools becomes a control variable in this first stage estimate of probability of participation. We also hypothesize that some schools within Appalachia will be more likely to participate than others. Those schools with lower than average test scores in math and science, for example, will have a stronger incentive to participate. The full set of first stage variables includes geographic location, participation in previous partnerships of this type (including the Appalachian Regional Systemic Initiative (ARSI)), the prior performance of the school, the change in performance in the two years prior to AMSP, the size of the school, and the student demographics of the school. We estimate the probability of participating in AMSP with a TOBIT model. This model allows estimation of dependent variables with a value of zero as well as positive continuous values.
The results of the TOBIT model are then used as an endogenously determined variable for AMSP participation in the second stage in which the effects of AMSP are estimated. In effect, the TOBIT estimates allow us to estimate the effects of AMSP as if all schools had an equal probability of participating in AMSP-i.e., as if we have a randomized trial.
To estimate the effects of AMSP participation on student outcomes, we use a fixed effects multiple regression model. Fixed effects models allow us to control for school level unobservable characteristics that may affect student outcomes. The dependent variable is school level math and science scores by school year for the years described earlier. Because Kentucky standardized tests are designed to generate different scores at the elementary, middle, and high schools, we estimate separate models for each level of school.
In addition to the AMSP participation variable, the model also controls for prior year student test scores, ARSI participation, student socioeconomic characteristics, school size, and teacher characteristics. These variables must be included as control variables for previous literature and theory suggest that they too will affect test scores of students. By controlling for other measurable influences on test scores, and controlling for unobservable school attributes through fixed effects, the estimate coefficient on the AMSP participation variable will provide an unbiased estimate of the effect of participation on student learning.
