The graph above shows the relationship between 2015 Milestones results and poverty and Limited English Proficiency (LEP) status in Atlanta Public Schools. The combined indicator for poverty and LEP is labeled "challenge index". The relationship is striking: schools on the left with a low challenge index have high test scores, while schools on the right with a high challenge index have low test scores. Scroll to the "Details" section below for more information on the metrics.
What does this graph tell us?
There are several take-aways.
1. The relationship between the challenge index and test scores is very strong. Poverty and LEP explain 94% of the variance in school-level test scores for elementary schools1. For middle schools, the challenge index explains 85% of the variance. However, most of the outliers from the middle school trend line are charter schools. When charter schools are removed the challenge index explains 95% of the middle school test score variance. (Elementary increases from 94% to 96% when charters are dropped.) The graph below displays middle school results with and without charter schools.
2. The difference between school-level and student-level variation is important. Although the challenge index explains most of the school-level variance in test scores, it only explains 20-25% of student-level variance2. This means there are some high-performing students at low-performing schools (and vice-versa). Our view of student achievement levels makes this clear.
3. Although the challenge index explains most of the variance in test scores, there are still meaningful differences between schools at the same challenge level. For example, consider Usher, which has a challenge index of 85.7% and is performing above the district trend line. We can compare Usher to Dunbar, which has a similar challenge index. Usher's proficiency rate is about five percentage points higher than Dunbar, and their percentage of students who are at least developing is thirteen percentage points higher.
4. This view provides a lot of information about school performance, but is not comprehensive. For example, the graphs would look different if we used a different challenge measure- such as including a measure for students with disabilities or mobility. Also, results do change from year to year. A school that is below the trend line this year might move up next year. Progress points on the CCRPI are a nice compliment to this view. And, perhaps most importantly, test scores are only one measure of a school's outcomes. It's also important to consider information like behavior, student engagement, or relationship with the community.
5. The challenge measure is a combination of poverty and LEP, but is mostly a poverty measure. Only three percent of APS students who took the 2015 End of Grade Milestones are LEP.
Usage
The visual at the top of the post is interactive. Mouse-over any school to see the NCE scores and challenge index, as well as metrics that are easier to interpret: the percentage of students who scored proficient or higher, and the percentage of students who scored developing or higher. The GADOE provides an explanation Milestones achievement levels here.
Use the options on the left to filter to a different school level, or to a specific grade or subject. Click on the cluster list to highlight a cluster, or hover over the details icons for more information. The cluster highlight is useful for schools that are difficult to find.
For a view of test scores that focuses on achievement levels, see this post, or use this link to see how schools compare to the state. To understand overall APS results, see the superintendent's blog post on Milestones results.
Details
The poverty component of the challenge index is the percentage of students who are directly certified for free or reduced priced lunch by the state because their parents or guardians qualify for other need based services (like TANF or SNAP). We use directly certified rates because schools that are part of the Community Eligibility Provision do not keep updated Free and Reduced Lunch percentages. The second is the percentage of students who are currently in the LEP program. The challenge index is the percentage of the student population in at least one of those two categories.
The test score measure used on the graph is a Normal Curve Equivalent score. A school's 'Average NCE' is the average of all students' NCE scores for the selected subjects. NCE scores range from 1-99, with a mean of 50, similar to percentiles. However, unlike percentiles, NCE scores maintain equal interval and can be meaningfully averaged and differenced. Using NCE allows us to average exams across grades and subjects more accurately. The NCE scores on the graph were calculated from the APS student population, so 50 is equal to the district average.
1. Variance explained is the R-squared of a regression of school-level NCE scores (averaged across grades and subjects) on challenge index, weighted by the number of students.
2. At the student level, the challenge index becomes a binary variable.
