Milestones School Level Results and Poverty

The graph above shows the relationship between 2015 Milestones results and poverty and Limited English Proficiency (LEP) status in Atlanta Public Schools. The combined indicator for poverty and LEP is labeled "challenge index". The relationship is striking: schools on the left with a low challenge index have high test scores, while schools on the right with a high challenge index have low test scores. Scroll to the "Details" section below for more information on the metrics.

What does this graph tell us?

There are several take-aways.

1. The relationship between the challenge index and test scores is very strong. Poverty and LEP explain 94% of the variance in school-level test scores for elementary schools¹. For middle schools, the challenge index explains 85% of the variance. However, most of the outliers from the middle school trend line are charter schools. When charter schools are removed the challenge index explains 95% of the middle school test score variance. (Elementary increases from 94% to 96% when charters are dropped.) The graph below displays middle school results with and without charter schools.


2. The difference between school-level and student-level variation is important. Although the challenge index explains most of the school-level variance in test scores, it only explains 20-25% of student-level variance². This means there are some high-performing students at low-performing schools (and vice-versa). Our view of student achievement levels makes this clear.

3. Although the challenge index explains most of the variance in test scores, there are still meaningful differences between schools at the same challenge level. For example, consider Usher, which has a challenge index of 85.7% and is performing above the district trend line. We can compare Usher to Dunbar, which has a similar challenge index. Usher's proficiency rate is about five percentage points higher than Dunbar, and their percentage of students who are at least developing is thirteen percentage points higher. 

4. This view provides a lot of information about school performance, but is not comprehensive. For example, the graphs would look different if we used a different challenge measure- such as including a measure for students with disabilities or mobility. Also, results do change from year to year. A school that is below the trend line this year might move up next year. Progress points on the CCRPI are a nice compliment to this view. And, perhaps most importantly, test scores are only one measure of a school's outcomes. It's also important to consider information like behavior, student engagement, or relationship with the community.

5. The challenge measure is a combination of poverty and LEP, but is mostly a poverty measure. Only three percent of APS students who took the 2015 End of Grade Milestones are LEP.

Usage

The visual at the top of the post is interactive. Mouse-over any school to see the NCE scores and challenge index, as well as metrics that are easier to interpret: the percentage of students who scored proficient or higher, and the percentage of students who scored developing or higher. The GADOE provides an explanation Milestones achievement levels here.

Use the options on the left to filter to a different school level, or to a specific grade or subject. Click on the cluster list to highlight a cluster, or hover over the details icons for more information. The cluster highlight is useful for schools that are difficult to find.

For a view of test scores that focuses on achievement levels, see this post, or use this link to see how schools compare to the state. To understand overall APS results, see the superintendent's blog post on Milestones results.

Details

The poverty component of the challenge index is the percentage of students who are directly certified for free or reduced priced lunch by the state because their parents or guardians qualify for other need based services (like TANF or SNAP). We use directly certified rates because schools that are part of the Community Eligibility Provision do not keep updated Free and Reduced Lunch percentages. The second is the percentage of students who are currently in the LEP program. The challenge index is the percentage of the student population in at least one of those two categories.

The test score measure used on the graph is a Normal Curve Equivalent score. A school's 'Average NCE' is the average of all students' NCE scores for the selected subjects. NCE scores range from 1-99, with a mean of 50, similar to percentiles. However, unlike percentiles, NCE scores maintain equal interval and can be meaningfully averaged and differenced. Using NCE allows us to average exams across grades and subjects more accurately. The NCE scores on the graph were calculated from the APS student population, so 50 is equal to the district average.


<sup>1. Variance explained is the R-squared of a regression of school-level NCE scores (averaged across grades and subjects) on challenge index, weighted by the number of students.

2. At the student level, the challenge index becomes a binary variable.</sup>

Milestones State Percentiles

The visual above shows scale score percentiles on Georgia end of grade exams. The percentiles show a school's relative position among all Georgia schools. For example, a percentile of 45 indicates a school's average scale score is higher than 45% of schools in Georgia.

The top part shows a heat map of performance by grade and subject, while the second part brings in additional years of data to show performance over time. 2015 data is from the Georgia Milestones, while previous years are from the CRCT. Using state percentile allows comparison across exams because although the exam changed, the comparison group does not- the comparison group is all other schools in the state in both cases. Click on the End of Course tab to see similar data for high schools. 

Scale score is used instead of proficiency rates to calculate state percentiles because scale scores are a continuous measure that capture all levels of achievement, while a proficiency rate is a binary measure for an individual student.

The filters in the top right allow the user to choose any public elementary or middle school in the state. This view shows that the default school, Garden Hills of Atlanta Public Schools, had higher performance relative to the state in grades 4 and 5 than in grade 3. Free lunch percentage is also reported at the bottom of the visual to provide additional context. Garden Hills has a high percentage of free and reduced lunch eligible students- 76.1%. (They also have a high English Language Learner percentage.) Schools with high free lunch percentages tend to have lower test scores. In this context, the grades and subjects where Garden Hills has a state percentile over 50 are quite high performing. A forthcoming post will explore the relationship between poverty and test scores in more detail. (Use the email subscription box on the right to be notified of this post, or follow us on Twitter.)

This view is helpful in conjunction with our view of achievement levels. The percentile view facilitates within school comparison across years, grades, and subjects by using state percentile as a single, consistent indicator. This makes it easier to identify areas of strength and areas for improvement. Meanwhile, the achievement levels view, by emphasizing achievement bands, helps us focus on our goals of preparing every student for college and a career.

For more information on the performance of APS students on Georgia Milestones, please see the superintendent's blog post on the topic.

Milestone End of Course Achievement Levels

The visual below shows achievement levels on the 2015 Georgia Milestones End of Course exams. For an explanation of this view, please see our post on the similar visuals for the End of Grade results. The results for the winter and spring administrations of the EOC are combined in this view. Also notice the "Within School" tab at the top, which changes the visual from an across-school comparison to a within-school comparison.

Milestones Achievement Levels

The graph above shows 2015 Milestones results by school. The default view averages across grades and subjects. Use the filters on the left to select a subject, grade, or grade level.

This visual provides both criterion-referenced and comparison data. Achievement bands color-coded in shades of blue- beginner, developing, proficient, and distinguished- show student performance according to state achievement criteria. This helps us understand how students are performing relative to a standard. The data sort, by percentage proficiency or higher, also allows the user to compare schools. This gives additional context to school performance. The sort-order can be changed to developing and above or distinguished using the drop-down in the top left. Over the next couple weeks, we'll release other dashboards that provide additional comparisons.

The superintendent provides more detail about district-level achievement results on her blog.

Filter to a specific subject to see that school performance is not uniform across subjects. Although Morningside has the highest rate of percentage proficient and higher when averaging across all subjects, Brandon has the highest rate for ELA, and Mary Lin has the highest rate for math (Morningside has the highest rate for both science and social studies).

The view below shows a within school comparison. Use the filter on the left to select a school. This view helps identify areas of strength and areas for improvement. For example, filter to Beecher Hills. This view shows that their achievement levels tend to be highest in third grade.

These views were created with public Milestones data from GADOE and the original files contain data from all schools in the state. We've left this data in our visuals; use the system filter to choose and view data for any district in the state.