Friday, January 20, 2017

APS Enrollment Data 1994-2016

The visual below shows enrollment trends at APS for the past 23 years. Each data point is from the GADOE October FTE count. Use the filters on the right to select a school or grade, or the tabs at the top to pick a different view.

APS enrollment increased for the fourth consecutive year to 51,927 students. To learn more about the types of insights available in this visual, see our enrollment blog post from last year.

Tuesday, January 3, 2017

2016 Student Growth Percentiles

The Georgia Department of Education (GADOE) calculates Student Growth Percentiles (SGP) for students who take the Milestones exams. SGPs indicate how much a student's achievement grew compared to academically similar peers. To calculate a 2016 SGP score, a student is grouped with other students with similar 2015 and 2014 scores, and then given a percentile rank based on how their 2016 score compares.

GADOE then uses SGP data to find the percentage of students making typical or high growth by school and district. "Typical or high growth" is defined as an SGP of at least 35. The state average percentage of students making typical or high growth is 65% (100 - 35).

The visual below shows the percentage of students making typical or high growth by grade and subject, using SGP results from GADOE. Use the filter on the left to choose a different school or district, or the tabs at the top to switch to high school or district-level views.

In this example, we can see that students at Inman Middle School show progress at rates above the state average in most grades and subjects. Seventh grade social studies was a relative weakness in 2014, but has since recovered to near the state average. Overall, math is a relative strength, where 80% of students are achieving typical or high growth.

This visual is helpful for schools or districts to use root-cause analysis. Working with other school or district leaders, identify which grade/subject combinations are high or low-performing, or have seen significant change. Then discuss the causes of those notable data points, and how that information can be used for future improvement.

The typical or high growth calculation is also used to determine progress points on the CCRPI. However, the CCRPI only includes "full academic year" students, while the SGP data files include all tested students. The results used for CCRPI progress points might vary from the SGP files by a few points but they show the same trends.

Confidence intervals1 are displayed as blue bands. Similar to the margin of error in a poll, the confidence interval tells us the range of possible values, but the middle value is most likely. Schools or districts with few students will have wide confidence intervals; large schools and districts have more precise growth estimates so their confidence intervals will be smaller.

Although progress data is very helpful for understanding school performance it does not give a complete picture. It is also important to consider overall achievement. This is especially true in elementary school, where progress is only measured in two of the six grade levels. A helpful compliment to the progress data is the Milestones versus Challenge Index graph below. This compares achievement for schools with similar populations, and better accounts for learning that occurred before 4th grade.

For more details on this visual, see our post from last year. This year mobility has been added to the challenge index because it adds additional predictive power2. Mouse over the details mark for more information on the new challenge index calculation.

(1) Confidence intervals are calculated at the 95% level using the standard error of the sample mean. This is only one of two sources of uncertainty. Each student SGP score is also an estimate and has associated uncertainty, but is not available to us. Hence, the confidence intervals are an underestimate of the total uncertainty. We compensate for this by using a relatively large confidence interval- 95%.
(2) Mobility improved R-square from 93.3 to 94.4 for elementary schools. Charter schools are not included because of different mobility patterns due to grade bands and school zones. Similar graphs that include charters for 2016 can be found at the bottom of this article.

Thursday, December 8, 2016

2016 CCRPI Results

The Georgia Department of Education released 2016 CCRPI results today (12/8). The district overall score decreased 1.8 points from 67.0 to 65.2.  Similarly, the overall score for the state of Georgia fell 1.9 points from 75.5 to 73.6. 

The view below is created using the state's results file and includes all Georgia public schools and districts. It is currently filtered to compare APS' 2015 and 2016 results by grade band and CCRPI component, highlighting that in each grade band APS maintained Achievement points but lost some ground in Progress points. 

Use the filters to compare results across Georgia districts or schools. When looking at district results, set the school filter to "All Schools". The "Schools Comparison" tab shows 2016 results for all APS schools and gains or losses over 2015 scores. Use the "Components" filter to see how schools resort based on individual CCRPI components.

Visit the GADOE CCRPI page for more information about CCRPI and to view detailed score reports. See more APS results, including Progress Point trends on our data report, or read the superintendent's blog.

The example below shows how the data visual can be used to view a school’s trend over time. Cleveland Elementary School has consistently grown their CCRPI since 2013, and earned 40 progress points in 2016, the maximum possible amount. Progress points are a useful component because they measure how much student achievement grew during the year, taking into account their scores from the previous year. Also note that the state’s CCRPI calculation changed in 2015 so progress became a larger component.

The next example shows how the data visual can be used to make cross-district comparisons. District-level elementary school CCRPI scores are displayed for six large metro Atlanta districts. Gwinnett County earned the most progress points of the six districts in 2016 and had the highest overall score.

CCRPI Components
Each school receives an overall Single Score. Each grade band within a school receives a CCRPI Score. These scores are made up of four components: Achievement (50), Progress (40), Achievement Gap (10) and a max of 10 Challenge Points. Challenge points include ED/EL/SWD Performance and Exceeding the Bar. Scores prior to 2015 should not be directly compared to scores from 2015 and later due to changes in calculations.

Tuesday, November 1, 2016

2016 Graduation Results

Graduation rates for cohort 2016 were released today (11/1). The view below shows the percentage of four-year graduates by school and district (orange). 

Overall, the district graduated 71.1 percent of its students in four years. This is compared to 71.5 percent in 2015. State-wide, the graduation rate is 79.2 percent.

Use the "Select Cohort" filter in the upper left-hand corner to see graduation rates from previous school years. To see one school's rates for the past six years, click on the "School Trends" tab then use the "Select a School" filter. 
Graduation rates for small APS schools that are now operating as one large school (Ex: Therrell Health, Law, and STEM) have been consolidated to more easily identify trends. 

Cohorts are based on the year students first enter ninth grade. The graduation rate is the percentage of these students who graduate within four years, adjusting for transfers. Read what Superintendent Carstarphen has to say about APS grad rates on her blog, and get grad rates for all Georgia schools and districts here

Friday, July 29, 2016

2016 Milestones Results

2016 Milestones results were released by GADOE this Tuesday (7/26). The visual below shows achievement levels by school.

Use the filters on the left to focus on a specific grade or subject, or to switch to elementary school. Use the tab at the top to switch to End of Course exams (high school). This view was created with the state data file, so users can also use the "System" filter to view other districts.

Notice that the view is sorted by the percentage of students scoring at the beginner level, but the relative ranking changes if a different category is used. Atlanta Classical Academy, which is our middle school with the lowest poverty rate, has the most students scoring above beginner. However, Inman middle school has a much larger percentage of students who reach the distinguished level. Achievement results tend to be highly correlated with family income- see last year's post on poverty and test scores.

The achievement level view is great for understanding how students compare to state standards and to other APS schools. The description of each state achievement level is available here. For a summary of overall APS performance, see the superintendent's blog.

The next view converts average test scores for each school into a state percentile. This view is more helpful for comparing performance over time, across grades and subjects, and to the rest of the state.

The percentiles show a school's relative position among all Georgia schools. For example, a percentile of 45 indicates a school's average scale score is as high or higher than 45% of schools in Georgia. For a more detailed description of the percentile view, see our previous percentile post

Note that the EOG percentiles view does not include advanced math students in eight grade who took the EOC, but they can be found on the EOC tab. This causes schools with a large advanced math program like Inman to have a lower 8th grade math percentile on the EOG view. 2016 is the first year that advanced math students did not also take the grade level exam.

To see more updates on district Milestones results, follow us on twitter or use the e-mail subscription on the left.

Finally, to indulge a data analyst who also thinks he's an artist, here's a picture of achievement for all elementary schools in the state: 

To see a more readable view of the same data, scroll to the achievement visual at the top of the post and select system = "All".

Tuesday, May 10, 2016

CCRPI State Percentile Ranks

The Georgia Department of Education released 2015 CCRPI results on May 3. The CCRPI is a great data set for understanding school performance. The state provides detailed school drill-downs here. However, for those who don't have a job that involves combing through CCRPI data, sometimes a single school report lacks context. For example, what is a "good" progress point score? These questions are even more relevant because both the state test and the CCRPI formula changed in 2015.

One way to add this context is to look at how a school's score compares to the rest of the state. As a supplement to the state data views, we made the view below that shows the state percentile rank for each CCRPI component.

Remember that each value is a state percentile, not their actual score. For example, the top result shows that Morningside's three-year average of achievement scores was in the 99th percentile in the state, and their three-year average of progress scores was in the 98th percentile. For more details on CCRPI indicators, see this page (the "General Resources" and "CCRPI Indicators" options on the right are particularly helpful). For more details about the visual, mouse-over the question-mark icon in the top right.

It is also helpful to look at this view in conjunction with other Milestones indicators. One such case is the progress component for elementary schools. The progress indicator only measures growth in 4th and 5th grades. Our view of test scores vs. challenge is a helpful compliment because it better captures the cumulative effect of the first four grades and makes comparisons to similar schools.

Three year averages are shown as the default view because the single-year fluctuations in a school's CCRPI tend to be temporary changes, as opposed to long-term trends. This is demonstrated in the correlation table below. If score changes were indicative of a long-term trend, then scores in adjacent years would be the most correlated. However, the correlation between 2012 and 2015, 0.86, is as high as any other correlation. This suggests that the majority of score changes are actually temporary fluctuations and that a multi-year average provides more information about school performance.

The CCRPI is composed of many different indicators and the data required is complex. In 2015 a few of our schools did not receive full credit due to data reporting problems. These errors were usually worth 1-3 CCRPI points. ANCS Middle and KIPP WAYS did not receive full credit this year for students who completed individual graduation plans and career inventories. Grady High School did not receive credit for the American Literature and Composition scores of students enrolled in AP English.

For more information about 2015 CCRPI results, see the superintendent's blog here, or our 2015 CCRPI data report. Or, let us know if you have questions on twitter.

Thursday, March 31, 2016

Milestones to National Percentile Conversion

The switch from CRCT to Georgia Milestones will benefit Georgia education in several ways. The more rigorous achievement levels will better inform schools and parents of their students' readiness for college and career, and the open-response questions will increase emphasis on writing instruction and critical thinking. 

In addition to increased rigor and open response questions, the 2015 Georgia Milestones results include national percentile scores. These scores are reported for each student as a supplement to the achievement and scale scores, calculated using a subset of 20 nationally-normed TerraNova items for each exam. This is a common psychometric method for creating a national percentile, but can yield odd results due to the number of questions and curriculum alignment. For example, the graph below shows the relationship between student's eighth grade science scale score and their national percentile.

Scale Score vs. National Percentile, 2015 APS Eighth Grade Science Milestones

Although students who have a high scale score tend to also receive a high national percentile, the graph shows substantial variation. Students who scored at the high end of Beginner (~470) received a national percentile of anywhere from 1 to 94. 

This variation is likely because only a subset of questions are nationally-normed. Although this method of calculating national percentiles is common, we recalculated national percentiles with a method that preserves students' scale score rank order. To do this, we used NAEP results to compare Georgia test score distributions to the national distribution, and then used that information, in conjunction with state means and standard deviations on the Milestones exam, to convert each scale score directly to a national z-score. We then used a cumulative normal distribution to convert from national z-score to national percentile.

We put the results in the visualization below. Select a grade and enter a student's Milestones score for each subject to see their national percentile. The visual also shows a confidence interval, based on both the Milestones scale score uncertainty, and the uncertainty in the relationship between Georgia's mean and the national mean. For example, the default view shows that a seventh grade student with an ELA scale score of 456 has an estimated national percentile of 19. The upper bound of the estimate is 30, and the lower bound is 11.

These percentiles were based on 2015 data, but should be mostly unchanged for 2016 results. As a final comparison, the graph below shows the relationship between scale score and the APS-estimated national percentile for each grade and subject. Notice that each scale score has only one estimated national percentile instead of a large range.

Let us know on twitter what you think, or if you'd like more details on the conversion formula. See the superintendent's blog for more information on Milestones performance in Atlanta Public Schools, or see previous posts on this blog.