There are few issues in education in DC right now that are as clearly problematic at the new OSSE school report cards and 5-star school ratings. For some background, I am a parent at Amidon-Bowen, but I am also an educational researcher who cleans and analyzes student achievement data as part of my day job.
The OSSE 5-star school ratings supposedly measure school quality, or how well schools are serving their students. But my analysis – and others’ – have shown that the 5-star school ratings are more strongly determined by which kids are in the schools than what the schools actually do to help students learn. Specifically, 64% of the variation in OSSE school ratings can be explained by student demographics alone. Let that sink in for a minute. We are rating schools on the basis of student demographics more so than school effectiveness.
If pressed, OSSE would respond (as they did in their report) by saying that once you control for the various STAR metrics, there is no relationship between student poverty and the school ratings. But that is a flawed argument because the school ratings are determined on the basis of the metrics. Metrics are ratings, they are one and the same. So if metrics are biased against high-poverty schools, then so are the ratings. This is not a matter of me, a member of the public, not understanding what OSSE is doing. This is a matter of OSSE leads not understanding the statistics behind the OSSE star ratings or the methodological implications of their decisions.
What are the implications of the star ratings? First, we’re giving low ratings to schools that serve predominantly at-risk and special education children, and we’re giving high ratings to schools that serve predominantly affluent children. Even though growth factors into a school’s overall score, we are still rating schools more so on student demographics than any other factors.
Second, with these school ratings, we’re telling principals and teachers whether they are doing a good job or not on, again, which students they serve. My advice to effective teachers in high-poverty schools shouldn’t be you’re doing a great job, ignore your state agency. OSSE should be on the front lines of identifying schools that are really moving the needle and making a difference for kids.
Third, we’re telling parents to steer away from the “bad” schools, you know the schools with too many at-risk and special education students. Better stick to the schools serving affluent children by moving into an expensive neighborhood or winning the lottery for a spot in one of the most sought-after charters serving more affluent students.
Fourth, and perhaps most problematic, is that schools with consistently low ratings will be threatened with take-over or closure, without any regard for to what extent these schools are helping kids learn or how students in these schools will fare in other places.
Let’s talk about the growth metrics that OSSE uses. The median growth percentile essentially compares achievement for students who start with the same score. But whose growth are we comparing against whose each year, and does that make sense? Median growth percentiles are also problematic because they may (a) fluctuate wildly from one year to the next and (b) are also correlated with student characteristics. It gets better – the creator of median growth percentiles specifically warned that median growth percentiles should not be used to determine school quality. So why are we going against the recommendations of the statistician who created the indicator in the first place?
How should school performance be gauged instead? We have to find valid measures that capture what schools actually do. Valid measures of growth, academic offerings and enrichment, and school climate. There are plenty of experts in this field who could help us get it right. And if the federal requirements are too restrictive, we can do what many states have already done and create an independent state accountability system instead.
This is really an exasperating issue if you understand the methodological problems. If you understand the stats behind this, and you are not speaking out either internally or externally, you are complicit in propping up a school system that further disadvantages our most disadvantaged children. Let’s not wait several years to fix it or discuss the unintended negative consequences of the new star ratings.
I’ll close by sharing several quotes from the feedback sessions that OSSE hosted about the new school report cards.
– “Do we have to use the stars? Do the stars simply reflect the economic status of students? Does this concern you?”
– “The proficiency and attendance measures reward schools with fewer students in poverty. This is not fair.”
– “At-risk demographics are a strong predictor of proficiency. We should do more to desegregate schools. The accountability system needs more growth, especially for low-proficiency schools.”
– “Proficiency is weighted too highly in the accountability system. Include higher weight on growth. Otherwise, schools have an incentive to push students out of the school.”
– “80% on PARCC proficiency and growth is too high. Add measures of social and emotional learning.”
– “Can growth be weighted higher than proficiency? Especially since we know that scores are strongly correlated to income?”
– “Have you tested the model on existing schools? Why not? That would enable us to see how this correlates to poverty.”
– “Think we should measure school environment/teacher quality/other things that are important.”
– “We should back map from what data we need to provide well-rounded measure of school quality. Don’t be scared about collecting new data if it’s the right data.”
– “Where is the focus on curriculum? How is a school supposed to know it’s important if you don’t even mention it?”
– “Why don’t you have a classroom observation measure for higher grades?”
– “We should use school climate surveys from students, teachers, parents, and administrators. Why was this measure rejected?”
– “There are more than 20 school climate surveys that the federal government have deemed valid. This is available.”
– “How is attendance a measure of school environment? I think that’s insufficient.”
– “Concern about how we will ensure kids are getting a well-rounded education when there’s such a strong focus on PARCC.”
– “Has there been consideration of including measures to encourage a well-rounded education?”
– “What about sports, arts, other subjects outside math/reading? Research shows enrichment, other offerings add to success of students academically.”
– “You should consider a rating in each different component of the school’s rating – not just one set of stars.”
– “We need to address core problems affecting students (hunger, suicide, poverty, trauma), not just make minor changes in accountability system.”
– “Why doesn’t high school system include growth?”
– “Why is there no growth metric in high school?”
– “Do you have mathematicians on your staff?”
Ward 6 Public School Parents Organization 