School Ratings Reflect Students More Than School Quality
Author: Massachusetts Institute of Technology
Published: 28 Mar 2024 - Updated: 8 Nov 2026
Publication Details: Peer-Reviewed | Informative
Table of Contents:
Synopsis - Definition - Overview - FAQs - Insights, Updates - Related Content
Synopsis
This research led by MIT economists, including Nobel laureate Josh Angrist, and published in the peer reviewed journal American Economic Review: Insights, used school assignment lotteries in Denver and New York City to compare schools as if in a randomized trial. It found that widely published school ratings largely reflect the preparation and family background of incoming students, and are strongly correlated with the share of white students, rather than measuring how much a school improves learning. Many lower-rated schools perform better than their ratings suggest. The findings matter to parents choosing schools, including families of children with disabilities, and to policymakers.*
At a Glance
- 1 - The paper is titled Race and the Mismeasure of School Quality.
- 2 - Denver data covered 2012-2013 to 2018-2019, and New York City data 2016-2017 to 2018-2019.
- 3 - Biased ratings can push up housing prices near highly rated schools, reinforcing inequality.
- 4 - The researchers propose a simple statistical adjustment that districts could use to rate schools more fairly.
Topic Definition
- Value-Added School Measure
A value-added school measure estimates how much a school contributes to students' learning by looking at changes in their test scores over time and adjusting for differences in the students who enroll, rather than simply reporting average scores. It aims to separate the effect of the school itself from the advantages or disadvantages students bring with them.
Overview
Are schools that feature strong test scores highly effective, or do they mostly enroll students who are already well-prepared for success? A study co-authored by MIT scholars concludes that widely disseminated school quality ratings reflect the preparation and family background of their students as much or more than a school's contribution to learning gains.
Indeed, the study finds that many schools that receive relatively low ratings perform better than these ratings would imply. Conventional ratings, the research makes clear, are highly correlated with race. Specifically, many published school ratings are highly positively correlated with the share of the student body that is white.
"A school's average outcomes reflect, to some extent, the demographic mix of the population it serves," says MIT economist Josh Angrist, a Nobel Prize winner who has long analyzed education outcomes. Angrist is co-author of a newly published paper detailing the study's results.
The study, which examines the Denver and New York City school districts, has the potential to significantly improve the way school quality is measured. Instead of raw aggregate measures like test scores, the study uses changes in test scores and a statistical adjustment for racial composition to compute more accurate measures of the causal effects that attending a particular school has on students' learning gains. This methodologically sophisticated research builds on the fact that Denver and New York City both assign students to schools in ways that allow the researchers to mimic the conditions of a randomized trial.
In documenting a strong correlation between currently used rating systems and race, the study finds that white and Asian students tend to attend higher-rated schools, while Black and Hispanic students tend to be clustered at lower-rated schools.
"Simple measures of school quality, which are based on the average statistics for the school, are invariably highly correlated with race, and those measures tend to be a misleading guide of what you can expect by sending your child to that school," Angrist says.
The paper, "Race and the Mismeasure of School Quality," was published in the American Economic Review: Insights. The authors are Angrist, the Ford Professor of Economics at MIT; Peter Hull, a professor of economics at Brown University; Parag Pathak, the Class of 1922 Professor of Economics at MIT; and Christopher Walters PhD '13, an associate professor of economics at the University of California at Berkeley. Angrist and Pathak are both professors in the MIT Department of Economics and co-founders of MIT's Blueprint Labs, a research group that often examines school performance.
The study uses data provided by the Denver and New York City public school districts, where 6th-graders apply for seats at certain middle schools, and the districts use a school-assignment system. In these districts, students can opt for any school in the district, but some schools are oversubscribed. In these circumstances, the district uses a random lottery number to determine who gets a seat where.
By virtue of the lottery inside the seat-assignment algorithm, otherwise-similar sets of students randomly attend an array of different schools. This facilitates comparisons that reveal causal effects of school attendance on learning gains, as in a randomized clinical trial of the sort used in medical research. Using math and English test scores, the researchers evaluated student progress in Denver from the 2012-2013 through the 2018-2019 school years, and in New York City from the 2016-2017 through 2018-2019 school years.
Those school-assignment systems, it happens, are mechanisms some of the researchers have helped construct, allowing them to better grasp and measure the effects of school assignment.
"An unexpected dividend of our work designing Denver and New York City's centralized choice systems is that we see how students are rationed from [distributed among] schools," says Pathak. "This leads to a research design that can isolate cause and effect."
Ultimately, the study shows that much of the school-to-school variation in raw aggregate test scores stems from the types of students at any given school. This is a case of what researchers call "selection bias." In this case, selection bias arises from the fact that more-advantaged families tend to prefer the same sets of schools.
"The fundamental problem here is selection bias," Angrist says. "In the case of schools, selection bias is very consequential and a big part of American life. A lot of decision-makers, whether they're families or policymakers, are being misled by a kind of naïve interpretation of the data."
Indeed, Pathak notes, the preponderance of more simplistic school ratings today (found on many popular websites) not only creates a deceptive picture of how much value schools add for students, but has a self-reinforcing effect - since well-prepared and better-off families bid up housing costs near highly-rated schools. As the scholars write in the paper, "Biased rating schemes direct households to low-minority rather than high-quality schools, while penalizing schools that improve achievement for disadvantaged groups."
The research team hopes their study will lead districts to examine and improve the way they measure and report on school quality. To that end, Blueprint Labs is working with the New York City Department of Education to pilot a new ratings system in 2024. They also plan additional work examining the way families respond to different sorts of information about school quality.
Given that the researchers are proposing to improve ratings in what they believe is a straightforward way, by accounting for student preparation and improvement, they think more officials and districts may be interested in updating their measurement practices.
"We're hopeful that the simple regression adjustment we propose makes it relatively easy for school districts to use our measure in practice," Pathak says.
The research received support from the Walton Foundation and the National Science Foundation.
Similar Content Viewed by Others
More on this topic: Disability Education Publications - Special Education Publications
Frequently Asked Questions
NOTE: Researched FAQs by Disabled World (DW)
What is selection bias in school ratings?
It occurs when schools look better or worse because of the kinds of students who choose to attend, rather than because of the teaching they provide.
How should parents of children with disabilities compare schools?
Ask about special education services, inclusion practices, staff training, class sizes and how individual education programs are carried out, and visit if possible.
What is a school choice lottery?
When more students apply to a school than there are seats, some districts use random lottery numbers to decide who gets a place.
Why do school ratings correlate with race?
Average test scores reflect family income and prior opportunities, which in the United States are linked to race because of long-standing inequalities.
What is a growth measure in education?
It tracks how much students' scores improve over time, which can show a school's effect better than a single average score.
Do higher-rated schools always serve students with disabilities better?
Not necessarily. A school's overall rating may say little about the quality of its special education and support services.
Where can families find information on special education quality?
State education departments publish special education data, and parent training and information centers can help families understand their options.
Will cities change how they rate schools?
The researchers were working with New York City on a pilot of a new rating system in 2024 and hope other districts will follow.
Insights, Analysis, and Developments
Editorial Note: Parents often choose a school, or even a home, based on ratings that seem objective, but this study shows that those numbers mostly describe who attends a school rather than how well it teaches. For families of children with disabilities, the more useful questions are about the quality of special education, inclusion, staff experience and support services, which raw test score averages rarely capture. Better ratings that measure growth could reward schools that help disadvantaged students most, and visiting schools and talking with other families remains one of the best guides.*
Attribution/Source(s): This peer-reviewed publication was selected for publishing by the editors of Disabled World (DW) due to its relevance to the disability community. Originally issued as a press release by Massachusetts Institute of Technology and published on 28 Mar 2024. Last updated on 8 Nov 2026. This content may have been edited for style, clarity, or brevity.
* Editorial additions by Ian C. Langtree.