You’ve probably heard that “what gets measured gets managed.” It’s a common phrase that has found a foothold in the world of education as part of the ever-changing landscape of assessment tools, big data and the categorization and sorting of schools across the world.
As superintendents, we live in a constant din of measurement: Which schools are performing best? What district has the highest paid staff? Which town spends the most on security? Which high school has the highest number of Ivy League acceptances? The answers to such questions ultimately impact hiring, retention, real estate value, property taxes and many other factors.
But what happens when school communities, school administrators and the measurement publications each have different criteria for excellence? This is the question that we puzzled over when we worked as district leaders in Madison Public Schools.
To develop an answer, we began a journey of evaluating various ranking systems, surveying our parents and staff, and delving into student performance metrics at Madison Public Schools. In 2018, we began sharing our results with our board members and school community through an annual “State of the Schools Address.” In 2022, we went a step further and presented our findings at the New Jersey School Boards Association’s Workshop to a standing-room only crowd of more than 200 board members, administrators and superintendents.
This article presents our work, so that school board members and district leaders can navigate the landscape of rankings and ratings systems to better communicate “school quality” to their stakeholders. As the findings indicate, this is a complex topic that does not provide a simple path forward. And “what gets managed gets measured” turns out to be a wholly insufficient paradigm for school or board leadership.
Rankings versus Ratings Systems Before delving into the specifics of individual publications or data sets, it is important to understand the framework for how these systems function. This means examining the differences and similarities between rankings systems and ratings systems on a basic definitional level.
Ratings systems typically assign numerical scores to entities based on a scale in which a higher score is generally considered better. You are probably familiar with ratings systems for consumer goods (e.g., Amazon), restaurants (e.g., Yelp), or travel (e.g., Orbitz) – and you recognize that if you’re wondering which bag to buy, which coffee shop to visit, or which hotel to book, you tend to favor the one with a 4.6 rating rather than a 3.9 rating. Each rating system creates a scale (e.g., one star through five stars), and the overall rating serves as a composite for a variety of subscores that are each given a particular weighting. In education, for example, the New Jersey Department of Education employs a ratings system for schools and assigns scores of 1 to 100 based on criteria such as New Jersey State Learning Assessments proficiency, graduation rates and chronic absenteeism.
Rankings systems, conversely, assign an order value based on the results of specific metrics, and lower values usually indicate a better ranking. Common rankings systems include athletics (e.g., the No. 1 team or player is considered the best overall), media (e.g., the No. 1 single in the “Top 40” or the No. 1 book on the New York Times Bestseller list is the most successful), or business (e.g., being ranked No. 1 in the “Best Employers in New Jersey” list is better than being ranked No. 30). Ranking systems feature prominently in the world of education with publications like U.S. News & World Report, Niche.com, and SchoolDigger.com, which evaluate schools using a wide range of factors such as test scores, college readiness, teacher retention and student progress.
Sources of Data Each ranking or rating system relies on a range of publicly available data sources to make its determination about the quality outcomes it publishes. In evaluating K-12 schools in New Jersey, these systems access the extensive data sets that are reported to the state and federal government for validation, analysis, evaluation and eventual publication. The data is collected in various sources, including the National Center for Education Statistics, Stanford Education Data Archive, state reporting frameworks and individual school district reports. The collections touch on an impressive list of items, including state assessment results, attendance records, suspension data, course enrollment, demographic information, graduation rates, postsecondary enrollment, staff data, budget data, surveys and many other factors.
This data is diverse, and each metric merits some examination. State assessment results, for example, provide a snapshot of student performance and growth in the core subjects of English language arts, mathematics, and science. Attendance records and tardiness data offer insights into how much time an average student spends in school and may serve as an analog regarding how committed a community’s students are to their education. Suspension data sheds light on the disciplinary aspects of a school’s environment. Demographic information provides an overview of the composition and diversity of the student body. Course enrollment shares data about the breadth of offerings available to students and the ways in which students engage with those offerings. Graduation rates and post-secondary enrollment offer a longitudinal perspective on a school’s success in preparing students for higher education. Staff data and budget allocations provide context for the resources available to educators and students. Surveys capture the sentiments of stakeholders and add a qualitative dimension to the sea of quantitative data.
Examining Specific Ratings and Ranking Systems The following section examines a number of ratings and rankings systems typically discussed in New Jersey. Each system possesses unique characteristics, methodologies and areas of focus that are worth considering.
NJDOE ratings:
- The NJDOE ratings system, published in the NJ School Performance Report, assigns summative scores from 1 to 99 for schools and considers criteria like NJSLA proficiency and growth, graduation rates, progress toward English language proficiency and chronic absenteeism. To find a school-specific score, click on the school, then “Detailed Report,” then, under the “Accountability” drop down menu, choose “Accountability Indicator Scores and Summative Ratings.”
- It also uses percentile rank to compare schools within grade bands (i.e., one high school versus another high school).
- The goal of the NJDOE ratings system is to provide a straightforward evaluation of schools within New Jersey, and parents and educators can leverage this system to understand the relative strengths and weaknesses of different schools. However, the reliance on only certain metrics might oversimplify the student experience within a particular school by not focusing on factors like teacher quality, extracurricular offerings and community engagement.
- A reference guide to the N.J. School Performance Report can be found at rc.doe.state.nj.us/Documents/ReferenceGuide.pdf.
GreatSchools.org ratings:
- The GreatSchools.org platform provides ratings for schools based on data sources such as test scores, student progress, college readiness and equity.
- The overall Summary Rating comprises a variety of subscores (“themed ratings”), including a Student Progress Rating or an Academic Progress Rating; a College Readiness Rating; an Equity Rating; and a Test Score Rating.
- GreatSchools.org offers transparency regarding its methodology and data sources. It also provides user-generated reviews, which are subjective and may reflect potential biases and inaccuracies.
U.S. News & World Report rankings:
- The U.S. News & World Report rankings system for districts and individual schools employs a unique methodology in its school rankings. Factors such as college readiness, math and reading proficiency and underserved student performance are considered.
- Six ranking indicators are used to determine a school’s overall score: College Readiness (30%), College Curriculum Breadth (10%), State Assessment Proficiency (20%), State Assessment Performance (20%), Underserved Student Performance (10%), Graduation Rate (10%).
- U.S. News & World Report provides transparency regarding its data, but it only focuses on high schools. Additionally, the emphasis on college readiness might not fully capture the many paths that students take after high school, such as vocational training or immediate entry into the workforce.
Niche.com rankings:
- Niche.com uses percentile ranks and incorporates survey data alongside various factors like academics, teacher quality, culture and diversity, and more.
- While the inclusion of survey data adds a qualitative dimension, it also introduces subjectivity. Also, its reliance on survey participation raises questions about the “representativeness” of the feedback received. Finally, fixed category weights might not adequately capture the evolving priorities and challenges of the education landscape.
SchoolDigger.comrankings:
- SchoolDigger.com adopts a narrow focus that exclusively considers student performance on state assessments (Average Standard Score). It provides percentile ranks over time and offers a longitudinal perspective on a school’s academic performance.
- While transparent about data sources, SchoolDigger.com’s limitation lies in its exclusive reliance on standardized test scores, which means that the long list of other aspects of a school’s success are not considered.
Key Considerations As you consider the wide range of factors that are used throughout the various ratings and rankings systems, you may recognize that “school quality” is not a simple and objective set of criteria. Given the complexity, we believe it is important to focus on a few key takeaways:
Rankings and ratings systems are developed by third parties:
Rankings and ratings are developed by third-party groups that have their own interests. While part of this work may be to provide a public service, other motivations may be to promote a brand or increase advertising sales. In many cases, these rankings and ratings are featured prominently on real estate websites, which can influence the decisions of homebuyers and renters and impact property values and community dynamics. Understanding the motivations and perspectives of these third parties is therefore crucial for a nuanced interpretation of rankings.
Variability: Different publishers use different calculations, which makes direct comparisons challenging. A school that performs well in one system might not fare as well in another due to differences in methodology, the criteria being considered, and how the factors are weighted. Being aware of this variability can help avoid making decisions in your district based solely on the rankings.
Winners and losers: Whether a system ranks schools from top to bottom or rates them on a given scale, these metrics create winners and losers among schools and districts, which can generate both pride and anxiety within communities. High rankings can boost a school’s reputation and attract positive attention, but they also may not be high enough for community members who demand the best. Lower rankings might lead to undeserved stigmatization, even though the student experience within a lower-ranked school might be very positive. Understanding the potential consequences of the use of these systems is essential for promoting a healthy educational environment.
Data lag: The data used in these rankings is subject to a time lag, which can be as long as four years in certain cases (i.e. the Civil Rights Data Collection). This lag can limit the usefulness of rankings, especially given the rapidly evolving educational landscape. Stakeholders should consider the relevance of historical data in making present-day decisions.
Community Engagement: As integral parts of their local community, schools play a significant role in shaping the identity of a town or neighborhood. Understanding the perceptions and expectations of the community and fostering community engagement can contribute to a more comprehensive evaluation of a school’s impact.
While “what gets measured gets managed” seems like a worthwhile adage, the truth of school rankings and ratings systems is far more complicated. These systems each offer some insight into school or district performance, but each must also be interpreted with caution and consideration regarding the needs and desires of the school community. More than anything, district leaders and board members would do well to provide as much information as possible about performance on these metrics to promote transparency, discuss data quality and address the biases that may be inherent in these systems.