ADDED: The committee had argued against using standardized test scores as the primary indication of student quality. Soft variables matter too, and U.S. News is rewarding the schools that mostly ignore those soft variables. The schools that resist U.S. News — as I well know! — take a hit in the rankings, while other schools scramble over them in the rankings by simply picking the highest test scores they can get. And with every passing year and climb in the rankings, those schools can get higher and higher scores, creating an upward spiral, as the U.S. News-resistant schools sink lower.
The committee’s report takes issue with the way the publication styles the rankings as the “Best Colleges” because the weights assigned to different metrics are essentially arbitrary.... One of their main concerns is that, given the diversity of higher education institutions, having a list of the “best” is impossible.That's lumping all schools together. Some emphasize scores more than others. Plus, it's a vicious circle. Who will stop first? If some schools backed off on test scores — as indeed, some schools already do — other schools would see an enhanced opportunity, snatch up the relatively high scorers that higher-ranked schools passed over, and the next ranking would show the results, punishing the school that tried to back off and redoubling the incentive to admit aggressively on test scores.
Robert Morse, who oversees the rankings for U.S. News, said .. [a]s long as colleges and universities continue to weight test scores and class ranking as a crucial component of admissions criteria... it is hypocritical for institutions to ask U.S. News not to do the same.
But what is the alternative? Other metrics of quality are worse, and there's no way to tell people to stop relying on the rankings or prevent U.S. News from continuing to process its imperfect information and publish it.