Bad metrics and strong incentives are a nasty combination

Bengt Holmstrom got a 2016 Nobel for this basic insight, which given how little I’ve heard about research metrics lately, may even have taken hold with our administrative leadership.

Now Stanford economists Caroline Hoxby and Sarah Turner have a new paper on the consequences for low SES students, as explained in Inside Higher Ed:

… The two charts below show how the adjusted gross family incomes of the two colleges’ enrolled students changed from 2008 (when pressure was just beginning to build on selective colleges to enroll low-income students) and 2016. In 2008 (the chart with the blue bars), the colleges’ enrollment of students just below and above the Pell Grant threshold (the red line) was roughly in proportion to what one would have expected based on what the researchers describe as the colleges’ “relevant pool” based on their geography and mission.

By 2016, in contrast, the colleges were admitting almost twice as many students with incomes just below the Pell threshold as their relevant pools would have predicted, while they were admitting far fewer (still needy) students with incomes above the Pell threshold.

These data showing a “large discontinuity” between students just under and just over the Pell Grant threshold pretty clearly suggest, Turner said, that these institutions are “actively targeting Pell Grant recipients.” Colleges like these are almost certainly giving significant financial support to those students to ensure they enroll, and because their financial aid dollars are limited, they are in all likelihood giving less financial aid to those low- and middle-income students who the graphs show to be enrolling at those colleges in smaller numbers than they were before.

But “if you’re a student who’s just above [the Pell threshold], you still need just as much financial aid as somebody who is just below,” Turner said. “We know that there’s distortion going on there” in the colleges’ aid policies.

Hoxby and Turner say they’re not trying to cast aspersions on the work done by the researchers who have focused on Pell eligibility or the bottom income quartile as benchmarks — especially since many of them are former students of Hoxby’s, she notes.

But by embracing flawed measures and deciding to “do something that’s a little bit sexy or prurient by putting people in rankings,” Hoxby said, these analyses “may not actually be helping the situation, but making the situation somewhat worse. These measures are not measuring what they’re supposed to be measuring.

Tagged . Bookmark the permalink.

One Response to Bad metrics and strong incentives are a nasty combination

  1. honest Uncle Bernie says:

    How feckless and even corrupt these colleges must be to change their enrollment policies in such a manifestly unfair and damaging way in response to this external “incentive.” Have these people no shame? Should the public trust these people to do right by their students?

Leave a Reply

Your email address will not be published.