Tag Archives: metrics

The Board of Trustees is meeting this week, and Chairman Chuck Lillis is obsessed with the idea that UO’s faculty are overpaid deadwood. So I’ve prepared this helpful Spreadsheet of Excellence. As a bonus I added another 20 excellent faculty at the bottom to get it to 120, to offset the fact that some of our most cited researchers are post-docs, retired, have left UO but haven’t updated their profile, or are just plain dead – which, according to this new economics paper can be good for their field, at least in the life sciences.

Of course this list only includes those who’ve added their profile to google scholar, which is easy. As of 9/3/2019, here.

If you’re interested in what’s wrong with these numbers and their inevitable misuse by UO administrators check out the posts and discussion in the metrics tag below. This post on the new Faculty Tracking Software – not a joke, but an actual initiative from UO Vice Provost Ellen Herman that will go out for bids soon – is a good place to start.

Or just indulge yourself in a little gratuitous ranking voyeurism:

SPREADSHEET OF FACULTY EXCELLENCE:

That would be UC-Santa Barbara:

Here at UO, the administration is about to decide that we should should spend ~$100K a year on “Faculty Tracking Software”.

Bengt Holmstrom got a 2016 Nobel for this basic insight, which given how little I’ve heard about research metrics lately, may even have taken hold with our administrative leadership.

Now Stanford economists Caroline Hoxby and Sarah Turner have a new paper on the consequences for low SES students, as explained in Inside Higher Ed:

… The two charts below show how the adjusted gross family incomes of the two colleges’ enrolled students changed from 2008 (when pressure was just beginning to build on selective colleges to enroll low-income students) and 2016. In 2008 (the chart with the blue bars), the colleges’ enrollment of students just below and above the Pell Grant threshold (the red line) was roughly in proportion to what one would have expected based on what the researchers describe as the colleges’ “relevant pool” based on their geography and mission.

By 2016, in contrast, the colleges were admitting almost twice as many students with incomes just below the Pell threshold as their relevant pools would have predicted, while they were admitting far fewer (still needy) students with incomes above the Pell threshold.

These data showing a “large discontinuity” between students just under and just over the Pell Grant threshold pretty clearly suggest, Turner said, that these institutions are “actively targeting Pell Grant recipients.” Colleges like these are almost certainly giving significant financial support to those students to ensure they enroll, and because their financial aid dollars are limited, they are in all likelihood giving less financial aid to those low- and middle-income students who the graphs show to be enrolling at those colleges in smaller numbers than they were before.

But “if you’re a student who’s just above [the Pell threshold], you still need just as much financial aid as somebody who is just below,” Turner said. “We know that there’s distortion going on there” in the colleges’ aid policies.

Hoxby and Turner say they’re not trying to cast aspersions on the work done by the researchers who have focused on Pell eligibility or the bottom income quartile as benchmarks — especially since many of them are former students of Hoxby’s, she notes.

But by embracing flawed measures and deciding to “do something that’s a little bit sexy or prurient by putting people in rankings,” Hoxby said, these analyses “may not actually be helping the situation, but making the situation somewhat worse. These measures are not measuring what they’re supposed to be measuring.

A letter from Provost Banavar, here:

The project, called Faculty Insights, will result in a sophisticated online system that enhances our ability to capture the wide range of research and creative activities that our faculty do. The primary purpose of the system will be to manage the faculty review process university-wide – including promotion, tenure, and post-tenure review – more efficiently and effectively. Introducing a Faculty Insights system at UO will enhance our ability to streamline faculty personnel processes and make the achievements and instructional activities of faculty in all the schools and colleges more visible, within the campus community and to the broader public. The system will also support the local metrics process and the production of annual unit-level research reports.

At this point the whole metrics fiasco is so toxic that (almost) everyone involved wants to just drop it. Yet it is a well-known fact that Johnson Hall never makes a mistake. What a dilemma!

At his recent metrics town hall, after hearing the litany of objections from faculty and heads, Provost Banavar offered the ingenious solution of saying that any proposal departments submit, even purely verbal descriptions of faculty research productivity that refuse to categorize journals and presses by quality, will count as “metrics”.

Dilemma resolved!

Meanwhile here’s the news from Italy – thanks to UO Psych department prof Sanjay Srivastava for the tip.

Self-citations as strategic response to the use of metrics for career decisions

Marco Seeber, Mattia Cattaneo, Michele Meoli, Paolo Malighetti

There is limited knowledge on the extent to which scientists may strategically respond to metrics by adopting questionable practices, namely practices that challenge the scientific ethos, and the individual and contextual factors that affect their likelihood. This article aims to fill these gaps by studying the opportunistic use of self-citations, i.e. citations of one’s own work to boost metric scores. Based on sociological and economic literature exploring the factors driving scientists’ behaviour, we develop hypotheses on the predictors of strategic increase in self-citations. We test the hypotheses in the Italian Higher Education system, where promotion to professorial positions is regulated by a national habilitation procedure that considers the number of publications and citations received. The sample includes 886 scientists from four of science’s main disciplinary sectors, employs different metrics approaches, and covers an observation period beginning in 2002 and ending in 2014. We find that the introduction of a regulation that links the possibility of career advancement to the number of citations received is related to a strong and significant increase in self-citations among scientists who can benefit the most from increasing citations, namely assistant professors, associate professors and relatively less cited scientists, and in particular among social scientists. Our findings suggest that while metrics are introduced to spur virtuous behaviours, when not properly designed they favour the usage of questionable practices.

It’s not just Nobel Prize winning economists and the UK Research Councils who think the administration’s research metrics plan is a mistake. Ken Calhoon, head of UO’s Dept of Comparative Literature, provides a less mathematical but no less thorough dissection:

February 27^th, 2018

Dear Friends and Colleagues,

Mozart wrote forty-one symphonies, Beethoven only nine. I have written none, but I offer these thoughts on metrics. I apologize in advance for the naiveté, as well as the pathos.

On September 14^th, at the beginning of the current academic year, University Provost and Senior Vice President Jayanth Banavar hosted a retreat for “academic leaders” in the EMU Ballroom. The highpoint of the assembly, in my view, was Jayanth’s own (seemingly impromptu) description of the research of David Wineland, the Nobel Laureate who recently joined the UO’s Department of Physics as a Knight Professor. In a manner that suggested that he himself must have been a gifted teacher, Jayanth provided a vivid and accessible account of Wineland’s signature accomplishment—speculative work aimed at increasing the computational speed of computers by “untrapping” atoms, enabling them to exist at more than one energy level at a time. With a humorous gesture to his own person, Jayanth ventured that it might be hard to imagine his body being in two rooms at once, but Wineland had figured out how, in the case of very small particles, this is possible. My own knowledge of quantum physics is limited to the few dismissive quips for which Einstein was notorious, e. g. “God is subtle but not malicious.” In any event, Wineland’s work was made to sound original and impressive. Equally impressive was the personable, humane and effective fashion in which Jayanth, with recourse to imagery and physical self-reference, sought to convey the essence of his fellow physicist’s work across all the disciplines represented in the room—and at the University.

I was inspired by the experience of seeing one person so animated by the work of another. However, my enthusiasm is measured today against the discouragement and disaffection that I and so many of my colleagues feel at the University’s current push, without meaningful debate, to metricize excellence—to evaluate our research in terms quite alien to the values our work embodies. As a department head with a long history at this institution, I must say that I feel helpless before the task of breaking our work down into increments and assigning numerical values to them. It can be done, of course, but the resulting currency would be counterfeit.

Over the course of my thirty-one-year career at the University of Oregon, I have presided over quite a few tenure and promotion cases and have been party to many more, both as departmental participant and as a member, for a two-year stint, of the Dean’s Advisory Committee in the College of Arts and Sciences. I am also routinely asked to evaluate faculty for tenure and promotion at other colleges and universities, where the process is more or less identical to ours. In past years I have been asked to write for faculty at Cornell, Harvard (twice), Johns Hopkins (twice), Washington University, University of Chicago, University of Pennsylvania, University of Minnesota (twice), Penn State, and Irvine, among others. I mention this not to boast—god forbid!—but to emphasize that institutions of the highest standing readily recruit faculty from the UO to assist in their internal decisions on professional merit and advancement.

For such decisions at the UO, department heads solicit evaluations from outside reviewers who are not only experts in the relevant field but are also well placed. They are asked to submit, along with their review, their own curriculum vitae and a biographical sketch. Reviewers are instructed to identify the most significant scholarly contributions which the individual under review has made, and to assess the impact of those contributions on the discipline. They are also asked to discuss the “appropriateness” of the publication venues, and also to “contextualize” their remarks with regard to common practices within the discipline or sub-field. They are asked to compare, “both qualitatively and quantitatively,” the work of the individual under review with that of other scholars in the field at comparable stages in their academic careers. Finally, the outside reviewers are asked to state whether the research record under consideration would meet the standards for tenure and promotion at their home institution. These instructions, which follow a template provided by Academic Affairs, differ little if at all from those I have received from other universities.

In response to these requests, we typically receive narratives, often three and four pages in length, in which reviewers—in accordance with the instructions but also with the conventions of professional service—not only discuss the candidate’s work in detail but also contextualize that work in relation, for example, to the evolving nature of the field, to others working on the same or similar material, not to mention the human content of that material. (I am usually asked to review the work of scholars working on the history of German literature and thought, as well as literary and film theory.) Looking back over the reports I have authored, I see that they contain phrases like “body of work,” “breadth of learning,” “intellectual energy,” “daunting command,” “surprising intervention,” “dazzling insight,” “staggering productivity,” etc. These formulations are subjective. As such, they are consistent with the process whereby one mind comes to grip with another. I am inclined to say that this process is particular to the humanities, but Jayanth Banavar’s lively and lucid presentation of David Wineland’s research would prove me wrong. It conveyed excitement.

What distinguishes the humanities from the sciences and many of the other, empirically oriented fields is that our disciplines are not consensus-based. We disagree among ourselves, often sharply, on questions of approach or method, on the validity and importance of the materials studied, on how arguments or interpretations should be structured or conceptualized. These disagreements may take place between departments at different universities, or within a single department. Disciplines within the humanities are in flux, and we suffer the additional burden of finding ourselves in a social and cultural world whose regard for humanistic work is markedly diminished. We often scramble to re-define our relevance while the ground shifts beneath our feet. To seek a stable set of ostensibly objective standards for measuring our work is to misrecognize the very essence of our work. These same standards risk becoming the instruments of this misrecognition.

In any case, the process of review for tenure and promotion, as formalized by Academic Affairs and by the more extensive guidelines which each unit has created, and for which each unit has secured approval both by its respective college and by Academic Affairs, already accounts for such factors as the stature of a press or journal, the rigor with which books and articles are reviewed, the quantity of publications balanced against their quality, and the impact which the faculty member’s research has had, or may be expected to have. But why the need to strip these judgments of their connective tissue? And for whom?

Curriculum vitae – “the course of [one’s] life.” When I was an undergraduate (at the University of Louisville, no less), I was greatly influenced by an historian of seventeenth-century Britain, Arthur J. Slavin. The dean of the college, he had been a friend of the mathematician Jacob Bronowski, recently deceased at the time, best known for his PBS series The Ascent of Man. One episode of the series begins with a blind woman carefully running her fingers over the face of an elderly, gaunt gentleman and speculating as to the hard course of his life. “The lines of his face could be lines of possible agony,” she says. The judgment is subjective, but accurate: The man, like Bronowski a Polish Jew, had survived Auschwitz, the remnants of which provide Bronowski with a physical backdrop for the dramatic and moving summation of an episode dedicated to the ramifications of the Principle of Uncertainty, which had been formulated by Werner Heisenberg just as all of Europe was about to fall victim to a despotic belief in absolute certainty. “It is said that science will dehumanize people and turn them into numbers. That is false: tragically false. Look for yourself…. This is where people were turned into numbers.”

I don’t mean to overdramatize the analogy, or even really to suggest one. I am more interested in Bronowski’s general statement that “[all] knowledge, all information between human beings, can only be exchanged within a play of tolerance. And that is true whether the exchange is in science, or in literature, or in religion, or in politics, or in any form of thought that aspires to dogma.” The dogma we are faced with today is that of corporate thinking, which is despotic in the sense that it mystifies. We in this country are inclined to think that people who have amassed great wealth know something we don’t—that they have the magic touch. It is from them and their public advocates that we hear the constant calls for governments, universities, prisons, hospitals, museums, utilities, national forests and parks to be run more like businesses. Why? (And which businesses? IBM? TWA? Pan Am? Bear Stearns? Enron? Wells Fargo?) Why is the business model the presumed natural guarantor of good organization? Why not a symphony? an eco-system? a cooperative? a republic? a citizenry? Why is the university not a model for business? Businesses certainly benefit from the talent we cultivate and send their way, outfitted with the knowledge, the verbal agility, the conceptual power that make up our stock in trade.

Our current national political scene presents us with constant images of promiscuous, self-reproducing wealth. Within this context, which is an extreme one, it is urgent that we as a collective make our case, and in terms commensurate with our self-understanding as researchers, thinkers, writers, fine artists, and teachers, not in terms that conform so transparently to the prevailing model of worker productivity.

Those who maintain that inert numbers are the only means we have for communicating our value have already been proven wrong by our own provost. I call upon our president, our provost and our many deans to bring their considerable talents, their public stature, as well as their commitment to the University, to bear on our cause. Many of us, I’m sure, are ready to support you.

With respect and thanks,

Ken

Dear Humanities and Social Science faculty,

Please join your colleagues Scott DeLancey (Linguistics), Spike Gildea (Linguistics), Volya Kapatsinski (Linguistics), Leah Middlebrook (Comparative Literature), Lanie Millar (Romance Languages), and Lynn Stephen (Anthropology) for a discussion of metrics for measuring our departmental research quality and the quality of our graduate programs. The panel will briefly summarize work done in some of our departments to identify what we value in our own work, ways to measure how well we achieve goals we value, and how we might take leadership in moving comparator institutions towards identifying and measuring their goals in comparable ways.

Tuesday, February 27 2:00-3:30 pm Gerlinger Lounge

Thanks to Lanie, Leah, Lynn, Scott, Spike, and Volya for their willingness to lead a timely discussion as we all consider how to create meaningful and useful metrics for our departments and disciplines.

Karen Ford and Phil Scher

InsideHigherEd’s interview with Jerry Muller about his new book. Published by the high impact-factor Princeton University Press. One excerpt:

Q: Some colleges, government agencies and businesses promote tools to evaluate faculty productivity — number of papers written, number of citations, etc. What do you make of this use of metrics?

A: Here too, metrics have a place, but only if they are used together with judgment. There are many snares. The quantity of papers tells you nothing about their quality or significance. In some disciplines, especially in the humanities, books are a more important form of scholarly communication, and they don’t get included in such metrics. Citation counts are often distorted, for example by including only journals within a particular discipline, thereby marginalizing works that have a transdisciplinary appeal. And then of course evaluating faculty productivity by numbers of publications creates incentives to publish more articles, on narrower topics, and of marginal significance. In science, it promotes short-termism at the expense of developing long-term research capacity.

More on the $600K Brad Shelton has dropped on Academic Analytics so far here.