Company that sells course evaluations writes report praising them

How surprising. As reported in InsideHigherEd here:

Student evaluations of teaching, or SETs, can provide a better understanding of what is working and what isn’t in classrooms. But gaining a “meaningful” understanding necessitates separating the “myths and realities” surrounding these evaluations, says a new report on the topic. That, in turn, requires data — lots of data.

So Campus Labs, a higher education assessment firm with 1,400 member campuses, opened its vault to create the new, myth-busting-style report. The study included more than 2.3 million evaluation responses from a dozen two- and four-year institutions that use Campus Labs’ course evaluation system, representing something of a national sample. All were collected in 2016 or later.


Philip Stark, professor of statistics at the University of California, Berkeley, and co-author of a major 2016 paper demonstrating gender bias in student evaluations, called the Campus Labs report “advertising, not science.”

“It’s particularly bad data analysis, including asking the wrong questions in the first place,” Stark said. Among his more specific criticisms was the lack of control group, conflating when students submitted their evaluations to when they were in class, and “no data on gender, ethnicity, grade expectations, grades or other measures of student performance.”

Based on existing research, “the strongest predictor of evaluations is grade expectations,” he said.

Students confuse actual learning with the feeling of learning

A new PNAS paper concludes:

… Comparing passive lectures with active learning using a randomized experimental approach and identical course materials, we find that students in the active classroom learn more, but they feel like they learn less. We show that this negative correlation is caused in part by the increased cognitive effort required during active learning. …

While this experiment was limited to entitled Harvard undergrads, the results should be of general interest to any university administrators who still don’t understand how student course evaluations create bad incentives for good teaching. Of course those deans who just want to use evaluations to measure “customer satisfaction” won’t care about the results:


Colleges Are (finally) Getting Smarter About Student Evaluations

That’s the news from the Chronicle of Higher Ed today here by Kristin Doerer (gated if off campus, some clips below) complete with a photo of some squirrelly looking economist:

Well, economists do have some experience with the misuse of metrics. From the article:

Emily Wu and Kenneth Ancell, two students at the University of Oregon, approached their honors research professor, Bill Harbaugh, a few years ago about studying the relationship between student evaluations and grade inflation. Harbaugh, a professor of economics, was enthusiastic. Wu and Ancell dived into the university’s extensive data on evaluation and transcripts, focusing on its two largest schools, journalism and business.

What they found surprised them.  Having a female instructor is correlated with higher student achievement,” Wu said, but female instructors received systematically lower course evaluations. In looking at prerequisite courses, the two researchers found a negative correlation between students’ evaluations and learning. “If you took the prerequisite class from a professor with high student teaching evaluations,” Harbaugh said, “you were likely, everything else equal, to do worse in the second class.”

The team found numerous studies with similar findings. “It replicates what many, many other people found,” said Harbaugh. “But to see it at my own university, I sort of felt like I had to do something about it.”

He did. In the spring of 2017, Harbaugh assembled a task force on the issue and invited Sierra Dawson, now associate vice provost for academic affairs, to join.

The UO Provost’s website on the reform process is here. We are piloting new surveys now and the Senate expects to have them in place by next fall. Back to the Chronicle article:

Legal Pressure

Doing nothing to revise or phase out student evaluations could be a risky proposition not just educationally, but also legally.

In August, an arbitrator ruled that Ryerson could no longer use student evaluations to gauge teaching effectiveness in promotion-­and-tenure decisions. The Ryerson Faculty Association brought the arbitration case and argued that because of the well-documented bias, student evaluations shouldn’t be used for personnel decisions.

“This is really a turning point,” said Stark, who testified on behalf of the Ryerson faculty group. He thinks the United States will see similar cases. “It’s just a question of time before there are class-­action lawsuits against universities or even whole state-university systems on behalf of women or other minorities, alleging disparate impact.” …

USC senate president to talk about teaching evaluation reform

This Wed, 9-10 in the library browsing room, and 3:20-3:50 at the Senate in the EMU Crater Lake room. From Around the O:

Ginger Clark, assistant vice provost for academic and faculty affairs at the University of Southern California, will discuss improving teaching evaluation methods during a campus appearance Nov. 28.

Clark, who is also a professor of clinical education and served as president of USC’s Academic Senate, has been central to her university’s efforts to advance student learning and develop better methods to evaluate teaching. Her lecture is sponsored by the UO committee on teaching evaluation.

She will speak at the Knight Library Browsing Room from 9 to 9:50 a.m. The event is open to the UO community and anyone can attend. Faculty members and staff are asked to sign up using MyTrack.

Clark also will address the University Senate later in the day in the Crater Lake rooms in the Erb Memorial Union from 3:20 to 3:50 p.m. Visitors are welcome.

“This is a chance for our campus to hear about how we can improve the classroom experience for our students and how we can help honor faculty for inclusive, engaged and research-led teaching,” said Sierra Dawson, associate vice provost for academic affairs. “Dr. Clark is considered a national leader in this area, and I’m looking forward to our faculty having the opportunity to learn more.”

Senate President Bill Harbaugh, an economics professor, said having Clark on campus will aid the effort to improve teaching evaluation.

“We all have an important stake in improving teaching and evaluating faculty,” he said. “Our current system relies too heavily on numerical course evaluations, which have been shown to be biased by gender and race, and uncorrelated with student learning.”

At the UO, the Senate has been working closely with the Office of the Provost to rethink the ways the university conducts teaching evaluations. The joint effort, which began in spring 2017, was designed to come up with a more effective way to define, develop, evaluate and reward teaching excellence.

The Senate created a task force to develop new ways to conduct evaluations at the UO. Currently, most evaluations are based on end-of-the-term ratings by students and a long-used faculty peer review system. Neither, the task force determined, fully gauge success in teaching nor the true ability of faculty members and how they design student learning experiences.

Recent research suggests such surveys include a bias against women and people of color, and they do little in the way of shedding light on teaching excellence or learning.

The Senate approved a motion to add a midterm student experience survey and an optional 10-minute instructor reflection at the end of the term as part of the evaluation process. The optional reflection would go to departments, ensuring that the instructor’s own voice can inform evaluators’ interpretation of student feedback.

Those tools are currently being tested across the UO and likely will become available campuswide for the 2019-20 academic year.

The Senate has been a driving force behind the potential improvements, establishing a new committee to work on continuous improvement and evaluation of teaching. The committee will regularly report to the Senate, and it plans to bring a new motion by the end of the academic year that would add a student experience survey at the end of each term, replacing the current student course evaluation.

To support the work, the provost has established a community for accelerating the impact of teaching focused on teaching excellence and evaluation. The community includes faculty members and local leaders from each school or college within the university, along with representatives from all three divisions of the College of Arts and Sciences.

Several faculty members from the group are testing the Senate-approved midterm student experience survey and the end-of-term instructor reflection, along with a novel end-of-term student experience survey. That effort includes faculty members from the Clark Honors College, the Lundquist College of Business, the School of Planning, Public Policy and Management in the College of Design and the human physiology and English departments in the College of Arts and Sciences.

The movement to reform teaching evaluation reaches beyond the UO. The Association of American Universities, with 60 member campuses in the United States and two in Canada, is spearheading an effort to connect schools working on teaching evaluation reform.

Across the globe, universities in Tasmania, Singapore, the United Kingdom and Sweden are also looking for ways to evaluate and celebrate successful instruction, along with creating innovative ways to evaluate teaching.

By David Austin, University Communications

UO Senate committee aims to improve student course evaluation process

Daily Emerald reporter Hannah Kanik has the story here:

The university faculty senate is working to create less biased and more informative course evaluations through a targeted task force and increased student input.

Last May, the University senate discovered sexist and racist correlations in the results of student course evaluations. Senate president Chris Sinclair appointed a task force to address the issue and conduct further research last June. The findings confirmed that course evaluations bred biased and misleading information regarding both student success rates and instructor performances.

… Sinclair emphasized the importance of student input to these evaluations as well. The senate is planning to include student input for the new evaluations and hopes to engage with their ideas in town hall meetings throughout the next year.