6/15/2015 update: That would be UC-Boulder. The Chronicle has the report here.
5/27/2015 update: Asking Too Much, or Not Enough?
Jake New has the latest in InsideHigherEd, here:
… The questions asked students if they had ever experienced a number of specific sexual activities without their consent, describing those actions with words and phrases such as “oral sex” and “penetration,” and defining the terms using definitions such as “when a person puts a penis, finger or object inside someone else’s vagina or anus.”
… At Penn, some students also complained that they didn’t realize the survey was about sexual assault, as it was referred to as a “climate survey,” as these kinds of surveys commonly are. Thinking the survey was about climate change, the students claimed, they deleted the email. The university declined to comment on the complaints, and said it does not plan on releasing its response rate until the fall. Harvard had a response rate of 52 percent, thanks in part to a large ad campaign on campus, including a video message from Harvard graduate Conan O’Brien.
I still think John Bonine’s Chronicle op-ed (below) is the most serious critique, because it points out that the AAU will hide the college identifiers that would allow researchers to figure out what policies are most effective at reducing sexual assaults. And the new report points to a new problem along those lines. The AAU has allowed campuses to use wildly different strategies to encourage students to complete the survey, and the resulting differences in response rates and who responds will further complicate any such efforts.
4/17/2015 update: VP Robin Holmes kicks off intentionally crippled $87K AAU rape survey
From: Vice President for Student Life Robin H. Holmes
To: University of Oregon Student
I’m writing to ask you to respond to a climate survey on sexual assault and sexual misconduct. The results will be used to guide policies to encourage a healthy, safe and nondiscriminatory environment at the University of Oregon. It is important to hear from you, even if you believe these issues do not directly affect you.
I know your time is valuable, but I hope you can find a few minutes to respond before the survey closes on Friday, May 8, 2015. By going to the website at the link below, you will be entered into a lottery to win $500 . We hope you will decide to complete the survey, but you are eligible for the lottery whether or not you complete the survey: [link deleted]
Your individual responses will be treated as confidential. Your participation in this survey is completely voluntary and will not affect any aspect of your experience at the University of Oregon. However, your response is important to getting an accurate picture of the experiences and opinions of all students.
Westat, a social science research firm, is administering the survey for us. If you have any questions about the survey or have difficulty accessing it, please send an e-mail to [email protected] or call 1 (855) 497-4787.
Thank you,
Robin H. Holmes
Vice President for Student Life
1/29/2015 update: Noted sociologist pays AAU $87,500 for intentionally crippled rape survey
More than half of the universities in the prestigious AAU have bailed on President Hunter Rawlings III’s politically motivated and intentionally crippled rape survey. Page down for John Bonine’s evisceration in the Chronicle of Higher Ed. Josephine Woolington has the latest in the Register Guard, here:
To ease concerns from some University of Oregon professors, UO Interim President Scott Coltrane has said the university could still back out of a nationalized sexual violence survey that has been criticized by dozens of scientists.
The UO, however, would likely be out $87,500 if Coltrane made that decision.
Coltrane announced in December that the university would develop its own campus “climate” survey in addition to using the national one overseen by the Association of American Universities, a nonprofit higher education trade group whose members include some of the nation’s top public and private universities, including the UO.
The UO agreed to pay for the AAU’s survey before an advisory committee appointed by Coltrane reviewed it and before the UO’s independent institutional review board evaluated it. …
UO psychology professor Jennifer Freyd and almost 60 other researchers from across the country say the AAU survey, which students will take in April, is flawed and could undermine federal efforts to understand the scope of sexual violence on college campuses.
The AAU has disputed such claims, noting that the survey will be based on a White House-recommended model and designed with a select group of 19 university professionals who have experience with survey research, sexual assault, gender studies and students affairs.
Experts say the survey lacks transparency because universities aren’t required to release the results to the public, a caveat that contrasts with recently proposed federal legislation that would require schools to conduct climate surveys and make the data publicly available online.
Why is Coltrane still participating in this charade?
1/18/2015 update: Almost half of AAU Universities just say no to Hunter Rawling’s campus rape survey
Michael Stratford has the story in InsideHigherEd:
More than two dozen of the nation’s top research universities have declined an offer by the Association of American Universities to anonymously survey their students about the prevalence of sexual assault on campus.
Twenty-six of the AAU’s 60 U.S. members told Inside Higher Ed this week that they had decided against participating in the association’s survey project, which some victims’ advocates and sexual violence researchers had criticized. …
Rumor is that AAU President Hunter Rawlings III (sic) has been calling the AAU presidents and engaging in some personal arm twisting. This makes his failure to get their consensus on how to spin the sexual assault problem even more embarrassing – if that’s possible, given the transparently political and anti-scientific nature of the AAU’s methodology.
And is UO going to be left holding a bigger financial bag because the cost of the consultant-prepared survey won’t be covered by enough schools? The one advantage of the AAU survey was consistency across schools. That was mostly eliminated by the AAU’s decision to strip school identifiers from the public data release. And now the data won’t even have much cross-school variation.
12/5/2014 update: Pres Coltrane will pay AAU $85K for problematic rape survey, and now seeks faculty buy-in.
Josephine Woolington has the report in the RG, here. John Bonine’s Chronicle op-ed below explains the political motivation for peculiar design of the AAU’s survey and its limited data-release plan, which violates the National Science Foundation’s guidelines. Other problems with its science are also documented in the letter to the AAU presidents from Jennifer Freyd and 60 or so other national researchers, linked to in Bonine’s piece below.
Now that Coltrane has made his decision, which ignored the recommendation of the UO Senate Task Force, he is trying to find “up to 10” UO administrators and faculty willing to serve on his alternative “administrative advisory group”, I assume in the hopes they will give him some advice he will like better, or at least buy him some cover for a while.
11/26/2014: Chronicle publishes John Bonine’s (Law) criticism of AAU rape survey secrecy
Here. In a nutshell, the AAU wants its member institutions to sign on to an expensive sexual violence student survey. Part of their pitch is that they will delay release of the results to allow for spin control, and permanently hide university identifiers from researchers. This will prevent cross-sectional or time-series studies aimed at understanding what policies and interventions reduce campus sexual assaults. UO President Scott Coltrane – a sociologist with expertise in the analysis of survey data – has to decide whether to spend $85K on this by Dec 1. As Bonine explains, it’s not a tough call:
… One might expect a prestigious group like the Association of American Universities to take a leadership role in these matters. In reality, however, the AAU wants member universities to commit by December 1 to a survey of sexual violence that will be kept largely secret from the public, including potential critics. Nearly 50 experts in sexual assault have criticized the plan in three open letters, but the AAU has continued to encourage presidents to spend $5-million on a survey that hasn’t even been written yet. …
When both the White House and Senators Kirsten Gillibrand, of New York, and Claire McCaskill, of Missouri, announced in the spring that they would push to require campus-climate surveys, the AAU sprang into action. But what were its motives? …
According to [AAU President] Rawlings last week, when institutions receive their campus results next summer, “AAU will require that universities agree NOT to publish or communicate survey findings internally (to the student population) or externally” until the AAU can engage its “national conversations” strategy. (The emphasis was provided by Rawlings.)
… Even so, the AAU is promising colleges that the results from their campuses will be kept secret from other colleges, the public, and politicians. It emphasizes in its letter to each president that “university-specific information will be shared only with that university.” The possibility of doing comparative analysis to determine which colleges have policies that actually work may become impossible.
John E. Bonine is a professor of law at the University of Oregon.
I actually find this entire debate to be very sad.
My own research is on sexual behavior, and same-sex sexual behavior in particular. I’m reminded of the debates that occurred in the late 1980’s and early 1990’s when there was opposition from both ends of the political spectrum about gathering reliable information on the sexual behavior: On the one side, you had conservatives who were concerned that if survey results showed that there were large numbers of people engaging in same-sexual behavior, then it would be more difficult to marginalize these populations. On the other side, activists on the left were concerned that if reliable survey results showed that there were fewer LGBTQ Americans, then it would be easier for bigots to justify discrimination.
I see a similar thing happening here: Rape apologists are reluctant to have these studies done because they will probably show how severe and systemic the problem of sexual violence is. On the other hand, we have activists who are concerned that if the numbers are “too low”, then it becomes easier to sweep the problem under the rug and for campuses to then do nothing.
I do not believe that either side’s intransigence is something that benefits survivors.
Something that I believe in fully, though, is that by doing high quality social scientific research, we can then use that data to inform better public policies. I hope we can move towards that as a goal.
You make way too many assumptions and ‘similar seeings’ that don’t necessarily translate to what’s happening now.
Is anyone actually reluctant to have studies done? Or is this about doing good studies, not bad ones that fail to measure accurately?
High quality research must certainly be the goal. But how do we know it is high quality if the sponsor is keeping the questions hidden? And how can we use the data for better policies if universities are allowed to keep the results hidden?
I wish people who don’t understand how unbiased social science research is done would stop with the spin behind this and saying that the questions are secret or that the data won’t be released. That is hogwash.
There are economies of scale involved in doing a study like this. So, if the UO were to do its own study, it would probably cost over 250K. Instead, we are getting better research done at 1/3 the cost by pooling resources with other Universities.
I completely admit that on a personal and political level that I am pro-LGBTQ and pro-survivor and this why the roadblocks to research that people who ostensibly advocate for the same things is so frustrating. When anti-science activists threw up these roadblocks to research on sexual behavior in the 90’s, every single day that the research didn’t get done it meant that we didn’t have the data and tools necessary to fight AIDS as a public health crisis. People actually died.
I see the same thing here: There are people who are ostensibly on the side of women and survivors (but are they really just on the side of getting funding for themselves?) putting up roadblocks to research. And, every single day that we don’t have this data to help us address sexual violence in meaningful way means that more women are assaulted. That’s heartbreaking to me at many different levels.
The AAU survey proposal is about politics, and it’s bad science.
Please read the Bonine op-ed before continuing to comment on this, as it addresses many of the questions you raise.
The AAU questions will not be confidential (though no one has seen them yet) but the survey will be “proprietary”. Presumably this means copyrighted, and therefore difficult to modify and re-use in the way that is typical for scientific research and progress.
The AAU explicitly promises universities that sign on that their data will not be released by the AAU, except after a delay, and even then with university identifiers removed. If this was an NSF or NIH sponsored research proposal, it would almost certainly be rejected for this reason.
Professor Freyd conducted the UO survey for approximately $25K (mostly participant payments) not $250,000, and she has released all the questions as open-source.
“saying that the questions are secret or that the data won’t be released. That is hogwash.” Really?
It looks like the questions are secret and the data won’t be released, according to quotations in the article. If the questions are not available for serious researchers in the field to study or suggest better ones, how can it be a reliable survey?
“There are people who are ostensibly on the side of women and survivors (but are they really just on the side of getting funding for themselves?) putting up roadblocks to research.”
Who is getting funding? Prof. Freyd said that she raised money and used it only for standard compensation for the time of those students who took the survey, not for herself.
“every single day that we don’t have this data to help us address sexual violence in meaningful way means that more women are assaulted.”
So on what basis do you conclude that the AAU data will be “meaningful”? Have you seen the questions? Inadequate questions result in meaningless data. And how will “we have this data” if each university is allowed to withhold it, according to the AAU?
The AAU admitted that its purpose in doing its survey, instead of getting behind the coming White House survey, is a political tactic, according to the article’s quotations from the AAU. That is what is “heartbreaking to me at many different levels.”
“There are people who are ostensibly on the side of women and survivors (but are they really just on the side of getting funding for themselves?) putting up roadblocks to research.”
That’s a pretty hefty accusation to throw around without an example. I have a mixed opinion of the survey, but stuff like this isn’t helpful. It’s good you clear up that you don’t mean Prof Freyd, but still. Are there any specific examples of this you’d like to point out?
“Please read the Bonine op-ed before continuing to comment on this, as it addresses many of the questions you raise.”
I’ve read it. I just strenuously disagree.
“The AAU questions will not be confidential (though no one has seen them yet) but the survey will be “proprietary”. Presumably this means copyrighted, and therefore difficult to modify and re-use in the way that is typical for scientific research and progress.”
I understand that it will be proprietary. That goes with any intellectual property. But, you can’t copyright a survey question. You can copyright the survey instrument itself. But, any single question or even a sub-section of questions can be re-used without permission.
“The AAU explicitly promises universities that sign on that their data will not be released by the AAU, except after a delay, and even then with university identifiers removed. If this was an NSF or NIH sponsored research proposal, it would almost certainly be rejected for this reason.”
In terms of the identifiers. One of the concerns I would have with releasing the data in this way is the it may be possible to identify respondents individually. This is particularly true if you are doing small area estimation for hard to find populations that may be differentially impacted by sexual violence (for example, trans men/women who are foreign students).
“Professor Freyd conducted the UO survey for approximately $25K (mostly participant payments) not $250,000, and she has released all the questions as open-source.”
Professor Freyd’s survey was a good start and certainly provided valuable information. But, I think we need a more robust study with a higher response rate to gather better information. That’s going to be expensive. I should say even if you were to double the response rate of Freyd’s study to 40% and not a single person in the next quintile of responders gave similar answers to the first respondents, then we are still looking at incredibly problematic patterns and rates of sexual violence.
This all misses the point of real, timely action on sexual assault.
One thing you are saying quite apparently is there is *money* in surveys, and this examples the conflicts others here are trying to highlight.
If they want to really follow the trends of rape in the AAU, then they can’t kick us out. He’s found a much cheaper way to keep is in the AAU? And one metric in which we’re likely above the median.
Wicked.
I’m surprised people are still pursuing what I see as both an anti-intellectual and also anti-woman crusade by trying to block the research.
In terms of the one commenter above who said, “It looks like the questions are secret and the data won’t be released, according to quotations in the article.” You have to separate the spin from the chaff….
Many of the “secret” questions were posted very early on online here:
https://www.insidehighered.com/sites/default/server_files/files/AAU_REQUEST%20FOR%20PROPOSALS_FINAL-3.pdf
Additionally, at this point, my understanding is that the Campus Advisory group has also seen the “not so secret” questions. The questions aren’t secret, that’s just another scare tactic.
In terms of the data not being publicly released, that will be up to the individual Universities. At the UO, I can’t see why they wouldn’t release the data. And, in the unlikely event that they chose not to release the data, there would be a variety of internal and external pressures that exist that would induce them to publicly produce the results.
One of the main things that I teach my students is that one of the key questions that we need to ask when doing sociological research is “Who benefits?” Similarly, if we are to ask “Who benefits from *not* doing this research?” And, the answer is simple: The Rapists. That’s not a side that I feel comfortable being on.
My feelings are a little more mixed. We have the Professor Freyd survey. It’s a good survey. Putting even 40k into it would be worthwhile. I think weighing the cost of the survey (trivial on university budget scales) and the harm of UO not being included in the survey (minimal) and the harm of not funding our own social science researchers but instead bringing in outside experts (absolutely massive in my book), I wish we instead funneled the money into Prof Freyd’s survey.
To clarify, this survey is much better than nothing, even at the cost, but I wish we had instead put our faith in our researchers, I can’t imagine what it’s like to put forth a strong research plan like Freyd did, and then see the university go with the outside research group.
That said, “intentionally crippled” is (for so many reasons) the kind of language it gets harder and harder to trust Harbaugh on. It’s clearly wrong, if he means it seriously it’s obnoxious, and the “it’s just jokes” routine gets more and more tired each time I see it (and I say this as a fan of the site).
Thanks for your comment.
I’m calling the AAU survey “intentionally crippled” because Hunter Rawlings promised participating universities that the AAU would strip the names of universities from the released data. This was done for political reasons.
Without this information researchers will be unable to use standard econometric methods to do the most important thing that could be done with this data: study the effectiveness of policies, interventions, and random events that vary across universities and time on rape. If this was an NSF funded research project, redacting this crucial data would mean it would almost certainly be rejected by peer review, and because it violates the NSF’s own data release guidelines.
It cripples the usefulness of the survey, and it was done intentionally.
The AAU survey will make it harder for real science to get done. They plan to send the survey to all students at the university, and then only analyze the data from a subset of these. I have never heard of anyone using a methodology like this before. Exposure to previous surveys induces bias that then other research have to deal with.
The AAU survey also begins by defining a number of terms related to sexual violence–and thus will undoubtedly impact how students define their own experiences.
The AAU will then not release data on a per school basis but only release certain statistics in aggregate (and it is not clear *what* statistics they will release).
In short, the survey isn’t better than nothing. Because, not doing the AAU survey isn’t nothing. Schools are going to do campus climate surveys: the white house has made sure of that. The question is: what survey and how? The AAU survey will not generate useful science (because of its methodological problems and because the AAU doesn’t even plan to release data to scientists) and will make it harder to do other surveys. How is that a good thing?
Am I the only grad student who didn’t get the email? Did I wrongly assume that all students would be included?
I assumed that “intentionally crippled” had to do with the fact that when I tried to take the survey I received a ‘session error’ message. It makes me really sad that my experiences at UO made me wonder if the link strategically went nowhere instead of just assuming that there was a tech problem.
If you have been to the counseling center they probably just used your records to answer the survey questions on your behalf.