Willingham’s literacy research methods declared invalid

For four months, UNC heard the same thing over and over again.

Sixty percent of 183 athletes admitted by the University between 2004 and 2012 read between a 4th grade and 8th grade level, according to a former athletic learning specialist.

But this narrative, first set in place by a CNN article, has been changed by three outside professors hired by UNC to investigate the claims.

The professors, from three out-of-state schools and each paid $5,000 in nonstate funds, separately determined that Mary Willingham’s methods of researching athlete literacy were not a valid gauge of the subject.

“It could have been hugely embarrasing for us, and it turns out that it wasn’t,” said Executive Vice Chancellor and Provost Jim Dean in an interview Sunday night.

About 6 or 7 percent of athletes in her sample had a vocabulary between a 3rd grade and an 8th grade level, according to the external findings of Nathan Kuncel with the University of Minnesota and Dennis Kramer with the University of Virginia, respectively. Lee Branum-Martin with Georgia State University also issued a report.

“If we accept the grade equivalents as meaningful (which I do not believe) then 6 percent of the total group would be the most accurate figure for students with grade equivalent vocabulary between 3rd and 8th grade,” Kuncel said in his findings.

The investigators, tasked with questions related to using the the Scholastic Abilities Test for Adults reading vocabulary as a measure of literacy, determined that the 25-question, multiple choice vocabulary subtest, is not a valid measure of reading grade level and literacy.

UNC announced it would pursue an outside evaluation of Willingham’s data in mid-January, and contracts were signed by the three researchers in mid-February.

In a statement, Willingham questioned the independence of the review and why the professors did not contact her or her co-investigator, an educational psychologist, about the findings.

“The fact that they engaged in this exercise without ever seeking input from me or my research partner, and without the raw scores, or an examination of the full battery of tests (on a majority of these same athletes) available in Accessibility Resources speaks volumes about the true motivations behind today’s press release,” she said.

The researchers were asked whether it was possible to determine a student’s reading grade level based on SATA RV and writing subtests, SAT verbal scores and one-on-one work with students — all methods Willingham said she used.

But the researchers say they were not given that data, only the SATA vocabulary scores.

“It was pretty clear that the claims about literacy were based on the SATA test. As far as we know, there’s no way to use SAT exams to get to a reading level, so we didn’t think there was any point in including that,” Dean said.

Another discrepancy is that Willingham has said her sample size was 183 athletes, but Kuncel said in his report that only 176 test scores were included in the data set.

Kuncel said he was approached by Debbi Clarke, who serves as an adviser to UNC’s Student-Athlete Academic Initiative Working Group , and had no opinions on the scandal before he did his research.

“I looked at the data that was provided to me and examined several of the claims that were made about it, so the report talks about what the data does and doesn’t say as far as I’m concerned,” Kuncel said in an interview. “I hadn’t followed sports news and such very much.”

Kramer said as far as independency, he evaluated the data for six weeks and did not know who the other investigators were until the release. Kuncel said he evaluated the data for a few weeks. Branum-Martin could not be reached for comment.

For four months, UNC’s administration has fought against the claims of this one employee, and Willingham has fought back.

Dean called the research “unworthy of our University” and a “travesty.” At packed faculty meetings, some faculty said the claims victimized athletes.

Even UNC point guard Marcus Paige weighed in. “Trust me, we all can read and write,” he said at a Board of Trustees meeting in March.Shortly after Willingham went public with her research findings in a CNN article, UNC’s Institutional Review Board determined that she needed to reapply for a license for her research, due to potentially violating FERPA and other privacy laws by possessing the names of athletes.

When Willingham applied for IRB approval in 2008 and 2013, she said she would not use names of athletes.

“This is private information,” said vice chancellor for research Barbara Entwisle in an interview in January. “I don’t know that they (student-athletes) had the opportunity to say that their data could be used in that way. I’m really very concerned about individuals in the data set.”

Willingham declined to comment on Friday’s developments beyond her initial statement.

“For now I will just say that I am disappointed that the university neglected to take even the most basic steps to ensure the integrity, impartiality, and fairness of its supposedly “independent” review of my data.”

Dean, however, said for now he feels relieved.

“I’m sure that we’ll hear back from people who have questions about the analysis, but we feel pretty confident that we’ve done this the right way,” he said.

“While this particular chapter may be closed, we know that we have a lot of work to do to get all these issues right.”

university@dailytarheel.com