For Online Courses, Questions Over How Success is Measured

The results of the University of Texas at Austin’s first full-semester foray into massive open online courses, or MOOCs, are in. But advocates for the classes, which are free or close to it, and open to anyone with an internet connection, recommend looking beyond the fall completion rates — ranging between 1 percent and 13 percent — to measure success. 

In October 2012, the University of Texas System partnered with edX, a Massachusetts-based startup, to develop MOOCs, which were being widely touted as a disruptive force in higher education delivery. 

The UT System invested $5 million in edX and committed to spending another $5 million on course development. It was in good company; Harvard University and the Massachusetts Institute of Technology had each ponied up $30 million to launch the service. Rice University, Texas’ elite private institution, also partnered with edX.

But the model has faced criticism for chronically low completion rates. And in those terms, UT-Austin’s early results, despite some courses being on the high end nationally, probably don't look particularly impressive to most.

Professor Michael Webber’s “Energy 101,” which had an enrollment that peaked at around 44,000 students, had 5,000 receive a certificate of completion — about 13 percent of the roughly 38,000 students who ultimately participated.

And a MOOC on drug development had a roughly 11 percent completion rate: Nearly 2,000 students received certificates, out of about 18,500 who participated.

Preliminary data on UT-Austin’s two other fall semester MOOCs show completion rates ranging from about 1 percent to 3 percent.

These rates aren't unusual. In December, at a conference in Austin, a research group from the University of Pennsylvania Graduate School of Education presented a study on 16 MOOCs offered by the University of Pennsylvania through a company called Coursera. They found that completion rates averaged at around 4 percent.

The study sparked a wave of negative headlines casting doubt on the future of such courses, which had been heralded mere months before. The backlash even sparked its own backlash. Given how easy it is to enroll in a MOOC, Kevin Carey, director of the Education Policy Program at the New America Foundation, compared relying on completion rates to deriving college graduation rates from the number of students who apply — as opposed to those who actually enroll. 

The measurement, he wrote, “mostly reflects how the traditional college mindset continues to dominate and limit the public understanding of what higher education can and should mean.”

At UT-Austin, university administrators and the professors running the courses similarly argue that completion shouldn't be the barmoter for success.  

Webber described his experience leading Energy 101 as “awesome.” For 5,000 students from all walks of life and many different countries to complete his course, he said, was no small thing. 

“I was elated with the numbers,” he said, “because it means I reached an order of magnitude or two orders of magnitude more people than I can reach in a classroom.”

Webber said the platform had limitations, including an inability to fully assess students. “I do not know if they learned the material, because I could not do assessment,” he said. “It was self-guided learning. It was edutainment. It was not quite a course. I didn’t expect it would be, so I was very satisfied.”

But he acknowledged that some of his colleagues had more negative experiences — in part, he speculated, because they may not have adjusted their expectations.

Harrison Keller, UT-Austin’s vice provost for higher education policy and research and the executive director of the Center for Teaching and Learning, said that going forward, it might be more productive for UT-Austin and other institutions offering MOOCs to tailor courses to a specific audience — high school students or working professionals, for example — rather than attempt a one-size-fits-all approach that’s largely based on the format of traditional courses.

“It’s pretty stunning that you’re having a pretty large number of people actually invest time in these courses,” he said. “That’s really exciting for the future in terms of what we can do with the MOOCs.”

While he said that some of the criticism of MOOCs has been overblown, Keller acknowledged that there are lingering questions about the concept, including the business model. He noted that the cost of producing UT-Austin’s first MOOCs ranged from $150,000 to $300,000 each.

At that price, he said, “you want to be able to take that content and use it on multiple platforms and be able to use it in different modes of delivery. That’s exactly the general direction we’re going in.”

One of the school’s priorities for this year, he said, is developing the infrastructure to support the dissemination of high-quality content through a variety of methods — not just MOOCs. 

“I think MOOCs are important. I think they’ve also in some ways been a distraction from a lot of other really interesting things,” Keller said. “I hope this year, not that MOOCs will go away, but that we’ll start seeing it as part of a bigger picture.”