In part 1, I began expanding on the poster session I presented at NERCOMP2013 on massive open online course (MOOC) pedagogy, technologies, and the role of the instructor, along with some basic history, the two types of MOOCs that have evolved to date, and some observations about technology and pedagogy. In part two, I aim to expand a bit more on my poster: the role of assessment, credentialing, copyright, and some of my MOOC principles.

Apostolos Koutropoulos’ 2-part article on MOOCs in Higher Education

Assessment and credentialing

When thinking of courses we invariably think of how we will assess mastery, or at least gain some understanding of those who complete it. This is also true for traditional courses that are offered for credit. Alongside assessment there is usually a credentialing issue, which in traditional education is taken care of by awarding degrees or certifications. Even if there is no summative assessment of mastery by an SME, there ought to be some sort of formative assessment to let the learner know how well he or she is doing in the course and what areas still need to be worked on.

cMOOC practice

Currently, cMOOCs generally don’t provide formal instructor assessment. (See Part 1 of this series if you need the definition of “cMOOC.”) There have, however, been mechanisms for peer review, feedback, and “remixing” of knowledge and information. These seem to be at the core of the cMOOC identity. These peer reviews are by identifiable peers. The technological decisions made in cMOOCs that allow distributed knowledge to be funneled into the course’s town square make it possible for participants in a cMOOC to demonstrate their work and get feedback, or to collaboratively work with one another.

cMOOCs typically don’t credential their learners; however, there have been some MOOCs that have begun offering badges as part of their of the learning process. These badges reward learners both for “staying the course,” (demonstrating certain behaviors, outcomes, or deliverables), and for deviating from prescribed paths (forging your own learning). Typically, Mozilla’s OBI is used, but Purdue’s Open Passport is also an option. Only one MOOC, mobimooc (2011), has awarded certificates of participation to learners that met certain criteria.

xMOOC practice

Assessment in xMOOCs utilizes automated testing and anonymous peer reviews. Formative assessment can be undertaken while students view course materials, but also as part of module quizzes. Quite a few courses seem to work on mastery grading, which allows learners a lot of attempts to pass a quiz in a satisfactory manner. There are also quite a few courses that give learners a limited amount of time and a limited amount of retries at these auto-graded assessments.

Peer assessments tend to be assignments of shorter length (250 to 700 words) around writing prompts. These assignments are graded anonymously by peers with a published rubric. In order for a learner to see their own grade on the assignment they need to review at least three peers. Anonymous peer reviews are matched automatically by the LMS and the feedback received can vary greatly. Rubrics for assignments are a way for learners to assess each other; however, if learners aren’t trained on how to provide good feedback, or don’t understand the rubric, they may not grade good assignments accurately, nor will they provide good feedback. They don’t always count participation in the discussion forums of the course as part of the assessment criteria for the course; but if it is counted, it’s only counted quantitatively, not qualitatively.  

xMOOCs commonly award a certificate of completion if a learner achieves over a certain score on assessments. The threshold for this score can vary, but it can start at 50 percent. Since the criteria vary on how to achieve a certificate of completion it’s hard to really compare what the value is, if any, of a certificate of completion.

Comparing practices

The challenge thus lies with the purpose of the course and the value of assessment. cMOOC learners should be able to demonstrate what they learned in manners that are meaningful to them and applicable in their environments. xMOOCs seem more focused on replicating existing structures, thus graded and time-constrained summative assessment seems to be a staple of the course. Assessment by an SME is difficult in massive environments. What it means to be assessed, how assessment is undertaken, and for what reasons needs a fundamental rethinking in the MOOC context. Replicating existing structures is not an appropriate course of action, and herein lays the major pitfall for MOOCs. If you replicate those structures too closely, free learners may demand free accreditation.

Copyright, Creative Commons, OER

Given that MOOCs base their genesis and existence on freely available materials, through Creative Commons (CC) licensing, Open Educational Resources (OER) repositories, or Open Access publishing, I firmly believe that you can’t call something a MOOC if there is a barrier to entry; it’s not “open” if you charge for it. Courses that charge to be part of the course don’t fit the MOOC model. They may be massive online courses, but they are definitely not “MOOCs.”

There is some disagreement among MOOC participants as to how free a course has to be in order to qualify to be a MOOC. For instance, some believe that if a paid textbook is required it disqualifies the course as a MOOC. I don’t. One can get the textbook from a library and still participate in the MOOC. Some might believe that Coursera’s signature track doesn’t qualify as a MOOC because it costs. I don’t; the signature track is above and beyond the free version. This could be thought of as freemium MOOC: the certification might cost, but the cost to get the knowledge from the course, sans certification, is still basically free. My final MOOC principle is that if you are using open resources, then any new knowledge generated should also be open.

With that said, cMOOCs seem to strive to be as open as possible. cMOOC creators try to use open content, as well as release their MOOC content under an open license, or leave the MOOC available after its end for anyone who wants to use these resources. This is encapsulated in the four types of activities of cMOOC: aggregate, remix, repurpose, and feed forward (Siemens et al, 2012). These four activities also encapsulate the ethos of the cMOOC. After all, aggregation, repurposing, and remixing is not easy when encumbered by traditional copyright. Without allowing open content to be fed forward, others cannot aggregate, remix, or repurpose.

xMOOCs tend to retain copyright of all material. A brief survey of the three big platforms (Coursera, Edx, Udacity) earlier this year shows that their terms of service are strictly in opposition of the four original MOOC tenets. xMOOC aren’t actively creating open materials. However, this may change. In March 2013, edX announced a change in its terms of service that make the default for its materials public domain. This may have to do with the fact that edX is a nonprofit venture, but it would be interesting to watch this space. The opportunity here is one of notoriety. If your institution is associated with quality open work, that could have a potential positive impact to your campus. However, good work of that magnitude could cost a lot depending on the discipline and the course.

Initiative: Personal or institutional?

In thinking about who initiates the MOOC, there are two usual suspects: the faculty and the institutional administration. This isn’t really a cMOOC/xMOOC question; rather, the idea here is to think a little about the implications underlying who initiates the conversation and any eventual implementations of a MOOC.

Major MOOC efforts need to be faculty led, not institutionally forced. This means that course design, technology selection, and implementation needs to come from faculty and instructional designers, and not based primarily on which provider the university has a contract with. However, MOOC efforts do need institutional support if they are going to succeed. A group of faculty can design, implement, and facilitate a great MOOC, but the university needs to create conducive environments for MOOCs to flourish, especially with interdisciplinary topics. Traditional courses are siloed into specific departments; however, MOOCs have an opportunity to break down those traditional barriers to create courses that touch upon many interconnecting disciplines.

MOOC efforts need time for design and internal reflection. What works for one MOOC in one discipline may not work for other courses in other disciplines. This means that you can’t force anyone to facilitate or develop a MOOC, and you can’t just take an existing course and put it in MOOC format. Also remember that MOOCs are experimental, and we should be sharing our findings with the community so that we can improve upon them. The challenge here, for institutions, is to enable MOOCs as a two-way learning tool and not worry about being left out of the party, as some institutions might feel.

Rules of thumb

Regardless of the MOOC you plan on offering, I propose four rules of thumb: Put learners first; do instructional design first; pick your faculty carefully; and don’t worry about the dropout rate!

Under no circumstances is it acceptable to just close the doors of the course because things aren’t working out. This is one important way in which you can think of your learners first. We saw a bad example of this early in 2013 when Fundamentals of Online Education closed its doors after just one week online (Kolowich, 2013a). The course did have issues, but instead of working through them and learning from the experience, the doors were closed with little notice to learners. Just like in traditional courses, if something isn’t working, modify it on the fly. In MOOC cases, your support team should be there to help the instructor(s) of the MOOC to resolve issues. Closing your course’s doors is bad for the learner, and subsequently bad for your own reputation.  

Second, instructional design should come first. While it’s healthy for educational technologists to know of the affordances of each platform, and how those affordances fit with pedagogical goals, there is no reason to go with one platform over another exclusively. We shouldn’t adopt the same stance with the MOOC-LMS as we did with our traditional course LMS. We ought to be open to educational experimentation for the benefit of teaching and learning. Sticking to one technology or provider is potentially detrimental to our learning process.

Third, when deciding who will facilitate the MOOC, if it’s institutionally supported, there needs to be some expectation setting. Recently a professor quit his MOOC because of philosophical differences over how the course should run (Kolowich, 2013b). While they didn’t shut down this course like the other course was, it’s still not great PR for the institution. There are institutions that do vet their online courses before they go live; I am sure that this is probably the case with xMOOCs as well in some institutions. Why not vet the instructor as well? Teaching online is different from teaching face-to-face, and facilitating a MOOC is different from both of them. The medium is experimental and instructors do need to adapt their teaching. This is how we will all learn more about teaching and learning in MOOCs.

Finally, have a better understanding of what the “dropout” numbers mean. I think that dropout is an inaccurate term because it lumps learners together who don’t belong together. For instance, those who were just window-shopping in the MOOC; those who know some of the materials and just want a refresher so they only participate sparingly; and those who are honestly interested in learning, but the course is failing them in some way or extracurricular issues are interfering with their participation. I am sure there are other categories as well.

The goals of each category of learner are different, and only the last category’s goals come close to approximating what I consider a traditional learner’s goals. Thus, we have to put aside the “funnels” (Sonwalkar, 2012) where we see many learners starting, but have few “completing,” and put aside snarky remarks comparing dropout rates between traditional online learning dropout rates to MOOC dropouts (Young et al, 2012) because there is no one-to-one correlation.

With this, go forth and MOOC, and remember, “If you're willing to fail interestingly, you tend to succeed interestingly” (Edward Albee).

References and resources

Kolowich, S. (2013a). Georgia Tech and Coursera Try to Recover From MOOC Stumble. Chronicle of Higher Education. Retrieved from:

Kolowich, S. (2013b). Professor Leaves a MOOC in Mid-course in Dispute Over Teaching. Chronicle of Higher Education. Retrieved from:

Lane, L. M. (2009) Insidious Pedagogy: How Course Management Systems Affect Teaching. First Monday, Volume 14, Number 10 - 5 October 2009 Retrieved from:

Siemens, G., Downes, S., and Cormier, D. (2012). How This Course Works.

Sonwalkar, N. (2012). Sustainability of Open Online Courses (MOOCs) in Higher Education. Keynote address presented at The Sustainability of MOOCs Forum. University of Massachusetts Boston. December 2012. Retrieved from:

Young, J., Cruz, J., Drimmer, A., Wilson, J., (2012). Plenary Panel: Evolution or Revolution? What's Happening with Traditional Online Learning? 18th Annual Sloan Consortium International Conference on Online Learning. Orlando, Fl. November 20 - 22, 2012.