In the summer of 2009, my friend Saul Carliner, now associateprofessor and provost fellow for eLearning at Concordia University in Montreal,was part of a discussion in TrainingMagazine about whether ROI (return on investment) was worthwhile formeasuring training’s value.
In that article, Carliner stated, “A 2003 meta-analysis (ananalysis of several studies of training practice) by Arthur, Bennett, Edens,and Bell, concluded that 78 percent of all training is assessed forsatisfaction. Their study also found that just 38 percent of programs areevaluated for learning, nine percent for transfer, and seven percent forimpact. A 2005 study by another research team had similar results.” (See“References” at the end of this article.)
A September 2010 eLearning Guild Research Report, Getting Started in e-Learning: Measuring Success, found that of all of the typesof measures used, almost 10% didn’t measure anything and almost 88% onlymeasured completion (Figure 1).
Figure 1: Measurement methods used by respondents in Getting Started in e-Learning: Measuring Success
More thought provoking still, the data in this survey alsoshowed that few stakeholders asked for proof that the instruction is effective(Figure 2).
Figure 2: Percentage of respondents in Getting Started in e-Learning: Measuring Success that were asked for proof ofeffectiveness
Some people think that doing return on investment (ROI)analysis is the answer. Carliner’s article says that C-level execs emphasizeperceptions over measures for a variety of reasons. One is that ROI accountinghas serious flaws.
Because much training is intendedto reduce errors, like scrap, the associated expenses are never incurred. Butaccounting systems—the ones ultimately used to track ROI—can only handleexpenses tracked on the books. Any resulting savings from reduced scrap is, atbest, a speculation and lacks credibility to many financial experts.Furthermore, many evaluation experts acknowledge that, even when performanceimproves, other factors could have contributed to, or been primarilyresponsible for, the resulting savings, such as new machinery that producesfewer flawed products.
And, he acknowledges, except for the largest traininggroups, few have the resources or skills for such evaluation activities. And,as you can see in the Guild’s GettingStarted in e-Learning: Measuring Success research report, they are notasking for it.
The Guild’s newest research report, Getting Started in e-Learning: Measuring Success, explainsthat the most fundamental problem with ROI is that it is based on measuringlearning “events,” and we now accept that the largest amount of learning in ourorganizations happens informally and in unstructured ways. Thereforeapplication of the kinds of models that measure ROI of learning events at bestprovides some indication of application and value gained from a small minorityof learning out of context.
Nic Laycock, the author of the report, agrees with Carlinerthat C-level execs are not nearly asinterested in ROI as we think that they are. When he asked a director in aprocess industry how we should measure L&D outcomes his response wasilluminating. “Nothing further than I have a comfort that my people are able,and mentally and physically well-equipped to do their work. I judge it by thelack of frequency with which my direct reports complain about not having thenecessary skills available to them. Of course I want to know we are spendingour limited resources in an efficient way…”
Laycock explains that a Corporate Leadership Council surveyin 2011 indicated that while 84% of executives were happy with training(Towards Maturity reports a contrasting 65% just two years later), less than15% would recommend the CLO and his or her team to colleagues as a support insolving performance problems. The Council’s more recent work indicates that asglobal growth gathers pace, performanceimprovements in the order of 10-20% will be required to ensure entitysurvival.
How does L&D begin to contribute to performance improvementsin the order of 10 – 20%? Beyond ROI: TheValue of Learning provides valuable approaches needed by all CLOs for establishingthe necessary partnerships and eliciting evidence of effective learningapplication, in context.
The report contains interviews with a small number ofC-level executives who gave their views about the learning function’simportance and contributions. They were also asked what they believe isimportant to measure in learning in an enterprise context. Finally, the reportcontains the views of statisticians and metrics experts concerning currentleading-edge models and methodologies.
References
Carliner, Saul. “Maybe ROI Really Is a Waste of Time.” Training Magazine. June 2009. https://www.nxtbook.com/nxtbooks/nielsen/training0609/ – /18
Shank,Patti. Getting Started in e-Learning:Measuring Success. The eLearning Guild. 3 September 2010. https://www.elearningguild.com/research/archives/index.cfm?id=145&action=viewonly&from=content&mode=filter&source=archives&showpage=3
