The Elusive ROI for Learning Through Technology

Fewwould dispute the convenience, low cost, and high efficiency of learningthrough technology. Whether eLearning, blended-learning, or mobile learning, itis usually just in time, just enough, and just for the user, which is the idealform of customization and convenience for participants. At the same time, forlarger audiences, eLearning represents a tremendous cost savings.

Executiveshave learned to love eLearning for its convenience, its timeliness, and its lowcost. Learning through technology makes it possible for large numbers of peopleto learn at the same time without leaving the workplace. What executive wouldnot love this? The problem is that the results are not always there.

Someprofessionals measure the ROI for eLearning on the basis of cost savings alone,comparing eLearning to the facilitator-led counterpart. When large numbers ofparticipants are involved, this cost savings is very dramatic. This isacceptable only if the outcome would be the same from learning throughtechnology when compared to the facilitator-led alternative. Unfortunately,that’s not often the case.

The challenge

Considerthis real example: Each year, the global salesteam at a major computer company launches over a hundred (the numbers arerounded for convenience) new products, services, and upgrades. The challenge isto prepare the sales force to sell them. A few years ago, facilitator-ledproduct training was offered regionally, monthly. Now, eLearning or mobilelearning is developed for each product. For every module, Level 3 (application)and Level 4 (impact) objectives are developed. For one product, a Level 4objective was for 80% of participants to sell the product in one month. For theclassroom version, this would mean that 20 participants out of 25 would sell theproduct in the first month. At this rate, a 150% ROI is delivered when comparedto costs. For the eLearning version, only 20% achieved a sale in one month.This would mean that five of the 25 participants would sell the product in thefirst month. However, because of the low cost of eLearning, the ROI is 450%(rounded). An obvious question is: Which is best? The answer is not socomfortable. A higher ROI is achieved with eLearning, but with less businessimpact. That is disturbing for the clients and the sponsors who must pay forthis.

Based on dozens of ROI studies at the ROI Institute,the results at the application and impact levels are usually less thantraditional facilitator-led versions. It does not have to be this way, but thestudies suggest that’s the case.

Typesof data

It is helpful to think about the types of data that youcan collect from any type of delivery process, whether it is facilitator-led, eLearning,blended learning, or mobile learning. These are shown in Figure 1.

Figure1: Status of measurement

Regardlessof mode of delivery, these measurement options are possible. In our studiesthrough the ROI Institute, we have seen mixed results in terms of Level 1, whencomparing eLearning and facilitator-led learning. Reaction measures withtechnology are lower, while learning measures with technology are often higher whencompared to facilitator-led. A significant drop off occurs at Level 3 (application).Participants in technology-based learning are less likely to actually use whatthey have learned. At Level 4 (impact), a reduction appears that parallels the Level3 reduction.

Causes

Whyis this the case? Our analysis points to these issues:

1. Facilitator-led learning is morelikely to have application and impact objectives (Levels 3 and 4). Afacilitator can teach to those levels of outcome as he or she describes the on-the-jobapplication and places emphasis on achieving impact.
2. In facilitator-led learning, apersonal bond is often developed with the facilitator and with other participants.This commitment to achieve results, beyond the classroom, is often an importantpart of the program and the learners are more likely to apply the content.
3. Application tools, built into theprogram, are more likely to be a part of facilitator-led learning thantechnology-based learning. These tools enable the participants to actually usethe content in the job setting.
4. In technology-based learning, theparticipant’s manager is left out of the process, for the most part while infacilitator-led learning, the role of a participant’s manager is often clearlydefined. In many studies, the manager is not even aware that the participant isin the technology-based program. Much research has shown that involvementbefore and after the program are the two most powerful interactions to transferthe learning to the job.
5. Because technology-based learning isconvenient and sometimes optional, learners do not always take it seriouslyfrom a learning perspective. Participants will take the required courses, and maytake them even if they are not required. Because of the impersonal nature ofthe interaction with the tablet, smartphone, laptop, or desktop, it is lesslikely that the participant will actually take action and make a difference.

Notonly have we seen this in studies with our clients, but we see it dramaticallyin our own work. We offer the ROI Certification as a facilitator-led five-daylearning program. This same program is available completely online and it takes26 weeks, with weekly modules that must be completed, live chats, and a messageboard. While the content is the same, the eLearning version dives more deeplyinto the content. With a test at the end of each module, the participants knowthat they have learned. However, the results at Level 3 are dramaticallydifferent. For those in the classroom learning, over 50% actually obtain thecertification, which means they have to complete an ROI study to achieveresults (Level 3). For the online learning version, the result is less than 5%.For the live version, about 700 people complete it each year and for the onlinelearning about 300 complete it—a total of 1,000. We have a consistent complaintfrom those who participate in the eLearning version, “I know much about thismethodology, but I’m not sure I know how to do an ROI study.” Although the samecontent and the same application tools are built in, what’s missing is theopportunity to interact with other participants, to work together on projects,to learn from each other, and to present their ROI plan in front of others. Withthe facilitator-led version, there is significant bonding with the facilitator.When there is a follow-up reminder, it is more personal because they know the facilitatorand they do not want to disappoint him or her and the rest of the group.

Whatis your experience?

We encourage others to conductstudies and compare eLearning with facilitator-led programs, evaluating at Levels3 and 4, to see the difference. These evaluations could be pushed all the wayto ROI (Level 5), if necessary. The methodology that should be used to measurethe application, impact, and ROI, is the ROI methodology. It is the most-usedevaluation system in the world. Figure 2 shows the ROI methodology.

Figure2: ROI model

Tips for designersand developers

To ensure that there isappropriate application and impact from technology-based learning, you need totake some specific steps. We know from our own experience that learning throughtechnology can drive business impact. We have ROI studies to show this. To doso requires addressing several issues. Here are six important areas:

1.Develop application and impact objectives.

Application objectivesdescribe what the participant should do with what they are learning. The impactobjective describes the consequence of the application. These are powerfulobjectives that link the program to the bottom line. Although rarely presentedin eLearning or mobile learning, they can be developed in almost any learningprocess.

2.Develop content to reflect application and impact objectives.

Because objectives havemoved beyond learning, it is critical to position the content to focus on applicationand impact, essentially relaying the message that the program is not successfulunless application occurs and there is a corresponding impact. This isreinforced with case studies, videos, demonstrations, exercises, games, andexamples to help with application and impact.

3.Design for application and impact.

As you design theprograms, build application tools into the process. Include action plans, checklists,guides, templates, and other tools. This is critical to ensure that applicationand impact occurs. These tools facilitate the use of what participants arelearning and suggest that application and impact are absolutely necessary.

4.Create an expectation for results.

Sometimes it is simpleto define success. Participants must know that success is not achieved when theyhave learned the content, but only if there is a corresponding application andimpact. This effort goes beyond developing objectives and focuses onemphasizing and describing program success. Part of this may include definingresponsibilities that shift the participant’s responsibility beyondparticipating in the program. Yes, it is important to participate in theprogram, complete it, and learn what is necessary to be successful. But it ismore important to actually apply it and have an impact. This is part of their responsibility.One organization included a statement of responsibilities at the beginning ofeach technology-based learning program that defined what the CEO saw as the participant’sresponsibility in the program.

5.Involve the participant’s manager.

Almost any research onthe transfer of learning to the job, will underscore the importance of the participant’smanager in achieving results. For the most part, technology-based learningprograms seem to ignore the manager. Sometimes, participants take programs ontheir own time and the manager does not know about it. Still, there are timeswhen it is helpful to let managers know that their employees will be involved. Providea few low-level activities to encourage manager involvement, both before participantsactually become involved in the program and after they have completed it.

6.Apply the principles of learning transfer.

Perhaps there is no areawithin learning and development that is as important as the transfer of learningto the job. Most typical actions work well in the learning-through-technology environment.Several valuable references on transfer strategies are available to ensure thatlearners use the learning on the job, such as Beyond Transfer of Training: Engaging Systems to Improve Performanceby Mary L. Broad and Making eLearningStick: Techniques for Easy and Effective Transfer of Technology-SupportedTraining by Barbara Carnes.

Summary

ROI for learning throughtechnology can be an elusive number, but it does not have to be. Unfortunately,many technology-based learning evaluations are revealing deficiencies with on-the-jobsuccess, making it difficult to convince management that eLearning or mobile learningis adding significant value. Learning through technology is convenient, timely,and in most cases, extremely low cost—sweet words to an executive. Howeverthese same executives want results. They need assurance that learning throughtechnology will drive application and impact.

By following someprescribed steps, designers and developers can make a huge difference in thesuccess of eLearning, pushing it to impressive levels of success forapplication and impact. When this is the case, then you can easily base the ROIon the cost savings generated when comparing learning through technology to afacilitator-led counterpart.