Mobile Reinforcement Crushes the Forgetting Curve

Mobile reinforcement can make learning stick, flatten the“forgetting curve,” and boost the effectiveness of training—all while reducingcosts.

If that sounds impossible, take a look at ECHO, an appcreated by SwissVBS; better yet, considertheir results. The creators of ECHO, Shahin Sobhani, president of SwissVBS, andBrett Smith, the director of learning solutions, wowed attendees at FocusOnLearning 2017—and walked away with the award for performance support. (Shortpresentations on ECHO and other DemoFest winners are available in the Best of FocusOn Learning DemoFest webinar.)

Sobhani and Smith presented the ECHO app as a case studywith General Electric’s Water and Power division. GE puts new sales hiresthrough a two-day, instructor-led training course on selling skills. Thecompany’s practice has typically included a benchmark test at the end of thetwo days, which measured learners’ comprehension and gave GE some idea of theinstructors’ effectiveness.

Focus on learning retention

The SwissVBS team leveraged GE’s approach with an app that tooklearning to a new level, focusing on reinforcing the training and making itstick. To be effective, Sobhani said, reinforcement has to be effortful, whichincludes what he calls “interleaving” topics; it also has to incorporatedelayed feedback, and it has to be based on a spaced learning model. ECHO incorporated these pedagogical principles and used theexisting GE training framework to test—and demonstrate—its effectiveness.

Following the standard post-training benchmark test, groupsof employees—just under 100 in all—were divided into test and control groups. Testgroups used the app; control groups did not. Learners with the app receivedmobile push reminders to do the reinforcement exercises three times a week forfour weeks. Each spaced learning session lasted about five minutes. Learnershad studied nine competency areas; the ECHO app covered all of these, interleavingquestions about different areas of competency and asking that learners applythe knowledge.

“In the case of reinforcement, first of all, there is a bigassumption: that learning has already happened somewhere else,” Sobhani said.“What you’re trying to do is reinforce the person so that they will retain thatinformation. Therefore, what you do with interleaving is, if you have taughtsomeone A, B, C, and D, you put it all in a bucket, and the brain’s got tofigure out how A, B, C, and D are connected.”

The SwissVBS team created a large pool of questions andtips, as well as flashcards, microlearning modules, and coaching podcasts thatlearners could review on their own. “When you ask questions, flashcards,tips—they have to figure out how everything is connected. And that exercise,which makes it effortful, helps reinforcement and retention,” Sobhani said.“Interleaving is actually one of the more powerful tools for reinforcement,” hesaid, though it is unpopular with learners, since it makes the exercises more difficult.

Each mini-session asked the learner to answer five challengequestions across the nine competencies. Learners received feedback appropriateto their responses; the “delayed feedback” aspect meant that feedback on allfive questions came at the end, not question by question. Delaying feedbackincreases the effort required, Sobhani said. It’s based on an illusion that weall have—we think we know more than we do, he explained. When learners getinstant feedback on whether they answered each question correctly, “it’sinstant gratification,” but “if the brain thinks, ‘I got this one; I got thisone …’ and then they find out no, they got one out of five right, the wholeperception of the brain changes.”

“Delayed doesn’t mean months, weeks, days, or hours; itcould be seconds,” he said. The added effort comes when the brain finally getsthe feedback and has to go back and “see what the right answers were and whythat was the right answer”—seeking the solutions, rather than having the rightanswer instantly appear.

Dynamic testing improves engagement—and results

In addition to answering challenge questions in ECHO, learnersgot coaching and suggestions for further resources. The app uses a dynamic testengine that collects and tracks learners’ responses—then targets coaching,resource recommendations, and future questions to their weak areas. “Thequestions that get sent to the learners are exactly the questions they need,based on where their skill set and knowledge retention is at that time,” Sobhanisaid.

The dynamic test engine is an element that Sobhani said wascritical; with it, “the messaging and tips can become much more fine-tuned tothe actual learner instead of being a general tip,” he said. The engine “allowsthe application to learn from the learner, so, as you continue to use the app,and it sees that you’re weak—let’s say there are five competencies that theorganization is trying to make you learn, and it sees you’re weak in three ofthem—the tips, the modules, the flashcards, the questions it starts giving toyou are going to be focused on your weaknesses.”

It seems logical that this approach would improveperformance. What might be less obvious is that it also increases engagement.“What we’ve seen is that adoption has gone up because of that. If you get anapp which is basically a polling app, and it just sends you a whole bunch ofquestions, and the next day it sends you a whole bunch more questions that areon the same topic that you already know, you’re going to start not using itanymore,” Sobhani said. “But if the app starts giving you stuff that you don’tknow, the chance of engagement and adoption just goes up.”

SwissVBS is constantly working to improve the mobilereinforcement app. Three reinforcement sessions a week seem to be about right;many customers choose a Monday-Wednesday-Friday reinforcement schedule. “It’snothing to do with delayed feedback, but giving that one day rest betweenreinforcement—the studies have shown that that helps as well,” Sobhani said. “Whathappens there is, the brain starts losing a little bit—memory leak—on thattopic and then immediately, the next day, it gets the topic back. That makes iteffortful; there is an exercise they have to do, and it forces the brain to seehow all these things are connected.” Some clients have tried usingreinforcement twice weekly, which works, but not as well, he said. “One time aweek? You might as well not do reinforcement.”

Four weeks after completing the in-person training, bothgroups of GE employees—test and control—took a second benchmark test. Controlgroup learners, who had not used the ECHO app, scored an average of 15 percentlower on the second test than they had on the first. But those who had used theapp three times a week scored, on average, 20 percent higher than they had onthe first exam—a net difference of 35 percent and evidence that, rather than succumbto the forgetting curve, those who used the app had actually improved theirrecall and understanding over the previous month.

A typical reinforcement program lasts four weeks, and somego as long as six, Sobhani said; length is partially dependent on the amount ofcontent, but, he cautions, “You can only do so much in a learning reinforcementplan. … You’re only reinforcing the core concepts; you’re not reinforcingeverything.”

It takes a few weeks to create a reinforcement program, soplanning should be concurrent with designing and developing training; SwissVBSlooks at the training date and works back from that to ensure that thereinforcement is ready to go the moment learners need it.

“The most important point about reinforcement is you haveto—you have to—I don’t know how much more to emphasize it—you have to doit literally the day after class is over. The studies are showing that evenwaiting two or three days, you’re losing so much,” Sobhani said. “Therefore, alot of our customers are seeing reinforcement as part of the entire ecosystemof learning and the learning journey. So it’s not three days of class andyou’re done; it’s three days of class and 30 days of reinforcement—and thenyou’re done, then you get your certificate, then you get recognition that youfinished this course.”

Sobhani is so convinced of the difference a reinforcementprogram makes that he believes there’s no point in investing in eLearning ortraining without it: “If you’re going to spend $200,000 on your learningcampaign, but you don’t have a reinforcement campaign? Don’t do it! Don’t wasteyour $200,000; 90 percent is gone within a week!”

Analytics sweeten the pot

The mobile reinforcement program demonstrably improvedlearners’ retention of the training. But that was not the only benefit thattheir managers noticed. Included with the app was a manager dashboard withsignificant analytics data—a key component for GE. “It was the analytics thatsold them on this,” Sobhani said.

The app targeted content to each learner’s weaknesses. Theanalytics data tracked individual learners’ performance and progress. It alsoprovided aggregate data on the learner cohort, far beyond whether learnerscompleted the training or answered questions.

Comprehensive data enables training managers to improve thetraining by looking at behaviors and patterns. “If 80 percent of your group gotthis competency wrong, it’s probably not them or the app; it’s the way youtaught it,” Sobhani said. “That’s a behavior. Now you can adjust.”

Improved training clearly benefits the next cohortsof learners; current learners win as well. Prior to using the ECHO app, anentire cohort of GE employees might attend follow-up training sessions thatcovered all nine competencies—even if there were areas where they did not needadditional training. With the data from the app, managers are starting toeliminate expensive training days, replacing them with coaching and one-hour webinarstargeted to learners’ weak areas.

Now, the plan is to pair learners who excel at a competencyarea with learners who are weak in that area for peer coaching. If largergroups of employees share a weak area, a one-hour webinar can cover only theneeded skill or information, reducing the need for longer, more general—andmore costly—training sessions. An additional benefit is that problem areas areidentified quickly, by studying learners’ progress data, whereas previously, GEtraining managers would react to mistakes employees made on the job over theseveral months following the initial training.

Putting business intelligence to work

“The analytics is very powerful at looking for behaviorchange and patterns. The retail industry has been using these analytics foryears,” Sobhani said. “We’re just using the power of business intelligence inthe learning space.”

Sobhani is excited about the success he’s seen with theSwissVBS app and the future potential. “Mobile reinforcement is so new thatpeople don’t even realize that they need it. Except that they’re dealing withthe crisis if you don’t have it: Why are people not getting this? Why arepeople making mistakes? How can they be forgetting it?”