You’ve Got a Ton of Learning Analytics Data. Now What?

You’ve got data. Lots of data. Your LMS and LRS collect allkinds of information about learners and their activities: You know whatlearning modules each learner has registered for, how much time they’ve spent,whether they watched an entire video or only the first minute, what their quizscores were, and who they chatted with during the virtual class session. Youknow who did what; in addition, you’ve got aggregate data about time spent andprogress recorded in eLearning department-wide—or even company-wide.

Great!

Now what?

Managers, instructional designers (IDs), and developers areconstantly told that they can use data to improve learners’ engagement, results,or job performance, to personalize eLearning, or to make learning stick. Arethese claims realistic? What data do managers or designers need, and how shouldthey use it to achieve any or all of these goals?

What data should you collect?

Data can help a manager or an ID improve learners’ jobperformance only if it is the right data and it is used in a strategic way. Tobegin, the manager has to have a clear goal in mind. What performance istargeted? What improvement is sought? What will change look like? The answersto these questions determine what data to collect—but even that is not enough.

Change has to be measured against something: currentperformance. You won’t know if employees’ performance improved after theeLearning if you don’t measure performance before the eLearning.

“To correlate to on-the-job performance, a baseline isneeded prior to the learning intervention,” said Sean Putman, an instructionaldesigner and xAPI expert at Learning Ninjas. “Managers can compare the baselineto the post-learning intervention data to help answer the question, ‘Did thelearning intervention work?’ With the right data collected, this comparisoncould be done—and is being done today—by companies.”

To further the goal of using learner data to evaluatewhether eLearning is effective, Jigsaw’s virtual classroom platform collectshundreds of “data points” on learners and their interactions with the learningtools on the platform and with one another. Jigsaw also helps managers evaluatedifferent kinds of learning.

“We’re looking at the value of engagement learning versuslearning by PowerPoint,” said Ginger Ackerman, Jigsaw’s vice president of salesand marketing. She distinguishes between lecture and PowerPoint slides—which shecalls “guides” for the instructor—and “learning tools,” which are activitiesthat foster learner engagement. On Jigsaw, these include chats in breakoutrooms, large-group discussions, collaborative work using shared whiteboardspace, and participation in role-playing exercises.

While Jigsaw—like other learning platforms—doesn’t actuallycollect job-performance data, the data it does collect helps managers decidewhether eLearning is effective. For example, managers measure pre- andpost-learning job performance; the learning platform captures data on whatactivities learners engaged with (and for how long), what they did in arole-play exercise, and their scores on quizzes. IDs, virtual classroominstructors, or learners’ managers can then “tie that together with applicationlearning,” Ackerman said. “Because we can do role-play reporting, because wecan do individual project work as well as breakout rooms, the results of thosepieces come back to the facilitator. And the facilitator can then review that withthe performance.” They “absolutely” can and do track and correlate individuals’learning activities with job performance, she said.

Creating a better learner experience

In addition to collecting data about learners’activities—time spent, material viewed, test results—learning systems generallycollect demographic data on the learners themselves. That might be conventionaldemographics like age, but what might be more relevant is job data: What is therole of the learner in the company? What tasks does she do? How much experiencedoes she have?

Mapping this data against baseline and post-learning jobperformance data can help determine whether an eLearning module correlates toimproved job performance. This analysis can also help managers map outeffective training plans for employees in different job roles or with varyingamounts of experience, thus personalizing and targeting the learning experiencefor future learners.

“When data is collected and compared to actual businessdata, we can see what actually works for a given demographic. When we havethe knowledge of what works, we can create learning paths to personalize the learningfor future learners by demographic,” Putman said. “As data is collected,patterns can be used to provide alternate paths through content based onanswers given or selections made.”

Ackerman said that the data collected and aggregated inJigsaw’s virtual classroom platform aids IDs in improving the learners’experience. “When you look at that information from an organizationaggregately, what it does is it provides the instructor the opportunity tounderstand what type of information they need to be building and designing fortheir corporate training going forward.”

Collecting and analyzing learner data can enhance thevirtual classroom experience for both instructors and learners, Ackerman said.For instance:

• Improving instructors’ skills—Jigsaw measures instructors’ use ofplatform tools. Managers and IDs can use that data to determine whether aninstructor is spending most of the virtual session lecturing—or engaging withlearners. They can then identify which instructors might need coaching to developa more engaging, activity-based teaching style.
• Using learners’ preferred tools—Dataabout which tools learners engage with most readily, or most often, helps IDsand virtual classroom facilitators create lessons that emphasize the tools thatlearners prefer and are willing to use. It can also highlight a need to teachlearners how to use other tools, helping them get more out of their eLearning.
• Providing useful job aids—Data collected about learners’ use ofmaterials, both within the virtual session and as downloaded job aids for usein the workflow, helps designers create tools that will improve learners’ jobperformance.

Not all data collected counts or measures activity. Qualitative data,such as comments gathered from chats, discussions, role-play exercises, orobservations of learner performance, also provides valuable information.Managers and IDs can study the vast amounts of quantitative and qualitativedata they’ve gathered to gauge the effectiveness of eLearning—and to improvefuture learners’ experience.