Performance Outcomes: You Have More Data Than You Think You Do

March 19, 2014

Things go wrong every day:

Support calls are flooding IT for help with whatis a relatively simple program
Employees routinely ignore an important company policy.
Few employees are using the costly equipment thatwas supposed to revolutionize your organization.

Sound familiar? Chances are highthat you’ve been called in to design training to solve one or more of these problems.And chances are also high that the content owner has a vague idea of theproblem, a firm idea of the solution, and a conviction that the “A” in ADDIE isunneeded in this case.

When you suggest conducting a needsassessment, the sponsor agrees that it could be valuable in differentcircumstances, BUT – metrics aren’t available, the only outcome data is a thinstack of Kirkpatrick Level 1 evaluations, and thus it would take too much timeto start from zero.

Don’t give up the analysis! Help is available!

Before you resign yourself to givinganalysis short shrift, consider: you have more data than you think you do. Theusers are communicating, the messages just aren’t neatly packaged for easytabulation. Five practical tips can get you started on an informed design that addressesreal problems.

1. Turn up the noise

A manager once responded to myquestions by identifying the goal of a requested training as “turning down thenoise.” Instead of avoiding complaints, crankup the dial. Open forums (online and in person), micro polls, and targeted surveysare good. Meeting one on one with the most vocal critics and resistant groupsis even better. Make it clear that you are there to understand, not to soothe—andtake copious notes.

2. Go with your gut, but don’t stop there

After conversations with thesponsor, critics, and stakeholders, you may feel you have a good handle onwhat’s going on. Realize that if they had a complete understanding of theproblem, they probably would have already solved it. Keep an open mind until the complexity shows up. Change champions,laggards, those who’ve attempted and failed, and successful users each have apiece of the puzzle. Don’t base your design on a single perspective, even yourown.

3. Seek out the workaround

If end users are bypassing thepolicy, the process, or the system that management wants them to use, it meansthey’re using something else. Leave your judgment at the door and discover the appeal of the workaround. Isit immediacy? Familiarity? Chunking? Personal contact? Simplicity? Users arewonderfully creative so you want to make sure your solution will do at least asgood a job of meeting their needs as their current workaround. The only way todo that is to acknowledge its advantages.

4. Codify the suffering

This will require a timeinvestment, but will yield by far the most valuable data. Who feels the pain? Whohears first-hand from unhappy users? Chances are it’s the Help Desk, ServiceCenter, and other front-line workers who pick up the phone when an end userhits the wall of frustration. Ask, “What have you heard a thousand times?” “Whatgets people angry?” “What have they given up doing?” If there is a ticketingsystem, pull the logs, but don’t stop there. End users are notorious forcalling or emailing instead of using the sanctioned system. Harvest that electronic crop. If time isshort, dump phone logs and emails into a document, and identify the mostfrequently occurring words. Entering the result into a word cloud tool like Tagxedoor Wordle will yield a powerful visual for your report to the project sponsor,as well as valuable direction for your design.

5. You are unique, like everyone else

Whatever the issue, you can betthat somewhere, someone else is already dealingwith it. Mine online tutorials, user communities, forums, blogs, and FAQs. Evenif your system or policy is homegrown, there are similar ones out there. Findthem and generate a bulleted list of reported outcomes and issues, then distributeit to users and stakeholders for a reality check. Reacting to a draft oftenclarifies hits and misses in a way that a blank slate can’t.