Certain words are like sparks in a puddle of gasoline. “Bias” is definitely one of those words—and for good reason. If there is something that we are doing, that we are unaware of, that is causing harm to others, then we definitely should be taking it seriously.

So, is there bias lurking in your learning data? Of course there is. Learning data is produced by humans (Shh; don’t tell the people who think that the LMS is doing all the work.) Where there are humans, there is bias, but to be able to do something about bias, you need to understand it. The terminology can be confusing so let’s get our terms straight.

Cognitive bias

Cognitive bias is a systematic error in thinking when evaluating information and acting on it.

Look at that word “systemic.” This isn’t about being mistaken or clumsy. This is about a shortcut that your brain makes on a regular basis that leads to errors. Your brain is always taking shortcuts. You have no choice. There’s just too much information in the world around you to take it all in without making some quick guesses about the scattered stimuli that you receive from your senses.

Magicians use your own cognitive biases against you to create their illusions. For more information about cognitive biases, check out Beau Lotto's TED talk. And, Jim Kline provides an example of a type of cognitive bias called a mental scotoma—the inability to see something if your brain can’t make sense of it—in a LinkedIn article.

This all may sound a bit innocuous. Where is the real evil specter of discrimination that we do without even knowing about it?

Implicit bias

Implicit bias (also called unconscious bias) results from a set of beliefs about the world that you are not consciously aware of that still affect the choices you make about what you perceive.

I grew up in the seventies watching a lot of television, and on every show about police, the criminals were Black. I don’t consciously believe that Black people are criminals, but those years of reinforcement in my impressionable years left an implicit bias that took me many years to notice. It takes a lot of work to undo these biases. There are plenty of resources for that.

Bias in L&D

But how would that manifest in L&D? I was once working with an L&D team that had spent several years getting their company to transition from instructor-led training to online training. I was helping them with a major data migration, and I asked about the records for people without computers.

They looked at me with bewildered expressions. “There are no learners without computers,” one of them said. When I pointed out that the company included facilities staff, manufacturing staff, and field workers who did not have computers, they were floored.

They had been so focused on online learning that they didn’t believe that anyone who didn’t have a computer would still be a learner. Think about how this mistake could affect decision making about deploying learning resources. Familiarizing yourself with different types of implicit bias is a good place to start.

How bias creeps into learning data—and what to do about it

Maybe you don’t believe this is a real issue. After all, learning data comes from an LMS. Users take courses and receive completions without human intervention; how could bias creep in? It’s just learning data. It’s not like we are running a statistical study.

Actually, these assertions are wrong. Humans have their fingers all over your “automated” data. Courses and learners are entered by admins. Completions that weren’t captured because of user or system errors are manually “corrected.” Reports are usually transferred to a spreadsheet so they can be “cleaned up.”

Learning data is not straightforward. In the age of Big Data, we make the mistaken assumption that we no longer need statistics because we now have all of the data. Look at the learning data in your company. Does it represent all of the learning that is happening? Is every learner represented?

Learning data collection and analysis is a statistical endeavor. We are observing a subset of activities and extrapolating to the whole. You might only be examining sums and averages instead of standard deviations and probabilities, but it is still statistics. Therefore we are introducing a third form of bias into the system.

Statistical bias

Bias in statistics is where the methods used to produce a statistical analysis result in an analysis that does not accurately describe the population.

Once you realize that you are dealing with statistical bias, you have a wealth of resources available for you to address the cognitive and implicit biases that may also be plaguing your system.

Selection bias

It may seem to you that you do not select sample populations for your learning data, but that is exactly what happens. Through course catalog design, evaluation design, communication, and the ways you make courses accessible, you have limited the pool of people who are reflected in your data.

Selection bias happens when the sample population doesn’t accurately represent the whole population. Look at the people with the most course completions in your LMS. Then look at the demographic distribution of your company. Do they match? Looking at your learning data this way is a good start to uncovering the results of unconscious bias.

Self selection is a subset of selection bias where the motivation for people to participate skews the results. This happens with course feedback. People provide feedback when they are highly motivated or highly disappointed. This excludes the majority of users who are more neutral in their reactions.

When you see big swings (good or bad) in the feedback data, take a look at the population to see if it is an accurate representation of your target audience. Find ways (like social media and departmental meetings) to reach out to people who are less likely to participate to make sure you get diverse viewpoints.

Observer bias

There is no such thing as a completely objective observer. When creating pilots, surveys, tests, and other user research, L&D departments usually lean on who they know to help. Who do L&D people know? Other L&D people and HR people.

When user research uses, as its base, information from L&D and HR people who already know the desired outcomes, bias can easily creep in. When conducting this type of research, reach out to people beyond your network. Use lists of people from projects unrelated to the work you are doing. Mailing lists of employee actions groups and special interest groups work great for this.

Time frame bias

Very often, courses have spikes in usage when they are launched and at the end of the performance review cycle. If you compare course usage but use cutoff dates that remove one of the course’s spikes, the data will be skewed. Adjust your usage data to account for deadlines and spikes that occur around reminders to complete training.

Funding bias

Okay, let’s be brutally honest. The recipients of all this data just happen to be the same people who approve your budget. Don’t get caught in the trap of trying to bring out the data that supports your cause. No one is really fooled.

The long-term viability of your team depends on its ability to produce results. If the data doesn’t demonstrate that, then you need to change your approach to the work. Use the data as a tool, not as an end product. Also, periodically review what data you are collecting and why, and ensure that you are collecting data that provides valuable, performance-related insights.

To learn more about statistical bias in business data analysis, check out this HBR article: 5 Types of Statistical Bias to Avoid in Your Analyses.

Where to start?

Back to our original question: Is there bias in your learning data. Yes, and you can do something about it—but only if you really look at how the data is created and you are open to seeing bias for what it is.

Start by looking at who is included and who is excluded and why. Then look at how the process affects those people from their perspective, not yours.

Without taking an honest look at how you collect learning data, your decisions will be based on flawed assumptions that perpetuate the status quo.

Explore leadership issues with your peers

Explore the strategies and skills required to navigate the needs of today’s ever-changing workplace with your learning leadership peers. Join the Learning Leaders Alliance, a vendor-neutral global community for learning leaders who want to stay ahead of the curve and for aspiring leaders wanting to build their skillsets.

The Learning Guild’s Alliance Membership package includes access to exclusive digital events and content curated for today’s modern learning leader, as well as opportunities to attend in-person learning leadership events held around the globe.