By Robyn Defelice
I still remember the first time I heard, “Those are vanity numbers. What am I supposed to do with those?”
That was 25 years ago, and I didn’t know what to do. I thought I was showing results that had value.
Stop Collecting ‘Vanity’ Metrics
It’s harder to show up as a trusted advisor to your stakeholders and leadership when the data does not support your recommendations or actions. If the data you collect cannot inform decisions, those decisions may not hold up. And right now L&D still often stops at completion, satisfaction, and usage data.
These vanity metrics look useful but cannot help you decide what to do next. Actionable insights do. Actionable insights are data that inform a course of action. (Need a refresher on this term? Check out Taking Stock of Data Literacy for L&D Professionals.)
As AI becomes part of the data conversation on learning teams, having data lead decision making is becoming more necessary. Teams are being asked to use data to bring content production in-house, compress development timelines, and do more with existing resources. That shift is putting more pressure on the data itself. If decisions are not founded on solid data, those decisions do not hold up.
How do you know if you have data right now that is actionable?
Ask—and answer—these three questions about the data you already have and on any data you are about to collect:
- What am I actually trying to solve?
- How do I know it’s worth solving?
- How will I know if I solved it?
They are straightforward, but they ask different things of the same data. They work on the data sitting in your LMS today, the survey you sent last quarter, and the report you put together for leadership last month.
Look at What You Already Have
The fastest way to put these questions to work is to point them at data you are already collecting. Pick something: The annual engagement survey. The post-training feedback form. The quarterly completion report. The dashboard you send leadership.
Then run these questions.
1. What Am I Actually Trying to Solve?
Look at what you are collecting and ask why you are collecting it—in other words, what problem is the data supposed to help with? If the answer is “we have always collected this” or “leadership asked for it once,” that tells you something. If the answer is “this helps us understand whether something changed on the job,” that tells you something different.
2. How Do I Know It’s Worth Solving?
Is the problem the data is pointing at one that needs your attention? Is solving it tied to performance, capability, or something the organization is asking the learning team to support? If the answer is yes, the data has a job to do. If the answer is unclear, the data may be collecting evidence of something that nobody is going to act on.
3. How Will I Know If I Solved It?
What would it take to know whether you have resolved the problem or are making progress? Time? Attention, respondent goodwill, your team’s capacity to analyze…?
If the cost is high and the outcome is low, the approach needs to be reconsidered. If the cost is reasonable and the outcome informs action, it is easier to justify continuing.
What This Looks Like in Practice
Let’s take a look at a common scenario for how these play out. As you read through it, consider what is functioning as a vanity metric, what is actionable, and what sits in between as directional data. Directional data suggests a relationship, but it does not confirm it.
You are asked to provide a report on annual compliance training. You have completion rates, satisfaction scores, and maybe a pass rate. It looks complete. Then you are asked how this training has impacted incident reports.
1. What Am I Actually Trying to Solve?
You’re being asked whether the training is aiding in the reduction of incidents, not just being completed to demonstrate that annual compliance training was done.2
2. How Do I Know It’s Worth Solving?
Incident rates connect to safety, regulatory exposure, and cost. If training is not helping to increase or reduce those rates, the company needs to know.
It also matters whether the training reduces incidents in a way that lasts over time. That inspires you to look at incident data from the last three years. Are incident rates tied to training interventions? Do incident rates improve in the first few weeks and then diminish over the final quarters of the year?
3. How Will I Know If I Solved It?
You need data that provides insight into incidents alongside the training data—preferably a baseline number on incidents prior to training versus post-training and tracking that continues past the immediate post-training window. Your team needs to have a person analyze that data and provide those correlations, if that data is even accessible or collected.
The same incident data can read as a vanity metric, directional data, or an actionable insight, depending on what else is being considered. Here are four ways this scenario can play out.
| Scenario | What the Data Shows | Signal Type | What You Can Conclude |
| Incidents dropped (only data considered) | A change occurred | Vanity | It shows something changed, but not what caused it or what to do next |
| Incidents did not reduce | No change in incidents despite completion | Directional | Completion alone may point to a gap in the training, not necessarily the cause |
| Incidents dropped (timing and focus considered) | Change in incidents in relation to training | Directional | You can begin to examine whether the training may have contributed, but it is not conclusive |
| Incidents drop, then increase over time | Short-term change, not sustained | Actionable & Directional | The annual training does have an effect, but it does not appear to sustain change over time; whether more training is needed is not known |
When you put those together, you can start to see whether anything changed. If incidents dropped and that is the only data being considered, it functions like a vanity metric. It shows something changed, but not what caused it or what to do next. This is where the baseline becomes important. The value of this actionable insight is not just in looking at the present, but in understanding what has happened over time.
If incidents did not reduce, the completion rate may point to a gap in the training, not necessarily the cause. If incidents dropped, your stakeholders may view the training as effective. If incidents drop in the weeks following but increase over the next two quarters, the training had an effect, but it was not enough. That is the difference between reporting activity and informing a decision.
The same logic applies to other metrics that learning teams collect regularly, as shown in the table below. The numbers in the left column show activity. On their own, they do not lead to a course of action.
The right column shows what those numbers look like when paired with data that does. That column also hints at all three questions: the problem being solved, the worth of solving it, and what effort is needed to know you’ve succeeded.
| Vanity Metric | What It Tells You Alone | Paired with Actionable Insight |
| 87% completion rate | 87% finished the training | Of the 87% who completed, 64% reported they did not know how to apply the content in their role |
| 312 attendees | 312 people showed up | 312 attended; managers reported behavior change in roughly 1 in 4 within 60 days |
| 92% pass rate | 92% passed the assessment | 92% passed; help desk tickets on the topic dropped from 40 per month to 12 per month within 120 days after rollout, data will be looked at again at 365 days after rollout |
| 4.2 out of 5 satisfaction | People rated the training favorably | Satisfaction was 4.2 out of 5; the same complaint about pacing came up in 47 of 80 open comments |
| 2,400 training hours delivered | 2,400 hours of training were completed | 2,400 hours delivered; time to productivity for new hires dropped from 6 weeks to 4 consistently across the last three new hire cohorts |
| 187 courses in the LMS | 187 courses are available | 187 courses available; 23 are accessed regularly, and 38 have not been opened in a year; the remaining 126 are flagged for review to determine relevance and use |
You can use this table with your own data. Replace the “Paired with Actionable Insight” column with the three questions for each metric. Not only can you see what data stories you are telling (vanity vs. actionable) you can begin to see the opportunities with and within your data.
Using These Questions Beyond Data
These questions do not apply only to the data you collect. They apply to the conversations around that data. When leadership asks for completion rates, the questions help you respond with more than a number and more than they anticipated! This is where you, as a trusted advisor, begin to shine.
When a stakeholder requests a survey, the questions help clarify the survey’s purpose. What problem is being addressed? How do we know it’s worth solving? How will you know if it’s been solved?
When our data produces actionable insights, a new conversation opens up. Who owns the data? Who has access to it? How is it being protected and used responsibly? In the next article, those questions become the focus as we look at data governance.
Image credit: BestForBest
