A Maturity Model Approach to Measuring Corporate Learning

In my last article (“SkillCamp: Linking Learning to Business Outcomes Through Performance Support”) I outlined the importance of performancesupport in learning, using the example of SkillCamp, a global learningframework for marketing and sales at Bayerʼs Crop Science Division. SkillCamphas established a stable approach for measuring success that I will share inthis article.

Context

SkillCamp is aglobally steered marketing and sales training initiative targeting severalthousand marketing and sales colleagues across many countries.

Bayer’s Crop Science Divisionis headquartered in Germany and is structured in global, regional, and localcountry organizations. Marketing and sales (M&S) activities are organizedas commercial excellence and partially supported by a global marketingdepartment. Figure 1 (later in this article) illustrates the scope of thecommercial excellence activity as it relates to our 12-Step Go-To-Marketapproach. Global training activities are steered from headquarters.

The cornerstone ofSkillCamp learning activities is our electronic performance support system (EPSS)—anelectronic repository for all relevant and actionable information that servesas a single point of reference and truth for all global colleagues in marketingand sales as well as their key interfaces. But it also covers regularinstructor-led trainings (ILTs), virtual instructor-led trainings (VILTs),webinars, newsletters, and individual learning paths.

SkillCamp metrics approach

The metrics approachwas initiated along with the setup of the SkillCamp program, and refined withthe growth of content and structure, covering tactical, functional, operational,and strategic elements as described below.

It targets all marketingand sales units and colleagues in our organization, spanning headquarters aswell as all our country organizations. It is globally integrated into learningfor these functions.

Overall, the metricsapproach can be broken down into three stages or levels of integration that I describein more detail in this article:

• Level 1: Initial SkillCamp objectives
• Level 2: Focusing on learner performance: ourfive SkillCamp goals
• Level 3: Focusing on impact: impactmeasurement and impact improvement in SkillCamp

These three stagesrepresent our SkillCamp Maturity Model for learning metrics.

Level 1: Initial SkillCamp objectives

In our stakeholderinterviews, we learned that a successful learning approach will need to providea common and consistently used terminology or language, close cooperationbetween marketing and sales, relevant examples from our and other industries,continuity, top-down support, and a close link to implementation.

Based on this insight wedefined our first milestones:

• Totrain around 4,500 colleagues in marketing and sales functions and their keyinterfaces with a SkillCamp foundational classroom training
• Toensure a common mindset and language around our 12-Step Go-To-Market approach
• Finally,to provide relevant and actionable processes and tools for the daily work ofthose colleagues in an EPSS

All these initialobjectives were achieved. After rolling out our SkillCamp Foundational Trainingand introducing performance support for marketing and sales, employees nowtruly use a common language and common processes for commercial excellence. Ifa marketing manager from Malaysia meets a colleague from India in a regionalconference, they can quickly align and exchange their activities along our12-Step Go-To-Market approach. Figure 1 shows these 12 steps and identifies thecontent or skill areas we cover with our commercial excellence initiative. Asales manager from the US can easily contact the right colleagues in the headquartersfor specific questions, and a manager from New Zealand can benefit from a casestudy provided in France. A common language, common basic approaches, examples,tools, concepts, videos, and contacts are shared in the community via our EPSS.

Figure 1: The12-Step Go-To-Market approach

Level 2: Focusing onlearner performance: our five SkillCamp goals

Once Level 1 wasachieved and SkillCamp was up and running, we launched Level 2 of themeasurement approach. Following our performance support approach, we wanted tomeasure learner performance. This would enable us to improve learnerperformance and adapt our learning and performance offerings accordingly.

We defined five clearobjectives for SkillCamp as a sustainable functional learning and performanceinitiative. For Level 2 our target group is really the individual learner.

While it is difficultto relate any learning or training initiative directly to business success astop management measures it (e.g., market share or turnover), it is neverthelesspossible to link individual learner performance to successful learning andtraining activities or initiatives. We conceive training and learning as aperformance improvement process of the individual. Their performanceimprovement objectives are derived from business process and thereforecontribute to business success. Business success on the individual level can,for example, mean:

• Greaterbusiness impact from training—for example, a sales or marketing manager gainsgreater customer understanding based on a better leverage of customer insights
• Increasedcapability to achieve business results from learning—for example, a sales ormarketing person can easily find and use resources he needs on the job
• Greatercapability to meet emerging business needs—for example, staff can accessemerging and existing trends for marketing campaigns as well as tools and how-toexplanations on where and how to use these tools

As a result, thefollowing five objectives are defined along the individual learner’s learningjourney:

1. SkillCamp provides sustainablelearning, i.e., learning that sticks
2. SkillCamp helps learners to useavailable resources, e.g., via an EPSS called “SkillCamp Online” that hostsresources like tools, examples, plans, or videos
3. SkillCamp aims to improve the performanceof individual learners and their communities, for example, their effectiveness,speed, depth of knowledge, or contextual understanding
4. SkillCamp aims to improve sharing andcommunity-building across our organization globally, regionally, and locally
5. SkillCamp continuously improves thecontent, learning material, know-how, and messaging of commercial excellence

We measure theseobjectives with specific indicators for each of our five goals.

The hardest part wasto identify realistic, helpful, and obtainable key performance indicators (KPIs)along our five overarching performance objectives. It was not theidentification of relevant metrics that was difficult. It was the availabilityof data that drove us toward a pragmatic approach. Some KPIs were discardedalong that road; others were taken as compromises. Overall, we found that manyquantifiable KPIs are very often just a good approximation of the actualmeasure we would be looking for. However, we decided to take the available dataand work with it rather than having nothing to work with.

Examples of such KPIsare:

• Numberof people trained (plan vs. actual)
• Satisfactionlevels for ILTs
• Numberof people attending webinars (plan vs. actual)
• Degreeto which learners changed their working behavior based on SkillCamp

Specifically for theEPSS KPI, examples are:

• Monthlynumber of unique users of the EPSS
• Monthlynumber of page views
• Topicsvisited
• Topdownloads, or
• Averagevisit duration (we aim at a duration of approximately one minute, as we wantthe learner/performer to quickly find what they need and go back to the workingprocess)
• Othermetrics include shared best practices, update cycles, or content quality

All those metrics arecollected and displayed in a dashboard on a monthly basis as well as in quarterlyinfographics. Examples are provided in the results section. The monthlydashboard meeting has become a kind of steering meeting, as the dashboardreview always results in specific improvement measures.

The investment inSkillCamp and SkillCamp Online cannot be justified only via classic feedbackmechanisms like participation rates and participant feedback sheets. We mustalso ask if participants actually apply the learning in their daily life, attheir moment of need, if they actively use SkillCamp (Online) resources, andhow they interact with our performance support and learning offers. Eventuallywe should be able to see if these measures lead to more success in our marketspace—via linking it back to individual performance improvements.

However, Level 2 ofour measurement approach did not aim at measuring financial impact or showing acorrelation between learning and business results. Level 3 of the measurementapproach changes that.

Level 3: Focusing onimpact: impact measurement and impact improvement in SkillCamp

Currently SkillCamp isin the process of understanding more concretely which specific metrics help usto comprehend individual performance improvements more deeply and to identify whatwe need to measure to really demonstrate impact.

This leads to apragmatic focus along our five goals that fit the current situation in ourbusiness, which is constantly undergoing a lot of change. It makes sense toadapt the metrics reporting and communication to these changing needs, while atthe same time sticking to a clearly defined approach to ensure validity ofcollected data over time and being able to understand trends (in our activitiesand their results) over several years.

Specifically, Level 3aims at taking those KPIs from Level 2 that measure impact and then adding KPIsand ways of measuring that enable us to improve impact.

We want to find outwhat business impact SkillCamp has on the individual level. To do that, weidentify those learners who report a high impact as well as those who report noimpact at all. As a result, we are able to:

• Showcasebest practices so other employees can learn from high performers
• Improvethe SkillCamp ecosystem to increase its impact on the company
• Identifyways to adapt our offerings to further fit the needs of the organization
• Quantifythe impact of SkillCamp and show how it has improved the business and financialperformance of the company

Specifically, theprocess—according to the “Success Case Method” by Robert O. Brinkerhoff—we are currentlypiloting is:

1. Build an impact map for a training todefine the desired learning and business outcome of the training and toidentify how exactly the training is linked to the overall business strategy
2. Execute a workshop or training event
3. Do pre-selection surveys to identifysuccess-case and non-success-case candidates
4. Run success and non-successinterviews: These are interviews with participants who implemented a lot afterthe workshop as well as interviews with those who had not changed anythingafter attending
5. Analysis and creation of the successand non-success cases
6. Use of the cases to verify our goal,change the workshop or learning setup, and intense stakeholder communication

Benefits

For Level 1: We managed to establish a sustainable learning initiative afterthoroughly analyzing the needs for nine months. SkillCamp has been running for fourto five years now and has created a common mindset for commercial excellence.The metrics approach is continuously evolving and improving. SkillCamp wasintegrated into the Crop Science Balanced Score Card.

For Level 2: We can report 100 percent coverage of the identified target groups withour SkillCamp Foundational Training. Sixty-seven percent of the trained peoplestate that they have changed their behavior and use the tools and SkillCampOnline. A global commercial excellence community has been established and isactively exchanging best practices and discussing commercial excellence bothoffline and online. SkillCamp reports to global marketing at the headquartersof Crop Science. It resides in the business itself. The performance supporttool (EPSS) is used heavily—doubling its usage from 2015 to 2016.

Entering Level 3: We will also be able to better shape ourprogram with learning paths for different roles as well as skill levels—e.g.,from a rookie level to a master level. As we are measuring this in the systemand for the individual learner (for example, with the help of badges), we areimmediately using the metrics data to improve learning in the organization.Level 3 is now starting to show that SkillCamp has a financial impact on theorganization.

Challenges

Overall, it makessense to start with a common base of content that is valid for the whole targetgroup before extending to other content areas or other target groups. Communityinvolvement and identifying multipliers is key. It is imperative to align allstakeholders when building or making big changes to a large training framework—e.g.,for EPSS changes, design changes, or new search strategies. It is difficult tokeep track of and prioritize all the communication topics that can or should becommunicated within a performance-support learning framework. But it is worththe effort. It is important to take enough time to set up and run awell-planned communication framework for your learning framework. It isimportant to provide enough human resources to run a performance-based learningframework (these can be internal or external resources). Getting the data toanalyze the impact is difficult. Focus on impact, however, is key. A good ideais to link learning metrics to company performance metrics.

Evolution

We are evolving from asimple spreadsheet-based analysis toward a more integrated tool-based approachby integrating relevant metrics and parts of the analysis into our EPSSstructure. This way analysis of our data becomes available in real time andaccessible for different user groups—in other words, metrics starts to becomescalable and available as a service—for example, for our country organizations.

One aspect that helpsus to implement this is a keyword-based automated search. The keywords andfaceted search are continuously optimized among others based on the results ofour metrics approach. Measured user behavior is collected, analyzed, and thenfed back into the system to improve usability.

Our measuring journey produced a path evolvingfrom traditional training metrics to clear objectives for a holistic learningframework involving a focus on the “Learning Moment of Apply” and our performance support approaches. Thethird stage of our maturity model focuses on understanding the impact oflearning on the individual level and how to use these insights to provebusiness effect while at the same time improving learning content andapproaches.