Dispatch from the Digital Frontier: Working the Numbers Game

Itwas hit or miss with Sir William Thomson, Baron Kelvin of Largs(1824-1907). Thanks to his work in thermodynamics, we understand thatthere are absolute temperatures, and we use his eponymous scale toexpress those. An Irishman by birth, he was opposed to Irish homerule, which pleased Queen Victoria and, along with his scientificachievements, gained him British knighthood. He declared X-rays ahoax, saw no future in radio, and flatly stated that “heavier thanair flying machines are impossible.”

Butit’s the measurement thing that has really caught on for most ofus; Lord Kelvin is the one who said, “To measure is to know,”and, more famously, “If you cannot measure it, you cannotimprove it.”

Theidea of using objective measurement as the basis for objective,fact-driven analysis and evaluation persists as a fundamental tenetof the hard and social sciences. By focusing on numbers, the theorygoes, we focus on facts; observations without numbers are merelyopinion.

Overthe years, science and technology have made more things observableand measurable. As new means of observation have become available,new methods of calculating the changes we can observe have emerged. Today, our fascination with numbers and measurements and what theytell us shows up, at least in the world of Internet-basedcommunications and social media, as metrics, and even more recently,analytics.

E-Learningprofessionals are a data- and measurement-oriented lot. We want toiterate and improve the learning products we produce, and we preferto do that based on objective data rather than guesswork. Whetherwe’re talking about front-end analysis, formative evaluation,post-implementation evaluation or any other form of examination of acommunication program, we continue, however, to encounter resistanceto looking at the numbers. Our colleagues in the world of Webanalytics have been successful in opening the minds of marketingprofessionals to the value of measurements and their analysis. In theworld of e-Learning, we have the opportunity to borrow from theirsuccess, while at the same time tailoring their metrics to our needs.

Wecannot just wholesale adopt marketing analytics for our own purposes.Regardless of the moniker, the answers we seek and the numbers weanalyze to tease them out are only as good as the questions we ask,the measurements we take, and the numbers we evaluate to arrive atour answers. If we want to understand the effectiveness of a Web pagedesign, we have to articulate our definitions of “effective” and“design,” ask questions that prompt measurements related to ourconcepts, and analyze the resulting numbers in ways that aremeaningful to our question.

Sometimes, thequestions we ask don’t automatically generate meaningful or usefulmeasurements. Indeed, the questions themselves can be elusive. Takeengagement, for example, a highly desired characteristicsought by almost every Web property. How can we determine whether alearning program is engaging, whether the learners are engaged? Andwhat do we want them to engage with – the site’s content, thesite’s publisher, other site users, or others unrelated to thesite? In defining the term, we begin to understand how important ourdefinition is.

Once a definitionis in place, we are in a much better position to figure out whatmeasurements to take. The measurements correspond with the indicatorsthat our definition points to. Those indicators reflect theobservable behaviors we would like to see our learners exhibit whenthey are taking the actions or being the characteristic we seek. Inthe case of engagement, we might agree that its indicators are:

• Recency – the last time Learner came to the site

• Frequency – how often Learner comes to the site

• Activity – number of actions/interactions Learner took in response to the content

• Duration – how long Learner stays on the site

• Virality – whether Learner forwards a link to the site and/or its content to others

• Ratings – the rating Learner gives the site and/or its content

Inexamining the numbers that measurements yield, it’s clear that theyonly become meaningful when analyzed in the context of each other.Lots of visitors to a site is great, but if their visits are briefand they never return, the site is probably not eliciting theengagement that the site owner desires. Likewise, receiving highratings from users is nice for the ego, but those ratings aren’thelpful to growing an audience if appreciative users don’t tellfriends and colleagues about the value they derive from the site inquestion.

We don’t all have to be “quant guys” or “numbers geeks.”But numbers matter – to our customers, to our learners, and to us.We won’t get the questions or the analysis right every time –LordKelvin sure didn’t – but metrics matter, and they are morereadily at hand all the time.