Great! You’ve made the decision and are ready to move forward with xAPI. You’ve decided you want better data about your learning programs. You know that most learning occurs outside the LMS and you need a way to track it. You believe the analytics that xAPI can give you will improve the way you design and manage learning. You’re ready to move forward and build your learning ecosystem. Now what?
You are going to need a learning record store (LRS) for all that xAPI data. Somehow you will need to integrate the LRS with other systems, such as your LMS and possibly your HRIS or a performance management system.
Unless you’re going to spin this up yourself, you are going to need a vendor or two (or more). But which ones? What questions should you ask? How do you know you can trust the answers?
In this article we will provide some help. We’ll give you questions to ask prospective vendors and describe what you should expect as an answer. So let’s get started!
General questions
Question 1
Please describe the extent of your xAPI support. Please provide a system documentation reference that describes your xAPI support.
Why ask this?
If a vendor has xAPI support, they should have documentation of it. Be wary of answers like “We support all the APIs” or “We can add it if you really need it.”
Question 2
Can you show me some xAPI statements with your product?
Why ask this?
Any vendor that truly supports xAPI should be able to show you statements either produced by their product (ex: authoring tools) or stored in their product (ex: LRS). If they cannot do that, you should walk away. If they can, you are in a position to evaluate the kinds of data being sent. Is it simplified like SCORM or does it offer a rich insight to the learning experience?
Question 3
What xAPI profiles are supported by your product(s)?
Why ask this?
xAPI profiles are simply a defined set of rules for using xAPI statements in certain scenarios. For example, if you want to track video consumption by your students, you will use the xAPI Video Profile. Why is this important for data providers? Using a profile ensures consistency across products. If you provide videos from Vendor A and others from Vendor B, you want the same data tracked by both. This helps to ensure consistent analytics.
Learning record store questions
Data receivers (learning record stores) should be able to receive data sent using any profile (or no profile at all), but they may have built-in reporting support for data sent using a profile.
Question 1
Do you have an LRS embedded in the LMS or do I need to have an external LRS in order to use your xAPI capabilities?
Why ask this?
Some systems have an LRS embedded with the LMS. This can be a plus because you only have one system, one place to go for data. But not everyone can run out and replace their LMS. Fortunately there are stand-alone LRSs available. The downside is that you now have two places storing learning data.
Question 2
Has your LRS passed the ADL conformance test?
Why ask this?
The Advanced Distributed Learning Initiative (ADL) is the group that brought us SCORM and xAPI. ADL has developed conformance tests for LRSs. ADL maintains a list of products that have passed the conformance tests. If a vendor’s LRS cannot pass these tests, then they are not conforming to the xAPI specification. You may not need or want full conformance, but it’s useful to know this answer so that you have your expectations in line.
Question 3
Is the LRS data portable / forwardable to another LRS?
Why ask this?
A conformant LRS should allow you to port your data to another one. No point in having all that data if it’s locked inside and you can’t get it out either now (for BI purposes) or later (when you move to a new system).
Question 4
How do you handle xAPI on a massive scale?
Why ask this?
xAPI systems can produce a lot of data. Unfortunately, xAPI is not the most efficient at very large scale. Providers that are doing more than just experimenting with xAPI have already had to deal with massive scale so their answer will give you insight to their thinking here. You also want limiters on the ability for users to inadvertently request so much data that they pull the system down.
Question 5
Are all LMS activities recorded as xAPI data?
Why ask this?
Some LMSs are recording all their data as xAPI and sending that out to an LRS. If the LMS does not contain an LRS, you will need to get one so that all your data can be in one place for reporting. If your LMS does not send data about LMS activities to an LRS, you’re going to have data in two places.
eLearning support
Question 1
Can eLearning published for xAPI or Tin Can be launched from the LMS?
Why ask this?
This is the most basic level of support for xAPI. The answer should be “yes” because the launch is going to look very similar to a SCORM launch. If you have an LMS that does not have an embedded LRS, your LMS should still support this launch and the xAPI statements should be stored in your stand-alone LRS.
Question 2
If I launch SCORM content, can your product generate equivalent xAPI statements for the SCORM data?
Why ask this?
Some LMSs will generate xAPI statements for SCORM content. Some will only generate a “complete” statement when the SCORM module is finished. However, if this is all they can do, there’s not a lot of value to the offering. Other systems will generate xAPI statements for test questions embedded in your content, which is more useful data.
Question 3
If eLearning published for xAPI or Tin Can and launched from the LMS sends back data that goes beyond typical SCORM data, will it be received and allowed by the LRS? (Example: “commented”)
Why ask this?
If all the system can do is offer SCORM-equivalent xAPI, then there isn’t much point in using xAPI (unless you have an external LRS). The rich data about what’s going on inside the eLearning is one of the value propositions for xAPI. Some of the systems that support xAPI ignore or (worse) reject data that isn’t on the short list of verbs they’ll receive. By the way, this means they are non-conformant and will not be able to pass the ADL tests we discussed earlier.
Question 4
Do you use the cmi5 profile for eLearning courses?
Why ask this?
cmi5 is the gold standard for using eLearning in an LMS with xAPI. Few LMSs use it so don’t take a “no” as an automatic disqualification, but if they say “yes” then you at least know that the basic functions of xAPI-based eLearning will be handled correctly in the LMS.
Leveling up: non-eLearning data
Question 1
Does the system accept inbound xAPI data from sources outside the LMS, assuming they are properly formatted and authenticated?
Why ask this?
This is a question for systems where the LMS and LRS are a single product. One of the key value propositions of xAPI is the ability to see learning and performance activity from outside the LMS. That requires your LRS (wherever it is) to be able to accept that data.
Question 2
Does the system allow the use of xAPI data by other learning and performance activities? For example, a performance support tool reaches into the LRS and can determine what learning activities a user has already seen.
Why ask this?
This is the inverse of the question above. If I write data to the LRS, can I get it back? There are many reasons for this question. First, you might want to feed data into a Business Intelligence tool so that you can compare learning data with data from other systems. For example, wouldn’t it be nice to compare your learning data about sales training with data from Salesforce, all in one dashboard? Second, ultimately you’ll want to be able to personalize learning experiences based on data from other learning and performance data in the LRS.
Question 3
Does the reporting function have access to xAPI data stored in the system? Is this limited to SCORM-like activity or can reports be run on data such as “commented?”
Why ask this?
There’s no point in storing the data if you can’t report on it.
Question 4
Can data stored in extensions be accessed from the reporting function?
Why ask this?
There’s a lot of rich data that can be stored in extensions…but it can be tricky for an LMS to report on it (SCORM LMSs never had to handle this stuff). There’s no point in storing the data if you can’t report on it.
Summary
Congratulations on your decision to move forward to xAPI! As you build your learning ecosystem, you will start getting better data about your learning programs. You’ll be able to track learning activities that occur outside the LMS, and you’ll get analytics to help improve the way you design and manage learning. We hope the questions and answers provided here will assist in choosing the right tools for your ecosystem.