Five Steps to Evaluate and Select an LMS: Proven Practices

(Editor’s Note: This is the first of five articles by Steve Foreman on learning management systems.)

Ifan organization is to evaluate learning management system (LMS) products andvendors effectively, what should it do? Finding the LMS that best meets your organization’sneeds is not easy. There are hundreds of LMS productsavailable. The investment of time and cost that organizations make in learningmanagement systems and related technologies is significant, as are the risks ofdisruption from selecting the wrong solution.

Once theLMS is in place, your organization is likely to be using it for the next threeto five years or more. You certainly don’t want to pour your efforts into aproduct that is poorly matched to your needs. A new research report fromThe eLearning Guild, Evaluating and Selecting a Learning Management System, gives insight into what organizationsthat have already been through the process do. In this article I offer the fivesteps that are proven to lead to the best decisions.

Somebackground: Learning management systems

A learningmanagement system (LMS) can help you manage the administration, delivery, tracking andreporting of instructor-led classes and eLearning programs. An LMSprovides automation that replaces rigorous and expensive manual work, savestime, and enables you to organize your content, data, and learner audiences. Itreports on data that can help you measure results and forecast demand.

If you already have an LMS and arenot satisfied with its features and capabilities, it may be time to considerpursuing a more suitable product. The LMS marketplace is always changing, withnew players entering the fray all the time. A product that is well suited toyour organization’s needs is probably out there.

If you don’t have an LMS, or youuse a custom-developed solution that is more than a few years old, yourorganization may be missing some beneficial features that are available incommercial products.

In The eLearning Guild’s new researchreport, survey responses suggest that the process an organization uses toevaluate and select an LMS can have a significant impact on the perceivedsuccess of the LMS implementation and the organization’s ultimate satisfactionwith the solution.

How to evaluate products andselect an LMS

Generally, the LMS evaluation andselection process should involve five major steps: needs analysis, requirementsdefinition, product vetting, product evaluation, and product selection (Figure1).

Figure 1: The five majorsteps in LMS evaluation and selection

Step 1: Needs analysis

A needs analysis is a great way toget started. Interview your leadership and conduct focus groups withstakeholders throughout the organization. If you are planning to replace yourcurrent LMS, ask training groups and users what they like most about thecurrent solution, and what they would most like to see improved. If yourorganization has no LMS, focus your questions on how the organization manages learningcurrently and what the key problems areas are—operationally and financially. Ask leadership broaderquestions related to the organization’s strategic goals and how humanperformance impacts them. Don’t forget to ask IT about any technical standardsor constraints you need to consider.

Once you have collected sufficientinput, you should be able to identify strategic, operational, and technicaldrivers that you can use as guideposts for defining your LMS requirements

Examples of drivers

Based on your data gathering, youmay identify the following types of drivers:

• Your leadership states that your organization isexpecting to undergo rapid global growth in the coming years—a strategic driver.
• Some of your training managers mention thatdelays of a day or more in updating learning records for training deliveredhalf-way around the world have been causing problems with report accuracy—an operational driver.
• Your IT department expresses concern about theaccessibility and data security regulations it must meet in certain countrieswhere users reside—a technicaldriver.

The strategic, operational, andtechnical drivers you discover during a needs analysis will bring your LMSgoals and requirements into focus. This approach will ensure that your LMSselection criteria align with your organization’s goals and priorities.

Step 2: Requirements definition

Requirements form the basis of yourLMS selection criteria. The clearer and more complete your LMS requirements,the easier it is to evaluate products. If you are an instructional designer,you may find that writing LMS requirements is somewhat similar to writinginstructional objectives. Try to focus on what learners or administrators mustbe able to do with the LMS. Eachrequirement should express a need, not a solution. Each requirement should bediscrete, without repeating or overlapping other requirements, and you shouldwrite it at the same broad-based level of detail. If you end up with somewherebetween 30 and 60 requirements, you’re probably at the right level of detail.

It is sometimes helpful to organizeyour requirements into three categories: functional, technical, and cost.Functional requirements describe what the system must be able to do from alearning management perspective. Technical requirements describe how the systemmust fit into the broader IT enterprise infrastructure. Cost requirementsdescribe how the system matches up with your organization’s budgetconsiderations, constraints, and expectations.

Examples of requirements

Based on the drivers listed in theprevious example, you may define the following requirements.

Functional requirements:

• LMS administrators must be able to manageonboarding curricula with automatic email reminders and progress tracking.
• Learners must be able to access the LMS and itscontent in their native language.
• The system administrator must be able toconfigure permissions and span of control for various learning administratorgroups so that enterprise administrators can manage enterprise-wide content,while local administrators work autonomously on localized content delivered inlocal time zones to local target audiences.

Technical requirements:

• The LMS must support specific countryregulations related to security and accessibility.
• The LMS must be easily scalable to handleincreasing numbers of users and content to keep in step with the growth of theorganization.

Cost requirements:

• The LMS must be compatible with a lower-priceddatabase management system.
• The LMS must provide cost-effective licensingoptions for the anticipated number of users in your organization.

It is important to prioritize yourrequirements—and you can use a number of methods to accomplish this. Oneeffective method is the analytic hierarchy process, because it offers a reliable, fun, and interactivemethod for achieving rapid group consensus. By prioritizing your requirements,you can use a weighted scorecard to evaluate how well various LMS products meeteach requirement and then calculate overall scores that indicate which productsalign best with your requirements and priorities.

Step 3: Product vetting

There are hundredsof LMS products available in the marketplace. Your first challenge is to narrowyour choices to a short list of products. One way to accomplish this is toidentify eight-to-ten distinctive requirements that can help you rule outnon-qualifying products. For example, not all LMS products support multiplelanguages. If this is critical for you, this makes the requirement formulti-language support a good vetting criterion. Similarly, not all productscomply with specific accessibility and security regulations; also good vettingcriteria. There are many different price ranges for products. Ballpark licensingcosts are another good vetting criterion.

Once you have defined a set ofvetting criteria, you can research product web sites and/or contact vendors toinquire about their product’s support for each vetting criterion. Select ashort list of products, perhaps eight-to-twelve, which meet the most vettingcriteria. You will evaluate these products in more detail.

Step 4: Product evaluation

The product evaluation processworks like a funnel. With each step in the process, you rule out non-qualifyingproducts and continue evaluating those that best meet your requirements. You willachieve the best results by performing all of the suggested evaluationactivities in the order shown in Figure 2.

Figure 2: The productevaluation process funnel

Request for information (RFI)

A good first step is to create arequest for information (RFI). An RFI contains a list of all your requirementswith from two to six questions per requirement for vendors to answer. Avoidasking yes or no questions. Instead, ask questions that will require vendors todescribe how they meet the requirement.

When your RFI is ready, contactyour short-listed vendors to ask if they are willing to respond. If so, sendthem the RFI along with a due date for their response. You may need to set someground rules to keep the process fair and equitable for all participatingvendors. For example, you may decide that no vendor will be told who the othervendors are; answers to any questions asked by one vendor will be distributedto all vendors; no RFI responses will be accepted after the due date, etc.

Evaluate the vendor responses witha scorecard that allows scorers to rate how well the vendor meets eachrequirement on a five-point scale (e.g., unsatisfactory, suboptimal,acceptable, strong, and optimal). You may decide to calculate scores for eachrequirement, each requirement category, and an overall score. Calculate thevendor scores by incorporating your requirement priority weighting so that the overallscore reflects how well the LMS meets your requirements, based on what is mostimportant to your organization.

After scoring the responses to theRFI, you can rule out some of the lower-scoring vendors and continue evaluatingthe top contenders.

Use case demonstrations

Invite each of the top contendersto visit your organization for a half day to demonstrate their product. A goodapproach is to send use cases to the vendors in advance and ask them todemonstrate how their product accomplishes each of your use cases. For thispurpose, a use case is a scenario or goal and set of parameters that wouldtypically be performed by a learner or administrator. Use cases should reflectthe way your organization manages learning. Select some common use cases aswell as some that are complex and challenging. This will enable you to compareall vendors based on the same demonstration criteria. Again, you can use ascorecard to evaluate the vendor demonstrations and rule out those vendorswhose products did not perform well in the demonstrations.

Example:

USE CASE#1: CREATE A LEARNING PATH

Background: Alearning path is provided for each key function in our organization. A learningpath consists of a number of learning requirements, each of which is supportedby one or more classes, materials, and assessments. An individual requirementwithin a learning path may have an expiration date, upon which the learner mustrefresh his or her skills by completing the most recent learning activities tofulfill the learning requirement.

Demonstration: Pleasedemonstrate how a learning path can be created in your product.

Instructions: Createa learning path mapped to a particular role or function. The learning pathshould contain at least four learning requirements, which must be taken in anenforced sequence. The first learning requirement is met by completing aninstructor-led class. The second learning requirement is met by accessing adocument containing policies and performance standards and getting electronicsignature sign-off from your mentor that you have reviewed the materials andmet with your mentor to discuss them. The third learning requirement iscompleted by achieving a passing score >= 80% in an online assessment. Thefourth learning requirement is completed by fulfilling the criteria of an insitu observation by your mentor. The learner must complete all four learningrequirements within a three-month timeframe to get credit for completing thelearning path. The second learning requirement is set to expire and requiresretraining annually.

Trial version or “sandbox”

As you continue to evaluate thefinalists, you may ask for a trial version of the LMS software or access to a “sandbox”installation where you can explore the finalist products. Hands-on explorationwill give you a better sense of the user-interface design, features, andcapabilities of the product.

Customer references

You may ask for customer referencesfrom each vendor. By checking with existing customers, you can find out moreabout the product, the vendor’s responsiveness, problems encountered and howthey were resolved, lessons learned, and more. Be aware that vendors are likelyto give you the names of their most satisfied customers. However, everycustomer has experienced bumps with their products. So, be inquisitive. Ask howtheir last upgrade went and what they would most like to see improved in theproduct.

Financial health

Finally, you may want to checkfinancial reports (public companyfinancial reports are readily available online, and for private companies youmay find information from Dun and Bradstreet. You may also want to checkinformation from industry analysts on the financial health and market share ofthe products you are evaluating. Some well-regarded learning-technologyindustry analyst groups include Gartner Group, Forrester Research, Brandon HallResearch, and Bersin by Deloitte. LearningSolutions Magazine regularly carries industry news and articles.

The strength of your vendor’sfinancial position and customer base helps ensure that the vendor will supportyour product well and will continue to evolve and improve with sufficientfunding for product research and development.

Step 5: Product selection

The final step is to settle on aproduct. If possible, you can place your organization in a strong position tonegotiate the best pricing and terms by selecting two or three finalistproducts, any of which will meet your needs. LMS vendors are eager to gain yourbusiness. To an LMS vendor, each sale represents a multi-year revenue stream.

Send a request for proposal (RFP)to each of your finalists asking for pricing quotes, implementation timeframes,and support options. Be sure to ask for hosting options and pricing if you haveelected to purchase a cloud solution. Work with your purchasing or procurementdepartments, if appropriate. Compare the responses, and don’t be afraid to goback to each vendor to ask them for a better price or more feature optionscomparable to incentives the other vendor finalists are offering.

For clarity, Table 1 explains the differencesbetween an RFI that you use in the product evaluation phase and an RFP that youuse in the product selection phase.

Concluding thoughts

Too often, organizations maketechnology decisions without adequate investigation of the strategic,operational, and technology drivers for their product choices. A thorough needsanalysis engages stakeholders and sponsors throughout your organization tocreate a big-picture view of your learning management system needs. Clearlydefined and documented requirements capture those needs and form the basis ofyour product selection criteria. A funnel approach narrows hundreds of productsdown to the handful that are best matched to your requirements. Use-casedemonstrations and sandbox exploration provide evidence of a product’scapabilities that help you either validate or debunk vendor promises before youmake a commitment. Getting bids from two or three vendors whose products areequally good candidates for your organization positions you to get the bestterms and/or feature set for your money.

Although this process may take moretime and effort than you may have anticipated, it helps ensure that you makethe optimum decision and end up with an LMS that is best suited to your needs. Ithas side benefits as well. Stakeholders who are involved in evaluationactivities tend to have more ownership of the resulting decision and becomeacclimated to the product in advance of implementation. Sponsors have increasedconfidence in the decision because of the due diligence involved in the process.The learning organization builds relationships with its customers, sponsors,stakeholders, and IT, all of which helps set the stage for implementation.

Buying an LMS is not easy; just askall the people and organizations who’ve tried (some more than once). Itrequires due diligence and smart, strategic thinking. Whatever yourorganization’s experience is or has been, you stand a much better chance forsuccess and satisfaction with your product if you follow proven practices inevaluating and selecting an LMS.