Beta Testing an Online Course

It’s ninety percent complete, this on-line course you’ve been working on for months. The ones you love are sick and tired of hearing about all the interesting things you have learned about your subject matter and you’re not sure if there is anything left to learn about it. Your eyes are bleary from looking into the unending depths of your computer screen. Your brain is crammed so full of the course content, images, navigation, and assessments that you don’t care if you ever see this course or anything that has to do with your topic again for as long as you live. If you are lucky, your course has nothing to do with chocolate or ice cream.

Editor’s Note: Parts of this article may not format well on smartphones and smaller mobile devices. We recommend viewing on larger screens.

At this point in the development process you’re ready to take your course out on the town, parade it around and get a little feedback. That’s right, it’s Beta Testing time. Pulses quicken, faces flush, hearts race at the fun times ahead. The excitement of the imminent vetting of your flawless work makes you giddy and high-spirited.

This is true unless, of course, you are in the normal, non-geeked-out, majority. You know, the rest of us who got into course development for reasons other than running tests, surveys, and data collection. Believe it or not, there are people who don’t get thrilled at the prospects of Beta Tests. People who are not even really sure how to run a Beta Test. People who get a little queasy about Beta Testing their baby in front of a bunch of strangers who have no comprehension of the amount of work it takes to put such a course together. Normal people.

For us normal folks, let’s take a look at Beta Testing and see if we can make it a bit easier to understand, perform, and use this process.

Beta means “not perfect”

Many developers believe that Beta Tests should only be conducted when the course reaches 100% completion. Our group at the California Department of Transportation (CalTrans) practices otherwise. For us, beta means not perfect. When we reach about 90% completion, we start talking Beta Test. This practice may have developed from compressed schedules, eagerness to capture feedback, or a desire to proceed with the process while the Web developers, Flash geniuses, and graphic artists were fine-tuning the last bits and pieces of the course. In our experience, early Beta Testing — before the product has reached 100% — has not significantly hindered our results and may give us other benefits with schedule and participant feedback.

When it comes to showing their work in progress, course developers are often like artists, no peeking until it’s finished. Most developers are hesitant to release an unfinished product for testing. This is a reasonable and normal concern. Nobody likes to be opened up to critical comments, no matter how helpful, when they are not even finished with the product. However, when it comes to Beta Testing, in our group unfinished is just fine and even preferred. There is still time to make a few tweaks, and developers can continue working on the project while the Beta Test is in process. In my experience, testers are more willing to make comments when they can see that the work is in progress and not a finished product waiting for their blanket stamp of approval.

Beta Tests for on-line courses are not meant to review the content. A subject matter expert (SME) provided the content, and probably double- and triple-checked it for accuracy. The purpose of the Beta Test for an on-line course is simply to check the effectiveness, usability, and functionality of the course from a typical user perspective. Is it easy to understand the navigation? Are the assessments intuitive or well explained? Do users get bogged down anywhere? Do the learners learn? Do the colors and layout work, or does neon green text and 80 images on the page actually hurt the user’s brain?

It’s important to make this distinction when you are preparing your users for their experience in the test. Let them know that the test is not about content. Tell them that they are helping you refine the functionality of the on-line experience. Explain that the questions afterward will be about the ease of use and functionality of the course, and not about the accuracy of the content taught in the course. When you make these distinctions before the testers begin your results will be more helpful and complete. These statements will also help ease the nerves of some of your testers who feel like they need to be experts in the subject matter of the course they are reviewing.

Picking the Beta Test group members

One of the key elements required when running a Beta Test is a group of willing victims (make that participants). It can be a real challenge for some developers to identify the right people to participate in the test. Many developers wonder how they can find 15 to 20 experts to review their on-line course. However, from the previous discussion you know that the test participants shouldn’t be experts in the subject matter.

Remember that we are simply testing the presentation of the material in the on-line environment. Since we are not testing the content we don’t need or want subject matter experts for the Beta Test. Test participants should possess about the same level of knowledge as the average future user of the on-line course. Since we typically invest in and build on-line learning for large audiences, this level of knowledge should be very easy to find in a willing group of 15 to 20 people.

If you work for a large organization you will probably have no problem identifying potential Beta Testers. If you are a smaller shop or a one man show you may have more of a challenge here. Smaller course development businesses should look to their client or potential clients for possible Beta Testers. You can always rely on family and friends for help if they possess the knowledge level of your typical target learner. However, if you’re building a course on concrete pavement design or high intensity LED optical testing, you’re going to have to go farther than the family reunion for a quality test group.

Once a participant pool is found it’s important to realize that there is a difference between identifying Beta Testers and getting them to participate in a test. Even in a larger organization it can be frustrating to find a group of people that will take the time to help you. Here at CalTrans we first began gathering groups by pulling people out of their cubes for a few hours, or sending out e-mails to different work groups asking for a big favor. Response to this method was less than stellar.

One solution to this challenge that we have adopted here in our big organization can be very effective for the small shop as well. Avoid the hunt all together and shift the burden to someone else. This timeless work-management technique is called “passing the buck,” and can be very effective.

Through experience, some developers have learned to include a paragraph in the project charter or contract that alerts the sponsor of the future Beta Test, and requires the sponsor to provide a group of 15 to 20 participants. A good clause might identify the participants as individuals at similar knowledge levels as the potential course users and provide an estimate of time required to complete the test.

This approach has worked wonders for the sanity of course developers who can become easily frustrated with busy people unable to find the time to complete the Beta Test. When the sponsor shoulders the burden to provide the Beta Test participants, the message to complete the on-line course is less of a favor and more of a mandate. When the sponsor is aware of this requirement early it is hardly an issue of concern.

Of course, this works well when you are building a course with a client or sponsor and a contract or charter. If you’re building a course on spec in hopes of selling it once it’s complete, you’ll have to find your Beta Test participants on your own. But their glowing comments in the feedback will be great marketing tools to help you sell your excellent product, right?

Test on-line, or in person?

So you have your group of testers. They are all eager to be the guinea pigs for your on-line course. Now you get to decide how to conduct your test. In a traditional classroom course there is really only one way to run a test; find a room, fill it with people, and teach your course. One of the many benefits of on-line training is the freedom to choose your testing methodology. You can use the traditional approach with a few adjustments, or you can go high-tech with on-line participation in multiple locations.

Classroom test

It’s only logical that the first Beta Tests for on-line courses that most of us run are hybrids of a traditional classroom method. We naturally extend our previous experience and knowledge into practice with the new technology. When we have taught or built courses for the classroom, and conducted Beta Tests in a classroom setting, we feel very comfortable with a live classroom test of our on-line course. This is not a problem. There are serious advantages to this method and some disadvantages that we’ll look at as well.

Process

When Beta Testing in this fashion, the project team finds a classroom full of computers, rounds up 15 to 20 people, all of whom will make their way through the on-line course individually. Observers hover around the room answering questions and taking notes. After most users are finished, a facilitator conducts an informal focus group, which is a fancy way of saying the facilitator asks specific questions, and has a guided conversation with participants about their experience in the on-line course environment. The facilitator and observers take lots more notes, and look for common concerns and threads of issues that will help refine the course.

Advantages

This is an excellent way of conducting a Beta Test. Advantages of this approach include timeliness and richness of information. When conducted in this manner, a Beta Test can be completed in one day. Your test takes up only one box on the calendar. The other method discussed below often takes one to two weeks. When you’re under a tight schedule, a one-day Beta Test is a huge benefit.

The quality feedback you get from this approach is also a plus. It’s been my experience that the feedback from these tests is rich in amount and detail. Because the facilitator conducts a guided conversation in a group format, the comments of participants prompt others to share more than they might in a one-on-one interview or survey situation. The focus group model allows the facilitator to dive deeper into any areas that present more concern or interest. It allows more freedom to immediately follow up on issues or concerns that rise in the course of the conversations, and allows the facilitator as well as every participant opportunities to seek clarity and understanding on questions and answers.

Disadvantages

While this is an effective way to perform a Beta Test, there are some disadvantages or challenges that need to be considered. This type of test is an event and comes with all the logistical issues implied of any event.

The first issue to consider is venue. Where will you conduct the Beta Test? You need a room with 10 to 20 computers. The size of the test group is limited by this constraint. If the available space only has five computers, then you can only have a test group of five or you must conduct multiple sessions. While many of us may know of computer labs at schools or businesses, the price or schedule may hinder our efforts. I work for a huge state government department with multiple computer labs and a contractual relationship with a nearby state college with computer labs and can still be challenged by scheduling issues.

Once you have a location all ready to go, you must get all your staff and Beta Testers into the same place at the same time. Make sure you provide clear instructions and a map to the location if needed. (See Figure 1.) It’s probable that two or three of your participants will not be able to make your test date or location for one reason or another. This isn’t a deal breaker, just don’t be surprised when your group of 15 turns into a group of 12.

Figure 1 This e-mailed flyer provides instruction and directions for Beta Test participants. Note the image of the cookies, a little bribe never hurts!

Set the stage for success by checking each computer, open the browser, navigate to your on-line course, ensure the mouse works, find headphones for each station if you have audio, and so forth. I also try to always bring a few plates of cookies and some napkins to make it feel more casual and friendly. How can you possibly go wrong with treats?

A traditional classroom-based test of your on-line course will have its challenges. However, the benefits are definitely there. Most of us are pretty comfortable with this approach, the feedback is rich and immediate, and since we can finish in one day, the classroom approach is a great way to go when the schedule is tight.

On-line test 

The second option for Beta Testing is perfect for the small operation without large resources capable of providing a classroom full of computers. This approach can be done with a team or with a one-person company. It may cost very little, or absolutely nothing depending on your current resources. Conducting an on-line test is also a natural choice for an on-line course. Using this method simply brings one more step of the process into the bright and shiny world of technology. Fun!

Process

With the on-line method, the nearly finished course is hosted on-line. The course is uploaded to a Web site that the testers will be able to access from their offices or homes. Once the course is on-line and ready for testing, the test facilitator sends out a notice to the test participants that informs them that the test is up and ready for their review. The notice gives the Web address of the site, and invites them to test the course at their convenience (within the next two days, two weeks, or whatever your time frame is). I typically give the participants one to two weeks to participate in the course, and I send reminders out daily to encourage participation.

Don’t forget the reason for a Beta Test is to gather feedback. When conducting an on-line Beta Test the easiest way to collect results is to continue the method with a simple on-line survey.

Our group uses Survey Monkey, an on-line survey service that provides a free account to anyone, with some minor limitations that are easy to accommodate. The free account allows you ten questions and 100 participants per survey. As you will see, the ten-question limit is a little malleable and just right for the kind of information being sought in a Beta Test. Of course, the 100-participant limitation has no bearing on Beta Tests. As mentioned earlier, a group of 15 to 20 users is more than adequate for most Beta Tests.

The Survey Monkey system is easy to use. You can build surveys for your Beta Test quickly, and made them available to your participants with a link in an e-mail. Build your survey before you release your Beta Test, and include the survey link in your original notice to the test participants. Let them know that when they are finished with the course they need to go to the survey to share their experience with you.

Another option is to imbed a link to the survey at the end of the on-line course. This is a temporary element, of course, and will be removed in the final product. This allows for a smoother transition, and possibly less confusion for the Beta Test participants.

Remember, the point of a Beta Test is to measure the functionality and usability of the course and not the content. The questions in the survey reflect these goals and do not test for learning of content. The validity of the content is tested by the assessments built into the course; games, quizzes, activities, tests, and so forth.

Due to the nature of the on-line course approach, most surveys for on-line course Beta Tests are very similar in our organization. Table 1 is an example of a survey we recently used for an on-line course Beta Test.

This test was created on a free Survey Monkey account. While Survey Monkey only allows for ten questions on a free account, we can massage our survey to include more. In this example we actually have 15 questions that the software only sees as nine. Notice that question 3 is a matrix of seven questions. Survey Monkey views this as only one question. This is a little trick that, I am sure, Survey Monkey graciously allows us to use for our benefit.

Take a look at question number 7. Remember, the instructions stressed that Beta Tests are not about content. However, no matter how many times you have told test participants this, there will be a few who found something they have issues with in the content. If you don’t give them an outlet for this information, they will e-mail the facilitator directly, place the comment in a different field, or simply speak disparagingly to others about the course and all the content flaws they seemed to find. Even if you know that the content is perfect and the comments about content are wrong, your testers still need an outlet for their concerns. They’ll be happier, and you’ll be less harassed. Plus, they might be right.

Two concerns should be on your mind when building a Beta Test survey: question quality and question quantity. I hesitate to use too many questions in fear of brutalizing the test participants. Our team prefers to target the areas of concern with a limited amount of good-quality questions, and allow comments to gather the thoughts of the test participants who need to share more. Table 1 is one example of this approach, and more can be found on-line. As you build your own survey instrument remember what your objectives are, and target your questions accordingly.

Advantages

On-line Beta Testing is the way to go when you are looking to save costs and accommodate participant schedules. Because the process eliminates the need for a classroom full of computers, we save on these resources. This method allows users to participate in the test from their own computers at their own locations. The logistics issues of bringing everyone together in a computer-equipped classroom are non-existent. No worries about maps, parking, security clearance into a building, or even providing cookies.

This method also caters to the varied schedules of the test participants. When given an extended period of time to complete the course, like one or two weeks, your users can log into the course at a time that is convenient for them. You and your users are not locked into a set two-hour period to participate. They may even have the option of testing the course from home in their bathrobe and bunny slippers. Who doesn’t want that?

This approach is also relatively easy for a course development team to implement. Simply host the course on-line and inform the participants of a time window to complete the course. The development team is free to continue on the project and simply wait for feedback to roll in through the on-line survey. Some of my team members wait with eager anticipation and check Survey Monkey daily to monitor the feedback as it posts.

Disadvantages

On-line Beta Tests are inexpensive, accommodating to participant schedules, and easy to conduct. However, an on-line test can take much longer than a one-day classroom approach. Even when testers are mandated by their supervisor or an SME, an on-line test is not often a high priority. In real life, other work issues bombard your testers and pull them away from you and your project timeline. Patience and communication will help you persevere. Daily friendly reminders to participants, SMEs, and other stakeholders are a valuable tool to reach the end of your test period successfully.

The richness of information gathered in an on-line Beta Test may also be lacking when compared with a traditional classroom test and feedback session. When you use a survey to gather your feedback you usually limit your data by the questions you ask. In a classroom approach, a focus group feedback session allows the facilitator to follow threads of discussion and delve deeper into possible areas of concern. This is not the case with a survey. Unless your participants volunteer information, you will only get feedback on issues you address in the survey questionnaire. Identifying your concerns and developing your questions are important tasks in this approach.

Using your results

Once your test is complete, whether it’s a classroom or on-line approach, you should have some valuable feedback that will guide you as you make the final touches on your course.

The in-course assessments will help you identify problem areas in the content. If you have students who consistently miss questions in specific knowledge areas you may need to reassess the content or the questions.

The feedback from the classroom focus group or the on-line survey will provide you with information on ways to improve the usability and functionality of the course. It may be challenging to sift through all the feedback, and identify the improvements you will make as well as the ones you won’t.

Since budget and schedule limit all of us, we must be frugal with the changes made at this point. Simple corrections and adjustments are fast, inexpensive, and easy to integrate. Large-scale system-wide changes are expensive, time-consuming, and, at this point, should be unnecessary. If the course design is sound and the content is correct, there should be very few changes needed.

Feedback that asks for color changes or changes in the instructional methodology is often ignored. Remember, a trained professional (you) has already used sound instructional design principles and a subject matter expert to build the course. Trust your experience and skill, trust your subject matter expert, and don’t try to please all the test participants.  Look for those few little feedback treasures that make sense, and which will enhance your course.

Conclusion

The purpose of the Beta Test for an on-line course is to simply check the effectiveness, usability, and functionality of the course from a typical user perspective. Course developers and project managers look for feedback from test participants to ensure that the course is effective and usable for a typical user. We looked at two different ways to conduct a Beta Test; the traditional classroom method and the online approach. Both techniques have their positives and negatives. The classroom method allows you to accomplish your Beta Test quickly with deep feedback but can be expensive. The online method is almost cost free but may take some time. The feedback from the online method hinges on your closing survey questions, with little or no opportunity to explore deeper issues.

For more information, I recommend reading Barb Lesniak’s great article, Putting It to the Test: Quality Control for e-Learning Courses from the July 30, 2002 Learning Solutions Magazine. Ms. Lesniak explains the Alpha and Beta Test processes, discusses why they are important, and provides tips on how to conduct them.

While some may take issue with this approach, I encourage Beta Testing even when you’re only at 90% complete. It doesn’t need to be perfect to check it for perfection. Beta Testing is a great way to double-check your work before releasing it for general use. The Beta Test allows you to look at the effectiveness of your instruction and the usability of your online structure. Proper preparation will yield valuable feedback that will guide you to make small changes with big effect. The changes you choose to make will enhance your final product and build your stakeholder’s confidence in you and your course.

January 11, 2010

Join or login to save this to your library

Contributor

Benjamin Martin

Project Manager, California Department of Transportation

Share

Category

Learning Solutions

Topics

Getting Started, Elearning Design Strategies, Elearning Development Strategies, Instructional Design, UX & UI