Chapter 11: Designing and conducting formative evaluations

“Studies have shown that thousands of the instructional products sold in the United States each year have not been evaluated with learners and revised prior to distribution (p. 283-284)

“Other studies have demonstrated that simply trying out materials with a single learner and revising the materials based on that data can make a significant difference in the effectiveness of materials.” (p. 284)

“We typically think about formative evaluation and revision of instructional materials as one major step. For the sake of clarity and to emphasize the importance of reexamining the whole instructional design process when instructional materials are to be revised, we separated the design and conduct of the formative evaluation study from the process of revising the instructional materials.” (p. 284)

“The major concept underlying this chapter is formative evaluation , which is the process designers use to obtain data for revising their instruction to make it more efficient and effective. Its emphasis is on the collection and analysis of data and the revision of the instruction.” (p. 284)

“There are three basic phases of formative evaluation. First, in one-to-one or clinical evaluation, the designer works with individual learners to obtain data to revise the materials. The second stage of formative evaluation is a small-group evaluation . A group of eight to twenty learners representative of the target population study the materials on their own and are tested to collect the required data. The third stage of formative evaluation is usually a field trial.” (p. 285)

As mentioned above, rarely do we get a chance to do a complete formative evaluation with these three steps. This is an ideal world, not the real one!

“What frame of reference can you use to design the formative evaluation? Keeping in mind the purpose of formative evaluation is to pinpoint specific errors in the materials in order to correct them, the evaluation design—including instruments, procedures, and personnel—must yield information about the location of and the reasons for any problems. Focusing the design only on the goals and objectives of the instruction is too limited. Data on learners’ achievement of goals and objectives is insufficient, although important, because these data only provide information about where errors occur rather than why they occur. Similarly, a shotgun approach to the collection of data is also inappropriate. Although collecting data on everything you can imagine produces a variety of information, it may yield some data that are irrelevant and incomplete.” (p. 285)

“Although the formative evaluation process focuses on the acquisition of data from learners, it is also important to have the instruction reviewed by specialists.” (p. 287)

“Formative evaluation of instructional materials is conducted to determine the effectiveness of the materials and to revise them in areas where they are ineffective. Formative evaluations should be conducted on newly developed materials as well as existing materials that are selected based on the instructional strategy. Evaluations are necessary for both mediated- and instructor-presented materials. The evaluations should be designed to produce data to pinpoint specific areas where the instruction is faulty and to suggest how it should be revised.” (p, 310)