By Terrence R. Redding, PH.D.
Abstract
Increasing emphasis is being placed on instructional designs that present material to be learned at the application level. One guideline reads, “All new self-study and classroom equivalent courses must use final exams and interactive inquiries that contain 70% application (as opposed to knowledge level) questions.” We have found that the subject matter experts working with the OLT instructional design team are unaware of the implications associated with the new requirements, and they have not been exposed to Bloom’s Taxonomy of Learning Objectives.
Thus, this paper explores Bloom’s Taxonomy in relation to highly effective instructional design for the delivery of distance education courses.
Introduction
OnLine Training Institute (OLT) actively seeks to share its knowledge and skills in instructional design with the larger community of online distance educators. In the past, we have presented comprehensive material on the mechanics of the iterative instructional design model and the underlying 23 factors associated with creating highly effective online courses. Most recently, we added “Priming,” a new factor in instructional design.
All courses developed by OLT follow stringent design models to ensure their effectiveness. Recently, OLT became aware that the National Association of Insurance Commissioners has begun the process of standardizing mid-level continuing education courses to include a minimum of 70% application-level material and questions.
This paper will review Bloom’s taxonomy, discuss developing knowledge and application-level questions, and provide examples of each.
Bloom’s Taxonomy
In 1956, Benjamin Bloom headed a group of educational psychologists who developed a classification of levels of intellectual behavior important in learning. Bloom found that over 95 % of the test questions students encounter require them to think only at the lowest possible level…the recall of information.
Bloom identified six levels within the cognitive domain, from the simple recall or recognition of facts as the lowest level, through increasingly more complex and abstract mental levels, to the highest order, which is classified as evaluation. Verb examples that represent intellectual activity on each level are listed here.
- Knowledge: arrange, define, duplicate, label, list, memorize, name, order, recognize, relate, recall, repeat, reproduce state.
- Comprehension: classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, restate, review, select, translate,
- Application: apply, choose, demonstrate, dramatize, employ, illustrate, interpret, operate, practice, schedule, sketch, solve, use, write.
- Analysis: analyze, appraise, calculate, categorize, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test.
- Synthesis: arrange, assemble, collect, compose, construct, create, design, develop, formulate, manage, organize, plan, prepare, propose, set up, write.
- Evaluation: appraise, argue, assess, attach, choose compare, defend estimate, judge, predict, rate, core, select, support, value, evaluate.
Formal education systems apply Bloom’s taxonomy as a hierarchy, moving students through the structure of the taxonomy in a deliberate fashion. However, in reality, they tend to be reiterative, moving students through the Knowledge level, extending them with Comprehension, and ultimately through at least the Application level. Online courses can easily do the same thing by covering the historical reasons certain facets of a particular profession exist (knowledge), covering the terminology or special language associated with a profession. Then, the same material is placed in context, and comprehension is tested to ensure comprehension is attained. Finally, case studies and situational examinations allow students to be evaluated on how well they can apply the material they have been exposed to.
Bloom’s Taxonomy provides a useful structure for categorizing test questions when assessing student learning. While not mentioned by the state regulatory agencies, it appears likely that Bloom’s Taxonomy is the foundational theory and guideline for incorporating more application-level test items.
The table below describes skills demonstrated for each level of thinking according to Bloom and question cues that can be used to elicit student responses within that level. The same content information can be assessed at different levels of cognition.
Here are some examples of test questions reflecting the six levels of learning according to Bloom.
Assessing Student Learning Using Bloom’s Taxonomy
| Competence | Skills Demonstrated |
| Knowledge | * observation and recall of information * knowledge of dates, events, places * knowledge of major ideas * mastery of subject matter Question Cues: list, define, tell, describe, identify, show, label, collect, examine, tabulate, quote, name, who, when, where, etc. |
| Comprehension | * understanding information * grasp meaning * translate knowledge into new context * interpret facts, compare, contrast * order, group, infer causes * predict consequences Question Cues: summarize, describe, interpret, contrast, predict, associate, distinguish, estimate, differentiate, discuss, extend |
| Application | * use information * use methods, concepts, theories in new situations * solve problems using required skills or knowledge Questions Cues: apply, demonstrate, calculate, complete, illustrate, show, solve, examine, modify, relate, change, classify, experiment, discover |
| Analysis | * seeing patterns * organization of parts * recognition of hidden meanings * identification of components Question Cues: analyze, separate, order, explain, connect, classify, arrange, divide, compare, select, explain, infer |
| Synthesis | * use old ideas to create new ones * generalize from given facts * relate knowledge from several areas * predict, draw conclusions Question Cues: combine, integrate, modify, rearrange, substitute, plan, create, design, invent, compose, formulate, prepare, generalize, rewrite |
| Evaluation | * compare and discriminate between ideas * assess value of theories, presentations * make choices based on reasoned argument * verify value of evidence * recognize subjectivity Question Cues: assess, decide, rank, grade, test, measure, recommend, convince, select, judge, explain, discriminate, support, conclude, compare, summarize |
Source: Much of this information was modified from a document written for the Online Education Resources of the Illinois Online Network. Copyright© 1998-2005, Illinois Online Network and the Board of Trustees of the University of Illinois. April 8, 2005, <http://www.ion.uillinois.edu/resources/tutorials/assessment/bloomtaxonomy.asp>.
The Department of Insurance, Texas, provides the following requirements and examples, which address the requirement for application-level questions.
Since January 7, 2003, all new self-study and classroom equivalent courses must use final exams and interactive inquiries that contain 70% application (as opposed to knowledge level) questions. All CE courses must be compliant by the later of January 7, 2004, or the expiration of a current certification period that began before January 7, 2003
Knowledge level questions involve the recall of specific facts, rules, dates, laws, and the like. These are items that can be memorized. An example of a knowledge-level question is:
What part of the PAP will cover damage caused by running into an animal?
a. Collision
b. Other than Collision
c. PIP
d. Uninsured-underinsured motorist
The student remembers that “contact with a bird or animal” is listed under other than collision.
Application-level questions call for the application of learned material in a new situation and usually deal with the application of rules, policies, methods, computations, laws, etc. Here are some examples that are based on actual test questions:
Jon and his passenger, Jim, are injured when Jon hits some ice on the road and skids into a tree. Jon has a PAP with basic Texas limits, PIP, and a $500 deductible for damage to his automobile. Jon’s injuries cost $2500 to repair. Jim’s injuries cost $12,000. The car repairs cost $10,500. The arborist who trims and dresses the damage to the tree charges $3000. How much is the insurance company going to pay?
a. $27,000
b. $24,000
c. $15,500
d. $13,000
The student remembers what coverages apply and calculates the amount.
Rhonda, a life insurance agent, re-reads her contract with the insurer. She realizes that it does not mention several of the activities, including meeting with prospects at their home, that she and her fellow agents perform. What type of authority do they actually have for these practices?
a. Judicial
b. Apparent authority
c. Implied authority
d. Express authority
The student takes the facts about authority and applies them to a situation.
Rhonda’s agency manager “looks the other way” when some of her fellow agents use an unethical and possibly illegal practice to increase their sales during a sales contest. By ignoring what he knows is happening, he has granted the agents ___________ authority to continue.
a. Judicial
b. Apparent authority
c. Implied authority
d. Express authority
The student again takes the facts about authority and applies them to a situation.
An agent’s authority to sell life insurance for a particular insurer is an example of ___________authority
a. Judicial
b. Apparent authority
c. Implied authority
d. Express authority
And again, the student takes the facts about authority and applies them to a situation. An alternate way of presenting these questions would be to combine the facts of the authority examples into one paragraph and then ask the questions.
A casualty insurer makes an offer to settle a law suit against their insured for $420,000. The offer is rejected and the jury determined that their insured’s liability was $600,000. The policy limits are $500,000. How much must the insurer pay to the plaintiff on behalf of the insured?
a. $420,000
b. $500,000
c. $600,000
d. $550,000
Juan is a third party claimant. His car was ruled a total loss in an accident caused by the first party’s negligence. Juan dearly loves his car, and wants the insurer to pay $12,000. He thinks $11,000 would repair it, and he wants the extra $1000 for his time and trouble in getting it repaired. The insurer declares it a total loss at $8,000, based on what similar cars are selling for. What fundamental principle of insurance is involved?
a. Fairness Doctrine
b. Indemnity
c. Collision
d. Liability
The student takes knowledge of the policy to understand what happened and applies knowledge of insurance principles to the situation.
Some providers have found it may be easier to write a series of questions, including some application level, by developing a short case study and asking the student to apply learned facts.
A scenario or situation question is often application-level, but just inserting a couple of names in a knowledge-level question does not automatically make it application-level. If the student has to recall facts about a policy, for instance, and apply those facts to a situation to answer the question, it probably is.
Extending the Continuing Education Course Model
It has been standard practice in CE courses to use case studies, or real life examples to develop a framework for a series of questions. The student is exposed to the scenario, retains the knowledge, and then demonstrates comprehension and the ability to apply that knowledge by responding to a series of questions based on the same scenario.
Under the new emerging guidelines, pre-licensing courses and state exams will use an increasing number of application-level questions (70%). Online courses developed by OLT will use the iterative model to incorporate the 24 factors associated with creating highly effective online courses and ensure that knowledge, comprehension, and application content and questions are used to provide high student success in each of our classes.
Reference:
Bloom, Benjamin S. (1956). Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain. New York: David McKay Co Inc.
July, 2007. http://www.coun.uvic.ca/learn/program/hndouts/bloom.html
July, 2007. http://www.tdi.state.tx.us/agent/agappllevelque.html
BIO
Terrence R. Redding, Ph.D., President and CEO of OnLine Training, has been designing and teaching adult education courses since 1968. Terry is a past W. K. Kellogg Fellow and has a doctoral degree in Adult and Higher Education. He has his Masters in the Psychology of Education and is an honors graduate with a Bachelor of Science in Education. His research has dealt with adult motivation theory and the origins of self-directed learning readiness. He has specialized in cognitive learning models and is the co-developer of the Honeycutt Redding Cognitive Task Analysis Model. For more than 28 years, he has been involved in an ongoing series of research projects associated with developing high self-directed learning readiness. Of great significance is the finding that children between ages 8 and 15 who become highly self-directed learners typically share a common childhood experience. Redding has labeled this experience “the first moment of lasting excitement.” High self-directed learners who share this common experience can remember some event that has captured their imagination for life. Typically, these individuals find their lifelong learning endeavors in some way tied to this “first moment of lasting excitement.”
