Writing quiz questions that assess student understanding and critical thinking
This article discusses the writing of effective multiple-choice style quiz questions that assess students' understanding and critical thinking skills, and guidance on how to design quiz questions that provide a reliable and valid measure of learning.
Designing questions to assess higher-order thinking
One of the greatest challenges of multiple-choice design is assessing higher-order thinking skills. Higher-order thinking is often used to refer to 'transfer', 'critical thinking' and 'problem-solving.'
When designing a quiz that assesses higher-order thinking skills, it is necessary to write questions/problems that require students to:
- use information, methods, concepts, or theories in new situations
- predict sequences and outcomes
- solve problems in which students must select the approach to use
- see patterns and organization of parts (e.g. classify, order)
- determine the quality/importance of different pieces of information
- discriminate among ideas
- examine pieces of evidence to determine the likelihood of certain outcomes/scenarios
- make choices based on reasoned argument.
The number of questions included in a quiz will vary depending on the content and purpose of the quiz. Generally, 10-20 questions are appropriate to ensure that students are only assessed on terms and concepts that are directly connected to learning outcomes. It is essential to consider including different question types to design a quiz that evaluates different levels of cognition and aligns with principles of academic integrity. Below the video there is an overview of the different question types available in Blackboard tests.
Parts of a Quiz
A typical multiple-choice question is comprised of 3 parts: the stem, the distractors and a single correct answer. For a simple, lower-order thinking or knowledge-based question this usually suffices.
However, an effective quiz question that addresses critical thinking should be made up of the following components:
- Context/introduction
- Question/problem (also known as the 'stem')
- Correct answer/s
- Distractors (wrong answer/s)
Adding a context or introduction is important. Because students are being asked to examine more information relevant to the question being asked they are much less likely to guess, or depend on the 'recall' of facts.
Increase question complexity
Making students select multiple possible correct answers increases the number of potential answers, requires students to exercise discrimination, and reduces the risk that students can "guess" the right answer.
The sections below outline how to write each component of an effective quiz question effectively.
1. Writing the context/introduction
It is critical to introduce the question and to relate it to a ‘real world’ context so that students can relate to what is being asked. A well-written context sets the stage for the question and should be phrased in a concise manner using familiar vocabulary.
Example:
After making a documentation error which action should the nurse take?
a) Use correcting liquid to cover the mistake and make a new entry
b) Draw a line through it and write error above the entry
c) Draw a line and write through it and write mistaken entry above
d) Draw a line and write through it and write mistaken entry and initials above
Imagine you are a nurse in the palliative care unit in a hospital, during your shift one of your patients has an allergic reaction to a new medication. As you are documenting the allergic reaction, you accidentally put down the wrong medication.
How do you correct the documentation error?
a) Use liquid paper correction fluid to cover the mistake and write the correct medication on top
b) Strikethrough the error and write the correct medication above
c) Draw a line and write mistaken entry and your initials then write the correct medication above
d) Always write in pencil and use an eraser to correct the entry
Using visuals to provide context
Images can help to convey context in very few words. Where appropriate, you could consider using charts and graphs to provide the context for the question as this will require students to carefully interpret the information.
2. Writing the question
A well-written question enables students to:
- Understand the question without reading it several times
- Answer the question without reading all the options
- Answer the question without knowing the answers to other quiz questions
The question should be:
- A direct/complete question rather than incomplete statements
- Concise and brief avoiding undue complexity, redundancy, and irrelevant information
- Stated in a positive form as negatively phrased questions are easily misunderstood
- Using familiar language, avoiding any unfamiliar terminology
To increase the validity and reliability of the quiz, it is also a good idea to randomize the position of the correct responses throughout the quiz. One way to ensure this is to consistently organize question-order alphabetically or from lowest-highest number. Blackboard also gives an option to randomize the order of questions and the order of answers for each student. Utilizing these features when designing a quiz is a way to further align it with SCU values connected to academic integrity.
Example:
A hotel attempting to match customers’ purchase patterns and their demand for guest rooms with future occupancy forecasts is known as:
a) Integrated management
b) Yield management
c) Sophisticated management
d) Reservations management
Imagine you are the manager at a busy hotel. As the manager, you are asked to match customer purchase patterns and demand for guest rooms with future occupancy forecasts.
What is this process known as?
a) Integrated management
b) Yield management
c) Sophisticated management
d) Reservations management
3. Writing the answer
A well-written answer enables students to select the right response without having to sort out complexities that have nothing to do with knowing the correct answer. However, students should not be able to guess the correct answer from the way the responses are written.
Quiz questions often have around 4 options for the students to choose from. However, there may be situations where it may make sense to have fewer or additional response options.
Example:
The best way to achieve more accurate and achievable budgets is to:
a) Have the budget committee monitor actual results on a frequent basis so that quick punitive action can be taken when actual results do not comply with budgeted expectations
b) Have the budget prepared by top executives only
c) Have all employees participated in the preparation of the budget
d) All of the above
What is the best way to achieve more accurate and achievable budgets?
a) Have a budget committee that frequently monitors results
b) Have a budget that is prepared by executives only
c) Have all employees participate in monitoring results
d) Have a budget that is created by employees
4. Writing effective distractors
It is critical to carefully write appropriate distractors of the same length and complexity, that are neither too similar that the answer is vague or arguable, nor too obvious that anyone could guess. An appropriate distractor:
- Mirrors the correct answer in length, style, complexity, phrasing and style
- Is plausible rather than exaggerated or unrealistic in a way that gives away the correct answer
Avoid the following
- Avoid distractors that contain minuscule or vague distinctions from the correct answer, as this may confuse the students (except where these distinctions are significant to demonstrating the unit learning outcomes).
- Using “all of the above” or “both a) & c),” make it easier for students to guess the correct answer with only partial knowledge. Instead, use a multiple answer question type or add more appropriate distractors.
- Verbal or grammatical clues that give away the correct answer.
- Using “none of the above” unless there is an objectively correct answer (e.g. a mathematics quiz).
Example:
Breathing rate may increase as a result of:
a) A small decrease in oxygen levels in the body
b) A small increase in carbon dioxide levels in the body
c) A decrease in blood pH
d) Both b and c
Imagine you are going for a run. As you run faster, you begin to breathe faster as well.
What reactions in the body causes the breathing rate to increase? (Select all that apply)
a) A decrease in oxygen levels in the body
b) An increase in carbon dioxide levels in the body
c) A decrease in blood pH
d) An increase in blood pH
Example:
All other things being equal, an increased seat turnover will:
a) Increase total revenue
b) Has no impact on the average check
c) Increase the average check
d) Decrease total revenue
If all other variables are unchanged, what will an increased turnover result in?
a) An increase in total revenue
b) An increase in the average check
c) A decrease in total revenue
d) A decrease in the average check
Reflecting on quiz questions used for assessing student learning
Once you have written a quiz question reflect carefully and analytically to ensure that:
- Students can't recall the answer from a case study already covered in class (where the question requires critical thinking)
- Check that the distractors are plausible.
- The correct answer is not notably longer or shorter than the others
- Check spelling and grammar and are correct
Also make sure to test the quiz out with others, especially those familiar with the unit content.
Analysing Blackboard Tests
Where you are using Blackboard tests for online quizzes it is possible to analyse the student responses to determine the discrimination and difficulty of a particular quiz. Please see the following article: Analysing questions/responses for a Blackboard test (quiz)
Further Resources
- Is This a Trick Question? A Short Guide to Writing Effective Test Questions
- Multiple choice questions - Charles Sturt University