What This Post Is About

Some strategies for developing effective quizzes to help engage learners and assess understanding/knowledge at the end of your lesson/content.

What This Post is Not About

This is not a detailed discussion on using frameworks (e.g. Bloom’s Taxonomy) to develop questions, although this is mentioned in passing.

Here is a Summary

Developing quizzes seems to be a perennial pain for instructional designers. Recently, I have seen some comments and posts bemoaning a sort of “writer’s block” to developing quizzes. Quite often, this occurs because IDs get very excited about developing the content, and forget that really, the quiz is the only thing that can tell us whether a learner has achieved their objectives.

Here Begins the Post

Nearly every instructional designer I know has at some time bemoaned the writing of quiz questions for learning content. Often, quizzes are seen as a “come-down” after the dizzying heights of scripting according to learning objectives and some overall content strategy (which determines the level and mode of interactivity to be used).  The race is often to get content that covers the objectives. Quiz questions (at best) are left to questions that are intended to elicit a response which is some sentence from the middle of the content (indeed, this sentence works hard for its existence, often being the feedback as well).

So, here are some strategies I employ for writing quiz questions…

Goals, Objectives, Content. Base your questions on these, in this order:

Goals – the overall goal should be a clear (although it may be complex) statement of some task the learner should be able to perform, or piece of information they should know. This is really the most important thing to test. But how do you test it, using Multiple Choice Questions, True or False, or Matching questions?

Refer to Blooms (or other) taxonomy for help to determine the type of question you should be asking.  Of course, you should have done this to define your objective – but let’s pretend you have done that.

Consider the verb:

  • What action is it that the learner must be able to perform?
  • Do you need them to recognise a statement of truth?
  • Do you need them to interpret data and identify the best statement to describe that?
  • Do you need them to interpret information (data/facts/statement) and identify a statement describing the consequence of that?
  • Do you need them to identify how certain concepts relate together?

Consider the conditions and information/tools to be used:

  • Do they need to know these, specifically?
  • Do they need to identify how conditions and information/tools change or should be considered with a different verb?
  • Do they need to identify how the conditions themselves will change the information/tools should be used?
  • Do they need to know the mechanism by which the information/tools should work?

Objectives are the simplified statements that make up the overall (complex) goal for a course.  Often (and it is certainly how I learned), it is the objectives that people tell you to quiz on. If you have well-defined objectives, your questions are simple: rewrite your objective so that a question mark can sit at the end. However, you should think about:

How different objectives combine to make up the overall goal – can you ask questions that relate to 2 or 3 objectives, and also attend to the overall goal? Isn’t this a better indication of whether the learner understands your content?

In what context are your objectives useful (again, you should be looking at the verb, conditions and information/tools required)? Your quizzes should test that a learner understands how to apply information (or use tools) in the right context. Quite often, an instructional designer will chase the ghost of a detailed, complex description of the operation of a tool, or the content of an idea. However, this could be quite as useless as knowing the innards of a cars engine, when actually what you need to know is when to change gear.

How can you “string” or “combine” questions, such that a series of questions can be used to take a learner through steps or the separate parts of a complex idea. This can always be a useful route to developing scenarios, which bring us closer to developing content, which makes the whole exercise more enjoyable for the instructional designer. Think about:

Whether there are a series of steps to be performed

What it might mean if a step is omitted – perhaps this can be a good way to highlight the result of an omission?

Whether it is important for the learner to understand why each step is important (and remember – sometimes it is not – for the purpose of completing a task, sometimes you only want a learner to know the requisite steps)

What about complex ideas? These might include interpreting an interface, alerts or other information that a learner may encounter. In these cases it is worth considering whether they need to know how different pieces of information relate to each other (e.g. if you get an alert here – do you need to check something else? How will inputting information at this point impact on some other system? What do you need to tell a customer about a process before you initiate it?).

Content – this is the heartbreaking work of staggering genius that you have devised to communicate important information to learners. Usually, I rail against asking questions based on content alone (from an ID point of view – always tell the learner you are asking questions based on the content – but from an ID point of view, to cohere to your syllabus and curriculum, you need to return to the objectives to base your questions).

However. (Always a however). The content can be used to guide how you put together questions and how the questions you write might relate to each other within the context of a quiz.

Using your content as a framework can help you to build the bridge between the objectives (as very specific pieces of information you need to ask about to confirm learning/understanding) and the goals (as the overall task or complex operation that the learner should be able to perform). This is where you can find the context to ask questions about how information or tools should be applied. Also, how you might string some questions together to cover more complex tasks, steps or information.

Also consider writing your questions backwards. Seriously, this works. Sometimes, having written your learning content, you go back to the beginning and can stare at the content and the objectives and wonder “What is it I was going to say?” On the other hand, if you start at the end, and move backwards toward the start, you can see new ways of considering the information and its importance. The information will also be fresher in your mind.

Well, sin é (Irish for “that’s it”). I hope some of these tips help. Please do leave a comment, question or query and I’ll answer as soon as possible.

Hello again, and thanks for dropping by. I appear to be blogging on the theme of questions in eLearning. This month, I’ll be talking about True or False questions – and what they can – and cannot – do. If you’d prefer to see more about developing content, downloads, supporting materials, integration for blended models, etc. please do let me know. I’m on questions at the moment because they appear to be poorly represented in eLearning blogs in general. Anyway, let’s get on.

What this post is about

The use, proper and poor, of that old cherry – the True or False? question. In this post, I’ll provide some advice and strategies for using True or False questions in your eLearning project. I hope to show there is more to True or False questions than many people assume (essentially – stating bald facts or bald lies to identify whether your learners can distinguish between them); and hopefully inspire some new thinking in the use of this question type.

What it is not about

This is not a post intended to give specific true or false questions.

Here is a Summary

Here begins the post

Recently, I was discussing True or False questions- from IRL (in-real-life) tests, actually – where the person I was talking to mentioned a third person who claims:
“True or false questions are useless! All one has to do is answer true – and you will pass. Because the statements used ALWAYS tend toward truth.” 

There may well be truth in this statement (or falsity!), but it is too much to deal with in one post – there are too many issues. Here are just some:

  • What is your passing grade (that allows someone to answer “True” for every question and still pass)?
  • What is your content (that the only manner in which to phrase the questions is such that they tend toward truth)?
  • What is the context (that you are using “True or False” questions as a form of assessment)?
  • What is your goal for your learners (that “True or False” questions are being used – is there a specific reason?)?
  • What is your learner’s goal (how does the use of “True or False” questions help them to achieve their aims, by applying the knowledge they are hopefully learning from your content)?
  • How is it that all your questions actually do tend toward truth (I must admit, in developing these questions I instinctively tend toward the opposite – there is no good reason for this, perhaps I am a man of falsehoods, I know not)?
So, let’s start by taking a look at the anatomy of a True or False Question.
At its heart, a True or False question is a simple multiple choice format. There is a statement, from which the learner selects an option.
Going a little deeper, a True or False question is a statement that either reflects a true state of events, or is a bald faced lie.
That’s it. In practice, it will look like this:
Statement: Some state of existence in the world.
  • True?
  • False?
The learner reads the statement, then selects True or False as a response, to indicate whether they believe that the statement does actually reflect some state of existence in the world, or does not.
So what’s wrong with it?
In many cases, it’s seen as far too simplistic (and because it is seen as simplistic, it is used simplistically). This is often true. Here’s an example. Imagine you read the Peter and Jane story to your learner (this is a story for children, along the lines of:
This is Peter. This is Jane. Peter and Jane are at the beach. Peter and Jane will build a sandcastle. Peter has a bucket. Jane has a shovel.)
Quite often – too often – the True or false question will be used to test the learner’s ability to remember a bald fact. For example:
True or false? Peter has a shovel.
Of course, we know Peter does not have a shovel.  The learner, irritated, clicks False and if they receive feedback, it annoys them that they are being told they were right (God help us all if they were wrong), which adds a delay (read: frustration) to their learning. I know not all statements of bald fact are this easy.
However, another issue creeps in: recognition (as opposed to understanding). When you make a statement that repeats (whether exactly to elicit “True” or incorrectly to elicit “False”), you run the risk of the learner recognising the statement. This is a memory of the language – not of the actual content, or meaning. The learner will not remember the importance of Peter’s bucket – they will simply recognise that the sentence they read did not say that Peter had a shovel. This is an important distinction, because in learning, we aim for retention of knowledge and understanding – not recognition of phrasing.
So, what’s right with it?
The True or False question can be a great way of asking a learner to demonstrate their understanding of a complex situation. This may appear to be a far cry from Peter and Jane, but it applies quite well. (I use the term “complex” here to mean a series or combination of simple statements, that combine to become complex). The True or False question can be used to take disparate concepts and join them together. Some applications include:
  • Between topics within a learning unit
  • As a pre-test to determine whether a learner can connect two objectives (i.e. apply knowledge they should have in a relevant context)
  • After a complex topic (i.e. where several concepts have been discussed) to determine whether a learner can put the concepts together, and understand their outcome
(This really is, just a few. There are many more applications, which I would tease out if there were a book as opposed to a blog to be written).
So what do these options mean in real life? 
Between topics within a unit – as part of a quiz, the True or False question can be used to link some concept in the previous topic to a concept in the next topic (e.g. In the last topic, we learned that Peter had a bucket and Jane had a shovel. We might also have learned that they intend to build a sandcastle – the mechanics of which will be dealt with in the next topic. So we might ask: True or False? A typical sandcastle cannot be built using a bucket and shovel.). If you’re using a True or False question in this way, it cannot be an assessment item. You can’t assess someone based on such a question without  presenting the information first, it’s simply not fair. However, if you aren’t assessing learners, this is an excellent way to focus their attention on what is coming next and to identify the key information from the previous topic that will be employed in the next topic.  Furthermore, if you are using feedback immediately after the question, it can be used as a learning event in itself (if you are comfortable with this, and your course is organised in such a way as to allow the learner to learn from the feedback). In short, you can ask about two disparate concepts, and use the feedback to demonstrate to the learner how they are connected. A note of caution – be careful with this approach. The learner should – based on the preceding content  – have some idea of the connection between the concepts. If something new is introduced out of the blue, the learner will lose faith in you. However, if the concepts have been dealt with thoroughly, a True or False question that challenges learners to connect them can be really useful in focusing their attention or making them think about the concepts in a new way.
As a pre-test, True or False questions can be used in a similar fashion -by asking about 2 seemingly disparate aspects to a complex procedure or body of knowledge to see if learners can connect them. For example: True or False? If Peter uses a shovel and Jane uses a bucket, they can build a sandcastle. In this example, it is true. We know from the story that this isn’t the way things worked out – but the learner should be able to extrapolate from the 2 actors and the 2 tools, the outcome should be achievable. This has been simplified – it’s not always the case that the statement will be true (see my final advice), but in this specific context, we are on safe ground. However, if one were training in health and safety, it might be the case that Peter and Jane need certificates to use the equipment they have. In such a situation, the very same question could be false – because only Peter has the certificate for use of a bucket and only Jane has the certificate for use of the shovel. Suddenly, something very simple has become quite complex in itself.
This leads me to the third point – the use of True or False questions after a complex subject, the use of True or False questions can be really effective. To build them, you need to phrase the statement very carefully, then ask whether it is true or false. Taking our previous example:
True or False? If Peter uses the bucket and Jane uses another bucket, they can build a sandcastle without a shovel at all.
Well, is this true or false? If we have only stated the facts as listed above, we cannot mark a learner down for saying it is false. All they know is that Peter and Jane have a bucket and a shovel and with these tools, they will build a sandcastle.
However, we may have explained the purpose of the bucket and shovel- to collect (shovel) and hold (bucket) sand. We may have gone further and explained the characteristics of a bucket and shovel, which means the learner should be able to identify that one bucket can be used to pick up sand – and the other can be used to hold it. If the learner is expected to be at a level whereby they can understand all this – then they should get this question right. We are testing their ability to recognise that one of the buckets can be used as a shovel, because it has characteristics that allow this. To get even more subtle – if the only way to build a sandcastle was to use a bucket and upend it into the sand – then the vice versa case is false – we cannot use 2 shovels to build a sandcastle, because a shovel (by its characteristics) cannot collect the amount of sand we require to be upended to create our sandcastle.
I hope these examples have provided some food for thought, and will extend the use of the True or False question type.
Some Final Advice/Thoughts. When using True or False questions:
The statement you write should contain all the information they need to answer true or false – based on their defined level of expertise and experience. As you may – or may not – have noticed, I hate “trick” questions. The truth (or falsity) of the statement you make should rely on itself alone.  You must remove all possibility where a learner thinks “Well, in this context, it might be true, but in general it is false”. The learner is attending to the question, and so must be able to answer the question based on the merits of the question provided (this may not be the case with all question types – but with True or False, I believe it is the case)
Your statement must be quite finely honed.  As an extension of the above – you must be careful to remove all uncertainty to the statement you make. Ensuring this will usually require a second pair of eyes to make sure there is nothing in your statement that might confuse them. This is really an issue for the editorial process, but it is important to point out.
Mix it up. Some statements are true, some are false. So, make some of your own statements false. My experience would echo the statement at the start of this post – that most True or False questions tend toward truth. However, if you do decide to tend toward false:
Don’t play with complex constructions that make the statement false (i.e. double, triple, negative falsehoods)
Don’t go with simple, bald false statements (see the earlier point about recognition vs understanding)
Make sure the false statement appears to be likely, but don’t try to trick the learner in this regard. Questions can be aloof from the learner and the general content;  a True or False question needn’t attempt to ‘catch a learner out’ by being especially inviting in any way. Remember – you have taught the content – the question can now be used to connect, combine or otherwise construct knowledge based on the individual parts.
If you use feedback – immediate feedback – True or False questions can be an opportune way to tease out differences. For example, if you have covered two disparate concepts that can be brought together in one True or False question, the feedback to the question can be used to provide further insight (the method or other technologies that Peter and Jane could use to build a sandcastle; how sandcastles can be seen as a metaphor for something else; etc.)
Here ends the post.
Thanks for reading. Please do leave a comment or email me if you’d like to talk about this more.