Advice on Quiz Assessments: 5 Things to Avoid

June 1, 2011

Hi there. Sorry I’m so late. I got caught in traffic.  From now on, I’m going to  try and get the train – this means posting once every month. Let’s see how that goes.

What this post is about: This post discusses some pitfalls I have come across when trying to create effective assessment or interactivity using quizzes/Q&As (and some that I have noticed in other people’s learning content).

What this post is not about: This is not an in-depth discussion on the various types and methods of assessment. Rather, it talks about what you might try to do (but really should avoid) when developing quizzes to test for knowledge retention and understanding.

Here is a summary: The last post talked about 5 things to do with quizzes, which I hope you found useful.  There are many temptations to over-reach when asking questions – leading to questions which you might think are more engaging and exercising, but in fact are irritating for learners, or do not add to the educational value of your content.

Here begins the post

Quizzes can be the hardest part of developing online learning. What’s a good question? The content seems so thin, how can I ever get X questions from it? How can I grab the learner’s attention with this? How can I avoid boring them?

You’re looking at a blank sheet of paper, wondering what you should be asking…

…then you get “creative.” You try to ask the same questions in different ways. This is not always a bad thing – especially where the same information might be applied in different contexts.  However, problem questions come from over-reaching. Here are 5 ways I’ve seen people over-reach, which (for me) just don’t hit the mark.

5 Things to Avoid

1. Avoid dealing in “semantics” and using “trick” questions

Both of these points relate to the language you use when asking questions.  Therefore the following is not necessarily true where your content is addressing comprehension, definitions, grammar or parts of language.

Using “semantics” means doing things like:

  • asking a question using a double negative
  • playing on words/puns or subtler meanings to ‘throw’ a learner
  • using options where a similarly-spelled word is used
  • using difficult sentence structures, especially where cause/effect or timelines are being asked about

This sort of tactic is just not fair to learners.  You aren’t testing them on their language skills; you’re testing them on the knowledge they are meant to be gaining as a result of your content.

Many argue that asking questions like this is a good way to “keep learners on their toes”, or “make sure they’re paying attention”. I’m not so sure.

  • First, keeping learners on their toes can be achieved by asking pop-questions within your content, or asking about subtle differences in application of knowledge – not by seeing whether they notice a minor aberration in your grammar, punctuation or spelling.
  • Furthermore, the quiz itself is a good way to make sure learners are paying attention; asking that they notice very subtle linguistic issues when they are concentrating on the concepts, theories or techniques you have been describing is off-putting.

Trick questions I have more difficulty with (not just in terms of their use, but also in describing them). These are questions where you actively try to lead a learner astray (i.e. away from the right answer).

I think quiz questions – whether for assessment or self reflection – should be fairly unbiased. For example, with multiple choice questions, they should present to the learner a series of equally likely (or unlikely) options.  If it’s skewed one way, it might be too easy (which doesn’t really exercise them) skewed another, it will feel to the learner like you are  trying to trick them (which you are!) The problem with tricking a learner is that you annoy them or (worse) they question the value of your content.

However, there is a subtlety in the “trickiness” here. Sometimes, you may want to ask about the differences between certain concepts or objects. Or, you may want to ask when a specific piece of information might be applied to a situation. In these cases, subtle differences can be a very useful way to tease out such differences (in fact, I’m a particular fan of “True/False” questions, where you ask whether a piece of information can be applied in the wrong context. The answer should be False, and the feedback can be used to explain the importance of context and information).

Be fair to your learners: They are there to learn from your content – not have their comprehension/concentration tested.

2. Avoid writing ‘to content’

You have objectives that you want your learner to achieve.  Often, these objectives will be well-used for proper assessment quizzes; but will be forgotten when trying to ask about the content a learner has just encountered. What often happens is that a sentence is lifted from the content to be reworked as a question.  Quite often, this will work for you (after all, your content is trying to drive your objectives).

However, this approach can lead to problems. If the sentence lifted from the content is ancillary (in that it explains something not directly addressing an objective, but supporting other information that directly addresses the objective), the learner may think this piece of information is more important than it actually is.

Another problem is where the sentence or piece of content you are basing your question on is within a context in the learning content. If the information is taken out of its context, it may lose meaning or importance. It may also make the information confusing, and therefore any question based on that information meaningless.

Finally, asking questions that speak to context can lead to asking unimportant questions. This is particularly the case where you have used case studies or examples in your content. Asking about a scenario you outlined, or the actions taken by some ‘pretend’ actor leads the learner completely astray from the point of your learning content. Of course, asking about the theory, concepts or steps that lie behind a scenario is fine. But all too often, I find people being asked “How much did Mary spend in the shop” rather than “In a shop, an apple costs 50c, a coke €1, a sandwich €5. If a customer buys 2 sandwiches 3 cokes and 4 apples, how much do they spend?”

Remember: your IDD should map the course goal into objectives, and those objectives to content. Both robust assessment and lighter self-reflective quizzes need to step back to the objectives in order to be really useful for learners.

Use your objectives to guide your content, rather than simply reviewing a topic for easy content to turn into questions.

3. Developing tests as a long series of simple multiple choice questions

This is the hardest “avoid” measure for me to defend. House style, SME opinions, certification demands and client requirements could make this point redundant. However, I shall argue on.

Being asked the same type of question over and over is pretty boring for learners. Click here, click there. One option, two options. Etc. Et cetera.  If you can at all, avoid doing this.

Most LMS and authoring tools now offer a range of question types that you can use. Try them out – don’t tie yourself to multiple choice simply because it seems to be the most robust type of question.  All types of questions can be used to ask about the truth of a statement or a definition. You could also consider:

  • Fill in the blanks for cause/effect, timeline, steps, definitions, conditions
  • Matching for the same as above, but also relationships
  • True/false for subtle differences arising from context, use of information, etc
  • Graphical questions excellent for relationships, screen-based information, etc

All these question types can also be used to provide scenarios or contexts within which you can really test your learners understanding.  There are of course many more contexts in which these questions are used – perhaps the subject for a series of posts; but I guess my advice is:

Avoid doing the same thing over and over again, try to be creative and consider new ways to ask about the information

A corrollorry of this is the way you frame your questions, where you are stuck with multiple choice alone. If the stem of your question is exactly the same over the course of 20 consecutive questions, your learners’ minds will start to stray. The ways to frame your question is bound only by your imagination. I also know that this advice could be seen to contradict my first point – but it shouldn’t. Reframing a question does not mean making it more obscure. Instead consider:

  • If you are asking about information, asking learners to identify a definition for something
  • Also with information, asking when information might be used
  • Also, providing a scenario and asking what information might be used
  • If you are asking about steps, providing one of the steps and asking the learner what happens previously or next
  • Also with steps, listing the steps in a process, then asking what was missing
  • Also, asking why a step is important

4. Getting ‘flashy’ for the sake of it

The exact opposite of the preceding point.  Trying to show off your content creation talents with wildly differing question types, formats and approaches could leave your learners with heads spinning.

It is tempting to try to “spice things up” or even “stir things up” using the vast selection of questions and content development tools out there. However, you should keep your learner in mind – they should be concentrating on getting to the right answers, rather than the question itself. Drawing too much attention to the format or bells and whistles attached to your question is something akin to writing a bad postmodern novel – those that appreciate the aesthetic of your great efforts may well miss the point of your making them.

Furthermore, great swings in the delivery of content or the way in which learners answer questions could lead to confusion on their part, and lead them astray from what is important in your learning content.

I’m not saying you should avoid being ambitious – rather, the manner in which you ask your question should suit the information you want the learner to work with. Sometimes the information you are testing on may require complex questions – but then you should make sure all the questions in your quiz work together in some way.

You want learners to be exercised by quizzes, but you don’t want them to be worn out by them.

5. Overcomplicating  your questions

Again, related to all of these points: keep it simple. Of course, some content is necessarily complex. But complexity is the addition of several simple pieces.

With the proliferation of available question types, you may be tempted to flex your muscles and shoe-horn a question into a question type. Alternatively, you may be tempted to create a large, complex scenario based quiz  that learners may not be able to follow. In short – these approaches add up to the same thing: Making the question difficult to comprehend. The longer a learner has to consider what a question means, the less effort they will put into actually thinking about the right answer.

This brings us back to the beginning – you’re not testing their language ability – you are testing their comprehension of the information you provide in your content.

Ask direct, pointed questions – so the learner concentrates on calling up the information, rather than working on understanding what the question is asking of them.

Here ends the post.

Talk to you next month! As always, please do leave a comment below.

Also, I’ve been toying with the idea of porting this whole project to a Tumblr blog, which might make it easier to start and engage in conversations. Please do let me know your thoughts on this.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: