Writing Listening Comprehension Items

What Information Should You Target?

The listening passage, the text, is the stimulus for the item. Items target specific information in the stimulus. There are many ways to determine what information to target, but how you write your items ultimately depends on what you want to test.

There are two main approaches to writing comprehension items.

Text-Based Item Writing

The characteristics of the spoken text can be used to construct the items. Generally, this means targeting the main points of the text. That is, what the speaker intends the listener to understand. Typically, items target comprehension of:

  • The gist or main idea of the speaker: What is the speaker talking about? What is the main point of the discussion?
  • The speaker's purpose or attitude: How does the man feel about x? What is the main intention of the speaker?
  • Significant supporting details: When will they go to lunch? Why was the man late?
  • Explicitly stated facts that are relevant to the main point: What time did the woman arrive? How many people came?
  • Inferences or conclusions: What will the woman do next? How did the man understand the problem?
  • Vocabulary, phrases, or idioms based on the context: What does the speaker mean by saying "its all over now"? What does "traumatic" mean?
  • What pronouns refer to: Who received the gift? What was a delicate situation?
  • Rhetorical or pragmatic purpose: Why does the professor mention onions? What is the speaker's purpose in comparing A to B?

Functionally-Based Item Writing

An alternative approach to writing items is to use a set of skills, functions or proficiency descriptors. The most common of these are the Common European Framework of Reference (CEFR), and the ACTFL/ILR descriptors.

Functional Descriptors

Here is a short set of functional descriptors that can be used to write items, taken from the ILR/ACTFL descriptors of language proficiency. The ability to understand:

  • Basic survival needs
  • Simple questions and answers
  • Courtesy expressions
  • Statements on familiar daily topics
  • Simple instructions and directions
  • Routine social demands or topics
  • Routine job requirements or topics
  • Simple descriptions
  • Narration in the past
  • Narration in the future
  • Instructions and directions
  • Implications and detect overtones
  • Interpret and relate ideas to an evaluative discourse
  • Unstated implications

The Common European Framework of Reference (CEFR)

The CEFR is a commonly used as a basis for writing items.

A1 Skills

  • Can follow simple texts that are very slow and carefully articulated. with long pauses to help assimilate the meaning.
  • Can understand instructions addressed carefully and slowly to him/her and follow short, simple directions.

A2 Skills

  • Can understand phrases and expressions related to areas of most immediate priority, (e.g. very basic personal and family information, shopping, local geography, employment) provided the speech is clearly and slowly articulated.
  • Can understand enough to meet needs of a concrete type, provided the speech is clearly and slowly articulated.
  • Can catch the main point in short, clear, simple messages and announcement.
  • Can understand simple directions relating to how to get from X to Y, by foot or public transport.
  • Can understand and extract the essential information from short recorded passages dealing with predictable everyday matters that are delivered slowly and clearly.
  • Can generally identify the topic of discussion around his/her when that is conducted slowly and clearly.

B1 Skills

  • Can understand the main points of clear standard speech on familiar matters regularly encountered at work, school leisure etc., including short narratives.
  • Can understand straightforward factual information about common everyday or job related topics, identifying both general messages and specific details, provided speech is clearly articulated in a generally familiar accent.
  • Can follow in outline straightforward short talks on familiar topics provided these are delivered in clearly articulated standard speech.
  • Can follow a lecture or talk within his/her own field, provided the subject matter is familiar and the presentation straightforward and clearly structured.
  • Can understand simple technical information, such as operating instructions for everyday equipment. Can follow detailed directions.
  • Can understand the main points of radio news bulletins and simpler recorded material about familiar subjects delivered relatively slowly and clearly.
  • Can understand the information content of the majority of recorded or broadcast audio material on topics of personal interest delivered in clear standard speech.
  • Can generally follow the main points of extended discussion around him/her, provided speech is clearly articulated in standard dialect.

B2 Skills

  • Can understand the main ideas of propositionally and linguistically complex speech on both concrete and abstract topics delivered in a standard dialect, including technical discussions in his/her field of specialization.
  • Can follow extended speech and complex lines of argument provided the topic is reasonably familiar, and the direction of the talk is sign-posted by explicit markers.
  • Can understand standard spoken language, live or broadcast, on both familiar and unfamiliar topics normally encountered in personal, social, academic or vocational life. Only extreme background noise, inadequate discourse structure and/or idiomatic usage influence the ability to understand.
  • Can follow the essentials of lectures, talks and reports and other forms of academic/professional presentations that are propositionally and linguistically complex.
  • Can understand announcements and messages on concrete and abstract topics spoken in standard dialect at normal speed.
  • Can understand most radio documentaries and most other recorded or broadcast audio material delivered in standard dialect and can identify the speaker's mood, tone etc.
  • Can understand recordings in standard dialect likely to be encountered in social, professional or academic life and identify speaker viewpoints and attitudes as well as the information content.
  • Can with some effort catch much of what is said around him/her, but may find it difficult to participate effectively in discussion with several native speakers who do not modify their language in any way.

C1 Skills

  • Can understand enough to follow extended speech on abstract and complex topics beyond his/her own field, though he/she may need to confirm occasional details, especially if the accent is unfamiliar.
  • Can recognize a wide range of idiomatic expressions and colloquialisms, appreciating register shifts. Can follow extended speech even when it is not clearly structured and when relationships are only implied and not signaled explicitly.
  • Can follow most lectures, discussions and debates with relative ease.
  • Can extract specific information from poor quality, audibly distorted public announcements e.g. in a station, sports stadium etc.
  • Can understand complex technical information, such as operating instructions, specifications for familiar products and services.
  • Can understand a wide range of recorded and broadcast audio material, including some non-standard usage, and identify finer points of detail including implicit attitudes and relationships between speakers.
  • Can easily follow complex interactions between third parties in group discussion and debate, even on abstract, complex unfamiliar topics
  • Can keep up with an animated conversation between native speakers.

C2 Skills

  • Has no difficulty in understanding any kind of spoken language, whether live or broadcast, delivered at fast native speed
  • Can follow specialized lectures and presentations employing a high degree of colloquialism, regional usage or unfamiliar terminology.

Item Types

There are a number of item types that allow for testing listening skills. The type of task used will be determined by the test takers' ability level, the constraints of the testing situation, and the best fit for the text-type used. Typical examples are:

  • Selected response: Binary choice (true-false, A or B),  Multiple choice (written text-based items, picture selection, etc.)
  • Constructed response: Short answer, Dictation, Information transfer, Body movement tasks (raise your right hand, close the window), Following instructions (draw a circle, color the bird)

What Makes a Good Test Item

Good listening comprehension items

  • Test salient information
  • Measure aural comprehension, exploiting the oral characteristics of the stimulus
  • Test comprehension based on the entire stimulus tested—not just a small portion
  • Allow test taker to demonstrate the targeted knowledge or skill Include, if necessary, reading, writing, or speaking skills at an appropriate level
  • Allow appropriate response time

The language of the items

  • Is clear and concise, directed and well-focused
  • Is easier or no more difficult than the language of the stimulus

The correct response

  • Is based on information presented in the stimulus
  • Cannot be determined by general prior knowledge

Constructed response items

  • Elicit a response of an appropriate length
  • Elicit a response that can be scored reliably

Selected response items

  • Contain one and only one answer
  • Are answerable without having to read the options first

Multiple-choice options

  • Are all plausible responses to question posed
  • Fit the stem grammatically
  • Are of similar length and structure
  • Are logically ordered

Fit logically within the context of the stimulus

Items sets

Often, a number of items are attached to one passage. These are called item sets. In that case:

  • Each item tests a distinct point or part of the stimulus
  • Items do not overlap with one another
  • The answer to one item does not help to answer another item
  • Jointly, the items test information from the entire stimulus
  • Items are presented in the same order as the information of the stimulus

Tips for Writing Good Listening Items

  • Target important information. Test something the speaker intended the listener to understand.
  • Do not test discrete, single points of information, such as small details. Test redundant, or repeated information.
  • To ensure that the point(s) tested are salient, listen to the stimulus, note what seems important to you, and test that
  • To ensure that questions are dependent upon the comprehension of the stimulus, give the items—without the stimulus—to a high ability listener. See whether they can answer correctly. They will probably guess a few correctly, but they should not be able to answer correctly based on prior knowledge. To ensure that the test taker has demonstrated evidence of comprehension, require responses in which the semantic content is expressed in a different form from that of the input. Do not allow them to repeat language from the stimulus.
  • Do not try to be clever, or tricky, in creating incorrect options. Do not focus on minor misunderstandings. Write distractors that are reasonable, given the topic, but which are clearly wrong given the text.