Part B

Item development

6. Approach

Throughout the item-production stages, the Numeracy team used the principles or guidelines outlined below as the basis for creating items. (Note: Appendix 1 includes sample items illustrating the type of stimuli and questions developed).

1. Items should cover as many aspects within each of the four facets of the numeracy construct as possible. Items were generated so as to require the activation of a broad range of skills and knowledge included in the construct of numeracy, as portrayed in the conceptual framework described in Part A and depicted in Table 1.
2. Items should include a realistic stimulus and one or more questions that adults might be expected to encounter, given such a stimulus. The aim was for the item content to be familiar and the questions purposeful to respondents across cultures. Most tasks were derived from real-life stimuli, from situations that are easily understood or that can be expected to be of importance or relevance in different cultures.
3. Items should have a free-response format. Items were structured to include a stimulus (e.g., a picture, drawing, display) and one or more questions, the answers to which the respondent writes in his own words.
4. Items should spread over different levels of difficulty. Items were produced to span the range of difficulty levels covered by the theoretical complexity factors outlined earlier. Attention was paid to generating some items at the lowest level of complexity, which are of interest in countries where policies and educational programs may be earmarked for low-skill populations.
5. Items should vary in the degree to which the task is embedded in text. While about 1/3 of items were embedded in or included relatively rich texts, about 2/3 of the items were designed to use little or no text, to reduce overlap with the Document and Prose literacy scales as they existed in IALS and continue in ALL.
6. Tasks should be relatively short. The use of short tasks enabled the inclusion of a large number of diverse stimuli and questions, thus allowing for coverage of many key facets of the numeracy framework. Items were not designed to simulate extended problem-solving processes, thus reducing overlap with the Problem-Solving scale.
7. Items should overcome issues associated with different unit systems. Items were designed so that different currency systems as well as different systems of measurement (metric or Imperial) could be applied to the numbers or figures used. Such items can retain equivalency with respect to their mathematical or cognitive demands and appear familiar to different populations even after being translated.