11.2 Difficulty levels in scale

A major goal of the ALL Main Survey is to profile the skill distribution of a country's population. The 42 items to be chosen to assess Numeracy in ALL (including the two Numeracy items in the Core screener) should represent a range of levels of difficulty, so that they can discriminate reliably between performance levels of respondents.

Tentative estimates of the performance level required of respondents to answer items in nationally representative samples in the Main study were derived on the basis of IRT analyses of the Pilot data and comparisons to data from the Prior IALS, and appear in Table 5. (While IALS and other large-scale comparative surveys of adults' skills represent item difficulty on a continuous scale, when reporting key results, they group items in terms of five difficulty levels, where level 1 refers to easy items and level 5 refers to difficult items). The 40 items (plus two Core items) selected for the Main appear to form a satisfactory distribution across five performance levels. The distribution of items in terms of difficulty levels provides the most information at the center of the expected population distribution, and thus promises a rich field of data from which to draw a profile of the population's numeracy.

11.3 Balance between blocks

Within the structure of the Main Survey, Numeracy items are organized by blocks. The BIB ("Balanced Incomplete Block") design adopted for the ALL survey, which was also used in IALS and other large-scale surveys, dictates that a block of Numeracy items be paired in booklets with a block of Prose and Document Literacy items or with a second block of Numeracy items. Average time requirements recorded during the pilot indicated that the allotment of 20 Numeracy items per block worked well and should be continued for the Main survey.